Reproducing 2D Only results on the Replica dataset

[shjung13]

Hi Phuc,

Thank you for the great work and deeply appreciate for providing a good code base to start with!

I am trying to run Open3DIS on the Replica dataset (the downsampled dataset you provided along with superpoints), but I could't reproduce your performance in the paper.

======== With code base based on commit 132fb69 =========
I am using the following configs, and ran scripts for grounding, generating instances, refining, and generating again to get the final 2D based 3D proposals.

At the end, I ran evaluation with inst_run_replica.py file.

However, I was able to get this number below, which is underperforming your reported performance of 18.1 AP.

Would you let me know if there is any step I need to do for reproducing Replica performance?
or Is the performance not reproducible with this version of code?

========= With latest code: 4b05043 ==========
I was able to successfully run the grounding part, but I can't run the generate_inst_3d.py file since I don't have post_filter and depth_thresh values that are required for running agglomerative clustering without spp.

Would you provide which values you used for them? + According to configs, you didn't use spp for the Replica dataset.

Is my understanding correct?

Thank you, and looking forward to hearing from you soon!

Best,
Sanghun

• Note: I found agglomerative clustering is not runnable for the Replica dataset since it does not return valid confidence (it returns None instead). But this confidence is necessary for getting point-wise CLIP embeddings. Would you also check this?
• I also found several variable errors (visi, num_points, etc) :)

[ngoductuanlhp]

Hi @shjung13 ,

Thank you for your interest in our work and for bringing up these points. You're correct; there were some errors introduced during the refactoring of the codebase.

• First, we indeed used superpoints on the Replica dataset. To replicate this, you can set the variable final_instance.spp_level to True. This will enable the model to run with superpoints as intended.

• Second, we found that using standard pairwise-matching to group instances between frames produced better results compared to agglomerative clustering. Therefore, by default, we have set the falg cluster.iterative to False in the current codebase.

I hope this addresses your questions.
Best.

[shjung13]

Hi @ngoductuanlhp

I appreciate for your quick response!

I just ran the code with spp_level set to True, and I got AP of 17.1 as below.

This is still 1% behind the reported number in the paper, but would this be within an error margin?

Thanks!

Best,
Sanghun