Some questions about training and evaluation

[wuhuateng]

1.During training, train the rpn+roi head to output bbox2d, but do not use it during inference. Instead, use Grounding Dino's bbox2d during inference. What is the purpose of training the pn+roi head to output bbox2d?
2.During training, Grounding Dino is not involved, but during inference and evaluation, Grounding Dino's bbox2d is used. Therefore, the bbox2d output by Grounding Dino is uncontrollable and will inevitably affect the 3D results. Why not train the bbox2d output by Grounding Dino to make it more accurate?
3.When evaluation, I want to use ground truth 2d for evaluation, however, even if I add gt oracle2D information using the mergr_oracle2d_to_detection_dicts function. When inference, the code still uses the bbox2d output from Grounding Dino. How can I replace the bbox2d output from Grounding Dino with gt bbox2d during inference?

[yaojin17]

Sorry for the late response.

1. The rpn + roi head preserves the detection ability on base categories. During inference, it can choose to use the box predicted by itself rather than Grounding DINO.
2. Our computational resources are constrained, so we just use the pretrained Grounding DINO. Also we want to prevent the model's performance degradation on in-the-wild images after fine-tuning on Omni3D.
3. The path of oracle 2D box file is set here. You can modify it to your path of ground truth 2D.

[wuhuateng]

thank you so much~