Large performance gap between my evaluation and results in the technical report

[madorism]

Hello,

I am currently evaluating the new MedGemma model on MIMIC-CXR report generation, but I observed a large performance gap compared to the results reported in the technical report.

I have tried to replicate the data cleaning and preprocessing described in the paper as closely as possible： Only use examples in the test set with non-empty findings, impressions, and indication sections. Use pretrained model instead of IT model for report generation task. Follow the prompt in the technical report.

Despite following these steps, my results differ significantly, where I got lower than 24 for radgraph F1 score.
Also I noted many repetition and even nonsense content.

Could you please clarify:
Which exact test set was used for the reported evaluation?
Were there any additional preprocessing steps not mentioned in the technical report?
Is it possible to access the exact evaluation split you used?

Thank you for your time and for releasing this work — it’s very valuable for the medical AI community.
Best regards,

[HuangChiEn]

same issue in text-only benchmark.
medgemma also s1k test-time scaling to boost the testing score, it may be 1 of gap to reproduced the results.

Even this issue, i think it still have gap between our reproduction & google internal production.