Difficulty Reproducing Reported Results on Aircraft Dataset with 5-Shot Setting

[happy-emart]

Hello Diff-II Research Team,

First and foremost, I want to express my sincere admiration for your Diff-II paper presented at CVPR 2025. The overall idea of your paper struck me as a very creative idea and was fun to read. Thank you for writing such a good paper.

So, I’ve been working to replicate the results outlined in your paper using the code provided here, specifically focusing on the Aircraft dataset with a 5-shot setting. Despite my efforts, I’ve encountered challenges in achieving the performance metrics reported, and I’d like to kindly ask for your guidance to ensure I’m not missing any critical steps.

I started by following the default instructions in the repository to run the 5-shot experiment on the Aircraft dataset.
However, the results I obtained didn’t align with the performance reported in your paper.

To troubleshoot, I adjusted the following settings:

• set inversion_steps to 25 on executing get_inversion.py
• set strength(i assumed it is a split ratio) to 0.3 on executing interpolation_le.py
• set syn_p(probably sampling rate) to 0.5 on executing train_classifier.py

Even after applying these changes and retraining, the results still fell short of what was reported. I’m concerned that I might have overlooked some additional configurations or steps necessary for successful reproduction.

Could you please assist me in pinpointing any errors or omissions in my approach? Here is my environment below:

Python version: 3.9.21
PyTorch version: 2.4.0
CUDA version: 12.6

If there are particular dependencies, environment setups, or subtle configurations critical to replicating your results, I’d greatly appreciate it if you could share those details.

Thank you once again for your outstanding contribution to the field—I’m truly excited to continue exploring it!

[scuwyh2000]

Hello Diff-II Research Team,

First and foremost, I want to express my sincere admiration for your Diff-II paper presented at CVPR 2025. The overall idea of your paper struck me as a very creative idea and was fun to read. Thank you for writing such a good paper.

So, I’ve been working to replicate the results outlined in your paper using the code provided here, specifically focusing on the Aircraft dataset with a 5-shot setting. Despite my efforts, I’ve encountered challenges in achieving the performance metrics reported, and I’d like to kindly ask for your guidance to ensure I’m not missing any critical steps.

I started by following the default instructions in the repository to run the 5-shot experiment on the Aircraft dataset. However, the results I obtained didn’t align with the performance reported in your paper.

To troubleshoot, I adjusted the following settings:

• set inversion_steps to 25 on executing get_inversion.py
• set strength(i assumed it is a split ratio) to 0.3 on executing interpolation_le.py
• set syn_p(probably sampling rate) to 0.5 on executing train_classifier.py

Even after applying these changes and retraining, the results still fell short of what was reported. I’m concerned that I might have overlooked some additional configurations or steps necessary for successful reproduction.

Could you please assist me in pinpointing any errors or omissions in my approach? Here is my environment below:

Python version: 3.9.21 PyTorch version: 2.4.0 CUDA version: 12.6

If there are particular dependencies, environment setups, or subtle configurations critical to replicating your results, I’d greatly appreciate it if you could share those details.

Thank you once again for your outstanding contribution to the field—I’m truly excited to continue exploring it!

Hey, thanks for following.

In the scripts/run.sh file, I mistakenly set the syn_p = 0 and split ratio s=1.0.

The correct parameters are mentioned in the paper: for the 5-shot aircraft
syn_p = 0.5
s = 0.3
inversion_steps = 200 for get_inversion.py and inversion_steps = 25 for interpolation_le.py

And the device is 8 3090 GPUs.

Try again, if you have any new questions, feel free to contact me by email or GitHub issue.

[happy-emart]

Thanks for detailed reply 😀 But, still I have some questions.

The correct parameters are mentioned in the paper: for the 5-shot aircraft syn_p = 0.5 s = 0.3 inversion_steps = 200 for get_inversion.py and inversion_steps = 25 for interpolation_le.py

I just tried a few settings for hyperparameters on your implementation, and get a result of about 47% of top-1 accuracy at below setting, which is much higher than the accuracy of 5-shot baseline sugges ted on the Diff-II paper(paper suggests 36%).

Dataset: Aircraft, and only 5 images per class (5-shot setting)
Model: ResNet50 (pretrained, served by Pytorch, unchanged from initial implementation)
Batch size: 8
Num of epoch: 150
Optimizer: SGD
Decay weight: 5e-4
Learning rate: 0.005

and with different setting adopting Adam optimizer, I could achieve similar accuracy.

Then, according to your answer above, I tried the original Diff-II method like this:

Image generation: Utilized "interpolation.py", with inversion step 200 and split ratio("strength" in the code) 0.3
Training Model: use same pretrained ResNet50 provided by Pytorch
(I referred Diff-Mix paper to set below figures)
Batch size: 64
Num of epoch: 128
Optimizer: SGD
Decay weight: 5e-5
Learning rate: 0.02

and you can see the graph here

Blue graph is first setting that I described in this post and red graph is the second setting that I described.

Could you please, explain what I am doing wrong?

[scuwyh2000]

Thanks for detailed reply 😀 But, still I have some questions.

The correct parameters are mentioned in the paper: for the 5-shot aircraft syn_p = 0.5 s = 0.3 inversion_steps = 200 for get_inversion.py and inversion_steps = 25 for interpolation_le.py

I just tried a few settings for hyperparameters on your implementation, and get a result of about 47% of top-1 accuracy at below setting, which is much higher than the accuracy of 5-shot baseline sugges ted on the Diff-II paper(paper suggests 36%).

Dataset: Aircraft, and only 5 images per class (5-shot setting) Model: ResNet50 (pretrained, served by Pytorch, unchanged from initial implementation) Batch size: 8 Num of epoch: 150 Optimizer: SGD Decay weight: 5e-4 Learning rate: 0.005

and with different setting adopting Adam optimizer, I could achieve similar accuracy.

Then, according to your answer above, I tried the original Diff-II method like this:

Image generation: Utilized "interpolation.py", with inversion step 200 and split ratio("strength" in the code) 0.3 Training Model: use same pretrained ResNet50 provided by Pytorch (I referred Diff-Mix paper to set below figures) Batch size: 64 Num of epoch: 128 Optimizer: SGD Decay weight: 5e-5 Learning rate: 0.02

and you can see the graph here

Blue graph is first setting that I described in this post and red graph is the second setting that I described.

Could you please, explain what I am doing wrong?

You can try the following hyperparameters:

1/ inversion: step = 200 and guidance strength is 1.0.
2/ generate: step = 25, guidance strength is 7.5, split ratio = 0.3.
3/ classification:

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --nproc_per_node=8 train_classifier.py
--pretrained
--arch="resnet50"
--epochs=128
--batch_size=256
--lr=0.1
--size=224
--seed=2020
--syn_p=0.5
--resize=256
--syn_dir='syn/aircraft/5shot/ours_0.3'
--datasets='aircraft'
--num_class=100 \

By the way, please tell me which result of the paper is 36% since you said ''paper suggests 36%'', you mean the ''original'' baseline?