result of train wan-animate by rcm

[renrenzsbbb]

Thanks for your great work. I find that when I adjust your code to train wan-animate, the result is bad than without training rcm.

https://github.com/user-attachments/assets/a7ad2ef8-4a74-4dd0-b5b8-f1906391666d
directly inference 4 step without training

https://github.com/user-attachments/assets/7f105535-e6b1-4d43-a30d-fbef16dc4a4d
train 15 step

https://github.com/user-attachments/assets/fb18c653-8852-4d25-b7d4-cd893589a65d
train 1500step

I set warmup to 100 and train in 8*A800 gpu, context_parallel_size=8 and state_t=8.
can you introduce me some advice? thanks in advance

[zhengkw18]

1. You can align your code with our newest version of training
2. JVP computation is complex. To avoid potential bugs, you can first try disabling the sCM loss (by setting config.loss_scale=0) and distill with pure DMD.
3. state_t=8 only corresponds to less than 1 second, but your inference has 14 seconds. It is better to align your training data and time length with your inference settings.

[wzr0108]

Have you solved this issue? I’m encountering a similar problem as well.

[zhengkw18]

I have added a discrete-time CM implementation https://github.com/NVlabs/rcm/blob/main/rcm/models/t2v_model_distill_rcm.py#L591, feel free to try it out by setting model.config.cm_type=dcm. In some cases, discrete-time CM performs more stably than sCM.