About Diffu-GRPO

[lkevinzc]

Hi team, thanks for the great work—seeing dLLM is RL-able is super exciting!

I noticed in your GRPO loss equation (screenshot below) that the loss is normalized by the response length |o|:

As we have discussed in sail-sg/understand-r1-zero, dividing by |o| can introduce a length bias during optimization. Given that dLLM operates with a fixed diffusion step T (more like fixed-horizon RL with eos being the absorbing state), could you clarify whether you're normalizing by |o| or by T in your actual implementation?

Also, do you expect this normalization choice to affect dLLM's behavior (e.g., output length) in a similar way to what we observed in Section 3.2 of this paper?

Thanks again for the awesome work!

[devaansh100]

Thank you for your interest in our work!

Given that dLLM operates with a fixed diffusion step T (more like fixed-horizon RL with eos being the absorbing state), could you clarify whether you're normalizing by |o| or by T in your actual implementation?

If I understand you correctly, you are asking if we normalize by number of effective tokens (|o|) or by the preset sequence length? If yes, we are doing the former.

do you expect this normalization choice to affect dLLM's behavior

We actually don't observe much of a trend between the length of right and wrong answers. This could be a consequence of the fact that we used a fixed, short sequence length of 256 for all the completions.

But I think we would need to run some ablations without the |o| to confirm this!

[lkevinzc]

@devaansh100 Thanks for your reply and looking forward to your results!

[BradKML]

@devaansh100 I am thinking of a few weird thoughts:

1. There are a few tools out there to make DLLMs behave more like AR-LLM, so it means de-biasing would be a good idea https://github.com/kuleshov-group/bd3lms https://github.com/Li-Jinsong/DAEDAL https://github.com/maomaocun/dLLM-cache https://github.com/LiangrunFlora/Slow-Fast-Sampling
2. Does the following paper correct for diffusion models with "incentivising correct reasoning"? https://arxiv.org/html/2506.14245v1
3. Have alternatives to Dr. GRPO been tried? https://arxiv.org/html/2503.14476v1 https://arxiv.org/html/2501.03262v1
4. Are "intuitors" and implicit rewards a viable way of stabilizing DLLM? https://arxiv.org/abs/2505.19590