cs.CVApr 20, 2026

UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models

Authors: Jiaqi WangHaoge DengTing PanYang LiuChengyuan WangFan ZhangYonggang QiXinlong Wang

Organizations: Beijing University of Posts and Telecommunications · Beijing Academy of Artificial Intelligence

Abstract

Uniform Discrete Diffusion Model (UDM) has recently emerged as a promising paradigm for discrete generative modeling; however, its integration with reinforcement learning remains largely unexplored. We observe that naively applying GRPO to UDM leads to training instability and marginal performance gains. To address this, we propose UDM-GRPO, the first framework to integrate UDM with RL. Our method is guided by two key insights: (i) treating the final clean sample as the action provides more accurate and stable optimization signals; and (ii) reconstructing trajectories via the diffusion forward process better aligns probability paths with the pretraining distribution. Additionally, we introduce two strategies, Reduced-Step and CFG-Free, to further improve training efficiency. UDM-GRPO significantly improves base model performance across multiple T2I tasks. Notably, GenEval accuracy improves from 69%69\% to 96%96\% and PickScore increases from 20.4620.46 to 23.8123.81, achieving state-of-the-art performance in both continuous and discrete settings. On the OCR benchmark, accuracy rises from 8%8\% to 57%57\%, further validating the generalization ability of our method. Code is available at https://github.com/Yovecent/UDM-GRPO.

Explore similar work

CardsList