RL for Diffusion Models

RL: Reinforcement Learning

Overview

The field studies how to adapt diffusion generators using task rewards, human preferences, or environment feedback rather than relying only on imitation or supervised denoising. Common approaches treat denoising steps as policy actions and use PPO or GRPO policy gradients, reward-weighted flow or score matching, distribution matching, and critic-guided control; distillation can make these updates cheaper. Applications span image and video alignment, reasoning in discrete diffusion language models, and robot policies, evaluated on benchmarks such as GenEval, GSM8K, and MuJoCo tasks. Key tensions include estimating useful learning signals across denoising steps, preventing reward hacking without losing diversity or fidelity, and achieving gains without expensive rollouts.

Latest papers

CardsList

192 papers11 papers in the last four weeks, down 65% on the four weeks before. 0.1% of all new papers.

  1. When Anchors Shift the Target: Advantage-Gated Adaptation for Diffusion Reinforcement Learning

    Nov 17, 2026Pu Cao, Min Zhou, Ying Shu +3Diffusion Model AdaptationRL for Diffusion Models

  2. ReMaRL: Learning to Unmask and Remask in Frozen Diffusion Language Models

    Nov 17, 2026Egor Shibaev, Danila Biktimirov, Ivan Klimov +3Diffusion Language Model InferenceDiffusion Language Model Decoding

  3. RiDEx: Reward-Informed Directional Exploration for Diffusion Reinforcement Learning

    Nov 17, 2026Song Yan, Chenfeng Wang, Wei Zhai +6RL for Diffusion ModelsDiffusion Model Alignment

  4. See Every Block Once: Variance-Reduced Post-Training for Masked Diffusion Language Models

    Nov 13, 2026Yunzhen He, Hiroaki Yamagiwa, Hidetoshi ShimodairaDiffusion Language Model TrainingMasked Diffusion Models

  5. FEROM: Frontier Endogenous Reveal-Order Marginal Policy Optimization for Masked Diffusion LMs

    Nov 7, 2026Zian Su, Ziyang Huang, Kaiyuan Zhang +1Masked Diffusion ModelsDiffusion Language Model Training

  6. FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning

    Nov 5, 2026Haitao Lin, Minsi Ren, Yang Li +4Diffusion Model Fine-TuningRL for Diffusion Models

  7. PBPO: Prefix-Block Policy Optimization for Diffusion Language Models

    Nov 5, 2026Katherine Tieu, Dongqi Fu, Si Zhang +6Diffusion Language Model TrainingRL for Diffusion Models

  8. Skillful and realistic precipitation forecasting with reinforced diffusion

    Oct 28, 2026Ozan Sener, Anand Avati, David Hafner +7Precipitation ForecastingDiffusion Models

  9. DTCA: Denoising Token Credit Assignment for Reinforcement Learning in Diffusion Language Models

    Oct 26, 2026Hongze Chen, Miao Pan, Lizhou Liang +7Diffusion Language Model TrainingRL for Diffusion Models

  10. Doob-Based Generative Policy Optimization

    Oct 20, 2026Maojiang Su, Qijie Zhu, Yu Xiao +4Policy OptimizationRL for Diffusion Models

  11. DualHeads: Reward-tilted few-step diffusion made simple

    Oct 20, 2026Yuanzhi Zhu, Eleftherios Tsonis, Simo Alami Chehboune +3Diffusion-Based Image GenerationDiffusion Model Fine-Tuning

  12. Beyond the Sequentiality Bottleneck: A Hybrid Diffusion-Autoregressive Framework for Efficient Multimodal Reasoning

    Oct 20, 2026Wei Cai, Jian Zhao, Yuchen Yuan +1Hybrid Language ModelsParallel Reasoning in Language Models

  13. Distribution-Based Regularization for Reward Fine-Tuning Diffusion and Flow Models

    Oct 17, 2026Romeo Passaro, Jeongjae Lee, Jong Chul Ye +4Generative ModelingDiffusion Model Alignment

  14. DiffusionRVGE: Reinforcement Learning via Regression-Based Value Gradient Estimation

    Oct 16, 2026Jiayuan Wang, Tongyao PangRL for Image GenerationRL for Diffusion Models

  15. Routing Policy Optimization for Online Diffusion Reinforcement Learning

    Oct 15, 2026Qiuhong Shen, Yiding Yang, Yuxin Zhang +8RL for Image GenerationRL for Diffusion Models

  16. Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond

    Oct 15, 2026Zhiyuan Ma, Jiabao Wei, Tianyu Wang +21RL for Diffusion ModelsRL for Generative Models

  17. ForkGRPO: Efficient Exploration and Fine-Grained Credit Assignment for Diffusion Language Models

    Oct 14, 2026Zhaoyan Jia, Chengwei Zhou, Gourav Datta +1RL for Language Model ReasoningRL for Diffusion Models

  18. A Few More Tokens Ahead: Reinforcing Parallelism in Diffusion LLMs

    Oct 13, 2026Jiahao Kuang, Yangfan Gao, Tao Gui +4Parallel Decoding for Diffusion Language ModelsParallel Decoding

  19. CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance

    Oct 10, 2026Shutong Ding, Zhaoyang Liu, Yimiao Zhou +3Diffusion PolicyRobotic RL

  20. DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models

    Oct 8, 2026Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang +6Diffusion Model Fine-TuningRL for Diffusion Models

  21. Reachability-Aware Diffusion Policy Optimization

    Oct 5, 2026Hikmet Simsir, Kutay Demiray, Ozgur S. OguzConstrained RLDiffusion Policy

  22. MGPO: Manifold-Guided Diffusion Alignment for Task-Aware Dataset Distillation

    Oct 4, 2026Yunyi Chen, Chenru Wang, Xinyi Ye +2Diffusion Model AlignmentDataset Distillation

  23. IDRF: Inverse-Distilled Reward Fine-tuning of Masked Discrete Diffusion Models

    Oct 2, 2026Vladislav Gromadskii, David Li, Samson Gourevitch +4Diffusion Model Fine-TuningMasked Diffusion Models

  24. Token-Level Video Reinforcement Learning

    Oct 1, 2026Yifan Wang, Gordon Guocheng Qian, Yanyu Li +2Video Diffusion ModelsRL for Video Generation

  25. iADD: Improving Alignment and Diversity in Diffusion Policy Optimization

    Oct 1, 2026Ashok Prasad Neupane, Saugat Adhikari, Pramish Paudel +2Diffusion Model AlignmentGenerative Modeling