RL for Diffusion Models

RL: Reinforcement Learning

Ask about this topic

Overview

The field studies how to adapt diffusion generators using task rewards, human preferences, or environment feedback rather than relying only on imitation or supervised denoising. Common approaches treat denoising steps as policy actions and use PPO or GRPO policy gradients, reward-weighted flow or score matching, distribution matching, and critic-guided control; distillation can make these updates cheaper. Applications span image and video alignment, reasoning in discrete diffusion language models, and robot policies, evaluated on benchmarks such as GenEval, GSM8K, and MuJoCo tasks. Key tensions include estimating useful learning signals across denoising steps, preventing reward hacking without losing diversity or fidelity, and achieving gains without expensive rollouts.

Latest papers

CardsList

182 papers27 papers in the last four weeks, up 80% on the four weeks before. 0.2% of all new papers.

  1. GDPO-SR: Group Direct Preference Optimization for One-Step Generative Image Super-Resolution

    Mar 16, 2026Qiaosi Yi, Shuai Li, Rongyuan Wu +3Image Super-ResolutionDirect Preference Optimization

    The Hong Kong Polytechnic University · OPPO Research Institute

  2. Diffusion Reinforcement Learning via Centered Reward Distillation

    Mar 14, 2026Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière +1RL for Diffusion ModelsDiffusion Model Alignment

    LIX, École Polytechnique, CNRS, IPP · Inria at Univ. Grenoble Alpes, CNRS, LJK

  3. REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning

    Mar 14, 2026Zhaoyuan Gu, Yipu Chen, Zimeng Chai +11Humanoid Loco-ManipulationDiffusion Policy

    Institute for Robotics and Intelligent Machines, Georgia Institute of Technology, Atlanta, GA 30332 USA

  4. Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models

    Mar 13, 2026David McAllister, Miika Aittala, Tero Karras +4RL for Image GenerationRL for Diffusion Models

    UC Berkeley · NVIDIA

  5. Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages

    Mar 13, 2026Vishnu Teja Kunde, Fatemeh Doudi, Mahdi Farahbakhsh +3RL for Diffusion ModelsDiffusion Language Model Training

    Department of Electrical & Computer Engineering, Texas A&M University, College Station, TX 77843

  6. GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learning

    Mar 10, 2026Haitong Ma, Chenxiao Gao, Tianyi Chen +2Reinforcement LearningRL for Diffusion Models

    Harvard University · Georgia Institute of Technology

  7. TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward

    Mar 8, 2026Yihong Luo, Tianyang Hu, Weijian Luo +1RL for Diffusion ModelsReinforcement Learning

    Hong Kong University of Science and Technology · The Chinese University of Hong Kong, Shenzhen · hi-Lab, Xiaohongshu Inc +1

  8. Diffusion Controller: Framework, Algorithms and Parameterization

    Mar 7, 2026Tong Yang, Moonkyung Ryu, Chih-Wei Hsu +4Diffusion Model AdaptationDiffusion Model Fine-Tuning

    Carnegie Mellon University; Intern at Google Research · Google · Yale University

  9. Optimizing 3D Diffusion Models for Medical Imaging via Multi-Scale Reward Learning

    Mar 6, 2026Yueying Tian, Xudong Han, Meng Zhou +3Medical Image Synthesis3D Diffusion Models

    University of Sussex · University of Toronto

  10. Stabilizing Reinforcement Learning for Diffusion Language Models

    Mar 6, 2026Jianyuan Zhong, Kaibo Wang, Ding Ding +5RL for Diffusion ModelsReinforcement Learning

    Huawei Foundation Model Department · The Chinese University of Hong Kong · The Hong Kong University of Science and Technology

  11. Diffusion Policy through Conditional Proximal Policy Optimization

    Mar 5, 2026Ben Liu, Shunpeng Yang, Hua ChenDiffusion PolicyRL for Diffusion Models

    Southern University of Science and Technology, Shenzhen, China · Limx Dynamics · Hong Kong University of Science and Technology, Hongkong, China +1

  12. LightningRL: Breaking the Accuracy-Parallelism Trade-off of Block-wise dLLMs via Reinforcement Learning

    Mar 4, 2026Yanzhe Hu, Yijie Jin, Pengfei Liu +2Parallel Decoding for Diffusion Language ModelsBlockwise Diffusion Language Models

    Shanghai Jiao Tong University · Huazhong University of Science and Technology

  13. Investigating Group Relative Policy Optimization for Diffusion Transformer based Text-to-Audio Generation

    Mar 2, 2026Yi Gu, Yanqing Liu, Chen Yang +1Text-to-Audio GenerationRL for Diffusion Models

    Tsinghua University · Microsoft

  14. Efficient Token-Level Policy Optimization for Masked Diffusion Models via Velocity Alignment

    Mar 2, 2026Chenxing Wei, Jiazhen Kang, Hong Wang +8RL for Diffusion ModelsPolicy Optimization

    Shenzhen University, China · Guangming Laboratory, China · ByteDance, China +5

  15. BeautyGRPO: Aesthetic Alignment for Face Retouching via Dynamic Path Guidance and Fine-Grained Preference Modeling

    Mar 1, 2026Jiachen Yang, Xianhui Lin, Yi Dong +4Image EditingRL for Image Editing

    School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, China · vivo BlueImage Lab, vivo Mobile Communication Co., Ltd., China · Guangdong Provincial Key Laboratory of Information Security Technology, China

  16. Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion

    Feb 24, 2026Sathwik Karnik, Juyeop Kim, Sanmi Koyejo +2Memorization in Generative ModelsT2I Diffusion Models

    Stanford University, Stanford, CA, USA · Yonsei University, Seoul, Korea

  17. DGPO: RL-Steered Graph Diffusion for Neural Architecture Generation

    Feb 22, 2026Aleksei Liuliakov, Luca Hermes, Barbara HammerGraph GenerationGraph Diffusion

    Machine Learning Group, Bielefeld University, Germany

  18. RIDER: 3D RNA Inverse Design with Reinforcement Learning-Guided Diffusion

    Feb 18, 2026Tianmeng Hu, Yongzheng Cui, Biao Luo +1Inverse Design3D Diffusion Models

    University of Exeter · Central South University

  19. Training-Free Adaptation of Diffusion Models via Doob's hh-Transform

    Feb 18, 2026Qijie Zhu, Zeqi Ye, Han Liu +2Diffusion Model AdaptationDiffusion Models

    Department of Statistics and Data Science, Northwestern University. · Department of Industrial Engineering and Management Sciences, Northwestern University. · Department of Computer Science, Northwestern University.

  20. LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models

    Feb 15, 2026Shufan Li, Yuchen Zhu, Jiuxiang Gu +6Unified Multimodal ModelsMultimodal Reasoning

    1Adobe · 2UCLA · 3Georgia Tech

  21. Horizon Imagination: Efficient On-Policy Rollout in Diffusion World Models

    Feb 8, 2026Lior Cohen, Ofir Nabati, Kaixin Wang +2Reinforcement LearningWorld Model Learning

    Technion · Microsoft Research

  22. Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design

    Feb 4, 2026Jaemoo Choi, Yuchen Zhu, Wei Guo +6T2I GenerationDiffusion Model Fine-Tuning

    Georgia Institute of Technology · National University of Singapore · Nanjing university.

  23. ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedule

    Jan 26, 2026Yilie Huang, Wenpin Tang, Xunyu ZhouDiffusion SamplingAdaptive Sampling

    Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027, USA. · Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University, New York, NY 10027, USA.

  24. ReDiF: Resource-Efficient Few-Step Diffusion Distillation via Reinforcement Learning

    Dec 28, 2025Amirhossein Tighkhorshid, Zahra Dehghanian, Hamid R. RabieeDiffusion DistillationRL for Diffusion Models

    Computer Engineering Sharif University of Technology

  25. Symmetry-Aware Steering of Equivariant Diffusion Policies: Benefits and Limits

    Dec 12, 2025Minwoo Park, Junwoo Chang, Jongeun Choi +1Diffusion PolicyRL for Diffusion Models

    Yonsei University, Seoul 03722, Republic of Korea · University of California, Berkeley, CA 94720, USA

  26. Multi-dimensional Preference Alignment by Conditioning Reward Itself

    Dec 11, 2025Jiho Jang, Jinyoung Kim, Kyungjune Baek +1Diffusion Model AlignmentDiffusion Model Guidance

    Seoul National University · Sejong University

  27. Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning

    Dec 1, 2025Sebastian Sanokowski, Kaustubh Patil, Majid KhadivRL ControlDiffusion Policy

    Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University Munich · Practical Project Student Exchange Program, Technical University Munich · MIT World Peace University

  28. Distribution Matching Distillation Meets Reinforcement Learning

    Nov 17, 2025Dengyang Jiang, Dongyang Liu, Zanyi Wang +12Diffusion Model AlignmentDiffusion Model Distillation

    1HKUST · 2Alibaba Group · 3SIAT, CAS +3

  29. Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization

    Oct 9, 2025Yuchen Zhu, Wei Guo, Jaemoo Choi +4RL for Language Model ReasoningPolicy Optimization

    Georgia Institute of Technology.

  30. Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models

    Sep 29, 2025Shuchen Xue, Chongjian Ge, Shilong Zhang +2Score MatchingDiffusion Model Alignment

    UCAS · Adobe Research · HKU +1