RL for Diffusion Models

RL: Reinforcement Learning

The field studies how to adapt diffusion generators using task rewards, human preferences, or environment feedback rather than relying only on imitation or supervised denoising. Common approaches treat denoising steps as policy actions and use PPO or GRPO policy gradients, reward-weighted flow or score matching, distribution matching, and critic-guided control; distillation can make these updates cheaper. Applications span image and video alignment, reasoning in discrete diffusion language models, and robot policies, evaluated on benchmarks such as GenEval, GSM8K, and MuJoCo tasks. Key tensions include estimating useful learning signals across denoising steps, preventing reward hacking without losing diversity or fidelity, and achieving gains without expensive rollouts.

Latest papers

182 papers27 papers in the last four weeks, up 80% on the four weeks before. 0.2% of all new papers.

DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
May 8, 2026Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade +4Group Relative Policy Optimization Credit Assignment in RL
The Ohio State University · Apple
MARBLE: Multi-Aspect Reward Balance for Diffusion RL
May 7, 2026Canyu Zhao, Hao Chen, Yunze Tong +3Multi-Objective Reinforcement Learning Diffusion Model Alignment
Zhejiang University · Work done during an internship at HiThink. · HiThink +1
MotionGRPO: Overcoming Low Intra-Group Diversity in GRPO-Based Egocentric Motion Recovery
May 7, 2026Nanjie Yao, Junlong Ren, Wenhao Shen +1Diffusion Model Guidance Markerless Motion Capture
The Hong Kong University of Science and Technology (Guangzhou), China · Nanyang Technological University, Singapore
Leveraging Verifier-Based Reinforcement Learning in Image Editing
Apr 30, 2026Hanzhong Guo, Jie Wu, Jie Liu +6Reward Modeling Reinforcement Learning
School of Computing and Data Science, The University of Hong Kong · ByteDance Seed · CUHK +1
AdvDMD: Adversarial Reward Meets DMD For High-Quality Few-Step Generation
Apr 29, 2026Xu Wang, Zexian Li, Litong Gong +2Diffusion Model Distillation Few-Step Diffusion Sampling
Shanghai Jiao Tong University · Alimama Tech
A Systematic Post-Train Framework for Video Generation
Apr 28, 2026Zeyue Xue, Siming Fu, Jie Huang +9Diffusion Model Alignment Video Diffusion Models
The University of Hong Kong · JD Explore Academy · Tsinghua University +2
V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think
Apr 25, 2026Bingda Tang, Yuhui Zhang, Xiaohan Wang +3Diffusion Model Alignment Group Relative Policy Optimization
Stanford University · Tsinghua University · Amazon FAR +1
ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control
Apr 22, 2026Shelly Golan, Michael Finkelson, Ariel Bereslavsky +2Multi-Objective Reinforcement Learning RL for Image Generation
Tel Aviv University · Lightricks · Adobe Research
FASTER: Value-Guided Sampling for Fast RL
Apr 21, 2026Perry Dong, Alexander Swerdlow, Dorsa Sadigh +1Diffusion Sampling Diffusion Policy
Stanford University
HP-Edit: A Human-Preference Post-Training Framework for Image Editing
Apr 21, 2026Fan Li, Chonghuinan Wang, Lina Lei +9Diffusion Model Alignment Image Editing Evaluation
Huawei Noah’s Ark Lab · Nankai University · Harbin Institute of Technology
Learning to Credit the Right Steps: Objective-aware Process Optimization for Visual Generation
Apr 21, 2026Rui Li, Ke Hao, Yuanzhi Liang +4Multi-Objective Reinforcement Learning Video Diffusion Models
University of Science and Technology of China HeFei, China · Shanghai Jiao Tong University ShangHai, China · TeleAI ShangHai, China
Guiding Distribution Matching Distillation with Gradient-Based Reinforcement Learning
Apr 21, 2026Linwei Dong, Ruoyu Guo, Ge Bai +3Diffusion Model Distillation Few-Step Diffusion Sampling
1Zhejiang University · 2Bytedance Inc. · 3Zhejiang Lab
Discrete Tilt Matching
Apr 20, 2026Yuyuan Chen, Shiyi Wang, Peter Potaptchik +2Masked Diffusion Models RL for Language Model Reasoning
1Harvard University · University of Oxford · 3IAIFI +1
UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models
Apr 20, 2026Jiaqi Wang, Haoge Deng, Ting Pan +5T2I Generation Group Relative Policy Optimization
Beijing University of Posts and Telecommunications · Beijing Academy of Artificial Intelligence
RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework
Apr 16, 2026Hao Gao, Shaoyu Chen, Yifan Zhu +4Diffusion-Based Planning RL for Diffusion Models
Huazhong University of Science & Technology · Horizon Robotics
Step-level Denoising-time Diffusion Alignment with Multiple Objectives
Apr 15, 2026Qi Zhang, Dawei Wang, Shaofeng ZouDiffusion Model Alignment RL for Diffusion Models
School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ 85281 USA
NaP-Control: Navigating Diffusion Prior for Versatile and Fast Character Control
Apr 15, 2026Chia-Wen Chen, Yan Wu, Korrawe Karunratanakul +1Diffusion Policy Diffusion Models for Motion Generation
ETH Zurich
Advantage-Guided Diffusion for Model-Based Reinforcement Learning
Apr 10, 2026Daniele Foffano, Arvid Eriksson, David Broman +2Model-Based RL Diffusion Model Guidance
Division of Decision and Control Systems of the EECS School at KTH Royal Institute of Technology, Stockholm, Sweden. · Digital Futures · Division of Software and Computer Systems of the EECS School at KTH Royal Institute of Technology, Stockholm, Sweden.
MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation
Apr 8, 2026Xiaoxiao Ma, Jiachen Lei, Tianfei Ren +6RL for Image Generation Autoregressive Diffusion
University of Science and Technology of China, Hefei, China · AMAP, Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China
FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling
Apr 8, 2026Yitong Li, Junsong Chen, Shuchen Xue +8Diffusion Model Alignment RL for Diffusion Models
NVIDIA · HKU · MIT
DARE: Diffusion Large Language Models Alignment and Reinforcement Executor
Apr 5, 2026Jingyi Yang, Yuxian Jiang, Xuhao Hu +3Diffusion Language Models Diffusion Model Alignment
Shanghai Artificial Intelligence Laboratory · Fudan University · Zhejiang University +1
Stepwise Credit Assignment for GRPO on Flow-Matching Models
Mar 30, 2026Yash Savani, Branislav Kveton, Yuchen Liu +5RL for Diffusion Models Step-Level Credit Assignment in RL
Carnegie Mellon University · Adobe Research
$R_\text{dm}$ : Re-conceptualizing Distribution Matching as a Reward for Diffusion Distillation
Mar 30, 2026Linqian Fan, Peiqin Sun, Tiancheng Wen +2Diffusion Model Distillation Distribution Matching Distillation
KlingAI Research · Tsinghua University
FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies
Mar 29, 2026Chenxiao Gao, Edward Chen, Tianyi Chen +1Flow-Based Policy Learning Diffusion Policy
Georgia Institute of Technology
PhySe-RPO: Physics and Semantics Guided Relative Policy Optimization for Diffusion-Based Surgical Smoke Removal
Mar 24, 2026Zining Fang, Cheng Xue, Chunhui Liu +3Geometry-Guided Image Restoration Image Restoration
School of Computer Science and Engineering, Southeast University · Zhongda Hospital,Southeast University · School of Future Technology, South China University of Technology
Path-Space Mirror Descent for On-Policy Reinforcement Learning under the Generalized Schrödinger Bridge
Mar 23, 2026Yuehu Gong, Zeyuan Wang, Yulin Chen +3Policy Optimization Continuous Control
School of Data Science, Fudan University · Laboratory for Big Data and Decision, National University of Defense Technology · ShanghaiTech University +2
Reward Sharpness-Aware Fine-Tuning for Diffusion Models
Mar 22, 2026Kwanyoung Kim, Byeongsu SimDiffusion Model Alignment RL for Diffusion Models
Department of AI Convergence, GIST · Samsung Research
CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think
Mar 19, 2026Zening Sun, Zhengpeng Xie, Lichen Bai +3Diffusion Model Alignment Diffusion Model Fine-Tuning
HKUST (GZ) · HIT (SZ)
dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models
Mar 19, 2026Wenxuan Zhang, Lemeng Wu, Changsheng Zhao +11Diffusion Language Models RL for Diffusion Models
Meta AI · KAUST
SHIFT: Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuning
Mar 18, 2026Xi Ye, Wenjia Yang, Yangyang Xu +4Diffusion Model Alignment Video Diffusion Models
Tsinghua University · Ant Group · University of Chinese Academy of Sciences

May 8, 2026J/K move · Enter open · S save

DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade, Lokesh Boominathan+3

The Ohio State University · Apple

Group Relative Policy Optimization Credit Assignment in RL Step-Level Credit Assignment in RL Step-Level Credit Assignment RL for Diffusion Models Diffusion Language Models+1

Open paper AskPDF arXiv

Diffusion large language models are a compelling alternative to autoregressive models, yet existing RL methods for diffusion treat all denoising steps as equally important and rely on biased, high-variance likelihood estimates. We identify two fundamental weaknesses: the absence of temporal credit assignment across the denoising trajectory, and the systematic bias of mean-field likelihood estimates used for policy optimization. To address these, we propose Denoising-Aware Credit Assignment for GRPO (DACA-GRPO), a lightweight, plug-and-play enhancement for any GRPO-style trainer. DACA-GRPO introduces two complementary mechanisms: Denoising Progress Scores, which extract per-token importance weights from intermediate predictions at no additional forward cost, and Stratified Masking Likelihood, which partitions token positions into strata so that each token is predicted with most of the sequence as context, reducing the mean-field bias. Applied on top of three GRPO base methods, DACA-GRPO achieves consistent improvements across seven benchmarks spanning mathematical reasoning, code generation, constraint satisfaction, and constrained generation, with gains of up to 5.6pp on math reasoning, 7.4pp on code generation, 36.3pp on constraint satisfaction, and 5.9pp on JSON schema adherence.