RL for Diffusion Models

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 467% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 120

All topics

Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance
May 28, 2026Shutong Ding, Zejia Zhong, Zhongyi Wang +4Diffusion Guidance Robotic RL
1ShanghaiTech University.
GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models
May 28, 2026Xiaohang Tang, Keyue Jiang, Che Liu +4Diffusion Model Distillation RL for Diffusion Models
UNIST · University of Basel
Explicit Critic Guidance for Aligning Diffusion Models
May 26, 2026Zhengyang Liang, Qihang Zhang, Ceyuan YangDiffusion Model Alignment Diffusion Model Guidance
University of Toronto · Vector Institute · The Chinese University of Hong Kong
Reinforcing Few-step Generators via Reward-Tilted Distribution Matching
May 25, 2026Yushi Huang, Xiangxin Zhou, Ruoyu Wang +3Diffusion Model Alignment Diffusion Model Distillation
Tencent Hunyuan · Hong Kong University of Science and Technology · Westlake University
DRM: Diffusion-based Reward Model With Step-wise Guidance
May 25, 2026Jaxon Zhang, Binxin Yang, Hubery Yin +2Diffusion Model Alignment Diffusion Model Guidance
Peking University · Work done during an internship at WeChat Vision, Tencent Inc. · WeChat Vision, Tencent Inc.
Reinforcement Learning from Denoising Feedback
May 25, 2026Qi He, Huan Chen, Ya Guo +3RL for Language Model Reasoning RL for Diffusion Models
Fudan University · Ant Group · Hong Kong University of Science and Technology
Score-Based One-step MeanFlow Policy Optimization
May 22, 2026Kyungyoon Kim, Donghyeon Ki, Hee-Jun Ahn +1Flow Matching MeanFlow
Korea University, Decision Making Lab · Gauss Labs Inc.
Stitched Value Model for Diffusion Alignment
May 19, 2026Hyojun Go, Hyungjin Chung, Prune Truong +8Diffusion Model Alignment Diffusion Model Guidance
1ETH Zurich · 2Google · University of Copenhagen
Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL
May 18, 2026Junyi Wu, Weijian Luo, Haoyang Zheng +2Diffusion Model Alignment One-Step Generative Modeling
Purdue University · hi-lab, Xiaohongshu Inc.
SafeDiffusion-R1: Online Reward Steering for Safe Diffusion Post-Training
May 18, 2026Komal Kumar, Ankan Deria, Abhishek Basu +3Diffusion Model Alignment Diffusion Model Fine-Tuning
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · Michigan State University (MSU), USA
Fine-tuning Pocket-Aware Diffusion Models via Denoising Policy Optimization
May 17, 2026Yuan Xue, Daniel Kudenko, Megha KhoslaMolecular Optimization Diffusion Model Fine-Tuning
L3S Research Center, Hanover, Germany · Delft University of Technology, Delft, Netherlands
Beyond Execution: Static-Analysis Rewards and Hint-Conditioned Diffusion RL for Code Generation
May 16, 2026Shuyin Ouyang, Zhaozhi Qian, Faroq AL-Tam +2RL for Code Generation Diffusion Guidance
King’s College London · Elm Europe
Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Models
May 16, 2026Siqi Luo, Jianghan Shen, Yi Xin +9Hierarchical RL Group Relative Policy Optimization
1Shanghai Jiao Tong University · 2Shanghai Artificial Intelligence Laboratory · 3Nanjing University +2
Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization
May 15, 2026Xiaoxuan He, Siming Fu, Zeyue Xue +9Diffusion Model Alignment Video Diffusion Models
Zhejiang University · Joy Future Academy · Independent Researcher +1
Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?
May 15, 2026Renye Yan, Jikang Cheng, Shikun Sun +7Diffusion Model Alignment Diffusion Sampling
1Peking University · 2Tsinghua University · 3Nanjing University +1
Video Models Can Reason with Verifiable Rewards
May 14, 2026Tinghui Zhu, Sheng Zhang, James Y. Huang +5Video Diffusion Models RL for Video Generation
University of California, Davis · Microsoft Research · University of Southern California +1
Controllable Molecular Generative Foundation Models
May 14, 2026Yihan Zhu, Yuhan Liu, Weijiang Li +2Molecular Optimization Molecular Graph Generation
University of Notre Dame
RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO
May 14, 2026Yanzuo Lu, Ronglai Zuo, Jiankang DengAutoregressive Diffusion Long-Horizon Video Generation
Imperial College London
CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL
May 14, 2026Zhenyang Ni, Yijiang Li, Ruochen Jiao +7Video Diffusion Models Reinforcement Learning
Northwestern University · University of California, San Diego · University of Oxford
Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Models
May 13, 2026Saba Ahmadi, Prasanna Parthasarathi, Yufei CuiRL for Language Model Reasoning RL for Diffusion Models
Noah’s Ark Lab
Image AID via continuous-time reinforcement learning
May 13, 2026Yilie Huang, Xun Yu ZhouDiffusion Model Guidance Masked Diffusion Models
Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027, USA. · Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University, New York, NY 10027, USA.
OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation
May 12, 2026Guohui Zhang, XiaoXiao Ma, Jie Huang +9Audio-Video Generation Audio-Visual Synchronization
University of Science and Technology of China · JD Explore Academy · Peking University
Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Models
May 12, 2026Yan Jiang, Ruihong Qiu, Zi HuangRL Benchmarks Reinforcement Learning
The University of Queensland
Efficient Adjoint Matching for Fine-tuning Diffusion Models
May 12, 2026Jeongwoo Shin, Dongsoo Shin, Yuchen Zhu +5Diffusion Model Alignment Stochastic Optimal Control
Seoul National University · Georgia Institute of Technology · Sungkyunkwan University
Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping
May 11, 2026Haoyuan Sun, Jing Wang, Yuxin Song +9Reward Hacking Preference Optimization
Nanyang Technological University · Baidu Inc. · Zhejiang University +3
Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models
May 11, 2026Andreas Bergmeister, Stefanie Jegelka, Nikolas Nüsken +2Diffusion Model Alignment Flow Matching
1TU Munich, MCML · 2MIT CSAIL · 3King’s College London +2
Relative Score Policy Optimization for Diffusion Language Models
May 11, 2026Zichao Yu, Shengze Xu, Bingqing Jiang +2RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
University of Science and Technology of China · The Chinese University of Hong Kong · The University of Hong Kong
TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment
May 9, 2026Jiaming Li, Chenyu Zhu, Nanxi Yi +9Diffusion Model Alignment Reward Hacking
1MAIR Lab, Huazhong University of Science and Technology · 2Kuaishou Technology · 3Nanyang Technological University +2
NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Models
May 8, 2026Wen Huang, Haoran Sun, Yongjian Guo +8Action-Conditioned Video Generation Action-Conditioned World Models
Tsinghua University · Peking University · JDT AI Infra +1
Guidance Is Not a Hyperparameter: Learning Dynamic Control in Diffusion Language Models
May 8, 2026Fan Zhou, Tim Van de CruysClassifier-Free Guidance Controllable Text Generation
KU Leuven

May 28, 2026, cs.ROJ/K move · Enter open · S save

Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance

Shutong Ding, Zejia Zhong, Zhongyi Wang, Ke Hu+3

1ShanghaiTech University.

Diffusion Guidance Robotic RL Diffusion Policy RL for Diffusion Models Sample-Efficient RL

Open paper AskPDF arXiv

Recent advances in reinforcement learning (RL) have achieved great successes by leveraging the multimodality and exploration capability of diffusion policies. Among these approaches, one representative branch focuses on the sampling-based policy optimization. This design enables better exploration capability of the diffusion model, particularly at the beginning of training, but suffer from low exploitation in Q-value information, resulting in a slow policy convergence. Another branch pays attention to gradient-based policy optimization, which sufficiently exploits the gradient of the Q function yet tends to collapse into a unimodal policy with low diversity. To address this issue, we propose CGPO, \textbf{C}ritic-\textbf{G}uided diffusion \textbf{P}olicy \textbf{O}ptimization, which effectively balances exploration and exploitation with the training-free guidance technique integrated into the denoising process of diffusion policy. Concretely, CGPO steers action generation toward high-value regions defined by the critic network and uses the guided actions as regression objectives. In this manner, CGPO reduces the time required to obtain high-quality actions and improves final performance with better balance between the exploration-exploitation tradeoff. We validate the effectiveness of CGPO on 5 MuJoCo locomotion tasks, and CGPO achieves state-of-the-art performance compared with existing diffusion-based RL methods. Notably, CGPO is the first success to incorporate diffusion policy into real-world RL, with its superior performance on Franka robot arm grasping tasks. Our official page is released at https://dingsht.tech/cgpo-webpage.

Momentum

Related topics

Latest papers 120