RL for Diffusion Models

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 467% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 120

All topics

A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models
Jul 16, 2026Zikun Zhang, Jiayuan Sheng, David D. Yao +1Masked Diffusion Models RL for Language Model Reasoning
Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027
Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation
Jul 13, 2026Runhui Huang, Qihui Zhang, Zhe Liu +3T2I Generation Multimodal Reward Modeling
The University of Hong Kong · Peking University · ByteDance Seed
A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer
Jul 12, 2026Haitong Ma, Haldun Balim, Yang Hu +2Sim-to-Real Transfer Diffusion Policy
1Harvard University. · 2Georgia Institute of Technology.
Reinforcing the Generation Order of Multimodal Masked Diffusion Models
Jul 9, 2026Yidong Ouyang, Zhe Wang, Sourav Bhabesh +1Text-Image Alignment RL for Diffusion Models
University of California, Los Angeles · AGI Foundations for AWS
Expressivity and Statistical Trade-offs in Diffusion Policy Learning
Jul 8, 2026Viet Vu, Renyuan Xu, Jiacheng Zhang +1Diffusion Policy RL for Diffusion Models
Stanford University · Chinese University of Hong Kong · Imperial College London
Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF
Jul 8, 2026Eric Zhu, Abhinav Shrivastava, Soumik MukhopadhyayDiffusion Model Alignment RL for Diffusion Models
Carnegie Mellon University · University of Maryland, College Park
Self-Improving Diffusion Classifiers with Minority Preference Optimization
Jul 4, 2026Hyunsoo Kim, Jungmyung Wi, Soobin Um +2Diffusion Model Alignment Long-Tailed Classification
1Korea University · 2Kook Min University · 3Kyung Hee University
Adaptive Reparametrized Time for Score-Based Diffusion Sampling
Jul 2, 2026Yilie Huang, Wenpin Tang, Xun Yu ZhouDiffusion Sampling Adaptive Sampling
Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong. · Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027, USA. · Department of Industrial Engineering and Operations Research and Data Science Institute, Columbia University, New York, NY 10027, USA.
PAPA: Online Personalized Active Preference Alignment
Jul 1, 2026Anindya Sarkar, Nasik Muhammad Nafi, Isaac Lyngaas +2Diffusion Model Alignment Diffusion Model Fine-Tuning
Washington University in St. Louis · Oak Ridge National Laboratory
SLIM-RL: Risk-Budgeted Random-Masking RL for Diffusion LLMs Without Trajectory Slicing
Jun 30, 2026Ruikang Zhao, Zhenting Wang, Han Gao +1Reinforcement Learning RL for Language Model Reasoning
Technical University of Denmark · MBZUAI Institute of Foundation Models · Iowa State University +1
Diffusion Fine-tuning with Rewarded Moment Matching Distillation
Jun 29, 2026Alexis Jacq, Guillaume Couairon, Valentin De Bortoli +3Diffusion Model Distillation Diffusion Model Fine-Tuning
1Google DeepMind
Qwen-Image-2.0-RL Technical Report
Jun 25, 2026Yixian Xu, Kaiyuan Gao, Yuxiang Chen +25Reward Modeling RL for Diffusion Models
Equal contribution. · Corresponding author.
SpatialFlow-GRPO: Where Spatial Credit Drives Image Editing
Jun 25, 2026Yankai Yang, Yancheng Long, Wei Chen +7Reward Modeling Group Relative Policy Optimization
Harbin Institute of Technology, Shenzhen · Kuaishou Technology
Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation
Jun 23, 2026Sijie Wang, Zhengyu Qing, Zhiqiang Tan +6Reinforcement Learning Distributed RL
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Ltd.
Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models
Jun 22, 2026Jiawei Xu, Minghui Liu, Aakriti Agrawal +2Masked Diffusion Models Language Model Decoding
University of Maryland, College Park · University of California, Los Angeles.
Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching
Jun 21, 2026Serge Thilges, Onur Celik, Denis Blessing +2Diffusion Policy Maximum Entropy RL
Autonomous Learning Robots, Karlsruhe Institute of Technology
Curvature-Adaptive Consistency Flow Matching: Autonomous Trajectory Optimization via Reinforcement Learning
Jun 21, 2026Songtao Tian, Guhan Chen, Bohan Li +2Curriculum RL Few-Step Diffusion Sampling
1Tsinghua University, Beijing, China · 2Huawei Large Model Data Technology Lab, Shenzhen, China
Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Training
Jun 17, 2026Ruiqi Lai, Dakai An, Wei Gao +6Diffusion Transformer RL for Image Generation
NTU Singapore · Hong Kong University of Science and Technology · Alibaba Group
STAR: SpatioTemporal Adaptive Reward Allocation for Text-to-Image RL Post-Training
Jun 16, 2026Jinjie Shen, Wei Deng, Xian Hu +2Diffusion Model Alignment Compositional T2I Generation
Trust-Region Diffusion Policies for Massively Parallel On-Policy RL
Jun 13, 2026Huy Le, Onur Celik, Denis Blessing +6Diffusion Policy RL for Diffusion Models
Bosch Center for Artificial Intelligence · Autonomous Learning Robots, KIT · University of Texas at Austin +1
Temporal Difference Learning for Diffusion Models
Jun 13, 2026Qizhen Ying, Yangchen Pan, Victor Adrian Prisacariu +1Few-Step Diffusion Sampling RL for Diffusion Models
Department of Engineering Science, University of Oxford, Oxford, United Kingdom · School of Computer Science, Carleton University, Ottawa, Canada.
QPILOTS: Efficient Test-Time Q-Steering for Flow Policies
Jun 11, 2026Yifan Ruan, Chenyang Cao, Andreas Burger +7Inference-Time Optimization Diffusion Policy
University of Toronto · Vector Institute · LG Electronics
DiPOD: Diffusion Policy Optimization without Drifting Apart
Jun 11, 2026Haozhe Jiang, Haiwen Feng, Pieter Abbeel +3Policy Gradient Methods RL for Diffusion Models
UC Berkeley · NVIDIA
A2D2: Fine-Tuning Any-Length Discrete Diffusion for Adaptive Decoding
Jun 11, 2026Sophia Tang, Yuchen Zhu, Molei Tao +1Diffusion Model Fine-Tuning Diffusion Language Model Decoding
Department of Computer and Information Science, University of Pennsylvania · School of Mathematics, Georgia Institute of Technology · Department of Bioengineering, University of Pennsylvania
Efficient Reinforcement for Visual-Textual Thinking with Discrete Diffusion Model
Jun 11, 2026Yoonjeon Kim, Yuhta Takida, Chieh-Hsin Lai +2Multimodal Reasoning RL for Diffusion Models
KAIST · Sony AI · KAIST & AITRICS +1
Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models
Jun 7, 2026Yawen Shao, Jie Xiao, Kai Zhu +6RL for Language Model Reasoning Credit Assignment in RL
University of Science and Technology of China · Tongyi Lab · Northeastern University
Read the Trace, Steer the Path: Trajectory-Aware Reinforcement Learning for Diffusion Language Models
Jun 3, 2026Anant Khandelwal, Manish GuptaRL for Language Model Reasoning Credit Assignment in RL
Microsoft AI, India
Fine-Tuning Diffusion Models for Molecular Generation via Reinforcement Learning and Fast Sampling
May 31, 2026Guang Lin, Shikui Tu, Lei XuMolecular Optimization Diffusion Model Fine-Tuning
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, Guangdong, China
Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies
May 31, 2026Hikmet Simsir, Ozgur S. OguzRobot Policy Adaptation Reinforcement Learning
Department of Computer Engineering, Bilkent University, Ankara, Türkiye.
Improving Visual Representation Alignment Generation with GRPO
May 30, 2026Shentong Mo, Sukmin YunDiffusion Model Alignment Diffusion Transformer
Carnegie Mellon University · Hanyang University

Jul 16, 2026, cs.LGJ/K move · Enter open · S save

A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models

Zikun Zhang, Jiayuan Sheng, David D. Yao, Wenpin Tang

Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027

Masked Diffusion Models RL for Language Model Reasoning Diffusion Model Fine-Tuning Continuous-Time RL RL for Diffusion Models Diffusion Language Models

Open paper AskPDF arXiv

We formulate reinforcement learning (RL) in continuous time with discrete state spaces and possibly arbitrary action spaces via a stochastic control approach, where the state dynamics are modeled as a controlled continuous-time Markov chain (CTMC). We consider policy optimization problems and derive the corresponding policy gradient methods, leading to continuous-time variants of proximal policy optimization (PPO) and group relative policy optimization (GRPO). As a primary application, we develop a complete continuous-time RL framework for fine-tuning score-based discrete diffusion models. The proposed framework enables reward-driven optimization without requiring differentiability on the reward signals. In contrast to the existing GRPO-based approaches that only rely on terminal rewards, our formulation allows intermediate reward or advantage signals to be incorporated throughout the denoising trajectory. Importantly, when specialized to masked diffusion models (MDMs), our framework encompasses a rich class of policy parameterizations over the vocabulary simplex with analytically tractable probability ratios, providing a unified perspective on exploration and policy optimization in MDMs. For masked diffusion large language models (dLLMs), we further propose trajectory subsampling techniques to efficiently estimate computationally prohibitive trajectory likelihoods, reducing the computational cost of computing per-position probability ratios. We showcase the effectiveness of our methods on both low-dimensional entropy-regularized optimization problems and RL post-training of dLLMs on mathematical reasoning and coding tasks.

Momentum

Related topics

Latest papers 120