Organizations: Huazhong University of Science and Technology · Beijing Institute of Technology · Tsinghua University · Hong Kong University of Science and Technology (GZ) · University of Chinese Academy of Sciences · Shanghai Jiaotong University · Fudan University & Shanghai AI Laboratory · Peking University · Tongji University · Wuhan University · Shanghai AI Laboratory · Tsinghua University & Shanghai AI Laboratory
Diffusion models (DMs) have steadily demonstrated outstanding performance in high-fidelity, controllable generation across diverse scenarios, following the "pretraining-finetuning" paradigm established in LLMs. Despite these advances, conventional DMs still face critical limitations in sophisticated reasoning, autonomous exploration of generation pathways, and adaptability to complex, human-aligned objectives. Inspired by RLHF's success in LLMs, RL has recently emerged as a pivotal post-training mechanism to iteratively refine DMs through reward-driven trajectory optimization. This paper presents the first comprehensive review of RL-driven DMs and an empirical analysis of how RL reshapes the diffusion process towards experience-centric, goal-directed generation. We propose a novel technical panorama elucidating RL's multifaceted roles in diffusion-based language, multimodal, robotic, and scientific models, with a granular focus on diffusion trajectory optimization and reward engineering. Through extensive experiments on multiple task-specific benchmarks, we systematically evaluate representative methods to intuitively highlight the advantages of different RL post-training strategies. Furthermore, we reveal and dissect three crucial bottlenecks that currently hinder the real-world deployment of RL-driven DMs: reward design fragility, exploration-exploitation trade-off in high-dimensional generation spaces, and training instability under sparse rewards. Based on these perspectives, we provide a principled roadmap for the integration of RL into DMs and outline promising future research directions for robust, interpretable RL-diffusion systems. We also construct a public repository to continuously track the latest developments in this rapidly evolving field: https://github.com/MAIR-Lab-HUST/Awesome-RL-for-Diffusion-Models.
Figures & tables
Fig. 1: A complete timeline of the development of DMs. At present, DMs have already passed the “Pretraining & Supervised Finetuning” era and entered the “RL-based Post-training” era. Our survey focuses on the latest development stage and reviews recent work on RL for DMs.
Fig. 2: The overview of text-guided visual content generation.
Fig. 3: A schematic comparison of the diffusion model denoising process and the RL problem formulated as an MDP.
Fig. 4: RL for DMs includes preference-/policy-based optimization.
Methods
Date
Backbone
Policy
Key Contributions
Primary Task
Links
GRPO-Related Post Training & Likelihood Estimation
d1 [ 22 ]
2025.04
LLaDA
On
First policy-gradient RL for dLLMs; masked-prompt likelihood approx
Math/Logic Reasoning
DiffuCoder [ 28 ]
2025.06
Qwen-2.5-Coder
On
Complementary masks for low-variance token likelihood
Code
wd1 [ 26 ]
2025.07
LLaDA
On
Weighted-likelihood objective; denominator-free update; much faster
Math Reasoning
BGPO [ 29 ]
2025.10
LLaDA
On
Boundary-guided lower bound enabling large MC with constant memory
Math/Code/Planning
DMPO [ 30 ]
2025.10
LLaDA
Off
Forward-KL via weighted denoising cross-entropy
Math Reasoning
TABLE I: Summary of reinforcement learning methods for diffusion-based LLMs as of 2026.08.
Fig. 5: An overview of representative works on RL4dLLMs.
Methods
GSM8K
MATH500
Countdown
Sudoku
HumanEval
MBPP
Non-RL diffusion-based LLMs
LLaDA-8B-Instruct [ 15 ]
78.2 †
36.2 †
20.7 †
11.7 †
49.4
41.0
RL-driven diffusion-based LLMs
d1 [ 22 ]
82.1
40.2
42.2
22.1
39.0
45.5
DiffuCoder [ 28 ]
N/A
N/A
N/A
N/A
73.2
78.6
wd1 [ 26 ]
82.3
39.0
51.2
76.4
39.0
39.2
TABLE II: Quantitative comparison of benchmark results for RL+dLLM methods. The values are rounded to three decimal places. For tasks with multiple sequence length evaluations, the best reported result is selected. Notably, † denotes third-party results. Other results are obtained from the original paper, where the best result across all reported evaluation protocols is chosen. The best and the second-best results of dLLMs in each column are demonstrated in green and gray .
Fig. 6: RLHF and DPO have been the two predominant RL methodologies for human alignment in recent years. In contrast, RLVR represents an emerging trend in RL for dMMs, significantly enhancing their capacity for complex task solving and scalable long-horizon reasoning.
Fig. 7: The overview of RL-driven diffusion multimodal tasks.
TABLE III: Image-generation alignment and related methods. Non-exclusive columns: Online refreshes samples; RLHF includes human/AI preference and quality feedback; DPO denotes direct-preference methods; RLVR uses target/rule feedback, possibly model-assisted. RePrompt trains a prompt policy; VARD’s offline stage pretrains its value model.
Fig. 8: Qualitative examples from SD-3.5-M, SD-3.5-M with Flow-GRPO, FLUX.1-dev, FLUX.1-schnell, and Nano Banana. Task labels indicate prompt categories, not benchmark evaluations. The Flow-GRPO checkpoint configuration is unspecified; these examples are illustrative, not controlled benchmark comparisons or evidence of limitations of outcome-based rewards.
TABLE IV: Video alignment and related methods; columns follow Table III . Online includes direct reward-gradient tuning. VideoRM reports DRaFT-V; PISA combines offline/online stages; VideoAlign uses Flow-DPO/Flow-RWR. † : reward-weighted regression. Align-A-Video performs per-instance editing; RLVR includes imperfect proxies.
Base Models
Methods
Metrics
Base
Aligned
Δ
Stable Diffusion v1.4
DDPO [ 66 ]
HPS-v2.1
0.239
0.297
+0.058
CLIP
0.363
0.381
+0.018
ReFL [ 63 ]
HPS-v2.1
0.239
0.357
+0.118
CLIP
0.363
0.418
+0.055
DanceGRPO [ 82 ]
HPS-v2.1
0.239
0.365
+0.126
CLIP
0.363
0.421
+0.058
TABLE V: Paired quantitative effects of representative RL/RL-family image generation methods. All values are collected from the reported tables in the original papers: DPOK’s Table 1 [ 65 ] , AlignProp’s Table 1 (v2) [ 68 ] , DanceGRPO’s Tables 2–4 [ 82 ] , and Flow-GRPO’s Tables 1–2 [ 81 ] , together with Qwen-Image-2.0-RL’s Table 1 and arena evaluation [ 109 ] . Rows should be interpreted as paired base-to-aligned improvements under the corresponding source protocol rather than as a single cross-paper leaderboard. SD v1.4 DDPO/ReFL/DanceGRPO HPS and CLIP results are DanceGRPO’s reward-specific runs; its Pick-a-Pic/GenEval results use joint rewards. DPOK is evaluated on training prompts. Flow-GRPO uses three separately trained task-specific models with KL regularization; OCR score is based on normalized edit distance, not the fraction of images with entirely correct text.
Base Models
Methods
Metrics
Base
Aligned
Δ
VideoCrafter2
VADER [ 127 ]
VBench-T
80.44
80.59
+0.15
VBench-Q
82.20
82.46
+0.26
VBench-S
73.42
73.09
-0.33
ModelScopeT2V
InstructVideo [ 126 ]
HPS-v2 reward (in-domain)
0.2506
0.2717
+0.0211
HPS-v2 reward (new animals)
0.2502
0.2645
+0.0143
HPS-v2 reward (non-animals)
0.2557
0.2682
+0.0125
TABLE VI: Paired quantitative effects of representative RL/RL-family video generation methods. All values are collected from the reported tables in the original papers: InstructVideo’s Table 1 [ 126 ] , VideoDPO’s Table 1 [ 72 ] , T2V-Turbo’s Table 1 [ 128 ] , DanceGRPO’s Table 5 [ 82 ] , and DenseDPO’s Table 1 [ 130 ] , together with VGGRPO’s Table 1 [ 138 ] . VADER results are taken from the same VBench comparison protocol reported in VideoDPO’s Table 1; DenseDPO uses the SFT baseline in its source table as the paired base. VBench-T/Q/S denote VBench Total, Quality, and Semantics; VQ/MQ/TA denote Visual Quality, Motion Quality, and Text Alignment. Original metric scales are retained; comparisons are within paired rows, not across papers. T2V-Turbo-VC2 uses a four-step VCM-VC2 baseline, not the full-step teacher.
Fig. 9: Overview of RL-driven Diffusion-based Robotics.
Fig. 10: Representative research about diffusion-based RL for robotics.
Task Name
Non-diffusion Methods
Diffusion-based Methods
TD3+BC [ 167 ]
BCQ [ 168 ]
BEAR [ 169 ]
CQL [ 170 ]
EDP [ 171 ]
IDQL [ 172 ]
Diffusion-QL [ 173 ]
DTQL [ 174 ]
DIVO [ 166 ]
halfcheetah-medium
48.3
47.0
41.0
44.0
52.1
51.0
51.1 (0.5)
57.9 (0.13)
67.71 (0.64)
hopper-medium
59.3
56.7
51.9
58.5
81.9
65.4
90.5 (4.6)
99.6 (0.87)
100.34 (0.44)
walker2d-medium
83.7
72.6
80.9
72.5
86.9
82.5
87.0 (0.9)
89.4 (0.13)
90.94 (1.63)
halfcheetah-medium-replay
44.6
40.4
29.7
45.5
49.4
45.9
47.8 (0.33)
50.9 (0.11)
55.53 (0.77)
hopper-medium-replay
60.9
53.3
37.3
95.0
101.1
92.1
101.3 (0.6)
100.0 (0.13)
101.28 (0.40)
TABLE VII: Representative offline RL methods on the D4RL Gym-MuJoCo and AntMaze benchmarks, adapted from DIVO [ 166 ] . All tasks use standardized D4RL v2 normalized scores, where higher is better. Gym-MuJoCo includes dense-reward locomotion tasks: HalfCheetah for planar running, Hopper for single-legged hopping and balance, and Walker2d for bipedal walking control; each is evaluated with medium, medium-replay, and medium-expert datasets. AntMaze includes sparse-reward long-horizon navigation tasks in U-Maze, Medium, and Large layouts, with play datasets collected from exploratory behavior and diverse datasets containing more varied goal-reaching trajectories. DIVO results are reported as mean ± standard error over the last 10 evaluations and 5 random seeds, and the aggregate rows sum normalized scores within each benchmark group.
Fig. 11: The visualization of four offline RL benchmarks for robot continuous control, including Ant, Hopper, Walker2d, and HalfCheetah.
Methods
Date
Backbone
Action Expert
Key Contributions / RL Signal
Primary Setting
Foundation VLA and Generative Action Experts
RDT-1B [ 192 ]
2024.10
Diffusion Transformer
MLP action head
Scalable diffusion foundation model for bimanual manipulation
Bimanual manipulation
π0 [ 193 ]
2024.10
VLM / LLM
Flow Matching
Continuous action generation conditioned on VLM features
General robot control
AgiBot World [ 194 ]
2025.03
VLM / LLM
Diffusion-based
Large-scale embodied data and generalist manipulation policy
Large-scale manipulation
π0.5 [ 195 ]
2025.04
VLM / LLM
Flow Matching
Co-training for open-world VLA generalization
Open-world robot control
RL, RFT, and World-Model-Based VLA Optimization
TABLE VIII: Diffusion-/Flow-based VLA methods with RL or RL-relevant fine-tuning signals.
Fig. 12: A summary of the overall five RL-driven scientific generation domains. Note that the representative methods listed here are not exhaustive; please refer to our main text for more details.
Reinforcement learning (RL) has been effective for post-training autoregressive (AR) language models, but extending these methods to diffusion language models (DLMs) is challenging due to intractable sequence-level likelihoods. Existing approaches therefore rely on surrogate likelihoods or heuristic approximations, which can introduce bias and obscure the sequential structure of denoising. We formulate diffusion-based sequence generation as a finite-horizon Markov decision process over the denoising trajectory and derive an exact, unbiased policy gradient that decomposes over denoising steps and is expressed in terms of intermediate advantages, without requiring explicit evaluation of the sequence likelihood. To obtain a practical and compute-efficient estimator, we (i) select denoising steps for policy updates via an entropy-guided approximation bound, and (ii) estimate intermediate advantages using a one-step denoising reward naturally provided by the diffusion model, avoiding costly multi-step rollouts. Experiments on coding and logical reasoning benchmarks demonstrate state-of-the-art results, with strong competitive performance on mathematical reasoning, outperforming existing RL post-training approaches for DLMs. Code is available at https://github.com/vishnutez/egspo-dllm-rl.
Reward models underpin the alignment of large language models, yet the dominant designs reduce each prompt--response pair to a point estimate or to a distribution from a fixed parametric family. This is at odds with human preference, which is inherently multimodal: the same response can be reasonably judged in many ways, and no single family covers all of them. To better fit this structure, we introduce DRM, a Diffusion Reward Model that recasts reward modeling as conditional density estimation over p(r∣x,y). Conditioned on a frozen LLM encoder, a lightweight Diffusion Transformer denoises Gaussian noise into a reward vector, placing no parametric assumption on the output distribution and naturally representing its multimodal structure. A single architecture handles both multi-attribute regression and pairwise preference data, and at inference N samples form an empirical reward distribution that can be aggregated into a scalar, a variance, or quantiles. Across five benchmarks, DRM matches or surpasses baselines under matched data and backbone, stays competitive with much larger discriminative, distributional, and generative RMs despite its modest training scale, and recovers multimodal reward structure where conventional heads collapse to a point. Uncertainty-aware rejection and lower-confidence-bound (LCB) aggregation further demonstrate that DRM can exploit distributional information beyond a scalar reward to improve reward-model decisions. Downstream RLHF experiments additionally show that using DRM as the training-time reward leads to improved policy performance, directly validating the practical benefit of diffusion-based reward modeling for RLHF training.
Xiangyang Wang, Bingxiang He, Zeyuan Liu +13
Tsinghua University · University of Illinois Urbana-Champaign
Policy loss estimation remains a fundamental and long-standing challenge in reinforcement learning (RL) for diffusion language models (DLMs). We introduce Reinforcement Learning from Denoising Feedback (RLDF), a novel training paradigm that leverages feedback obtained from rollout and training processes to facilitate accurate and efficient policy loss estimation. To balance the trade-off between computational efficiency and estimation effectiveness, RLDF optimizes the model toward the clipped clean state from intermediate noisy states, combined with weighted timestep sampling over denoising timesteps. Extensive experiments demonstrate that RLDF achieves consistent and substantial improvements in both performance and generalizability across two representative DLM architectures, LLaDA and Dream, on multiple reasoning benchmarks. Our work lays a principled foundation for scalable reinforcement learning in diffusion language models. We build Drift, a training framework for DLMs, available at https://github.com/ant-research/Drift.
Qi He, Huan Chen, Ya Guo +3
Fudan University · Ant Group · Hong Kong University of Science and Technology