Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond

Zhiyuan Ma; Jiabao Wei; Tianyu Wang; Hanqing Wang; Jiadong Pan; Haowen Xu; Liangliang Zhao; Ruixun Liu

cs.AIOct 15, 2026

Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond

Authors: Zhiyuan Ma, Jiabao Wei, Tianyu Wang, Hanqing Wang, Jiadong Pan, Haowen Xu, Liangliang Zhao, Ruixun Liu,

Organizations: Huazhong University of Science and Technology · Beijing Institute of Technology · Tsinghua University · Hong Kong University of Science and Technology (GZ) ·

Abstract

Diffusion models (DMs) have steadily demonstrated outstanding performance in high-fidelity, controllable generation across diverse scenarios, following the "pretraining-finetuning" paradigm established in LLMs. Despite these advances, conventional DMs still face critical limitations in sophisticated reasoning, autonomous exploration of generation pathways, and adaptability to complex, human-aligned objectives. Inspired by RLHF's success in LLMs, RL has recently emerged as a pivotal post-training mechanism to iteratively refine DMs through reward-driven trajectory optimization. This paper presents the first comprehensive review of RL-driven DMs and an empirical analysis of how RL reshapes the diffusion process towards experience-centric, goal-directed generation. We propose a novel technical panorama elucidating RL's multifaceted roles in diffusion-based language, multimodal, robotic, and scientific models, with a granular focus on diffusion trajectory optimization and reward engineering. Through extensive experiments on multiple task-specific benchmarks, we systematically evaluate representative methods to intuitively highlight the advantages of different RL post-training strategies. Furthermore, we reveal and dissect three crucial bottlenecks that currently hinder the real-world deployment of RL-driven DMs: reward design fragility, exploration-exploitation trade-off in high-dimensional generation spaces, and training instability under sparse rewards. Based on these perspectives, we provide a principled roadmap for the integration of RL into DMs and outline promising future research directions for robust, interpretable RL-diffusion systems. We also construct a public repository to continuously track the latest developments in this rapidly evolving field: https://github.com/MAIR-Lab-HUST/Awesome-RL-for-Diffusion-Models.

Figures & tables

Fig. 1: A complete timeline of the development of DMs. At present, DMs have already passed the “Pretraining $\&$ Supervised Finetuning” era and entered the “RL-based Post-training” era. Our survey focuses on the latest development stage and reviews recent work on RL for DMs.

Fig. 2: The overview of text-guided visual content generation.

Fig. 3: A schematic comparison of the diffusion model denoising process and the RL problem formulated as an MDP.

Fig. 4: RL for DMs includes preference-/policy-based optimization.

Methods	Date	Backbone	Policy	Key Contributions	Primary Task	Links
GRPO-Related Post Training & Likelihood Estimation
d1 [ 22 ]	2025.04	LLaDA	On	First policy-gradient RL for dLLMs; masked-prompt likelihood approx	Math/Logic Reasoning
DiffuCoder [ 28 ]	2025.06	Qwen-2.5-Coder	On	Complementary masks for low-variance token likelihood	Code
wd1 [ 26 ]	2025.07	LLaDA	On	Weighted-likelihood objective; denominator-free update; much faster	Math Reasoning
BGPO [ 29 ]	2025.10	LLaDA	On	Boundary-guided lower bound enabling large MC with constant memory	Math/Code/Planning
DMPO [ 30 ]	2025.10	LLaDA	Off	Forward-KL via weighted denoising cross-entropy	Math Reasoning

TABLE I: Summary of reinforcement learning methods for diffusion-based LLMs as of 2026.08.

Fig. 5: An overview of representative works on RL4dLLMs.

Methods	GSM8K	MATH500	Countdown	Sudoku	HumanEval	MBPP
Non-RL diffusion-based LLMs
LLaDA-8B-Instruct [ 15 ]	78.2 †	36.2 †	20.7 †	11.7 †	49.4	41.0
RL-driven diffusion-based LLMs
d1 [ 22 ]	82.1	40.2	42.2	22.1	39.0	45.5
DiffuCoder [ 28 ]	N/A	N/A	N/A	N/A	73.2	78.6
wd1 [ 26 ]	82.3	39.0	51.2	76.4	39.0	39.2

TABLE II: Quantitative comparison of benchmark results for RL+dLLM methods. The values are rounded to three decimal places. For tasks with multiple sequence length evaluations, the best reported result is selected. Notably, † denotes third-party results. Other results are obtained from the original paper, where the best result across all reported evaluation protocols is chosen. The best and the second-best results of dLLMs in each column are demonstrated in green and gray .

Fig. 6: RLHF and DPO have been the two predominant RL methodologies for human alignment in recent years. In contrast, RLVR represents an emerging trend in RL for dMMs, significantly enhancing their capacity for complex task solving and scalable long-horizon reasoning.

Fig. 7: The overview of RL-driven diffusion multimodal tasks.

Methods	Training Strategy		Feedback Paradigm			Links
Methods	Online	Offline	RLHF	DPO	RLVR	Links
Outcome-level Alignment (Global Reward Optimization)
Diffusion-DPO [ 70 ]		✓	✓	✓
DPOK [ 65 ]	✓		✓
DDPO [ 66 ]	✓		✓		✓
LargeRL [ 92 ]	✓		✓		✓

TABLE III: Image-generation alignment and related methods. Non-exclusive columns: Online refreshes samples; RLHF includes human/AI preference and quality feedback; DPO denotes direct-preference methods; RLVR uses target/rule feedback, possibly model-assisted. RePrompt trains a prompt policy; VARD’s offline stage pretrains its value model.

Fig. 8: Qualitative examples from SD-3.5-M, SD-3.5-M with Flow-GRPO, FLUX.1-dev, FLUX.1-schnell, and Nano Banana. Task labels indicate prompt categories, not benchmark evaluations. The Flow-GRPO checkpoint configuration is unspecified; these examples are illustrative, not controlled benchmark comparisons or evidence of limitations of outcome-based rewards.

Methods	Training Strategy		Feedback Paradigm			Links
Methods	Online	Offline	RLHF	DPO	RLVR	Links
Temporal & Cross-modal Alignment (Motion Quality & Coherence)
InstructVideo [ 126 ]	✓		✓
VideoDPO [ 72 ]		✓	✓	✓
VADER [ 127 ]	✓		✓		✓
T2V-Turbo [ 128 ]	✓		✓

TABLE IV: Video alignment and related methods; columns follow Table III . Online includes direct reward-gradient tuning. VideoRM reports DRaFT-V; PISA combines offline/online stages; VideoAlign uses Flow-DPO/Flow-RWR. † : reward-weighted regression. Align-A-Video performs per-instance editing; RLVR includes imperfect proxies.

Base Models	Methods	Metrics	Base	Aligned	$\Delta$
Stable Diffusion v1.4	DDPO [ 66 ]	HPS-v2.1	0.239	0.297	+0.058
	DDPO [ 66 ]	CLIP	0.363	0.381	+0.018
	ReFL [ 63 ]	HPS-v2.1	0.239	0.357	+0.118
	ReFL [ 63 ]	CLIP	0.363	0.418	+0.055
	DanceGRPO [ 82 ]	HPS-v2.1	0.239	0.365	+0.126
	DanceGRPO [ 82 ]	CLIP	0.363	0.421	+0.058

TABLE V: Paired quantitative effects of representative RL/RL-family image generation methods. All values are collected from the reported tables in the original papers: DPOK’s Table 1 [ 65 ] , AlignProp’s Table 1 (v2) [ 68 ] , DanceGRPO’s Tables 2–4 [ 82 ] , and Flow-GRPO’s Tables 1–2 [ 81 ] , together with Qwen-Image-2.0-RL’s Table 1 and arena evaluation [ 109 ] . Rows should be interpreted as paired base-to-aligned improvements under the corresponding source protocol rather than as a single cross-paper leaderboard. SD v1.4 DDPO/ReFL/DanceGRPO HPS and CLIP results are DanceGRPO’s reward-specific runs; its Pick-a-Pic/GenEval results use joint rewards. DPOK is evaluated on training prompts. Flow-GRPO uses three separately trained task-specific models with KL regularization; OCR score is based on normalized edit distance, not the fraction of images with entirely correct text.

Base Models	Methods	Metrics	Base	Aligned	$\Delta$
VideoCrafter2	VADER [ 127 ]	VBench-T	80.44	80.59	+0.15
		VBench-Q	82.20	82.46	+0.26
		VBench-S	73.42	73.09	-0.33
ModelScopeT2V	InstructVideo [ 126 ]	HPS-v2 reward (in-domain)	0.2506	0.2717	+0.0211
		HPS-v2 reward (new animals)	0.2502	0.2645	+0.0143
		HPS-v2 reward (non-animals)	0.2557	0.2682	+0.0125

TABLE VI: Paired quantitative effects of representative RL/RL-family video generation methods. All values are collected from the reported tables in the original papers: InstructVideo’s Table 1 [ 126 ] , VideoDPO’s Table 1 [ 72 ] , T2V-Turbo’s Table 1 [ 128 ] , DanceGRPO’s Table 5 [ 82 ] , and DenseDPO’s Table 1 [ 130 ] , together with VGGRPO’s Table 1 [ 138 ] . VADER results are taken from the same VBench comparison protocol reported in VideoDPO’s Table 1; DenseDPO uses the SFT baseline in its source table as the paired base. VBench-T/Q/S denote VBench Total, Quality, and Semantics; VQ/MQ/TA denote Visual Quality, Motion Quality, and Text Alignment. Original metric scales are retained; comparisons are within paired rows, not across papers. T2V-Turbo-VC2 uses a four-step VCM-VC2 baseline, not the full-step teacher.

Fig. 9: Overview of RL-driven Diffusion-based Robotics.

Fig. 10: Representative research about diffusion-based RL for robotics.

Task Name	Non-diffusion Methods				Diffusion-based Methods
Task Name	TD3+BC [ 167 ]	BCQ [ 168 ]	BEAR [ 169 ]	CQL [ 170 ]	EDP [ 171 ]	IDQL [ 172 ]	Diffusion-QL [ 173 ]	DTQL [ 174 ]	DIVO [ 166 ]
halfcheetah-medium	48.3	47.0	41.0	44.0	52.1	51.0	51.1 (0.5)	57.9 (0.13)	67.71 (0.64)
hopper-medium	59.3	56.7	51.9	58.5	81.9	65.4	90.5 (4.6)	99.6 (0.87)	100.34 (0.44)
walker2d-medium	83.7	72.6	80.9	72.5	86.9	82.5	87.0 (0.9)	89.4 (0.13)	90.94 (1.63)
halfcheetah-medium-replay	44.6	40.4	29.7	45.5	49.4	45.9	47.8 (0.33)	50.9 (0.11)	55.53 (0.77)
hopper-medium-replay	60.9	53.3	37.3	95.0	101.1	92.1	101.3 (0.6)	100.0 (0.13)	101.28 (0.40)

TABLE VII: Representative offline RL methods on the D4RL Gym-MuJoCo and AntMaze benchmarks, adapted from DIVO [ 166 ] . All tasks use standardized D4RL v2 normalized scores, where higher is better. Gym-MuJoCo includes dense-reward locomotion tasks: HalfCheetah for planar running, Hopper for single-legged hopping and balance, and Walker2d for bipedal walking control; each is evaluated with medium, medium-replay, and medium-expert datasets. AntMaze includes sparse-reward long-horizon navigation tasks in U-Maze, Medium, and Large layouts, with play datasets collected from exploratory behavior and diverse datasets containing more varied goal-reaching trajectories. DIVO results are reported as mean

\pm

standard error over the last 10 evaluations and 5 random seeds, and the aggregate rows sum normalized scores within each benchmark group.

Fig. 11: The visualization of four offline RL benchmarks for robot continuous control, including Ant, Hopper, Walker2d, and HalfCheetah.

Methods	Date	Backbone	Action Expert	Key Contributions / RL Signal	Primary Setting
Foundation VLA and Generative Action Experts
RDT-1B [ 192 ]	2024.10	Diffusion Transformer	MLP action head	Scalable diffusion foundation model for bimanual manipulation	Bimanual manipulation
$\pi_{0}$ [ 193 ]	2024.10	VLM / LLM	Flow Matching	Continuous action generation conditioned on VLM features	General robot control
AgiBot World [ 194 ]	2025.03	VLM / LLM	Diffusion-based	Large-scale embodied data and generalist manipulation policy	Large-scale manipulation
$\pi_{0.5}$ [ 195 ]	2025.04	VLM / LLM	Flow Matching	Co-training for open-world VLA generalization	Open-world robot control
RL, RFT, and World-Model-Based VLA Optimization

TABLE VIII: Diffusion-/Flow-based VLA methods with RL or RL-relevant fine-tuning signals.

Fig. 12: A summary of the overall five RL-driven scientific generation domains. Note that the representative methods listed here are not exhaustive; please refer to our main text for more details.

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

RL for Diffusion Models

Reinforcement learning for diffusion models adapts denoising and sampling to rewards, preferences, and task outcomes.

See all 179 papers

Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages
Mar 13, 2026Vishnu Teja Kunde, Fatemeh Doudi, Mahdi Farahbakhsh +3RL for Diffusion Models Diffusion Language Model Training
Department of Electrical & Computer Engineering, Texas A&M University, College Station, TX 77843
Diffusion Reward Models
Sep 27, 2026Xiangyang Wang, Bingxiang He, Zeyuan Liu +13Reward Modeling Multimodal Reward Modeling
Tsinghua University · University of Illinois Urbana-Champaign
Reinforcement Learning from Denoising Feedback
May 25, 2026Qi He, Huan Chen, Ya Guo +3RL for Language Model Reasoning RL for Diffusion Models
Fudan University · Ant Group · Hong Kong University of Science and Technology

Mar 13, 2026J/K move · Enter open · S save

Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages

Vishnu Teja Kunde, Fatemeh Doudi, Mahdi Farahbakhsh, Dileep Kalathil+2

Department of Electrical & Computer Engineering, Texas A&M University, College Station, TX 77843

RL for Diffusion Models Diffusion Language Model Training RL for Language Model Reasoning Diffusion Language Models

Open paper AskPDF arXiv

Reinforcement learning (RL) has been effective for post-training autoregressive (AR) language models, but extending these methods to diffusion language models (DLMs) is challenging due to intractable sequence-level likelihoods. Existing approaches therefore rely on surrogate likelihoods or heuristic approximations, which can introduce bias and obscure the sequential structure of denoising. We formulate diffusion-based sequence generation as a finite-horizon Markov decision process over the denoising trajectory and derive an exact, unbiased policy gradient that decomposes over denoising steps and is expressed in terms of intermediate advantages, without requiring explicit evaluation of the sequence likelihood. To obtain a practical and compute-efficient estimator, we (i) select denoising steps for policy updates via an entropy-guided approximation bound, and (ii) estimate intermediate advantages using a one-step denoising reward naturally provided by the diffusion model, avoiding costly multi-step rollouts. Experiments on coding and logical reasoning benchmarks demonstrate state-of-the-art results, with strong competitive performance on mathematical reasoning, outperforming existing RL post-training approaches for DLMs. Code is available at https://github.com/vishnutez/egspo-dllm-rl.

Abstract

RL for Diffusion Models

Explore similar work

Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages

Diffusion Reward Models

Reinforcement Learning from Denoising Feedback