Scaling Reinforcement Learning for Diffusion Models via Velocity Matching

Jaemoo Choi; Wei Guo; Yuchen Zhu; Arash Vahdat; Molei Tao; Julius Berner; Yongxin Chen

cs.CVAug 24, 2026

Scaling Reinforcement Learning for Diffusion Models via Velocity Matching

Authors: Jaemoo Choi, Wei Guo, Yuchen Zhu, Arash Vahdat, Molei Tao, Julius Berner, Yongxin Chen

Organizations: Georgia Institute of Technology · NVIDIA

Abstract

Reward-based fine-tuning of diffusion models has largely inherited likelihood-based policy optimization developed for autoregressive large language models (LLMs). Diffusion models, however, are natively trained through velocity regression and do not directly provide the likelihood of a generated sample. Existing methods address this using transition likelihoods along stochastic denoising trajectories or evidence lower bounds (ELBOs) to approximate generated-sample likelihoods. These approximations arise from applying likelihood-based updates to models whose native training and generation operate through velocity fields. We instead take velocity matching as the starting point for reward fine-tuning. We propose reward-based velocity matching (RVM), which weights the velocity-matching loss by reward with an additional anchor regression term, requiring neither likelihood estimation nor likelihood ratios. RVM recovers Reinforce Adjoint Matching at the update level and contains DiffusionNFT as a special case, while ELBO-based methods are closely related through the same velocity-regression structure. This unified view isolates reward design and anchor velocity as principal design choices in velocity-based fine-tuning. Across text-to-image, text-to-video, and image-to-video generation, RVM matches or outperforms the evaluated trajectory-based methods. On Wan2.1-T2V-1.3B, it achieves the highest VBench Overall among the evaluated methods, with substantially lower estimated training costs than the trajectory-based approaches. For video fine-tuning, we introduce a dynamic-tracking reward that provides explicit motion feedback and improves both Dynamic Degree and overall VBench performance.

Figures & tables

Figure 1: Overview of reward-based velocity matching (RVM). Left: Autoregressive LLMs operate naturally through token probabilities, whereas diffusion models are trained through velocity matching, motivating reward fine-tuning directly in velocity space. Right: RVM generates samples, scores them with reward models, and forward-noises each to an intermediate state. RVM objective combines reward-weighted velocity matching with anchor regression, requiring neither likelihood-ratio estimation nor stored rollout trajectories. This formulation supports different rollout samplers, applies to image and video generation, and recovers RAM and DiffusionNFT as special cases.

Figure 2: We illustrate both updates under the same rollout and compare their loss targets. Given a generated endpoint

\bm{x}_{0}

with positive reward , (a) the trajectory-based loss uses the target

\bm{v}_{\mathrm{traj}}

determined by the realized noisy transition from

\bm{x}_{t}

\bm{x}_{t-\Delta t}

, whereas (b) velocity-based losses, such as RVM, use the endpoint-defined target

\bm{v}_{\mathrm{tgt}}=\bm{\epsilon}-\bm{x}_{0}

. Arrows indicate denoising directions. Negative rewards reverse the corresponding attraction into repulsion. See Sec. B.4 for details.

Panel of Figure 3 — Figure 3: Qualitative comparison between the base model and RVM on Wan2.1-T2V-1.3B and SkyReels-I2V-V1. Additional examples are provided in Sec. G.3 .

Trajectory-based
	Reward				VBench
Method	TA	VQ	MQ	HPSv3	Dyn. Degree	Quality	Semantic	Overall
Wan2.1-1.3B (w/o CFG)	$2.53$	$2.33$	$0.35$	$1.85$	$54.17$	$78.59$	$65.73$	$76.02$
Wan2.1-1.3B (w/ CFG)	$5.50$	$4.08$	$0.97$	$7.85$	$65.28$	$83.72$	$80.64$	$83.10$
FlowGRPO	$2.48$	$2.24$	$0.27$	$1.62$	$55.56$	$78.67$	$64.87$	$75.91$
DanceGRPO †	$-1.17$	$3.41$	$0.30$	–	$58.33$	$82.81$	$66.06$	$79.46$

Figure 4: Evaluation results on Wan2.1-T2V-1.3B. We report reward scores including VideoAlign, HPSv3 and VBench on VBench dataset. Dyn. Degree is the VBench dynamic-degree dimension, one of the seven quality dimensions already averaged into Quality , reported on its own because it is the only one that measures how much the video moves. Blue shading marks the first and second highest scores per column, including ties. † cited numbers from Li et al. (2026b) .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Sampling policy	Anchor $\bm{v}_{\mathrm{anc}}$	Scale $c(r)$	Reach $A(r)$
RAM	$\pi_{\theta}$	$\bm{v}_{\mathrm{ref}}$	$r_{\mathrm{ram}}+1$	$\tfrac{{r^{i}_{\mathrm{ram}}}}{{r_{\mathrm{ram}}+1}}$
DiffusionNFT	$\pi_{\mathrm{old}}$	EMA velocity	$\bar{\beta}$	$(2r_{\mathrm{nft}}-1)/\bar{\beta}$
RVM	$\pi_{\mathrm{old}}$	any	${\beta}{\big({r_{\mathrm{rvm}}}/{\beta}+1\big)}$	$\tfrac{{r_{\mathrm{rvm}}/{\beta}}}{{r_{\mathrm{rvm}}/\beta+1}}$

Appendix

Table 1: Velocity-based objectives through the shared regression form 5 . With

\pi_{\mathrm{old}}:=\pi_{\theta}

\beta=1

, and

\bm{v}_{\mathrm{anc}}:=\bm{v}_{\mathrm{ref}}

, RAM’s gradient matches that of RVM. With

\bm{v}_{\mathrm{anc}}:=

EMA velocity,

r_{\mathrm{rvm}}=2r_{\mathrm{nft}}-1

, and

\beta=\bar{\beta}-r_{\mathrm{rvm}}

, DiffusionNFT equals RVM up to

\theta

-independent constants.

Hyperparameter	Wan2.1-T2V-1.3B	SkyReels-I2V	SD3.5-M (OCR)
Task	text-to-video	image-to-video	text-to-image
Resolution ( $H\times W$ )	$480\times 832$	$400\times 640$	$512\times 512$
Frames / fps	$53$ / $15$	$53$ / $15$	—
Sampler	DPM-2 ODE	DPM-2 ODE	DPM-2 ODE
Sampling steps	$16$	$16$	$10$
Flow-matching shift	$8.0$	—	—

Appendix

Table 2: Default hyperparameters for the RVM (reward-weighted velocity matching) runs on each base model. All runs share the optimizer, DPM-2 ODE sampler, and CFG-free setting described above; the two video runs share the reward mixture of Sec. C.3 , while the OCR run uses the reward of Sec. C.2 .

Figure 11: Few-step inference on Wan2.1-T2V-1.3B. VBench Total against the number of inference steps, under the same timestep schedules.

	VBench-I2V
Method	Dyn. Degree	Quality	I2V	Overall
SkyReels-I2V-V1 (w/o CFG)	$54.10$	$75.85$	$89.74$	$82.80$
SkyReels-I2V-V1 (w/ CFG)	$38.62$	$76.92$	$91.32$	$84.12$
RVM	$49.19$	$80.50$	$94.67$	$87.58$

Appendix

Table 3: Evaluation results on SkyReels-I2V at native spatial resolution We report VBench-I2V scores on the VBench-I2V dataset. Dyn. Degree is the VBench dynamic-degree dimension, one of the seven quality dimensions already averaged into Quality .

Method	PickScore	ClipScore	HPSv2.1	Aesthetic	ImgRwd
FlowGRPO †	$23.50$	$0.280$	$0.316$	$5.90$	$1.29$
DiffusionNFT †	$23.61$	$0.288$	$0.344$	$6.04$	$1.46$
PEPG †	$23.68$	$0.296$	$0.325$	$6.06$	$1.45$
RVM	$23.30$	$0.289$	$0.333$	$5.82$	$1.38$

Appendix

Table 4: Evaluation Results on SD3.5-M (DrawBench) We report the text-to-image PickScore fine-tuning task, evaluated on DrawBench with the protocol of Choi et al. (2026) . Training uses PickScore, CLIPScore, and HPSv2.1. Blue shading marks the first and second highest scores per column, including ties. † cited numbers from Choi et al. (2026) .

Panel of Figure 12 — Figure 12: Per-reward training curves on Wan2.1-T2V-1.3B Each panel shows one reward component across training iterations; dashed lines mark the untrained base.

Panel of Figure 13 — Figure 13: Additional qualitative comparison on Wan2.1-T2V-1.3B Frames $1,14,27,40,53$ of each method for the same prompt; top to bottom: Wan2.1-1.3B (w/o CFG), Wan2.1-1.3B (w/ CFG), DiffusionNFT, RAM, Ours (RVM).

Panel of Figure 14 — Figure 14: Additional qualitative comparison on Wan2.1-T2V-1.3B . Frames $1,14,27,40,53$ of each method for two prompts; within each group of three rows, top to bottom: Wan2.1-1.3B (w/o CFG), Wan2.1-1.3B (w/ CFG), Ours (RVM).

Panel of Figure 15 — Figure 15: Additional qualitative comparison on Wan2.1-T2V-1.3B Frames $1,14,27,40,53$ of each method for two prompts; within each group of three rows, top to bottom: Wan2.1-1.3B (w/o CFG), Wan2.1-1.3B (w/ CFG), Ours (RVM).

Panel of Figure 16 — Figure 16: Additional qualitative comparison on SkyReels-I2V Frames $1,14,27,40,53$ for three prompts; within each pair, SkyReels-I2V-V1 (top) and Ours (RVM) (bottom).

Panel of Figure 17 — Figure 17: Additional qualitative comparison on SkyReels-I2V Frames $1,14,27,40,53$ for three prompts; within each pair, SkyReels-I2V-V1 (top) and Ours (RVM) (bottom).

Explore similar work

Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models
Apr 19, 2026Jeongjae Lee, Jinho Chang, Jeongsol Kim +1Diffusion Model Alignment Diffusion Model Fine-Tuning
Graduate School of AI KAIST, Korea
Efficient Adjoint Matching for Fine-tuning Diffusion Models
May 12, 2026Jeongwoo Shin, Dongsoo Shin, Yuchen Zhu +5Diffusion Model Alignment Stochastic Optimal Control
Seoul National University · Georgia Institute of Technology · Sungkyunkwan University
Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models
May 11, 2026Andreas Bergmeister, Stefanie Jegelka, Nikolas Nüsken +2Diffusion Model Alignment Flow Matching
1TU Munich, MCML · 2MIT CSAIL · 3King’s College London +2

Apr 19, 2026J/K move · Enter open · S save

Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models

Jeongjae Lee, Jinho Chang, Jeongsol Kim, Jong Chul Ye

Graduate School of AI KAIST, Korea

Diffusion Model Alignment Diffusion Model Fine-Tuning Score-Based Generative Modeling

Open paper AskPDF arXiv

Reward-based fine-tuning steers a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model. Although existing methods are derived from different perspectives, we show that many can be written under a common framework, which we call reward score matching (RSM). Under this view, alignment becomes score matching against a value-guided target, and the main differences across methods reduce to the construction of the value-guidance estimator and the effective optimization strength across timesteps. This unification clarifies the bias-variance-compute tradeoffs of existing designs, and distinguishes core optimization components from auxiliary mechanisms that add complexity without clear benefit. Guided by this perspective, we develop simpler, more efficient redesigns across representative differentiable and black-box reward alignment tasks. Overall, RSM turns a seemingly fragmented collection of reward-based fine-tuning methods into a smaller, more interpretable, and more actionable design space. Code is available at https://github.com/jaylee2000/rsm

Abstract

Explore similar work

Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models

Efficient Adjoint Matching for Fine-tuning Diffusion Models

Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models