Scaling Reinforcement Learning for Diffusion Models via Velocity Matching

Jaemoo Choi; Wei Guo; Yuchen Zhu; Arash Vahdat; Molei Tao; Julius Berner; Yongxin Chen

cs.CVAug 24, 2026

Scaling Reinforcement Learning for Diffusion Models via Velocity Matching

Authors: Jaemoo Choi, Wei Guo, Yuchen Zhu, Arash Vahdat, Molei Tao, Julius Berner, Yongxin Chen

Organizations: Georgia Institute of Technology · NVIDIA

Abstract

Reward-based fine-tuning of diffusion models has largely inherited likelihood-based policy optimization developed for autoregressive large language models (LLMs). Diffusion models, however, are natively trained through velocity regression and do not directly provide the likelihood of a generated sample. Existing methods address this using transition likelihoods along stochastic denoising trajectories or evidence lower bounds (ELBOs) to approximate generated-sample likelihoods. These approximations arise from applying likelihood-based updates to models whose native training and generation operate through velocity fields. We instead take velocity matching as the starting point for reward fine-tuning. We propose reward-based velocity matching (RVM), which weights the velocity-matching loss by reward with an additional anchor regression term, requiring neither likelihood estimation nor likelihood ratios. RVM recovers Reinforce Adjoint Matching at the update level and contains DiffusionNFT as a special case, while ELBO-based methods are closely related through the same velocity-regression structure. This unified view isolates reward design and anchor velocity as principal design choices in velocity-based fine-tuning. Across text-to-image, text-to-video, and image-to-video generation, RVM matches or outperforms the evaluated trajectory-based methods. On Wan2.1-T2V-1.3B, it achieves the highest VBench Overall among the evaluated methods, with substantially lower estimated training costs than the trajectory-based approaches. For video fine-tuning, we introduce a dynamic-tracking reward that provides explicit motion feedback and improves both Dynamic Degree and overall VBench performance.

Figures & tables

Figure 1: Overview of reward-based velocity matching (RVM). Left: Autoregressive LLMs operate naturally through token probabilities, whereas diffusion models are trained through velocity matching, motivating reward fine-tuning directly in velocity space. Right: RVM generates samples, scores them with reward models, and forward-noises each to an intermediate state. RVM objective combines reward-weighted velocity matching with anchor regression, requiring neither likelihood-ratio estimation nor stored rollout trajectories. This formulation supports different rollout samplers, applies to image and video generation, and recovers RAM and DiffusionNFT as special cases.

Figure 2: We illustrate both updates under the same rollout and compare their loss targets. Given a generated endpoint

\bm{x}_{0}

with positive reward , (a) the trajectory-based loss uses the target

\bm{v}_{\mathrm{traj}}

determined by the realized noisy transition from

\bm{x}_{t}

\bm{x}_{t-\Delta t}

, whereas (b) velocity-based losses, such as RVM, use the endpoint-defined target

\bm{v}_{\mathrm{tgt}}=\bm{\epsilon}-\bm{x}_{0}

. Arrows indicate denoising directions. Negative rewards reverse the corresponding attraction into repulsion. See Sec. B.4 for details.

Panel of Figure 3 — Figure 3: Qualitative comparison between the base model and RVM on Wan2.1-T2V-1.3B and SkyReels-I2V-V1. Additional examples are provided in Sec. G.3 .

Trajectory-based
	Reward				VBench
Method	TA	VQ	MQ	HPSv3	Dyn. Degree	Quality	Semantic	Overall
Wan2.1-1.3B (w/o CFG)	$2.53$	$2.33$	$0.35$	$1.85$	$54.17$	$78.59$	$65.73$	$76.02$
Wan2.1-1.3B (w/ CFG)	$5.50$	$4.08$	$0.97$	$7.85$	$65.28$	$83.72$	$80.64$	$83.10$
FlowGRPO	$2.48$	$2.24$	$0.27$	$1.62$	$55.56$	$78.67$	$64.87$	$75.91$
DanceGRPO †	$-1.17$	$3.41$	$0.30$	–	$58.33$	$82.81$	$66.06$	$79.46$

Figure 4: Evaluation results on Wan2.1-T2V-1.3B. We report reward scores including VideoAlign, HPSv3 and VBench on VBench dataset. Dyn. Degree is the VBench dynamic-degree dimension, one of the seven quality dimensions already averaged into Quality , reported on its own because it is the only one that measures how much the video moves. Blue shading marks the first and second highest scores per column, including ties. † cited numbers from Li et al. (2026b) .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Sampling policy	Anchor $\bm{v}_{\mathrm{anc}}$	Scale $c(r)$	Reach $A(r)$
RAM	$\pi_{\theta}$	$\bm{v}_{\mathrm{ref}}$	$r_{\mathrm{ram}}+1$	$\tfrac{{r^{i}_{\mathrm{ram}}}}{{r_{\mathrm{ram}}+1}}$
DiffusionNFT	$\pi_{\mathrm{old}}$	EMA velocity	$\bar{\beta}$	$(2r_{\mathrm{nft}}-1)/\bar{\beta}$
RVM	$\pi_{\mathrm{old}}$	any	${\beta}{\big({r_{\mathrm{rvm}}}/{\beta}+1\big)}$	$\tfrac{{r_{\mathrm{rvm}}/{\beta}}}{{r_{\mathrm{rvm}}/\beta+1}}$

Appendix

Table 1: Velocity-based objectives through the shared regression form 5 . With

\pi_{\mathrm{old}}:=\pi_{\theta}

\beta=1

, and

\bm{v}_{\mathrm{anc}}:=\bm{v}_{\mathrm{ref}}

, RAM’s gradient matches that of RVM. With

\bm{v}_{\mathrm{anc}}:=

EMA velocity,

r_{\mathrm{rvm}}=2r_{\mathrm{nft}}-1

, and

\beta=\bar{\beta}-r_{\mathrm{rvm}}

, DiffusionNFT equals RVM up to

\theta

-independent constants.

Hyperparameter	Wan2.1-T2V-1.3B	SkyReels-I2V	SD3.5-M (OCR)
Task	text-to-video	image-to-video	text-to-image
Resolution ( $H\times W$ )	$480\times 832$	$400\times 640$	$512\times 512$
Frames / fps	$53$ / $15$	$53$ / $15$	—
Sampler	DPM-2 ODE	DPM-2 ODE	DPM-2 ODE
Sampling steps	$16$	$16$	$10$
Flow-matching shift	$8.0$	—	—

Appendix

Table 2: Default hyperparameters for the RVM (reward-weighted velocity matching) runs on each base model. All runs share the optimizer, DPM-2 ODE sampler, and CFG-free setting described above; the two video runs share the reward mixture of Sec. C.3 , while the OCR run uses the reward of Sec. C.2 .

Figure 11: Few-step inference on Wan2.1-T2V-1.3B. VBench Total against the number of inference steps, under the same timestep schedules.

	VBench-I2V
Method	Dyn. Degree	Quality	I2V	Overall
SkyReels-I2V-V1 (w/o CFG)	$54.10$	$75.85$	$89.74$	$82.80$
SkyReels-I2V-V1 (w/ CFG)	$38.62$	$76.92$	$91.32$	$84.12$
RVM	$49.19$	$80.50$	$94.67$	$87.58$

Appendix

Table 3: Evaluation results on SkyReels-I2V at native spatial resolution We report VBench-I2V scores on the VBench-I2V dataset. Dyn. Degree is the VBench dynamic-degree dimension, one of the seven quality dimensions already averaged into Quality .

Method	PickScore	ClipScore	HPSv2.1	Aesthetic	ImgRwd
FlowGRPO †	$23.50$	$0.280$	$0.316$	$5.90$	$1.29$
DiffusionNFT †	$23.61$	$0.288$	$0.344$	$6.04$	$1.46$
PEPG †	$23.68$	$0.296$	$0.325$	$6.06$	$1.45$
RVM	$23.30$	$0.289$	$0.333$	$5.82$	$1.38$

Appendix

Table 4: Evaluation Results on SD3.5-M (DrawBench) We report the text-to-image PickScore fine-tuning task, evaluated on DrawBench with the protocol of Choi et al. (2026) . Training uses PickScore, CLIPScore, and HPSv2.1. Blue shading marks the first and second highest scores per column, including ties. † cited numbers from Choi et al. (2026) .

Panel of Figure 12 — Figure 12: Per-reward training curves on Wan2.1-T2V-1.3B Each panel shows one reward component across training iterations; dashed lines mark the untrained base.

Panel of Figure 13 — Figure 13: Additional qualitative comparison on Wan2.1-T2V-1.3B Frames $1,14,27,40,53$ of each method for the same prompt; top to bottom: Wan2.1-1.3B (w/o CFG), Wan2.1-1.3B (w/ CFG), DiffusionNFT, RAM, Ours (RVM).

Panel of Figure 14 — Figure 14: Additional qualitative comparison on Wan2.1-T2V-1.3B . Frames $1,14,27,40,53$ of each method for two prompts; within each group of three rows, top to bottom: Wan2.1-1.3B (w/o CFG), Wan2.1-1.3B (w/ CFG), Ours (RVM).

Panel of Figure 15 — Figure 15: Additional qualitative comparison on Wan2.1-T2V-1.3B Frames $1,14,27,40,53$ of each method for two prompts; within each group of three rows, top to bottom: Wan2.1-1.3B (w/o CFG), Wan2.1-1.3B (w/ CFG), Ours (RVM).

Panel of Figure 16 — Figure 16: Additional qualitative comparison on SkyReels-I2V Frames $1,14,27,40,53$ for three prompts; within each pair, SkyReels-I2V-V1 (top) and Ours (RVM) (bottom).

Panel of Figure 17 — Figure 17: Additional qualitative comparison on SkyReels-I2V Frames $1,14,27,40,53$ for three prompts; within each pair, SkyReels-I2V-V1 (top) and Ours (RVM) (bottom).

Explore similar work

Apr 19, 2026

Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models

Reward-based fine-tuning steers a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model. Although existing methods are derived from different perspectives, we show that many can be written under a common framework, which we call reward score matching (RSM). Under this view, alignment becomes score matching against a value-guided target, and the main differences across methods reduce to the construction of the value-guidance estimator and the effective optimization strength across timesteps. This unification clarifies the bias-variance-compute tradeoffs of existing designs, and distinguishes core optimization components from auxiliary mechanisms that add complexity without clear benefit. Guided by this perspective, we develop simpler, more efficient redesigns across representative differentiable and black-box reward alignment tasks. Overall, RSM turns a seemingly fragmented collection of reward-based fine-tuning methods into a smaller, more interpretable, and more actionable design space. Code is available at https://github.com/jaylee2000/rsm

Jeongjae Lee, Jinho Chang, Jeongsol Kim +1

Graduate School of AI KAIST, Korea

Diffusion Model Alignment Diffusion Model Fine-Tuning Score-Based Generative Modeling

Ask PDF

May 12, 2026

Efficient Adjoint Matching for Fine-tuning Diffusion Models

Reward fine-tuning has become a common approach for aligning pretrained diffusion and flow models with human preferences in text-to-image generation. Among reward-gradient-based methods, Adjoint Matching (AM) provides a principled formulation by casting reward fine-tuning as a stochastic optimal control (SOC) problem. However, AM inevitably requires a substantial computational cost: it requires (i) stochastic simulation of full generative trajectories under memoryless dynamics, resulting in a large number of function evaluations, and (ii) backward ODE simulation of the adjoint state along each sampled trajectory. In this work, we observe that both bottlenecks are closely tied to the non-trivial base drift inherited from the pretrained model. Motivated by this observation, we propose Efficient Adjoint Matching (EAM), which substantially improves training efficiency by reformulating the SOC problem with a linear base drift and a correspondingly modified terminal cost. This reformulation removes both sources of inefficiency; it enables training-time sampling with a few-step deterministic ODE solver and yields a closed-form adjoint solution that eliminates backward adjoint simulation. On standard text-to-image reward fine-tuning benchmarks, EAM converges up to 4x faster than AM and matches or surpasses it across various metrics including PickScore, ImageReward, HPSv2.1, CLIPScore and Aesthetics.

Jeongwoo Shin, Dongsoo Shin, Yuchen Zhu +5

Seoul National University · Georgia Institute of Technology · Sungkyunkwan University

Diffusion Model Alignment Stochastic Optimal Control Diffusion Model Fine-Tuning RL for Diffusion Models T2I Diffusion Models

Ask PDF

May 11, 2026

Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models

Diffusion and flow-matching models scale because pretraining is supervised regression: a clean sample is noised analytically, and a model regresses against a closed-form target. RL post-training aligns the model with a reward. In image generation, this makes samples compose objects correctly, render text legibly, and match human preferences. Existing methods rely on costly SDE rollouts, reward gradients, or surrogate losses, sacrificing pretraining's regression structure. We show that the structure extends to RL post-training. Under KL-regularized reward maximization, the optimal generative process tilts the clean-endpoint distribution towards samples with higher reward and leaves the noising law unchanged. Combining this with the adjoint-matching optimality condition and a REINFORCE identity, we derive Reinforce Adjoint Matching (RAM): a consistency loss that corrects the pretraining target with the reward. At each step, we draw a clean endpoint from the current model, evaluate its reward, noise it as in pretraining, and regress. No SDE rollouts, backward adjoint sweeps, or reward gradients are required. Like the pretraining objective, RAM is simple and scales. On Stable Diffusion 3.5M, RAM achieves the highest reward on composability, text rendering, and human preference, reaching Flow-GRPO's peak reward in up to

50\times

fewer training steps.

Andreas Bergmeister, Stefanie Jegelka, Nikolas Nüsken +2

1TU Munich, MCML · 2MIT CSAIL · 3King’s College London +2

Diffusion Model Alignment Flow Matching RL for Diffusion Models RL for Generative Models RL Post-Training

Ask PDF