GuidanceNFT: NFT-Based Reward Fine-Tuning of Guided Diffusion Models

Jiayin Liu; Zeyu Michael Li; Bingshuo Qian; Xiang Cheng

cs.LGOct 11, 2026

GuidanceNFT: NFT-Based Reward Fine-Tuning of Guided Diffusion Models

Authors: Jiayin Liu, Zeyu Michael Li, Bingshuo Qian, Xiang Cheng

Organizations: Tsinghua University · Duke University

Abstract

DiffusionNFT (NFT) fine-tunes diffusion models toward task-specific rewards, but optimizing these rewards does not guarantee fidelity to the full prompt. Although classifier-free guidance (CFG) strengthens prompt alignment, NFT is formulated as a CFG-free method, and applying CFG to the fine-tuned velocity field can reduce the optimized reward. This motivates optimizing rewards over the guided output distribution while seeking to retain its prompt alignment. However, applying standard NFT directly to guided samples can cause reward degradation or training collapse. We trace this failure to a mismatch between the reference velocity used in the NFT loss and the score of the guided sample distribution, which biases the improvement direction. We propose GuidanceNFT, which estimates the matching reference velocity through online flow matching on guided samples and uses it to correct both regression terms. With an exact reference, the corrected objective recovers NFT's intended update. In our experiments, the learned correction sustains reward improvement under both explicit and built-in guidance. On SD3.5-M, GuidanceNFT achieves higher optimized rewards than NFT with and without CFG across GenEval, text-rendering, and multi-reward objectives, together with better overall prompt alignment as measured by off-domain evaluation metrics. Experiments on models with built-in guidance, including Model Guidance and ELF, further demonstrate improved ImageNet generation, a better perplexity-entropy tradeoff on OpenWebText, and higher accuracy on GSM8K.

Figures & tables

Figure 1: Updating the guided field improves reward while retaining data structure. (a–b) Pretrained samples without and with CFG. (c) NFT moves samples toward the rewarded branches, but (d) adding CFG afterward pushes many samples off the tree. (e) Updating the guided field better combines reward targeting with tree fidelity. Panels (c–e) zoom into the dashed region; blue branches mark the reward target and orange points mark off-tree samples.

Figure 2: Reference error distorts the reward-driven update. (a) A matched reference spreads samples across the rewarded branches through the intended update in (b). (c) An incorrect reference concentrates samples on fewer branches. (d) The reference error adds a displacement that (e), applied alone, moves samples away from the reward target. Hit reports the fraction reaching the reward target and cells the spatial diversity.

Figure 3: Reference correction sustains reward improvement under CFG. (a) At evaluation CFG 1.5, SiT NFT loses reward even after restoring the pretrained empty-condition prediction; GuidanceNFT continues to improve. (b) With CFG-4 rollouts, SD3.5-M NFT and analytic correction collapse, whereas GuidanceNFT improves GenEval reward. (c) Faster collapse accompanies larger reference gaps at noise level

\sigma=0.5

. Horizontal axes show training updates. Bars in (a) show evaluation-seed std; bands in (c) span the mean to one prompt std above it.

Training objective	Method	Train reward	Held-out evaluation
			GenEval	OCR	PickScore	CLIPScore	HPSv2.1	Aesthetic	ImgRwd
—	Pretrained w/o CFG	—	0.283	0.113	20.69	0.232	0.199	4.990	-0.594
	Pretrained w/ CFG	—	0.636	0.532	22.38	0.278	0.276	5.252	0.820
GenEval	NFT w/o CFG	0.900	0.855	0.434	21.70	0.265	0.234	4.886	0.271
	NFT w/ CFG	0.657	0.565	0.624	21.19	0.271	0.216	4.773	0.148
	GuidanceNFT	0.914	0.877	0.674	21.83	0.277	0.250	4.947	0.682

Table 1: SD3.5-M reward improvement and quality retention. W/o and w/ CFG denote evaluation scales 1 and 4. Fine-tuned rows average evaluation seeds 42 and 123; pretrained rows use seed 42. Train reward measures the optimized objective on its test set. Gray marks training criteria; bold marks the best within each objective. Seed stds are given in Appendix D .

Figure 4: GuidanceNFT retains prompt content during reward specialization. Selected SD3.5-M examples show object and spatial alignment after Learned3 fine-tuning (first four columns), and readable text within the requested scenes after OCR fine-tuning (last four). Rows compare NFT at CFG 1, NFT at CFG 4, and GuidanceNFT at CFG 4.

Figure 5: GuidanceNFT avoids degradation with built-in guidance. (a) ImageNet reward continues to improve on Model Guidance. (b) On ELF-B, it maintains text entropy on OpenWebText at SCCFG 3. (c) On the Qwen-based ELF model, it retains GSM8K accuracy at SCCFG 2. Standard NFT degrades in all three settings. Horizontal axes show training updates; faint traces and bold curves show raw and smoothed values.

Figure 6: GuidanceNFT produces more detailed class-conditional images. Selected ImageNet examples compare NFT (top) and GuidanceNFT (bottom). (a) SiT uses CFG 1 for NFT and CFG 1.5 for GuidanceNFT. (b) Model Guidance uses built-in guidance with external CFG 1.

Method	Pass@1	Pass@2	Pass@4	Pass@8
Pretrained	39.17 $\pm$ 1.04	49.82 $\pm$ 1.15	59.20 $\pm$ 1.21	67.32 $\pm$ 1.29
NFT	41.68 $\pm$ 1.08	51.53 $\pm$ 1.17	59.78 $\pm$ 1.22	67.02 $\pm$ 1.30
CEDR	42.16 $\pm$ 1.11	51.36 $\pm$ 1.19	59.40 $\pm$ 1.24	66.26 $\pm$ 1.32
GuidanceNFT	43.03 $\pm$ 1.10	52.66 $\pm$ 1.16	61.24 $\pm$ 1.20	68.61 $\pm$ 1.26

Table 2: GSM8K with SCCFG-2 evaluation. Pass@

k

(%): at least one correct answer in

k

attempts. Selected checkpoints are within 500 updates, including NFT’s best retained point. Values are mean

\pm

problem-bootstrap std (Appendix G ).

Figure 7: GuidanceNFT improves the language quality–diversity tradeoff. Left: ELF-B outputs remain varied with GuidanceNFT but become repetitive with NFT; both train and sample at SCCFG 3. Right: GuidanceNFT improves the perplexity–entropy tradeoff over the pretrained model and CEDR-style reconstruction. GuidanceNFT and reconstruction train at SCCFG 3 and 1, respectively. Markers vary evaluation SCCFG, with endpoint

w

values labeled. The dotted line marks entropy 5.15 nats for comparing perplexity at matched diversity.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Freeze the rollout policy and baseline prediction $v^{\rm old}$ .
	Generate endpoints $X$ with the specified guidance and sampler; evaluate and freeze reward weights $r$ .
	Draw fresh $(t,\epsilon)$ and construct $(X_{t},U)$ by Equation 14 .
	Refresh $v_{\phi}$ with unweighted FM on the endpoint population used by the NFT objective.
	Freeze $v_{\phi}$ and form the policy displacement $d=v_{\theta}-v^{\rm old}$ .
	Form $\mathcal{C}_{\phi}=\operatorname{sg}(v_{\phi}-v^{\rm old})$ and regress $v^{\rm old}+\mathcal{C}_{\phi}\pm\beta d$ in the two terms of Equation 12 .
	Update the policy using the NFT regressions and the regularization penalty (Appendix C ); refresh the rollout policy according to its schedule.

Appendix

Algorithm 1 GuidanceNFT (one outer iteration)

Method	Reward $\uparrow$	Hit $\uparrow$	On-tree $\uparrow$
Base	0.755	0.254	0.848
CFG	0.721	0.347	0.946
NFT	0.956	0.630	0.805
NFT + CFG	0.985	0.851	0.715
Guided-field NFT	0.996	0.971	0.907

Appendix

Table 3: Endpoint metrics for the toy experiment in Appendix B.1 . Figure 1 displays the same five configurations.

Figure 8: Reference error redirects samples or reduces reward-target coverage. Rows compare the intended update with reference-error displacements, applied alone or together with it. Columns show (a) the reference field, (b) reference-error magnitude and update direction, (c) the resulting field and rollout density, and (d) final samples. Error alone redirects samples away from rewarded branches; combined with the intended update, it narrows their spatial coverage. Blue branches mark the reward target.

Method	Rollouts	Loss reference	Loss prediction
SD3.5 and SiT: explicit CFG
NFT	$q_{1}$	Conditional output	Conditional output
NFT + pretrained empty branch	$q_{1}$	Conditional output	Conditional output
NFT (uncorrected)	$q_{w}$	Conditional output	Conditional output
NFT (analytic correction)	$q_{w}$	Analytic CFG field	Conditional output
GuidanceNFT	$q_{w}$	Fitted reference	Conditional output

Appendix

Table 4: Rollout distribution, loss reference, and loss prediction.

q_{1}

denotes unguided rollouts and

q_{w}

guided rollouts (

q_{\rm CFG}

in Section 3 ), including built-in guidance with external CFG 1. SD3.5 uses

w=4

, SiT uses

w=1.5

, and the Qwen-based ELF reasoning model uses native SCCFG 2. Conditional output denotes

v_{\theta,c}

; the native guided field is the model’s built-in guided prediction. Loss references use detached old outputs or a fitted reference. For CEDR, the reconstructed reference and loss prediction apply its detached SCCFG correction. NFT + pretrained empty branch ( Phunyaphibarn et al., 2026 ) replaces only the empty-condition prediction

v_{\varnothing}

(the empty branch) at inference.

Setting	Retained NFT candidates	NFT selected	GuidanceNFT reported
SiT / ImageNet	100–400, every 50 updates	200	400
Model Guidance / ImageNet	200, 400	200	400
Qwen / GSM8K	50–400, every 50 updates	200	500

Appendix

Table 5: Checkpoints used for the main comparisons. NFT is selected by PickScore for SiT/MG and by Pass@1 under the 64-step asynchronous protocol for GSM8K. GuidanceNFT uses the reported fixed checkpoints. Candidate grids differ across methods; reported uncertainty is conditional on the selected checkpoints.

Metric	Comparator	GuidanceNFT $-$ comparator (mean $\pm$ std)
GenEval dense	NFT, CFG 4	$+0.2462\pm 0.3182$
GenEval dense	NFT, CFG 1	$+0.0143\pm 0.1737$
GenEval dense	NFT + pretrained empty branch, CFG 4	$+0.1043\pm 0.2719$
GenEval task macro	NFT, CFG 4	$+0.3053\pm 0.1353$
GenEval task macro	NFT, CFG 1	$+0.0269\pm 0.0256$
GenEval task macro	NFT + pretrained empty branch, CFG 4	$+0.1337\pm 0.0406$

Appendix

Table 6: SD3.5 at update 200. Stds describe paired differences across 553 prompt means for GenEval dense reward, six task means for GenEval task-macro success, and 200 prompt means for Learned3. Each GenEval prompt retains four images; each Learned3 prompt retains ten images across two evaluation seeds. Learned3 is PickScore/26 + CLIPScore + HPSv2.1. These stds describe spread across units; they measure neither training variation nor the uncertainty of the mean.

Objective	Method	CFG	Train reward	GenEval	OCR	PickScore	CLIPScore	HPSv2.1	Aesthetic	ImgRwd
GenEval	NFT	1	0.900 $\pm$ 0.0024	0.855 $\pm$ 0.0077	0.434 $\pm$ 0.0059	21.70 $\pm$ 0.0021	0.265 $\pm$ 0.0002	0.234 $\pm$ 0.0001	4.886 $\pm$ 0.0132	0.271 $\pm$ 0.0178
GenEval	NFT	4	0.657 $\pm$ 0.0139	0.565 $\pm$ 0.0003	0.624 $\pm$ 0.0144	21.19 $\pm$ 0.0061	0.271 $\pm$ 0.0014	0.216 $\pm$ 0.0006	4.773 $\pm$ 0.0116	0.148 $\pm$ 0.0224
GenEval	GuidanceNFT	4	0.914 $\pm$ 0.0015	0.877 $\pm$ 0.0013	0.674 $\pm$ 0.0016	21.83 $\pm$ 0.0099	0.277 $\pm$ 0.0004	0.250 $\pm$ $<0.0001$	4.947 $\pm$ 0.0128	0.682 $\pm$ 0.0153
OCR	NFT	1	0.971 $\pm$ 0.0014	0.035 $\pm$ 0.0042	0.971 $\pm$ 0.0008	19.61 $\pm$ 0.0118	0.208 $\pm$ 0.0009	0.137 $\pm$ 0.0006	4.208 $\pm$ 0.0036	-1.679 $\pm$ 0.0183
OCR	NFT	4	0.192 $\pm$ 0.0067	0.265 $\pm$ 0.0064	0.235 $\pm$ 0.0070	20.21 $\pm$ 0.0188	0.242 $\pm$ 0.0001	0.188 $\pm$ 0.0008	4.596 $\pm$ 0.0051	-0.919 $\pm$ 0.0642
OCR	GuidanceNFT	4	0.975 $\pm$ 0.0010	0.488 $\pm$ 0.0099	0.974 $\pm$ 0.0009	21.83 $\pm$ 0.0293	0.274 $\pm$ 0.0003	0.256 $\pm$ 0.0022	5.120 $\pm$ 0.0072	0.425 $\pm$ 0.0426

Appendix

Table 7: SD3.5 rewards and evaluation-seed variation. Each cell shows the mean on the first line and

\pm

sample std on the second, using two evaluation seeds per fixed update-200 checkpoint. Train reward uses on-domain test evaluations; the remaining metrics follow Table 1 . All evaluations use 40 sampling steps at

512\times 512

; PickScore is unscaled. Bold marks the best mean within each objective. These stds measure sampling variation, not training variability.

Task	Prompts	NFT (%)	GuidanceNFT (%)	Difference $\pm$ std
color_attr	100	22.25	74.25	$+52.00\pm 39.99$
colors	94	61.70	82.98	$+21.28\pm 39.99$
counting	80	58.75	92.19	$+33.44\pm 39.17$
position	100	56.50	87.00	$+30.50\pm 46.00$
single_object	80	87.19	99.06	$+11.88\pm 26.38$
two_object	99	59.34	93.43	$+34.09\pm 40.00$

Appendix

Table 8: GenEval success by task. Results use update 200, evaluation CFG 4, one evaluation seed, and four images per prompt. Differences and sample stds are in percentage points across paired prompt means within each task. These stds describe prompt variation at fixed checkpoints.

Figure 9: Learned3 fine-tuning retains more requested content with GuidanceNFT. Six selected SD3.5-M examples compare object, spatial, and text alignment. Columns show different prompts; rows show NFT at CFG 1, NFT at CFG 4, and GuidanceNFT at CFG 4, from top to bottom.

Figure 10: GuidanceNFT combines readable text with the requested scene. Six selected SD3.5-M OCR examples compare NFT at CFG 1 (top), NFT at CFG 4 (middle), and GuidanceNFT at CFG 4 (bottom). NFT often omits scene content, while adding CFG disrupts text legibility. Column headings give the target text.

Figure 11: GuidanceNFT preserves color and shading during GenEval fine-tuning. Selected SD3.5-M examples compare NFT at CFG 1 (top), NFT at CFG 4 (middle), and GuidanceNFT at CFG 4 (bottom). Adding CFG to NFT darkens images and obscures detail; GuidanceNFT retains more visible color and shading. Column headings summarize the prompts.

Figure 12: A fitted reference better represents Model Guidance rollouts. (a–b) On NFT and GuidanceNFT samples, fitted references have lower flow-matching error than the native field; zero denotes a separately fitted reference. (c) GuidanceNFT continues to improve reward as NFT declines; the vertical line marks update 200. Bands show mean

\pm

one std across classes.

Figure 13: GuidanceNFT limits reference mismatch as training progresses. Columns compare NFT, analytic correction, and GuidanceNFT on SD3.5-M. Top: disagreement with a fitted rollout reference across noise levels. Bottom: flow-matching error relative to that reference; negative values indicate lower error. From update 50 (dashed) to 100 (solid), mismatch grows for NFT and analytic correction but remains small for GuidanceNFT. Bands show mean

\pm

one std across prompts.

Task	Method	$n$	Rollout/ reward	Reference	Policy	Iteration median [IQR]	Ratio
GenEval	NFT	193	113.6	0.0	274.3	387.9 [387.2, 388.7]	1.00 $\times$
GenEval	GuidanceNFT	184	182.2	150.6	339.7	672.4 [671.1, 674.3]	1.73 $\times$
Learned3	NFT	94	92.5	0.0	274.5	367.3 [366.7, 367.8]	1.00 $\times$
Learned3	GuidanceNFT	96	160.5	150.5	339.2	650.2 [649.9, 650.6]	1.77 $\times$

Appendix

Table 9: Recorded SD3.5 training costs. Host seconds on two H200 GPUs, FP16, microbatch 24 per GPU, and 1,152 rollout endpoints per update.

n

counts eligible complete iterations; IQR is the 25th–75th percentile. Ratios divide iteration medians by NFT within each task. The total includes rollout, reward scoring, reference fitting, and policy updates; component medians need not sum to the total median. Known evaluation/checkpoint and restart boundaries are excluded. Unmarked long pauses remain (2/3 GenEval and 1/0 Learned3 intervals exceed five times the respective median for NFT/GuidanceNFT). Reference initialization is excluded. These are recorded per-iteration latencies, not total training costs; deployment requires no auxiliary-reference pass.

Method	SiT, CFG 1	SiT, CFG 1.5	Model Guidance
Pretrained	18.849 $\pm$ 0.012	19.287 $\pm$ 0.015	19.239 $\pm$ 0.003
NFT	20.161 $\pm$ 0.019	20.476 $\pm$ 0.025	20.010 $\pm$ 0.023
GuidanceNFT	21.465 $\pm$ 0.006	21.579 $\pm$ 0.013	21.580 $\pm$ 0.018

Appendix

Table 10: Unscaled PickScore (

\uparrow

), mean

\pm

sample std. Each checkpoint uses three evaluation seeds and 6,000 images. NFT uses its best observed checkpoint (200); GuidanceNFT uses 400. MG has external CFG 1. Bold marks the best mean. Stds are conditional on the selected checkpoints and do not measure training variation.

Method	CFG	100	150	200	250	300	350	400
NFT	1	$19.392$ $\pm 0.025$	$19.791$ $\pm 0.007$	$\mathbf{20.161}$ $\pm 0.019$	$19.419$ $\pm 0.023$	$19.013$ $\pm 0.009$	$18.477$ $\pm 0.003$	$18.317$ $\pm 0.003$
	1.5	$19.814$ $\pm 0.006$	$20.179$ $\pm 0.010$	$\mathbf{20.476}$ $\pm 0.025$	$19.747$ $\pm 0.026$	$19.094$ $\pm 0.006$	$18.390$ $\pm 0.003$	$18.273$ $\pm 0.004$
GuidanceNFT	1	$19.442$ $\pm 0.016$	$19.880$ $\pm 0.015$	$20.375$ $\pm 0.016$	$20.853$ $\pm 0.003$	$21.157$ $\pm 0.009$	$21.345$ $\pm 0.008$	$\mathbf{21.465}$ $\pm 0.006$
	1.5	$19.924$ $\pm 0.005$	$20.319$ $\pm 0.003$	$20.728$ $\pm 0.008$	$21.097$ $\pm 0.003$	$21.325$ $\pm 0.007$	$21.474$ $\pm 0.016$	$\mathbf{21.579}$ $\pm 0.013$

Appendix

Table 11: SiT-XL/2 unscaled PickScore (

\uparrow

) at every retained checkpoint (columns 100–400). Each cell shows the mean with

\pm

sample std beneath it across three evaluation-seed means, conditional on a fixed checkpoint. Each seed covers all 1,000 ImageNet classes with two images per class (2,000 images per seed; 6,000 images per entry), using ten-step SDE sampling. NFT trains with CFG-1 rollouts. GuidanceNFT first fits its reference for 100 updates and then trains with CFG-1.5 rollouts. Bold marks the best checkpoint within each method and evaluation-CFG row.

Update	50	100	150	200	250	300	350	400	450	500
NFT	$32.12$ $\pm 0.34$	$34.22$ $\pm 0.46$	$34.77$ $\pm 0.38$	$\mathbf{36.57}$ $\pm 0.31$	$35.00$ $\pm 0.88$	$29.54$ $\pm 1.18$	$19.84$ $\pm 1.35$	$13.65$ $\pm 0.35$	—	—
GuidanceNFT	$31.39$ $\pm 0.27$	$31.99$ $\pm 0.20$	$33.21$ $\pm 0.35$	$34.37$ $\pm 0.12$	$34.14$ $\pm 0.12$	$35.08$ $\pm 0.62$	$34.60$ $\pm 0.49$	$35.18$ $\pm 0.30$	$\mathbf{35.38}$ $\pm 0.24$	$35.28$ $\pm 0.29$

Appendix

Table 12: Synchronous SCCFG-2 evaluation. The sweep uses 1,319 GSM8K questions, 32 steps, synchronous clocks

(1,1,1)

, external CFG 1, ELF/prompt EMA

0.99/0.99

, and three evaluation seeds. Each cell shows mean accuracy (%) with

\pm

sample std beneath it, across evaluation seeds for one training run. Bold marks each method's largest observed mean. NFT training stopped after its rollouts produced only incorrect answers, so update 400 is its last checkpoint (dashes). This protocol differs from the 64-step asynchronous main-table evaluation.

Explore similar work

Oct 8, 2026

DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models

Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA) off-policy training. We show that NFT-style objectives contain a hidden forward--reverse consistency force that joint positive--negative training cancels only partially, and that EMA stabilizes training through restoration rather than off-policy rollouts. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which aggregates each prompt's rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models (DiffusionART-Pairwise). Across image and video generation, DiffusionART trains efficiently and stably without per-task tuning. In particular, on HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, respectively, while achieving 28.8% and 53.0% higher mean training rewards.

Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang +6

HPC-AI Lab, National University of Singapore · Kling AI · HKUST

Diffusion Model Fine-Tuning RL for Diffusion Models RL for Image Generation Diffusion Model Alignment RL for Video Generation

Ask PDF

Jun 16, 2026

NoiseTilt: Noise-Tilted Reverse Kernels for Diffusion Reward Alignment

We introduce the Noise-Tilted Reverse Kernel (NTRK), a reward-guided diffusion sampler that injects reward gradients through the noise term, leaving the pretrained reverse kernel unchanged and requiring only a single sample per step. Reward-guided sampling at inference time has greatly expanded the versatility of pretrained diffusion models. Yet existing methods face a trade-off. Gradient-based guidance shifts the reverse mean, steering generation but pushing intermediate states outside the region that the model was trained on and degrading quality. Search-based methods preserve quality but gain no gradient signal. No prior method achieves both. NTRK resolves this by keeping the reverse mean fixed and biasing the noise term toward high reward. This is enabled by a whitening operator, the central mechanism behind NTRK, which converts reward gradients into noise-compatible perturbations without losing their guiding signal. Across various reward alignment tasks, NTRK outperforms recent state-of-the-art baselines without losing sample quality. Remarkably, on aesthetic generation, NTRK surpasses the reward of the best baseline at 500 NFEs using only 25 NFEs, a 20 times reduction in compute.

Jisung Hwang, Yunhong Min, Jaihoon Kim +2

1KAIST · †This work was done in part while the author was a visiting researcher at The University of Tokyo. · 2The University of Tokyo

Diffusion Model Alignment Diffusion Model Guidance Diffusion Guidance Diffusion Sampling

Ask PDF

Aug 24, 2026

Scaling Reinforcement Learning for Diffusion Models via Velocity Matching

Reward-based fine-tuning of diffusion models has largely inherited likelihood-based policy optimization developed for autoregressive large language models (LLMs). Diffusion models, however, are natively trained through velocity regression and do not directly provide the likelihood of a generated sample. Existing methods address this using transition likelihoods along stochastic denoising trajectories or evidence lower bounds (ELBOs) to approximate generated-sample likelihoods. These approximations arise from applying likelihood-based updates to models whose native training and generation operate through velocity fields. We instead take velocity matching as the starting point for reward fine-tuning. We propose reward-based velocity matching (RVM), which weights the velocity-matching loss by reward with an additional anchor regression term, requiring neither likelihood estimation nor likelihood ratios. RVM recovers Reinforce Adjoint Matching at the update level and contains DiffusionNFT as a special case, while ELBO-based methods are closely related through the same velocity-regression structure. This unified view isolates reward design and anchor velocity as principal design choices in velocity-based fine-tuning. Across text-to-image, text-to-video, and image-to-video generation, RVM matches or outperforms the evaluated trajectory-based methods. On Wan2.1-T2V-1.3B, it achieves the highest VBench Overall among the evaluated methods, with substantially lower estimated training costs than the trajectory-based approaches. For video fine-tuning, we introduce a dynamic-tracking reward that provides explicit motion feedback and improves both Dynamic Degree and overall VBench performance.

Jaemoo Choi, Wei Guo, Yuchen Zhu +4

Georgia Institute of Technology · NVIDIA

Diffusion Model Fine-Tuning RL for Diffusion Models RL for Video Generation Video Diffusion Models

Ask PDF