Video diffusion alignment has been heavily relied on scalar rewards. These rewards are typically derived from learned reward models in human preference datasets, requiring additional training and extensive collection. Moreover, scalar rewards provide coarse, global supervision, offering limited prompt-generation mismatch credit assignment and making models prone to reward exploitation and unstable optimization. We propose Diffusion-DRF, a free, rich, and differentiable reward framework for video diffusion fine-tuning. Diffusion-DRF employs a frozen, off-the-shelf Vision-Language Model (VLM) as the critic, eliminating the need for reward model training. Instead of relying on a single scalar reward, it decomposes each user prompt into multi-dimensional questions with freeform dense VQA explanation queries, yielding information-rich feedback. By direct differentiable optimization over this rich feedback, Diffusion-DRF achieves stable reward-based tuning without preference datasets collection. Diffusion-DRF achieves significant gains both quantitatively and qualitatively, outperforming state-of-the-art Flow-GRPO by 4.74% in overall performance on unseen VBench-2.0.
Figures & tables
Figure 2: Gradient Analysis. In contrast to the latest scalar-based VideoAlign [ 24 ] and our Vanilla-DRF baseline built on a simple yes/no text-video alignment VLM feedback, our Diffusion-DRF leverages free and rich structured feedback to empirically yield more semantically coherent and spatially concentrated gradients, with heatmaps that better align with regions where the generated video appears to violate the input prompt. We present more visualizations in Section C.1 .
Figure 3: Pipeline of Diffusion-DRF. Given a reference video and its caption as the prompt, our Diffusion-DRF framework queries a frozen VLM with structured multi-dimensional questions spanning text-video alignment (TA), physical fidelity (Phy), and visual quality (VQ). For each dimension, the VLM produces a Yes/No binary judgment together with a dense free-form explanation. The logit difference through next-token prediction defines a differentiable reward for each query, and the aggregated multi-dimensional rewards are backpropagated through the VAE decoder and the final K denoising steps to update the latent diffusion model (e.g., DiT [ 32 ] ).
Figure 4: Automatic Annotation Pipeline. Instead of relying on vague global questions or human preference pairs, we construct structured queries directly from captioned videos. Given a video caption as the prompt, we extract atomic key points and formulate targeted questions across three domains: text-video alignment (TA), physical fidelity (Phy), and visual quality (VQ). Each question is posed in a minimal and unambiguous Yes/No format with a constrained free-form explanation, encouraging precise and localized judgments from the frozen VLM. For Phy and VQ, the same queries are additionally applied to caption-matched real videos to obtain reference answers that anchor physical and perceptual assessment. Detailed prompts and domain facets are provided in I .
Method
VBench-2.0
Overall
Creativity
Common
Controllability
Human
Physics
Material
Dynamic
Motion
Complex
Cameras
Sense
Fidelity
Attribute
Rationality
Landscape
Motion
Wan2.1-1.3B-T2V [ 42 ]
52.99
53.79
55.52
26.59
80.65
48.40
36.23
37.00
37.36
17.33
20.68
Flow-DPO [ 24 ]
50.27
46.77
50.28
27.76
71.76
54.78
69.24
40.29
35.06
18.44
22.24
Flow-GRPO [ 23 ]
50.64
44.71
50.85
25.48
77.80
54.37
69.07
36.63
36.21
14.89
19.32
PickScore [ 19 ]
49.62
35.97
55.23
23.88
81.89
51.13
66.67
39.19
36.78
16.89
22.84
Table 1: Quantitative results on VBench-2.0. In addition to the overall score, we report representative sub-dimension metrics closely related to text-video alignment and physical fidelity. The best results are highlighted in bold , and the second best are underlined. For completeness, we evaluate Diffusion-DRF with three different VLM backbones: InternVL3-1B [ 62 ] , Qwen2.5-VL-3B, and Qwen2.5-VL-7B [ 1 ] , denoted as Diffusion-DRF (1B), (3B), and (7B), respectively. All methods are trained with Wan2.1-1.3B-T2V as base model on the same datasets for fair comparisons.
Figure 5: Qualitative Comparison. All videos are generated under the same configuration and random seed. Across both examples, prior reward-tuning baselines either miss key entities or relations in the prompt, or noticeably degrade video quality, while our method better preserves semantic fidelity and visual coherence. For example, competing methods fail to correctly model two different dogs under complex lighting in the left case and omit the child in the right case, whereas our approach faithfully captures both the described motion and entities. See supplementary videos for full video comparisons across methods.
Metrics
Win (%)
Tie (%)
Lose (%)
VideoAlign
53.3
11.0
35.7
Human
29.0
54.0
17.0
Table 2: Pairwise evaluation of Diffusion-DRF vs. the base model, including VideoAlign-based automatic preference and human judgments.
Figure 6: Training dynamics of the same base model optimized with different rewards.
Figure 7: Qualitative results of single-prompt reward tuning. The base model, VideoAlign [ 24 ] , and PickScore [ 19 ] all exhibit noticeable defects. In contrast, Diffusion-DRF produces the most faithful result without introducing visual distortions.
Question sets
VBench-2.0
Baseline
52.99
TA Only
52.15
VQ Only
52.45
Phy Only
51.81
TA + Phy
53.55
TA + VQ
53.92
Table 3: Ablation studies on question sets and styles.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
class
config
value
VAE
temporal_compression_ratio
4
spatial_compression_ratio
8
Text Encoder
tokenizer
google/umt5-xxl
text_length
512
vocab size
256384
dim
4096
Appendix
Table 4: Training Configure.
Method
GPUs
Peak VRAM / GPU
Sec. / step
Selected step
GPU-hours
VBench-2.0
VideoAlign [ 24 ]
16 × A100-80GB
72 GB
22.50
500
50.00
53.55
Flow-GRPO [ 23 ]
16 × A100-80GB
68 GB
75.00
500
166.67
50.64
Diffusion-DRF (7B)
16 × A100-80GB
75 GB
23.73
1500
158.17
55.38
Appendix
Table 5: Training cost of online reward-tuning methods. All methods fine-tune Wan2.1-1.3B-T2V on the same 16 × A100-80GB node; “Selected step” is the checkpoint reported in Table 1 . GPU-hours are computed as GPUs×sec./step×selected step/3600 . Flow-DPO is omitted because it trains offline on pre-collected preference pairs at a single sampled timestep and does not roll out videos during optimization.
Figure 8: Gradient heatmap visualizations for different reward formulations. Original video failed to satisfy the color requirement of the turtle’s shell (left) and two men (right).
Objective
All defects
Split by caption grounding
Unmentioned background
Caption-grounded foreground
Random masks (control)
1.0 ×
1.00 ×
1.00 ×
Answer only
1.3 ×
1.34 ×
1.45 ×
TA only
–
1.05 ×
2.34 ×
Phy only
–
2.00 ×
2.00 ×
VQ only
–
2.15 ×
2.12 ×
Appendix
Table 6: Gradient localization on synthetic defects. Localization enrichment (higher is better) is the fraction of absolute reward-gradient mass that falls inside the defect mask divided by the mask’s spatio-temporal volume fraction; 1.0× corresponds to a spatially uniform gradient. Left: 200 held-out clips with localized blur, noise, or frame-freeze corruptions inserted into foreground or background regions. Right: a second set of 200 held-out clips whose captions contain no background description, split by whether the corrupted region is caption-grounded foreground or caption-unmentioned background. “Random masks” are size-, shape-, and duration-matched control masks.
Method
VBench-2.0
Overall
Creativity
Common
Controllability
Human
Physics
Sense
Fidelity
CogVideoX
49.52
37.88
54.87
24.56
78.10
52.18
w/ Diffusion-DRF
51.89
40.63
57.75
27.11
81.28
52.69
Appendix
Table 7: Generalization to CogVideoX. We report VBench-2.0 results of CogVideoX and of CogVideoX fine-tuned with Diffusion-DRF.
Figure 9: Pair-wise evaluation on VideoGen-Eval. With the same prompt and configuration, we perform pairwise comparisons of generated videos using the VideoAlign scores of text-video alignment (TA), visual quality (VQ), motion quality (MQ) and overall (OA). We compare the Diffusion-DRF with the base model (left) and the Flow-GRPO (right) respectively.
Method
VBench-2.0
Overall
Creativity
Common Sense
Controllability
Human Fidelity
Physics
Baseline
52.99
53.79
55.52
26.59
80.65
48.40
TA Only
52.15
49.41
53.80
27.03
74.88
55.64
VQ Only
52.45
50.53
53.22
27.14
80.22
51.28
Phy Only
51.81
37.04
57.82
27.14
79.46
56.28
TA + Phy
53.55
50.56
60.12
25.72
75.57
55.78
Appendix
Table 8: Detailed results of ablation studies on question sets and styles.
Method
VBench-2.0
Training
Overall
Creativity
Common
Controllability
Human
Physics
VRAM
Sense
Fidelity
Usage (Gb)
Nf=2
50.83
48.43
48.10
27.50
80.96
54.14
62
Nf=4
51.45
43.22
53.51
27.64
78.4
54.45
65
Nf=6
52.19
43.85
55.23
29.25
78.88
53.75
68
Nf=8
54.80
54.50
55.81
28.10
79.61
56.03
71
Appendix
Table 9: Additional ablation studies on input frames. We report the results of the ablations on changing the number of input frames for the VLM on VBench-2.0. All models are trained with the same configuration as Diffusion-DRF.
Target format
Single run
Three seeds (mean ± std)
Answer only
53.30
–
Shuffled explanation (length-matched)
52.47
–
Explanation → Answer
55.24
55.31 ± 0.16
Answer → Explanation (default)
55.38
55.44 ± 0.26
Appendix
Table 10: Explanation ordering and content controls (VBench-2.0 Overall). All variants use the same queries, data, and optimization settings and differ only in the supervised target sequence. The shuffled control keeps the answer tokens and the explanation length but pairs each query with an explanation drawn from another sample in the same supervision group with the same answer label. Three-seed statistics are reported where available ( Section D.4 ).
Setting
Overall
Diversity
Base model
52.99
58.6
PickScore [ 19 ]
49.62
29.5
VideoAlign [ 24 ]
53.55
52.4
SFT on reference videos (matched budget)
53.42
–
Diffusion-DRF, no reference
53.15
–
Diffusion-DRF, shuffled reference
53.06
–
Appendix
Table 11: Role of the caption-matched reference video (VBench-2.0). “Shuffled reference” pairs each caption with a real video from a different caption; “SFT” directly fine-tunes the generator on the same 5K real videos with a matched update budget. Diversity is the VBench-2.0 diversity score (higher means more cross-sample variation).
Training setting
Reference
Explanation
OpenVid-1M
VideoUFO
Base model (Wan2.1-1.3B-T2V)
52.99
52.99
Answer only, no reference
53.02
53.10
Full feedback, no reference
✓
53.15
53.22
Answer only, with reference
✓
53.30
53.54
Full Diffusion-DRF
✓
✓
55.38
55.15
Interaction (difference-in-differences)
+1.95
+1.49
Appendix
Table 12: Reference conditioning versus explanation supervision (VBench-2.0 Overall). A 2×2 factorial over (i) whether Phy/VQ queries are conditioned on the caption-matched reference video and (ii) whether free-form explanations are supervised in addition to the Yes/No answers, on two training corpora. “Answer only” and “Full” use the same TA/VQ/Phy query set. The VideoUFO column uses 5,000 caption–video pairs from VideoUFO [ 43 ] with the same filtering, query pipeline, TA:Phy:VQ ratio, initialization, and optimization settings. Rows within ±0.26 of the base model (one run-to-run standard deviation, Section D.4 ) are not distinguishable from it.
Figure 10: Additional Samples of Single Prompt Finetuning.
Figure 11: Additional Gradient Heatmaps - Part1.
Figure 12: Additional Gradient Heatmaps - Part2.
Figure 13: Prompts used for the caption decomposition. Part 1
Figure 14: Prompts used for the caption decomposition - Part2.
Figure 15: Prompts used in the training for text-video alignment (TA).
Figure 16: Prompts used in the training for physical fidelity (Phy).
Figure 17: Prompts used in the training for visual quality (VQ).
Achieving simultaneous preference alignment and distillation acceleration in video diffusion models remains an open challenge. Existing methods optimize the two objectives over mismatched representation spaces, where improving one objective often compromises the other. To overcome this, we propose Reward Lightning, a unified framework that aligns and accelerates a video diffusion model within a single shared representation. Its central principle is homology: both objectives are evaluated on identical latent features, which mitigates the gradient conflicts that arise when they are optimized over disjoint representations. As a foundational component, we first introduce a latent reward model (LRM) that scores videos directly in the latent space, without decoding back to the pixel space. Building on the LRM, homologous preference distillation (HPD) reuses this shared backbone to perform adversarial distillation and preference alignment jointly, yielding few-step generators that remain faithful and well aligned. Extensive experiments demonstrate that the LRM surpasses pixel-level and latent-level reward baselines by 11.0% and 14.7% in preference accuracy, and that Reward Lightning generates high-fidelity videos in merely 1 to 4 steps, improving the average VBench score by 2.1% while leading in text alignment, motion quality, and visual quality. Project page: https://reward-lightning.github.io.
Efficiently aligning large-scale video diffusion models with human intent requires a scalable and trajectory-aware pathway that bridges the inherent discrepancy between training noise distributions and practical inference trajectories. While existing paradigms such as Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO) attempt to address this, they are often hindered by either reliance on bias-prone, complex reward models or suboptimal timestep sampling. In this paper, we propose Diffusion-APO (Aligned Preference Optimization), a trajectory-aware algorithm that resolves this misalignment by synchronizing training noise with inference-time denoising paths to maximize gradient signal efficacy. To translate this algorithmic innovation into a practical solution, we introduce a unified and modular RLHF framework that integrates online ranking, half-online anchoring, offline refinement, and distillation-aware drift correction. This framework enables flexible, multi-stage preference alignment across diverse data and computational constraints without relying on scalar-reward-based policy gradients. Through extensive experiments, we demonstrate that Diffusion-APO consistently outperforms standard baselines in visual quality and instruction following, while effectively preserving generative fidelity during model acceleration, providing a robust, end-to-end pathway for scalable video diffusion alignment.
Image-conditioned video diffusion models achieve impressive visual realism but often suffer from weakened motion fidelity, e.g., reduced motion dynamics or degraded long-term temporal coherence, especially after fine-tuning. We study motion alignment in video diffusion models post-training. To address this, we introduce pixel-motion rewards based on pixel flux dynamics, capturing both instantaneous and long-term motion consistency. We further propose Smooth Hybrid Fine-tuning (SHIFT), a scalable reward-driven framework that unifies supervised fine-tuning and advantage-weighted fine-tuning. Benefiting from novel adversarial advantages, SHIFT improves convergence speed and mitigates reward hacking. Experiments show that our approach efficiently resolves dynamic-degree collapse in modern video diffusion models supervised fine-tuning. Project page: https://xiye20.github.io/projects/SHIFT/.
Xi Ye, Wenjia Yang, Yangyang Xu +4
Tsinghua University · Ant Group · University of Chinese Academy of Sciences