Aligning video generation models with human preference depends critically on the quality of the reward model that scores generated videos. Current video reward models, however, source their reward signal from a model's representational rather than generative capacity: they either repurpose vision-language models as pixel-space scorers, or train auxiliary scoring heads on top of frozen generator features. The generator's actual ability to predict velocities, denoise latents, and produce videos plays no role in scoring its own generations. We argue that this overlooks a structural property of offline preference learning. Any preference optimization objective that approximates an offline preference dataset mathematically targets a strictly defined KL-constrained optimal policy. By isolating the reward term from this analytical solution and mapping the exact generation log-likelihood to its flow-matching surrogate, the empirical reward emerges deterministically as the relative difference in continuous-time velocity-prediction errors. We formalize this equivalence into GenRM-Flow, a framework where a preference-tuned video generator inherently operates as a process-aware reward model directly in the noisy latent space. We further show theoretically that distinct fine-tuning objectives, whether contrastive like DPO and IPO or non-contrastive like RWR, are alternative optimization pathways toward the same optimal policy, each inducing a reward readout from the generator's velocity-prediction errors. Furthermore, the extracted process reward seamlessly substitutes external VLM scorers in FlowAWR. Operating purely on native latent outputs without auxiliary reward networks, this generative reward drives measurable improvements in downstream video generation.
Figures & tables
Figure 1: Videos generated by Wan2.1-T2V-14B fine-tuned with FlowAWR using the GenRM-Flow reward, exhibiting high motion quality and dynamic fidelity.
Figure 2: The pipeline of GenRM-Flow . (Panel A) Traditional paradigms rely on representational capacity , utilizing a frozen backbone (e.g., a VLM or base generator) as a feature extractor coupled with an auxiliary discriminative head. (Panel B) The proposed framework sources the reward directly from generative capacity . Following preference optimization, the implicit reward RD(x,c) is algebraically extracted from the velocity-prediction errors. The scalar score is computed natively by contrasting the active policy (flame, θ⋆ ) against a frozen reference (snowflake, θref ), requiring no architectural modifications or auxiliary heads.
Table 1: Three preference fine-tuning objectives as optimization pathways to obtain the trained weights θ⋆ . While their loss formulations LF(θ) vary significantly—ranging from bounded contrastive margins to reference-free likelihood maximization—they all structurally aim to approximate the optimal offline policy π∗ . Consequently, during the evaluation phase, they implicitly define reward readouts corresponding to their theoretical likelihood shifts: the relative readout of Eq. ( 6 ) for contrastive objectives and a reference-free readout for RWR.
Figure 3: (Left) Pairwise preference prediction accuracy during training. (Middle, Right) HPSv3 reward trajectories during FlowAWR training on the 1.3B and 14B backbones. The implicit process reward yields higher sustained HPSv3 evaluation scores compared to PAVRM.
Base Model
Method
[0, 0.2]
(0.2, 0.4]
(0.4, 0.6]
(0.6, 0.8]
(0.8, 1.0]
Avg
Qwen3-VL-8B
VideoAlign [ 28 ]
–
–
–
–
–
78.26
Wan2.1-T2V-1.3B
PAVRM [ 36 ]
77.19
77.50
78.27
76.11
71.25
76.06
GenRM-Flow
87.77
88.08
88.49
88.61
81.00
86.79
Wan2.1-T2V-14B
PAVRM [ 36 ]
81.87
83.85
84.79
84.69
78.02
82.64
GenRM-Flow
89.10
89.94
90.46
90.75
84.55
88.96
Table 2: Pairwise preference prediction accuracy (%) on the held-out evaluation set. The performance of our implicit process reward is compared against baseline reward models (VideoAlign and PAVRM) across five uniformly spaced noise timestep intervals t∈[0,1] . Bold denotes the highest accuracy per column within each backbone scale.
Figure 4: Qualitative comparison between PAVRM and GenRM-Flow as the per-step reward signal in FlowAWR with a 1.3B backbone. Both runs use FlowAWR with the same configuration, differing only in the reward signal. Red boxes mark localized structural deformations, spatial inconsistencies, and temporal artifacts produced by PAVRM. Across scenes, our reward yields visibly fewer geometric defects and better spatial and temporal continuity.
Base Model
Method
Inner Test Set
VBench [ 10 ]
MS ↑
DD ↑
SC ↑
HA ↑
MS ↑
DD ↑
SC ↑
HA ↑
Wan2.1-T2V-1.3B
Pretrain
98.31
59.07
97.02
88.15
98.52
65.19
97.56
94.00
VideoAlign [ 28 ]
98.58
60.68
97.22
89.96
98.73
67.52
97.64
95.32
PAVRM [ 36 ]
98.56
63.59
97.18
89.73
98.71
67.83
97.52
94.68
GenRM-Flow
98.88
72.43
98.28
93.16
98.95
73.46
97.70
96.25
Wan2.1-T2V-14B
Pretrain
98.12
61.18
97.18
89.32
98.30
65.46
97.52
95.40
Table 3: Comprehensive quantitative comparison of reward-guided generation on our inner test set and the out-of-domain VBench benchmark. Bold denotes the highest metric value per backbone.
Normalization Mechanism
Training Status
MS ↑
DD ↑
SC ↑
HA ↑
Raw rt (no normalization)
Policy Collapse ( ∼ 250 steps)
N/A
N/A
N/A
N/A
Z-score Normalization
Converged
98.74
69.85
97.61
95.61
Softmax (Ours)
Converged
98.95
73.46
97.70
96.25
Table 4: Ablation on intra-group normalization mechanisms during FlowAWR training (Wan2.1-T2V-1.3B, evaluated on VBench). “Policy Collapse” indicates training collapse. Bold denotes the best results among all.
Video generation is central to AI-powered content creation. Aligning generated videos with human preferences is a key criterion for evaluating generation quality. Despite significant progress in visual quality, three key challenges remain. First, the reliability of reward signals is constrained by the quality of human preference data, which is often affected by subjective noise and bias. Second, standard scalar reward models collapse multi-aspect human preferences into a single value, leading to the loss of dynamic trade-offs across multiple preference dimensions. Third, in policy optimization, the widely adopted KL divergence imposes primarily local constraints and may fail to capture the global structure of human preferences. To address these challenges, we propose a unified preference-aware learning framework for video generation. First, we introduce elite-guided filtering to calibrate preference data and construct reliable supervision for reward model training. We then model video quality as a multidimensional reward distribution to capture the uncertainty inherent in human preferences, and use the Wasserstein distance to align the learned reward distribution with the empirical human preference distribution. Finally, we introduce Wasserstein-based distributional alignment into GRPO, guiding policy optimization to better match the global structure of human preferences over videos. Experiments on reward modeling and video generation demonstrate that our approach improves the reliability of reward signals and the perceptual consistency of generated videos. Our code is available at https://github.com/alignhs26/ahs.
Nai-Xin Zhai, Weihua Cheng, Dexu Yu +9
City University of Hong Kong · DeciLix Lab · ShanghaiTech University +7
Video reward models are trained and evaluated on human preferences between videos that often differ simultaneously on properties such as prompt alignment, visual fidelity, motion, and generator identity. As a result, aggregate accuracy does not reveal which video properties a reward model can reliably distinguish. We construct preference pairs by controlled interventions that change one property of a generated video while holding the rest fixed. The resulting pairs, by construction, provide preferences for how the videos differ along a specific dimension, without requiring any human annotation. We propose IsoVRBench, a video generation reward modeling benchmark of 8,000 such pairs across eight interventions, and show that state-of-the-art models separate broad quality differences between generators but miss important features such as temporal order. The best reward model reaches only 60.8% on reordered frames against 100% for humans, and rankings on existing benchmarks do not predict this failure. Using the synthetic isolated intervation pairs as training data leads to 1) IsoVRM, a reward model outperforming its counterpart trained with a larger human labeled preference dataset by 3.8 points on VBench-2.0, 2) IsoWan, the video generator post-trained with IsoVRM, outperforming its human-data counterpart in human preferences, and 3) gains that stack when combined with human data. Because intervention pairs are automatically generated rather than collected, the benchmark and training data can be easily extended as generators evolve.
Shuyue Stella Li, Jacqueline He, Jiahui Chen +5
University of Washington · 2Meta Superintelligence Labs
Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.
Zhenchen Tang, Yang Li, Songlin Yang +6
New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent +1