Aligning video generation models with human preference depends critically on the quality of the reward model that scores generated videos. Current video reward models, however, source their reward signal from a model's representational rather than generative capacity: they either repurpose vision-language models as pixel-space scorers, or train auxiliary scoring heads on top of frozen generator features. The generator's actual ability to predict velocities, denoise latents, and produce videos plays no role in scoring its own generations. We argue that this overlooks a structural property of offline preference learning. Any preference optimization objective that approximates an offline preference dataset mathematically targets a strictly defined KL-constrained optimal policy. By isolating the reward term from this analytical solution and mapping the exact generation log-likelihood to its flow-matching surrogate, the empirical reward emerges deterministically as the relative difference in continuous-time velocity-prediction errors. We formalize this equivalence into GenRM-Flow, a framework where a preference-tuned video generator inherently operates as a process-aware reward model directly in the noisy latent space. We further show theoretically that distinct fine-tuning objectives, whether contrastive like DPO and IPO or non-contrastive like RWR, are alternative optimization pathways toward the same optimal policy, each inducing a reward readout from the generator's velocity-prediction errors. Furthermore, the extracted process reward seamlessly substitutes external VLM scorers in FlowAWR. Operating purely on native latent outputs without auxiliary reward networks, this generative reward drives measurable improvements in downstream video generation.
Figures & tables
Figure 1: Videos generated by Wan2.1-T2V-14B fine-tuned with FlowAWR using the GenRM-Flow reward, exhibiting high motion quality and dynamic fidelity.
Figure 2: The pipeline of GenRM-Flow . (Panel A) Traditional paradigms rely on representational capacity , utilizing a frozen backbone (e.g., a VLM or base generator) as a feature extractor coupled with an auxiliary discriminative head. (Panel B) The proposed framework sources the reward directly from generative capacity . Following preference optimization, the implicit reward RD(x,c) is algebraically extracted from the velocity-prediction errors. The scalar score is computed natively by contrasting the active policy (flame, θ⋆ ) against a frozen reference (snowflake, θref ), requiring no architectural modifications or auxiliary heads.
Table 1: Three preference fine-tuning objectives as optimization pathways to obtain the trained weights θ⋆ . While their loss formulations LF(θ) vary significantly—ranging from bounded contrastive margins to reference-free likelihood maximization—they all structurally aim to approximate the optimal offline policy π∗ . Consequently, during the evaluation phase, they implicitly define reward readouts corresponding to their theoretical likelihood shifts: the relative readout of Eq. ( 6 ) for contrastive objectives and a reference-free readout for RWR.
Figure 3: (Left) Pairwise preference prediction accuracy during training. (Middle, Right) HPSv3 reward trajectories during FlowAWR training on the 1.3B and 14B backbones. The implicit process reward yields higher sustained HPSv3 evaluation scores compared to PAVRM.
Base Model
Method
[0, 0.2]
(0.2, 0.4]
(0.4, 0.6]
(0.6, 0.8]
(0.8, 1.0]
Avg
Qwen3-VL-8B
VideoAlign [ 28 ]
–
–
–
–
–
78.26
Wan2.1-T2V-1.3B
PAVRM [ 36 ]
77.19
77.50
78.27
76.11
71.25
76.06
GenRM-Flow
87.77
88.08
88.49
88.61
81.00
86.79
Wan2.1-T2V-14B
PAVRM [ 36 ]
81.87
83.85
84.79
84.69
78.02
82.64
GenRM-Flow
89.10
89.94
90.46
90.75
84.55
88.96
Table 2: Pairwise preference prediction accuracy (%) on the held-out evaluation set. The performance of our implicit process reward is compared against baseline reward models (VideoAlign and PAVRM) across five uniformly spaced noise timestep intervals t∈[0,1] . Bold denotes the highest accuracy per column within each backbone scale.
Figure 4: Qualitative comparison between PAVRM and GenRM-Flow as the per-step reward signal in FlowAWR with a 1.3B backbone. Both runs use FlowAWR with the same configuration, differing only in the reward signal. Red boxes mark localized structural deformations, spatial inconsistencies, and temporal artifacts produced by PAVRM. Across scenes, our reward yields visibly fewer geometric defects and better spatial and temporal continuity.
Base Model
Method
Inner Test Set
VBench [ 10 ]
MS ↑
DD ↑
SC ↑
HA ↑
MS ↑
DD ↑
SC ↑
HA ↑
Wan2.1-T2V-1.3B
Pretrain
98.31
59.07
97.02
88.15
98.52
65.19
97.56
94.00
VideoAlign [ 28 ]
98.58
60.68
97.22
89.96
98.73
67.52
97.64
95.32
PAVRM [ 36 ]
98.56
63.59
97.18
89.73
98.71
67.83
97.52
94.68
GenRM-Flow
98.88
72.43
98.28
93.16
98.95
73.46
97.70
96.25
Wan2.1-T2V-14B
Pretrain
98.12
61.18
97.18
89.32
98.30
65.46
97.52
95.40
Table 3: Comprehensive quantitative comparison of reward-guided generation on our inner test set and the out-of-domain VBench benchmark. Bold denotes the highest metric value per backbone.
Normalization Mechanism
Training Status
MS ↑
DD ↑
SC ↑
HA ↑
Raw rt (no normalization)
Policy Collapse ( ∼ 250 steps)
N/A
N/A
N/A
N/A
Z-score Normalization
Converged
98.74
69.85
97.61
95.61
Softmax (Ours)
Converged
98.95
73.46
97.70
96.25
Table 4: Ablation on intra-group normalization mechanisms during FlowAWR training (Wan2.1-T2V-1.3B, evaluated on VBench). “Policy Collapse” indicates training collapse. Bold denotes the best results among all.
New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent +1