IsoVRBench: Video Reward Modeling Without Human Feedback

Shuyue Stella Li; Jacqueline He; Jiahui Chen; Sreya Dutta Roy; Yulia Tsvetkov; Luke Zettlemoyer; Marjan Ghazvininejad; Xiaochuang Han

cs.AIOct 12, 2026

IsoVRBench: Video Reward Modeling Without Human Feedback

Authors: Shuyue Stella Li, Jacqueline He, Jiahui Chen, Sreya Dutta Roy, Yulia Tsvetkov, Luke Zettlemoyer, Marjan Ghazvininejad, Xiaochuang Han

Organizations: University of Washington · 2Meta Superintelligence Labs

Abstract

Video reward models are trained and evaluated on human preferences between videos that often differ simultaneously on properties such as prompt alignment, visual fidelity, motion, and generator identity. As a result, aggregate accuracy does not reveal which video properties a reward model can reliably distinguish. We construct preference pairs by controlled interventions that change one property of a generated video while holding the rest fixed. The resulting pairs, by construction, provide preferences for how the videos differ along a specific dimension, without requiring any human annotation. We propose IsoVRBench, a video generation reward modeling benchmark of 8,000 such pairs across eight interventions, and show that state-of-the-art models separate broad quality differences between generators but miss important features such as temporal order. The best reward model reaches only 60.8% on reordered frames against 100% for humans, and rankings on existing benchmarks do not predict this failure. Using the synthetic isolated intervation pairs as training data leads to 1) IsoVRM, a reward model outperforming its counterpart trained with a larger human labeled preference dataset by 3.8 points on VBench-2.0, 2) IsoWan, the video generator post-trained with IsoVRM, outperforming its human-data counterpart in human preferences, and 3) gains that stack when combined with human data. Because intervention pairs are automatically generated rather than collected, the benchmark and training data can be easily extended as generators evolve.

Figures & tables

Figure 1: Controlled intervention data generation procedure. A prompt is rendered once to create an anchor video. A controlled intervention (e.g., shuffling segments of a video) automatically produces a modified counterpart with a predetermined preference label. The resulting pairs serve reward model evaluation ( IsoVRBench ), reward model training, and video generator post-training.

Panel of Table 1 — Table 1: Interventions in IsoVRBench , with their descriptions and evaluation dimensions. IsoVRBench draws $1{,}000$ pairs from each intervention, resulting in $8{,}000$ total samples.

Figure 2: IsoVRBench interventions on a representative prompt. The top row is the anchor prompt and every row below is a corresponding clip generated from each intervention. See Table 1 for intervention descriptions.

Panel of Table 2 — Table 2: Per-intervention accuracy (%) on IsoVRBench , sorted in descending order by average performance. Ties are counted as incorrect; LiFT, VideoScore2, and VisionReward tie on $61\%$ , $27\%$ , and $21\%$ of pairs, and PickScore and VideoReward rarely tie (Appendix D ). Both UnifiedReward models also return no judgment on $78$ pairs, which count as wrong. Accuracy over decided pairs, those on which a model prefers one side, is in Appendix A .

Figure 3: Accuracy on each published benchmark against IsoVRBench accuracy. Each reward model is a dot, and each horizontal axis uses the benchmark’s reported metric. Blue is above chance on , red is below, and the dashed line is at chance.

\rho

is the Spearman rank correlation down each panel, and none differs from zero under an exact permutation test. Details in Appendix N .

Figure 4: Accuracy (%) against the frame count on two interventions that reuse the anchor’s frames. Absent points mark frame counts a model cannot be scored at (Appendix L ). Seven of the eleven sit at or below chance at every frame count on at least one of the two interventions, and VisionReward falls on as it is given more frames.

Table 7

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 5 — Table 5: Accuracy over decided pairs only, with the share of pairs on which each model expressed no preference. Compare Table 2 , which counts those pairs as wrong.

Panel of Figure 5 — Figure 5: Accuracy on decided pairs against the number of denoising steps on Reduced Steps, with prompt, seed, anchor and every other sampling parameter held fixed. Eight of the nine models gain as the budget falls from $16$ to $8$ denoising steps, by $7.4$ to $14.2$ points. PickScore, in red, gains least. LiFT, dashed, falls by $5.8$ . Tie rates are in Table 5 .

Panel of Table 6 — Table 6: Frames sampled per clip against accuracy on the two interventions that reuse the anchor’s frames. Clips are $81$ frames at $16$ fps. Underlined entries fall below chance. Frame count orders neither column. Accuracies count a tie as wrong, as in Table 2 , and so run lower than in Table 14 , which drops ties as VideoGen-RewardBench does.

Setting	Value
Backbone	Qwen2-VL-2B-Instruct
Initialization	VideoReward, or the bare backbone
Reward head	scalar, read at three per-dimension marker tokens
Objective	Bradley-Terry with ties
LoRA	rank $64$ , $\alpha$ $128$ , dropout $0.05$
LoRA exclusions	lm_head, rm_head, embed_tokens, visual

Appendix

Table 7: Training configuration for both released reward models.

Panel of Table 8 — Table 8: The two released reward models on IsoVRBench , ties counted as wrong, with VideoReward from Table 2 for reference.

Benchmark	Generators	Generator identity	VideoReward	LiFT
VideoGen-RewardBench	$12$	$77.4$	$73.2$	$56.8$
GenAI-Bench	$10$	$74.1$	$69.9$	$50.7$
VBench-2.0	$4$	$78.2$	$50.6$	$58.9$

Appendix

Table 9: Accuracy (%) of predicting each benchmark’s human preference from the identities of the two generators alone, by five-fold cross-validation, next to the reward models with the highest and lowest average on IsoVRBench . Chance is

50

. Every column uses the pairs of Table 14 , which for VBench-2.0 are the

2{,}494

decided pairs of its four dimensions.

Generator	In VideoReward’s	VBench-2.0	VideoReward		IsoVRM -VR
	training data	people	win	$\Delta$	win	$\Delta$
Kling	$59.3$ / $80.9$	$64.6$	$70.9$	$+6.3$	$81.2$	$+16.6$
Sora	absent	$61.9$	$47.9$	$-14.0$	$70.2$	$+8.3$
HunyuanVideo	absent	$47.6$	$41.1$	$-6.5$	$42.9$	$-4.7$
CogVideo	$44.8$	$28.1$	$40.9$	$+12.8$	$9.7$	$-18.4$

Appendix

Table 10: Percentage of its comparisons each generator wins, by human count and as each model’s scores imply, with each model’s deviation from the human value. The first column is that generator’s standing in VideoGen-RewardBench, the data VideoReward was trained on. Kling has two entries in VideoGen-RewardBench and VBench-2.0 uses a third version, so its training-set entry is the least certain of the four.

Panel of Table 11 — Table 11: Mean accuracy over the three generator-substitution interventions, which carry an overall label because a weaker generator degrades every dimension at once, against accuracy on . Underlined entries are below chance. LiFT is the only model to hold its rank, at the bottom of both.

Design property	IsoVRBench	TemporalBench	TempCompass
Original target	Video-generation reward models	Video text-generation and embedding models	Video and image language models
What the model ranks	One prompt, rank two videos	One video, rank binary captions	One video, rank two captions
What changes	One named video property	A caption detail	The candidate caption
Human role	Validate each intervention, with no per-item label	Write and refine positive captions, then review generated negatives	Annotate video metadata, then review generated instructions
Frame-order isolation	Same prompt and frames, reordered	None, since the evaluated video is fixed	None, since the evaluated video is fixed

Appendix

Table 12: Differences among the benchmark designs. The TemporalBench and TempCompass columns describe the settings evaluated here.

Model	TemporalBench	TempCompass
VideoReward	$\mathbf{21.3}$	$69.5$
UnifiedReward-Flex-9B	$13.3$	$63.3$
PickScore	$13.5$	$61.0$
UnifiedReward-2.0-8B	$20.9$	$68.3$
VideoScore v1	$7.0$	$40.1$
VideoScore v1.1	$6.9$	$43.0$

Appendix

Table 13: Accuracy on the caption-ranking settings of TemporalBench and TempCompass. All values are percentages. Bold marks the best result in each column.

	Published Benchmarks					IsoVRBench
Model	VideoGen-RewardBench	GenAI-Bench	VBench-2.0	TemporalBench	TempCompass	Temporal Shuffle	Frame Hold
UnifiedReward-2.0-8B	$\mathbf{77.5}$	$80.5$	$63.3$	$20.9$	$68.3$	$53.1$	$54.6$
UnifiedReward-Flex-9B	$74.5$	$\mathbf{80.8}$	$63.4$	$13.3$	$63.3$	$48.5$	$51.8$
VideoReward	$73.2$	$69.9$	$50.6$	$\mathbf{21.3}$	$\mathbf{69.5}$	$61.0$	$71.9$
VisionReward	$69.0$	$73.8$	$63.3$	$2.7$	$47.4$	$47.4$	$56.9$
VideoScore2	$67.7$	$68.5$	$53.5$	$12.2$	$57.1$	$54.0$	$54.3$

Appendix

Table 14: Accuracy on five published benchmarks against accuracy on the two IsoVRBench interventions that reuse the anchor’s frames. All columns use each benchmark’s reported metric, and the IsoVRBench columns use the decided-item protocol of Table 5 . Underlined entries fall below the corresponding random baseline. The last two rows give the Spearman rank correlation down each benchmark’s column. Bold marks the best model in each published-benchmark column.

Panel of Table 15 — Table 15: Spearman rank correlation between accuracy on each published benchmark and accuracy on each IsoVRBench intervention, over the 9 reward models common to all columns. Each benchmark uses its reported metric. Bold marks a correlation an exact permutation test separates from zero at $p<0.05$ . VBench-2.0 correlates negatively with Frame Hold, at $-0.61$ and $-0.80$ without LiFT, and TemporalBench’s largest correlation is $+0.67$ , with Attribute Edit.

Panel of Figure 6 — Figure 6: Interventions ordered by median reward-model accuracy. Each bar spans the nine reward models from worst to best, the dot marks their median, blue above chance and red below, and the diamond marks human agreement on the same $45$ pairs per intervention, scored the same way, which stays between $84$ and $100$ . On the two frame-reuse interventions the median model reaches $39.8$ and $47.7$ , against $93$ and $100$ for people. People score above the best reward model on every intervention, by more than the error of the $45$ rated pairs on all but ( $84$ against $80.5$ ) and ( $98$ against $89.6$ ).

Preferred generator	Win Rate	Decisions	Agreement	$\bm{\kappa}$
IsoVRM -Q2 over VideoReward	$56.2$	$240$	$63.3$	$0.26$
IsoVRM -VR over VideoReward	$57.1$	$240$	$59.2$	$0.17$
IsoVRM -Q2 over IsoVRM -VR	$54.6$	$240$	$55.8$	$0.11$

Appendix

Table 16: Blinded human preference comparisons on

80

prompts. Win rate pools the three annotation passes. Agreement is the mean exact agreement over the three pairs of passes on the same comparison, and the final column gives Fleiss’

\kappa

across the three passes.

	Cyclic Prompts	Prompts	Cyclic (%)
Pass 1	$1$	$80$	$1.2$
Pass 2	$4$	$80$	$5.0$
Pass 3	$0$	$80$	$0.0$
All passes	$5$	$240$	$2.1$
Majority of passes	$2$	$80$	$2.5$

Appendix

Table 17: Transitivity of the human comparisons. A prompt is cyclic for a pass when its three pairwise choices there form a cycle, such as IsoVRM -Q2 over IsoVRM -VR, IsoVRM -VR over VideoReward, and VideoReward over IsoVRM -Q2. Random choices would give a cycle on

25\%

of prompts. The last row applies the majority of the three passes to each comparison.

Dimension	Videos	Pretrained	VideoReward	IsoVRM -Q2	IsoVRM -VR
Camera_Motion	$108$	$15.7$	$7.4$	$\mathbf{16.7}$	$10.2$
Complex_Landscape	$30$	$14.7$	$\mathbf{16.7}$	$14.7$	$16.0$
Complex_Plot	$61$	$13.5$	$\mathbf{14.5}$	$11.7$	$11.1$
Composition	$53$	$\mathbf{49.1}$	$47.2$	$43.0$	$48.2$
Diversity	$10$	$47.6$	$64.3$	$60.9$	$\mathbf{81.0}$
Dynamic_Attribute	$94$	$\mathbf{12.8}$	$11.7$	$7.4$	$8.5$

Appendix

Table 18: Every VBench-2.0 dimension for the four generators, in percent. Videos is the count each is scored on, held equal across all four dimension by dimension. Bold marks the best in a row. Against the VideoReward-supervised run, the IsoVRM -VR run leads on 11 dimensions and trails on 6; the median difference is

+0.8

and a bootstrap over dimensions puts the mean difference in

[-1.9,+7.8]

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models
May 8, 2026Yuancheng Wei, Linli Yao, Lei Li +4Reward Modeling Multimodal Reward Modeling
South China University of Technology · Peking University · The University of Hong Kong +1
VGA-BenchV2: An Expanded Unified Benchmark and Multi-Model Framework for Evaluating Video Aesthetics and Generation Quality
Aug 26, 2026Longteng Jiang, DanDan Zheng, Qianqian Qiao +7Video Generation Evaluation Video Quality Assessment
Ant Group, Beijing, China · Beijing Film Academy, Beijing, China · State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China
V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation
Jun 4, 2026Tao Liu, Leela Krishna, Gouti Pavan Kumar +2Video-Reference Video Generation Video Editing Benchmarks
Centific Global Solutions Inc.

May 8, 2026J/K move · Enter open · S save

Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models

Yuancheng Wei, Linli Yao, Lei Li, Haojie Zhang+3

South China University of Technology · Peking University · The University of Hong Kong+1

Reward Modeling Multimodal Reward Modeling Video Reasoning

Open paper AskPDF arXiv

Multimodal reward models have advanced substantially in text and image domains, yet progress in video understanding reward modeling remains severely limited by the lack of robust evaluation benchmarks and high-quality preference data. To address this, we propose a unified framework spanning benchmark design, data construction, and reward model training. We introduce Video Understanding Reward Bench (VURB), a benchmark featuring 2,100 preference pairs with long chain-of-thought reasoning traces (averaging 1,143 tokens) and majority voting evaluation across general, long, and reasoning-oriented video tasks. We further construct Video Understanding Preference Dataset (VUP-35K) via a fully automated pipeline, providing large-scale high-quality supervision for video reward training. Building on the data, we train VideoDRM and VideoGRM, a discriminative and a generative reward model, both achieving state-of-the-art performance on VURB and VideoRewardBench. Further analysis confirms that VUP-35K enhances both reward performance and model reasoning capability, while VideoDRM and VideoGRM yield significant gains under best-of-

N

test-time scaling.

Abstract

Reward Modeling

Explore similar work

Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models

VGA-BenchV2: An Expanded Unified Benchmark and Multi-Model Framework for Evaluating Video Aesthetics and Generation Quality

V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation