FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling

Peiyuan Zhang; Xiangyu Zhao; Hongbo Liu; Xiaoxing Hu; Mingxin Liu; Shuran Ma; Yunhang Shen; Jian Hu

cs.CVAug 22, 2026

FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling

Authors: Peiyuan Zhang, Xiangyu Zhao, Hongbo Liu, Xiaoxing Hu, Mingxin Liu, Shuran Ma, Yunhang Shen, Jian Hu,

Organizations: Shanghai Jiao Tong University · Tencent Youtu Lab · Tongji University

Abstract

Reliable reward models are essential for text-to-video evaluation and alignment. However, the trade-off between evaluation accuracy and inference efficiency places high demands on the quality of training supervision. Existing approaches often rely on holistic judges with fixed rubrics or open-ended reasoning, leading to incomplete inspection, unfaithful justification, and entangled attribution. We introduce FIRM-Video, a unified checklist-driven data construction framework based on a check-before-score principle: construct dimension-specific checklists, verify each criterion against temporal visual evidence, and aggregate only verified decisions. For Instruction Following, FIRM-Video decomposes prompts into weighted atomic requirements; for World Coherence, it constructs prompt-calibrated, target-specific checks grounded in visible entities and actions; and for Perceptual Quality, it applies a generic taxonomy of visual defects. The verified criteria and scores are further transformed into natural-language analyses for end-to-end reward modeling. Subsequently, we construct FIRM-Video-90K with 88,044 dimension-specific instances from 29,348 videos, and introduce FIRM-Video-Bench with 750 point-wise human annotations across 250 videos. The Qwen3-VL-based FIRM-Video-8B achieves the best overall MAE on FIRM-Video-Bench while consistently delivering the highest VBench Total, Quality, and Semantic Scores in Best-of-8 sampling across three video generators.

Figures & tables

Figure 1: Qualitative evaluation case of FIRM-Video-8B , illustrating its improved alignment with human judgments.

Figure 2: Statistics of the FIRM-Video-90K dataset. Left: distributions across evaluation dimensions and score ranges. Right: distributions of video duration and prompt length.

Figure 3: Overview of the FIRM-Video data construction framework and reward model training. Given a prompt and its generated video, FIRM-Video applies a unified check-before-score pipeline to construct fine-grained supervision for instruction following, world coherence, and perceptual quality. The resulting dimension-specific analyses and scores form FIRM-Video-90K, which is then used to train the FIRM-Video-8B reward model.

Model	Instruction Following				World Coherence				Perceptual Quality				Overall
Model	MAE $\downarrow$	STD $\downarrow$	Acc./Relaxed Acc. $\uparrow$	SRCC $\uparrow$	MAE $\downarrow$	STD $\downarrow$	Acc./Relaxed Acc. $\uparrow$	SRCC $\uparrow$	MAE $\downarrow$	STD $\downarrow$	Acc./Relaxed Acc. $\uparrow$	SRCC $\uparrow$	MAE $\downarrow$	STD $\downarrow$	Acc./Relaxed Acc. $\uparrow$
Closed-source models
GPT-5	0.62	0.71	0.50/0.88	0.74	1.66	1.19	0.20/0.48	0.49	0.95	0.86	0.34/0.76	0.51	1.08	1.03	0.35/0.71
Gemini-3.1-Pro	0.77	0.76	0.40/0.86	0.71	1.27	1.08	0.28/0.62	0.54	1.44	1.12	0.21/0.59	0.41	1.16	1.04	0.30/0.69
Doubao-Seed-2.0-Lite	0.80	0.83	0.41/0.84	0.67	1.56	1.24	0.24/0.53	0.44	0.80	0.75	0.38/0.84	0.55	1.05	1.03	0.35/0.73
Open-source models

Table 1: Performance comparison on FIRM-Video-Bench across three evaluation dimensions: Instruction Following (IF), World Coherence (WC), and Perceptual Quality (PQ). The two values under Acc./Relaxed Acc. denote strict and relaxed accuracy, respectively. The row marked with

\dagger

reports the performance of direct data construction pipeline and is excluded from ranking.

Sampling Strategy	Total Score	Quality Score	Semantic Score	Video Quality				Video–Condition Consistency
Sampling Strategy	Total Score	Quality Score	Semantic Score	Subject Consistency	Background Consistency	Temporal Flickering	Imaging Quality	Multiple Objects	Color	Spatial Relationship	Appearance Style
T2V model: LaVie-Base
Random	79.42	81.34	71.77	92.09	97.64	97.63	65.92	37.73	89.65	43.30	23.94
By Qwen3-VL-8B	80.03	81.61	73.73	92.64	97.56	97.60	66.88	50.91	83.11	47.32	24.27
By InternVL3-8B	80.45	82.10	73.82	92.94	97.56	97.56	67.02	50.30	85.85	45.81	24.10
By VideoScore2	79.95	81.61	73.33	92.95	97.80	97.76	66.54	53.73	86.86	45.70	23.87

Table 3: Best-of-N performance comparison of FIRM-Video-8B and other sampling strategies on VBench. The best and second-best results under each T2V generator are highlighted in bold and underlined , respectively.

Dimension	Aggregation	MAE $\downarrow$	STD $\downarrow$	Acc. $\uparrow$	Relaxed Acc. $\uparrow$	SRCC $\uparrow$
IF	mean	0.68	0.70	0.44	0.88	0.69
IF	Importance weighted (ours)	0.68	0.68	0.43	0.90	0.69
WC	mean	0.86	0.83	0.37	0.82	0.61
WC	Importance weighted (ours)	0.80	0.83	0.41	0.84	0.63

Table 4: Aggregation ablations on FIRM-Video-Bench. STD is computed over absolute errors; relaxed accuracy allows a one-point deviation from the expert rating. Best results within each dimension are bold.

Dataset	Dimension	Score=1	Score=2	Score=3	Score=4	Score 5	Total
FIRM-Video-90K	IF	5,104 (17.4%)	8,900 (30.3%)	5,551 (18.9%)	3,806 (13.0%)	5,987 (20.4%)	29,348
	WC	4,795 (16.3%)	11,129 (37.9%)	5,649 (19.2%)	2,510 (8.6%)	5,265 (17.9%)	29,348
	PQ	874 (3.0%)	3,749 (12.8%)	4,257 (14.5%)	18,331 (62.5%)	2,137 (7.3%)	29,348
	All	10,773	23,778	15,457	24,647	13,389	88,044
FIRM-Video-Bench	IF	28 (11.2%)	73 (29.2%)	63 (25.2%)	59 (23.6%)	27 (10.8%)	250
	WC	48 (19.2%)	76 (30.4%)	48 (19.2%)	44 (17.6%)	34 (13.6%)	250

Table 5: Score distributions of FIRM-Video-90K and FIRM-Video-Bench.

Reward	VBench Dimension	VBench Group
IF	Dynamic Degree	Quality
	Overall Consistency	Cond. Consist.
	Object Class	Cond. Consist.
	Multiple Objects	Cond. Consist.
	Human Action	Cond. Consist.
	Color	Cond. Consist.

Table 6: Dimension mapping from VBench dimensions to FIRM-Video reward dimensions. “Quality” and “Cond. Consist.” denote VBench’s Video-Quality and Video–Condition Consistency groups, respectively.

T2V Model	Sampling Strategy	Subject Consistency	Background Consistency	Temporal Flickering	Motion Smoothness	Dynamic Degree	Aesthetic Quality	Imaging Quality
LaVie-Base	Random	92.09	97.64	97.63	96.63	55.56	64.53	65.92
	Qwen3-VL-8B	92.64	97.56	97.60	96.62	55.56	64.93	66.88
	InternVL3-8B	92.94	97.56	97.56	96.69	61.11	64.74	67.02
	VideoScore2	92.95	97.80	97.76	96.39	56.94	64.27	66.54
	FIRM-Video-8B	93.66	97.91	98.12	96.58	56.94	64.17	67.46
CogVideoX-2B	Random	93.47	96.66	96.90	97.23	73.61	58.47	59.72

Table 7: Best-of-8 results on the video-quality subdimensions of VBench.

T2V Model	Sampling Strategy	Object Class	Multiple Objects	Human Action	Color	Spatial Relationship	Scene	Appearance Style	Temporal Style	Overall Consistency
LaVie-Base	Random	93.12	37.73	92.00	89.65	43.30	52.33	23.94	24.73	27.20
	Qwen3-VL-8B	91.61	50.91	96.00	83.11	47.32	52.69	24.27	25.25	27.75
	InternVL3-8B	95.81	50.30	96.00	85.85	45.81	51.02	24.10	25.03	27.46
	VideoScore2	91.06	53.73	93.00	86.86	45.70	50.80	23.87	25.16	27.33
	FIRM-Video-8B	90.51	58.99	93.00	88.20	51.20	52.03	24.40	25.19	27.57
CogVideoX-2B	Random	76.11	49.16	87.00	88.43	61.30	39.90	22.95	23.77	24.43

Table 8: Best-of-8 results on the video–condition consistency subdimensions of VBench.

Figure 5: Human annotation interface for FIRM-Video-Bench. Annotators rate instruction following, perceptual quality, and world coherence on a five-point scale.

Figure 6: Qualitative example of FIRM-Video-8B evaluation (1)

Figure 7: Qualitative example of FIRM-Video-8B evaluation (2)

Figure 8: Qualitative example of FIRM-Video-8B evaluation (3)

Figure 9: Qualitative example of FIRM-Video-8B evaluation (4)

Figure 10: Qualitative example of FIRM-Video-8B evaluation (5)

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 313 papers

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation
Oct 1, 2026Yu Huang, Jungang Li, Zhiyuan Wang +8Agentic Video Generation Video Generation
Department of Data Science, City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Westlake University +3
VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing
Aug 6, 2026Ziyun Zeng, Zixuan Wang, Yongsheng Yu +2Video Editing Benchmarks Video Editing
University of Rochester
Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models
May 8, 2026Yuancheng Wei, Linli Yao, Lei Li +4Reward Modeling Multimodal Reward Modeling
South China University of Technology · Peking University · The University of Hong Kong +1

Oct 1, 2026J/K move · Enter open · S save

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

Yu Huang, Jungang Li, Zhiyuan Wang, Yonghua Hei+7

Department of Data Science, City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Westlake University+3

Agentic Video Generation Video Generation Video Generation Evaluation

Open paper AskPDF arXiv

Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standards. Existing evaluation benchmarks, however, predominantly assess visual quality, aesthetic appeal and physical plausibility, while paying limited attention to text, an essential medium for conveying information in everyday scenes. A generated video may appear visually compelling and feature lifelike subjects, yet still render the text within the scene incorrectly. To address this overlooked dimension, we introduce VTR-Bench, a systematic benchmark for evaluating the Visual Text Rendering capabilities of video generation models. VTR-Bench situates text within concrete application scenarios, such as advertisements and scientific videos, with 300 carefully constructed prompts spanning five scenario categories. We develop an automated evaluation pipeline with human alignments that separately assesses text fidelity through carrier-specific transcription and scene and motion requirements through a prompt-specific chain of query. Beyond evaluation, we introduce a Keyframe-Guided Agentic Framework in which a Director agent coordinates image and video generation with visual evaluation, guiding iterative refinement and candidate selection through visual feedback. Experiments on 11 state-of-the-art models reveal widespread difficulties in accurately rendering scene text, with the best-performing model recording an overall word error rate (WER) of 0.250. We further analyze text rendering failures to characterize the challenges faced by current video generation models. These findings highlight visual text rendering as a key challenge for video generation and demonstrate a practical path toward improvement. Code is available at https://github.com/hardenyu21/VTR-Bench.

Abstract

Reward Modeling

Explore similar work

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing

Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models