Q-Save: Towards Scoring and Attribution for Generated Video Evaluation

Xiele Wu; Mingtao Chen; Zicheng Zhang; Yixian Liu; Yiming Liu; Jianxiang Lu; Jin Wang; Shushi Wang

cs.CVNov 24, 2025

Q-Save: Towards Scoring and Attribution for Generated Video Evaluation

Authors: Xiele Wu, Mingtao Chen, Zicheng Zhang, Yixian Liu, Yiming Liu, Jianxiang Lu, Jin Wang, Shushi Wang, +6 more

Organizations: Shanghai Jiao Tong University · Hunyuan · Shanghai Innovation Institute

Abstract

Evaluating AI-generated video (AIGV) quality hinges on three crucial dimensions: visual quality, dynamic quality, and text-video alignment. While numerous evaluation datasets and algorithms have been proposed, existing approaches are constrained by two limitations: the absence of systematic definitions for evaluation dimensions, and the isolated treatment of the three dimensions in separate models. Therefore, we introduce Q-Save, a holistic benchmark dataset and unified evaluation model for AIGV quality assessment. The Q-Save dataset contains nearly 10,000 video samples, each annotated with Mean Opinion Scores (MOS) and fine-grained attribution explanations across the three core dimensions. Leveraging this attribution-annotated dataset, we train the proposed Q-Save model, which adopts the SlowFast framework to balance accuracy and efficiency, and employs a three-stage training strategy with Chain-of-Thought (COT) formatted data: Supervised Fine-Tuning (SFT), Grouped Relative Policy Optimization (GRPO), and a final SFT round for stability, to jointly perform quality scoring and attribution generation. Experimental results demonstrate that Q-Save achieves superior performance in AIGV quality prediction while providing interpretable justifications. Code and dataset will be released upon publication.

Figures & tables

Figure 1: Scoring and interpretation performance of Q-Save-tuned Qwen3-VL-8B-Instruct, compared to the baseline.

Dataset/ Benchmark	Year	Closed Source	Annot. Type	Scale	#Dims	Attrib.
VideoScore2	2025	No	MOS	27k	3	Yes
VideoGen- RewardBench	2025	Yes	SBS	108k	4	No
T2VQA-DB	2024	No	MOS	10k	1	No
VideoPhy-2	2025	No	MOS	6.8k	2	No
AIGVE-Bench	2025	No	MOS	2.4k	9	No
Q-Save	2025	Yes	MOS	10k	3	Yes

Table 1: Comparison of representative datasets/benchmarks for AIGV evaluation. SBS denotes side-by-side preference annotation and MOS denotes mean opinion score annotation.

Figure 2: Overview of the human annotation process. Step 1: raters are trained using text-defined rating levels (simulated via a rating-based syllabus for LMMs); Step 2: raters assign scores and select reasons for low ratings (Bad–Fair); Step 3: ratings are aggregated into MOS and reasons are converted into descriptions (via a probability-based inference method for LMMs).

Figure 3: Model architecture and video preprocessing. Our evaluator is based on a fine-tuned Qwen3-VL-8B-Instruct backbone and uses SlowFast-style preprocessing to extract spatiotemporal features (FPS = 8).

Figure 4: Overview of the three-stage hybrid training pipeline. Blue denotes SFT data/models and orange denotes RL data/models (Cold Start $\rightarrow$ Warm-Up $\rightarrow$ Cool-Off).

Model	Visual Quality				Dynamic Quality				Alignment
	Instance-level		Model-level		Instance-level		Model-level		Instance-level		Model-level
	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC
FastVQA	0.087	0.062	-0.103	-0.091	0.125	0.073	-0.542	-0.376	0.294	0.271	0.513	0.432
SimpleVQA	0.152	0.119	-0.076	-0.048	0.193	0.138	0.089	0.021	0.316	0.295	0.547	0.479
Dover	0.238	0.214	0.169	0.157	0.286	0.263	0.227	0.201	0.352	0.334	0.569	0.513
Q-Align	0.275	0.253	0.198	0.184	0.312	0.291	0.254	0.232	0.378	0.359	0.583	0.536

Table 2: Performance comparison on the Q-Save test set across visual quality, dynamic quality, and text-video alignment. Best in bold , second best underlined .

Method	Preference Accuracy (%)			MOS PLCC ( $\times 100$ )
	VideoGen RewardBench	GenAI Bench	T2VQA-DB Preference	VideoPhy2 -test	VideoScore-Bench-v2
	VideoGen RewardBench	GenAI Bench	T2VQA-DB Preference	VideoPhy2 -test	Visual	Align	Phy
VideoReward	/	72.89	36.15	34.54	46.36	48.31	35.23
VisionReward	54.31	73.30	37.64	20.89	46.85	45.32	38.25
Q-Align	42.05	64.25	43.24	4.11	54.71	34.01	37.78
AIGVE-MACS	37.09	53.21	36.91	10.00	27.30	6.90	13.03

Table 3: Cross-dataset validation on out-of-domain benchmarks. Left: preference accuracy (%) on pairwise benchmarks. Right: PLCC (

\times 100

) on MOS-style point-score benchmarks. Bold denotes the best model and underlined denotes the second best. “/” denotes the method trained on the benchmark’s in-domain training set (excluded from comparison).

Strategy	Visual Quality				Dynamic Quality				Alignment
	Instance-level		Model-level		Instance-level		Model-level		Instance-level		Model-level
	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC
Stage 1 w/o SlowFast	0.50	0.51	0.58	0.57	0.58	0.57	0.54	0.58	0.63	0.61	0.79	0.75
Stage 1	0.57	0.54	0.61	0.60	0.61	0.60	0.52	0.51	0.66	0.65	0.84	0.81
Stage 1+2	0.61	0.60	0.71	0.63	0.64	0.63	0.74	0.65	0.67	0.66	0.87	0.86
Stage 1+2+3	0.63	0.65	1.00	0.91	0.68	0.67	0.94	0.86	0.70	0.70	1.00	0.97

Table 4: Ablation study on training strategies. We report SRCC and PLCC at both instance-level and model-level across visual quality, dynamic quality, and text-video alignment. Best in bold , second best underlined .

Training Configuration	Wan2.2	Hunyuan1.5-video
w/o RL	66.76	55.86
RL with HPSv3	68.10	56.23
RL with Q-Save-v1.5 (ours)	69.95	56.68

Table 5: Human evaluation scores after reinforcement learning fine-tuning of video generation models in the game-CG scenario. Best in bold , second best underlined . HPSv3 is an open-source reward model.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Word cloud of prompts and overview of video generation model sources

Category	Subcategory	Count
Style and atmosphere	Atmosphere	109
	Sad	16
	Romantic	13
	Calm/Tranquil	19
	Other	2
	Fantasy	11

Appendix

Table 6: Prompt Elements Categorized by Style, Lens Parameters, and Spatial Relationships

Category	Subcategory	Count
Scene	Human scenes	136
	Indoor	49
	Outdoor	52
	Famous humanities classics	35
	Virtual Scene	102
	Abstract Scene	18

Appendix

Table 7: Categorized Statistics of Prompt Elements Across Scene and Subject Types

Category	Quantity range
Category	1-3	4-6	7-9	10 or more	unknown
Human	278	25	20	8	7
Animal	161	49	18	41	15
Plant	34	16	15	5	76
Object	434	24	19	65	213
Virtual Subject	82	45	14	18	13

Appendix

Table 8: Distribution of Prompts by Entity Category and Quantity Range. Each cell shows the number of prompts containing a specific category (e.g., Human, Animal) within the indicated quantity range.

Figure 6: MOS distributions for visual quality, dynamic quality, text-video alignment, and overall quality of generated videos in the Q-Save dataset.

Figure 7: Distribution of Mean Opinion Scores (MOS) across six generative models (Veo2, Kling, Kling2.0, Hunyuan, Dreamina, Wan), evaluated along four quality dimensions: Text Alignment (TA), Motion Quality (MQ), Visual Quality (VQ), and Overall Quality (All). Each subplot represents a model-dimension pair, with MOS scores (1–5) on the x-axis and frequency counts (0–500) on the y-axis. Blue bars denote histogram counts, while overlaid curves represent smoothed kernel density estimates.

Method	Visual Quality		Text Alignment		Overall
	w/ Ties	w/o Ties	w/ Ties	w/o Ties	w/ Ties	w/o Ties
VisionReward	35.89	59.03	44.86	61.15	54.31	67.58
Q-Align	32.01	52.98	35.77	51.06	42.05	52.52
AIGVE-MACS	38.05	30.80	30.76	11.66	37.09	37.08
VideoPhy2	-	-	37.04	22.14	30.75	26.41
Dover	39.34	68.87	38.01	55.65	54.27	68.58

Appendix

Table 9: Full evaluation results on VideoGen-Reward-Bench . Bold denotes the best model and the underlined denotes the second best.

Method	PLCC
	Semantic	Physical	Avg
VisionReward	28.11	13.67	20.89
Q-Align	5.52	2.70	4.11
AIGVE-MACS	8.09	11.90	10.00
VideoPhy2	38.64	29.84	34.24
Dover	3.85	1.15	2.50

Appendix

Table 10: Full evaluation results on Video-Phy2-test . Bold denotes the best model and the underlined denotes the second best.

Figure 8: Qualitative comparison of text-video alignment.

Figure 9: Qualitative comparison of visual quality.

Figure 10: Qualitative comparison of dynamic quality.

Figure 11: Qualitative comparison of CoT outputs.

Explore similar work

Comparison Drives Preference: Reference-Aware Modeling for AI-Generated Video Quality Assessment
Apr 18, 2026Minghao Zou, Gen Liu, Guanghui Yue +5Video Quality Assessment Video Generation Evaluation
Cardiff University Cardiff, UK · Shandong University of Science and Technology Qingdao, China · Shenzhen University Shenzhen, China +2
VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluation
Apr 11, 2026Longteng Jiang, DanDan Zheng, Qianqian Qiao +7Video Generation Evaluation Video Quality Assessment
Ant Group · Beijing Film Academy · State Key Laboratory of General Artificial Intelligence, BIGAI
Multi-Dimensional Quality Assessment for AI-Generated Human-Centric Videos: Dataset and Model
Jul 18, 2026Sijing Wu, Yunhao Li, Huiyu Duan +4Mixture of Experts Multimodal Large Language Models
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai, China · USC-SJTU Institute of Cultural and Creative Industry, Shanghai Jiao Tong University, Shanghai, China · Polytech Nantes, Universit´e de Nantes, France

Apr 18, 2026J/K move · Enter open · S save

Comparison Drives Preference: Reference-Aware Modeling for AI-Generated Video Quality Assessment

Minghao Zou, Gen Liu, Guanghui Yue, Baoquan Zhao+4

Cardiff University Cardiff, UK · Shandong University of Science and Technology Qingdao, China · Shenzhen University Shenzhen, China+2

Video Quality Assessment Video Generation Evaluation

Open paper AskPDF arXiv

The rapid advancement of generative models has led to a growing volume of AI-generated videos, making the automatic quality assessment of such videos increasingly important. Existing AI-generated content video quality assessment (AIGC-VQA) methods typically estimate visual quality by analyzing each video independently, ignoring potential relationships among videos. In this work, we revisit AIGC-VQA from an inter-video perspective and formulate it as a reference-aware evaluation problem. Through this formulation, quality assessment is guided not only by intrinsic video characteristics but also by comparisons with related videos, which is more consistent with human perception. To validate its effectiveness, we propose Reference-aware Video Quality Assessment (RefVQA), which utilizes a query-centered reference graph to organize semantically related samples and performs graph-guided difference aggregation from the reference nodes to the query node. Experiments on existing datasets demonstrate that our proposed RefVQA outperforms state-of-the-art methods across multiple quality dimensions, with strong generalization ability validated by cross-dataset evaluation. These results highlight the effectiveness of the proposed reference-based formulation and suggest its potential to advance AIGC-VQA.

Abstract

Explore similar work

Comparison Drives Preference: Reference-Aware Modeling for AI-Generated Video Quality Assessment

VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluation

Multi-Dimensional Quality Assessment for AI-Generated Human-Centric Videos: Dataset and Model