VGA-BenchV2: An Expanded Unified Benchmark and Multi-Model Framework for Evaluating Video Aesthetics and Generation Quality
Authors: Longteng Jiang, DanDan Zheng, Qianqian Qiao, Heng Huang, Huaye Wang, Yihang Bo, Bao Peng, Jingdong Chen, +2 more
Organizations: Ant Group, Beijing, China · Beijing Film Academy, Beijing, China · State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China
We introduce VGA-BenchV2, an extended human-aligned benchmark and optimization framework for jointly evaluating and improving video generation quality and aesthetic value. Built upon VGA-Bench, VGA-BenchV2 preserves the original fine-grained taxonomy with two primary dimensions-Aesthetic and Generation-and 52 sub-dimensions. Guided by this taxonomy, we curate 1,016 diverse prompts and collect over 60,000 videos generated by 12 mainstream video generation models. More importantly, VGA-BenchV2 substantially expands human-labeled supervision by adding 36,000 task-level annotations, including 16,200 for aesthetic quality, 13,200 for aesthetic tagging, and 6,600 for generation quality, corresponding to 13.46x, 11.15x, and 1.55x scale-ups over VGA-Bench, respectively. Leveraging this enlarged annotation corpus, we develop a hybrid evaluator architecture consisting of VAQA-Net for continuous aesthetic scoring and two Qwen-based Large Vision-Language Model evaluators, VTag-Net and VGQA-Net, for aesthetic tagging and generation quality assessment. Extensive experiments demonstrate strong alignment with human judgments across diverse generation models. Beyond evaluation, VGA-BenchV2 further introduces an evaluation-to-optimization pipeline, where the learned aesthetic evaluator serves as a reward model for reinforcement learning-based generator fine-tuning. This closes the loop from benchmark construction and human supervision to automated evaluation and model optimization, enabling video generators to improve not only in realism but also in aesthetic quality and human preference alignment. Resources are available at https://huggingface.co/datasets/BestiVictoryLab/VGA-Bench.
Figures & tables
Figure 1: Overview of VGA-BenchV2. Left: expanded human annotations supporting evaluator training; middle: hybrid evaluator architecture combining VAQA-Net, VTag-Net, and VGQA-Net; right: holistic evaluation of 12 state-of-the-art video generation models, visualizing performance across 10 representative dimensions selected from the 52 sub-dimensions of VGA-Bench. The bottom panel illustrates the evaluation-to-optimization pipeline using RL.
Figure 2: Comparison between VGA-Bench (V1) and VGA-BenchV2 (V2). Both include human annotations and multi-model evaluation; V2 expands annotation scale, adds a hybrid evaluator, and supports optional RL optimization.
Benchmark
Aesthetic Dims
Evaluator Architecture
Annotation Usage
RL Opt.
V-Bench Huang et al. (2024)
1
Off-the-shelf Expert Models
Alignment Only
✗
V-Bench2.0 Zheng et al. (2025)
2
Off-the-shelf Expert Models
Alignment Only
✗
T2V-CompBench Sun et al. (2025)
0
MLLM-based
Alignment Only
✗
ChronoMagic-Bench Yuan et al. (2024)
0
Hybrid
Alignment Only
✗
StoryEval Wang et al. (2025b)
0
MLLM-based
Alignment Only
✗
VGA-Bench Jiang et al. (2026)
21
Specialized Nets
Training & Alignment
✗
Table 1: Comparison with existing video generation benchmarks. VGA-BenchV2 preserves the fine-grained aesthetic and generation taxonomy of VGA-Bench while further extending it with hybrid human-aligned evaluators and an evaluation-to-optimization pipeline.
Figure 3: Prompts for the three core dimensions, their corresponding sub-dimensions, and example generated videos.
Figure 4: Examples of human annotations for the three core dimensions.
Task
VGA-Bench Supervision
New V2 Annotations
V2 Total
Scale-up
Aesthetic Quality
1,300
+16,200
17,500
13.46 ×
Aesthetic Tagging
1,300
+13,200
14,500
11.15 ×
Generation Quality
12,000
+6,600
18,600
1.55 ×
Total
14,600
+36,000
50,600
3.47 ×
Table 2: Human annotation expansion in VGA-BenchV2. We report task-level supervision for aesthetic quality, aesthetic tagging, and generation quality. VGA-BenchV2 substantially increases the amount of human-labeled data used for evaluator training and alignment.
Figure 5: Overview of the hybrid evaluation networks. VAQA-Net predicts scalar aesthetic scores via a pre-trained encoder and regression head, while VTag-Net and VGQA-Net leverage a LVLM backbone via instruction tuning for aesthetic tagging and quality reasoning.
VAQA-Net (SROCC)
VTag-Net (Accuracy)
Dimension
Score
Dimension
#Cls.
Acc.
Overall Score
87.6
Color Temp. (ColT)
3
64.8
Composition (Com)
86.9
Saturation (Sat)
3
71.9
Shot Size (SS)
87.4
Brightness (Bri)
3
70.3
Lighting (Lig)
87.2
Contrast (Con)
3
75.0
Visual Tone (VT)
87.4
Light Quality (LQ)
2
66.3
Table 3: Validation of Aesthetic Evaluators. We report the SROCC (%) for VAQA-Net and Classification Accuracy (%) for VTag-Net. Specifically, we adopt Top-1 Accuracy for dimensions with ≤4 classes and Top-2 Accuracy for those with >4 classes (marked with *) to account for varying label space sizes.
Dim.
Acc.
Dim.
Acc.
Dim.
Acc.
Dim.
Acc.
Cha-T
70.5
Sty-T
69.6
Sce-R
89.2
Glo-R
69.8
Act-T
70.7
Spd-T
76.6
Obj-R
81.2
Ima-R
59.2
Obj-T
75.2
Rig-R
62.2
Cha-R
56.8
Noi
58.3
Sce-T
74.7
Flu-R
75.6
Wea-R
66.8
AbL
85.7
Pos-T
64.3
Gas-R
78.3
Tim-R
83.1
Cla
66.4
Cam-T
72.5
Gra-R
80.4
Lig-R
80.5
Sta
50.5
Table 4: Validation of VGQA-Net. We report the Accuracy (%) across 31 sub-dimensions. The suffixes -T and -R denote Video-Text Consistency and Reality/Plausibility, respectively.
Figure 6: RL Aesthetic Optimization Pipeline. We utilize the Flow-GRPO algorithm to fine-tune video generation models, employing VAQA-Net as the reward model to enhance artistic quality.
Model
Aes. Score
Tag Cla.
Gen. Level
SVD Blattmann et al. (2023a)
0.20
0.60
0.69
AnimateDiff Guo et al. (2023)
0.36
0.57
0.67
LaVie Wang et al. (2025a)
0.34
0.58
0.68
Show-1 Zhang et al. (2025)
0.29
0.60
0.04
ModelScope Wang et al. (2023a)
0.31
0.51
0.64
CogVideoX Yang et al. (2024)
0.41
0.56
0.70
Table 5: Performance comparison of state-of-the-art text-to-video generation models on aesthetic score (Aes. Score), tag classification accuracy (Tag Cla.), and generation level (Gen. Level) metrics.
The rapid advancement of AIGC-based video generation has underscored the critical need for comprehensive evaluation frameworks that go beyond traditional generation quality metrics to encompass aesthetic appeal. However, existing benchmarks remain largely focused on technical fidelity, leaving a significant gap in holistic assessment-particularly with respect to perceptual and artistic qualities. To address this limitation, we introduce VGA-Bench, a unified benchmark for joint evaluation of video generation quality and aesthetic quality. VGA-Bench is built upon a principled three-tier taxonomy: Aesthetic Quality, Aesthetic Tagging, and Generation Quality, each decomposed into multiple fine-grained sub-dimensions to enable systematic assessment. Guided by this taxonomy, we design 1,016 diverse prompts and generate a large-scale dataset of over 60,000 videos using 12 video generation models, ensuring broad coverage across content, style, and artifacts. To enable scalable and automated evaluation, we annotate a subset of the dataset via human labeling and develop three dedicated multi-task neural assessors: VAQA-Net for aesthetic quality prediction, VTag-Net for automatic aesthetic tagging, and VGQA-Net for generation and basic quality attributes. Extensive experiments demonstrate that our models achieve reliable alignment with human judgments, offering both accuracy and efficiency. We release VGA-Bench as a public benchmark to foster research in AIGC evaluation, with applications in content moderation, model debugging, and generative model optimization.
Longteng Jiang, DanDan Zheng, Qianqian Qiao +7
Ant Group · Beijing Film Academy · State Key Laboratory of General Artificial Intelligence, BIGAI
Despite rapid advancements in video generation models, aligning their outputs with complex user intent remains challenging. Existing test-time optimization methods are typically either computationally expensive or require white-box access to model internals. To address this, we present VQQA (Video Quality Question Answering), a unified, multi-agent framework generalizable across diverse input modalities and video generation tasks. By dynamically generating visual questions and using the resulting Vision-Language Model (VLM) critiques as semantic gradients, VQQA replaces traditional, passive evaluation metrics with human-interpretable, actionable feedback. This enables a highly efficient, closed-loop prompt optimization process via a black-box natural language interface. Extensive experiments demonstrate that VQQA effectively isolates and resolves visual artifacts, substantially improving generation quality in just a few refinement steps. Applicable to both text-to-video (T2V) and image-to-video (I2V) tasks, our method achieves absolute improvements of +11.57% on T2V-CompBench and +8.43% on VBench2 over vanilla generation, significantly outperforming state-of-the-art stochastic search and prompt optimization techniques.
Video-to-video (V2V) generation is difficult to evaluate because outputs must both follow editing instructions and preserve frame-level correspondence with the source video, which existing T2V and I2V metrics do not capture. We introduce V2V-Bench, a 11-dimension benchmark organized into five categories: temporal alignment, structural fidelity, transformation quality, video quality, and semantic alignment. V2V-Bench pairs diverse source videos with challenging editing tasks and evaluates two commercial models, Grok Imagine and Gemini Veo3, and one open-source model, Open Sora 2. Results show complementary model strengths: Grok performs better on editing fidelity, while Veo3 achieves stronger visual quality. On six V2V-specific dimensions, V2V-Bench reaches a Spearman correlation of 0.905 with human judgments.