Human evaluation of AI-generated videos has so far relied on explicit human feedback, such as judgments or ratings. While these signals capture the conclusions viewers reach, they fail to offer insights into the perception that leads to them. This paper investigates eye tracking as a complementary human evaluation signal for AI-generated videos. We conducted two Visual Turing Test (VTT) experiments with a total of 90 participants on videos generated by latest generative models, in which participants watched each video twice and then gave their feedback. Across the experiments, we collected 4,286 valid eye-tracking trials on 438 videos. Our analysis indicates that gaze patterns offer informative evaluation signals that align with explicit human feedback. Building on the findings, we propose the first computational method GazeEval to predict the perceived visual realism of AI-generated videos from eye-tracking data. It achieves 71.3% accuracy in prediction, providing evidence that eye tracking data carries informative signals about how people evaluate a video. We release the study data to support further research on eye tracking for evaluating AI-generated video.
Figures & tables
Figure 1. Our study investigates eye tracking as an evaluation signal for video generation. Left : The first frames and text descriptions of real videos are used for video generation with generative models, producing an AI-generated video set of physical scenes. Middle : Participants watch a mix of real and AI-generated videos while their eye movements are recorded under the Visual Turing Test (VTT): Participants are asked to judge whether each video is AI-generated and rate the visual realism after watching the video twice. Right : We analyze predicted saliency maps and eye-tracking data from the first and second watches. We train GazeEval that predicts percieved visual realism from eye tracking data. Our findings indicate that eye tracking data from two-pass VTT can serve as an effective signal for video evaluation.
Figure 2. Group comparisons for the two-pass Visual Turing Test. (a–c) report the four saliency evaluation metrics for the three attention-map comparisons; (d) report the four eye movement statistics for the second watch. Each panel carries two comparisons: AI vs. real clips and judged AI vs. judged real. Brackets mark Wilcoxon signed-rank tests: ∗p<.05 , ∗∗p<.01 . The full report can be found in Appendix A .
Figure 3. An illustration of five frames (one per second) from a video generated by the prompt “ A glass beverage dispenser pouring grapefruit juice into a glass that contains some water. ” The clip is decomposed second by second here to make the temporal pattern visible. The top row shows the bottom-up dynamic video saliency predicted by SalFoM, focusing on the motion area. The other two rows shows the aggregated attention maps from the first and second watch. The human attention map during the first watch aligns more closely with the bottom-up saliency map, while the attention map from the second watch shows greater divergence, as viewers attempt to identify anomalies. In this video, the error is that while the glass dispenser pours out water, the water level remains unchanged. During the second watch, viewers focused more on the water level.
Figure 4. Sample videos from five video generative models with human attention maps from our data collection. All the study videos and human attention maps can be found in the supplemental material.
Figure 5. The Spearman correlation was calculated between each saliency evaluation metric and viewers’ own five-point visual realism ratings across all (viewer, clip) pairs for the 300 AI-generated clips. The analysis includes three types of attention-map comparisons. Six metrics (CC, SIM, NSS, AUC, sAUC, IG) measure agreement, while two (KL and EMD) measure distance, leading to an expectation of opposite signs for their correlations. In all cases, a greater consistency in gaze patterns corresponds with videos that are perceived as less realistic.
Figure 6. From eye tracking data to visual realism prediction. Top: This presents a running example. Accompanying the stimulus keyframe are the attention maps from viewers’ first and second watches, each generated by pooling the entire clip. On the right, SalFoM’s prediction for the same clip is shown, derived from the raw pixels. The three comparisons form the basis for the model inputs: the first watch compared with the second watch, and each of the two watches compared against SalFoM’s prediction. Middle: Each comparison is evaluated using eight saliency metrics: CC, SIM, EMD, KL, NSS, AUC, sAUC, and IG, resulting in the 24 video-level features. Since SalFoM generates a density map without attached fixations, the directed metrics are applied in one direction only. The resulting vector feeds into a classifier from the six candidate families listed, utilizing cross-validated ROC-AUC within each training fold. Bottom: This histogram represents the held-out probabilities for all 300 clips, categorized by the ratings the clips actually received. The bars above the line indicate the 165 clips with a mean rating exceeding 3, while the bars below indicate the 135 clips that did not. Each clip is scored based on the fold that held it out within a single scene-grouped partition.
Method
#Features
Accuracy
Bal. Accuracy
ROC-AUC
Trivial baseline
–
0.550
0.500
0.500
Stimulus-only baseline (VBench features)
7
0.600±0.039
0.614±0.044
0.660±0.060
Eye movement statistics
8
0.503±0.069
0.522±0.065
0.496±0.096
Saliency evaluation metrics (1st vs. SalFoM)
8
0.617±0.073
0.618±0.063
0.665±0.076
Saliency evaluation metrics (2nd vs. SalFoM)
8
0.667±0.059
0.665±0.057
0.689±0.039
Saliency evaluation metrics (1st vs. 2nd)
8
0.687±0.034
0.690±0.034
0.751±0.046
Table 1. Predicting perceived visual realism from gaze at video level. The input features are from video stimulus and eye tracking data. Scores are the mean ± standard deviation over 5 held-out folds. #Features is the number of features. Within each metric, the best score is colored in dark blue and the second is in light blue.
Figure 7. Accuracy against the number of viewers per video for GazeEval with saliency evaluation metrics (1st vs. 2nd vs. SalFoM). Accuracy rises monotonically with the number of viewers k from 56.0% ( k=1 ) to 71.7% ( k=8 ). In our dataset, some clips have fewer viewers than k , so k=7 and k=8 average 6.88 and 7.60 viewers.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 8. Group comparisons for the two-pass Visual Turing Test. (a–c) report the eight saliency evaluation metrics for the three attention-map comparisons; (d–e) report the four eye movement statistics for each pass. Each panel carries two comparisons: AI vs. real clips and judged AI vs. judged real. Brackets mark Wilcoxon signed-rank tests: ∗p<.05 , ∗∗p<.01 .
Recent advances in model architectures, compute, and data scale have driven rapid progress in video generation, producing increasingly realistic content. Yet, no prior method systematically measures how faithfully these systems render human bodies and motion dynamics. In this paper, we present HumanScore, a systematic framework to evaluate the quality of human motions in AI-generated videos. HumanScore defines six interpretable metrics spanning kinematic plausibility, temporal stability, and biomechanical consistency, enabling fine-grained diagnosis beyond visual realism alone. Through carefully designed prompts, we elicit a diverse set of movements at varying intensities and evaluate videos generated by thirteen state-of-the-art models. Our analysis reveals consistent gaps between perceptual plausibility and motion biomechanical fidelity, identifies recurrent failure modes (e.g., temporal jitter, anatomically implausible poses, and motion drift), and produces robust model rankings from quantitative and physically meaningful criteria.
AI video generation has advanced rapidly and entered widespread commercial use. As a result, quality differences among videos produced by state-of-the-art AI video generation models~(AIVGMs) have become increasingly difficult to discern using conventional evaluation criteria, such as visual fidelity and semantic instruction following. Meanwhile, human evaluation now requires more expertise and sustained attention, substantially increasing annotation costs. This calls for automated evaluation that can reliably distinguish fine-grained differences among advanced AIVGMs with minimal human intervention. To address this challenge, we present RAVEN-Eval, a rubric-guided automated evaluation framework for AIVGMs, built primarily on the LMM-as-a-judge paradigm. Through an automatic task curation and quality-filtering pipeline, RAVEN-Eval curates 150 text-to-video~(T2V) tasks and 100 image-to-video~(I2V) tasks, and systematically collects more than 4,500 AIGVs. At its core, RAVEN-Eval adopts rubric-guided automated LMM preference judgement, in which LMM judges conduct pairwise comparisons according to task-specific rubrics. It further introduces an anchor-based model insertion approach to reduce the evaluation cost of incorporating new models. Finally, we evaluate 20 high-performance AIVGMs, as well as the judging capabilities of 13 LMM judges, and establish the RAVEN-Eval Leaderboards. Overall, RAVEN-Eval paves a scalable path for automatic and trustworthy evaluation of rapidly evolving AIVGMs.
Ziheng Jia, Jiaying Qian, Zicheng Zhang +3
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of visual reasoning capabilities of video generation models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our analysis further explore the output failure modes, input condition sensitivity, performance transfer boundary from synthetic fine-tuning, and internal denoising perspective revealing limited self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-bench will help stimulate the development of next-generation video generation models. Website: https://hexuan21.github.io/VGI-Bench/
Xuan He, Cong Wei, Yuhao Cheng +20
University of Illinois Urbana Champaign · University of Waterloo · Vector Institute +6