Evaluating Visual Realism of AI-Generated Videos with Eye Tracking

Danqing Shi; Lan Jiang; Jake Lyons; Miri Zilka

cs.HCOct 28, 2026

Evaluating Visual Realism of AI-Generated Videos with Eye Tracking

Authors: Danqing Shi, Lan Jiang, Jake Lyons, Miri Zilka

Organizations: Lund University Sweden · University of Cambridge United Kingdom

Abstract

Human evaluation of AI-generated videos has so far relied on explicit human feedback, such as judgments or ratings. While these signals capture the conclusions viewers reach, they fail to offer insights into the perception that leads to them. This paper investigates eye tracking as a complementary human evaluation signal for AI-generated videos. We conducted two Visual Turing Test (VTT) experiments with a total of 90 participants on videos generated by latest generative models, in which participants watched each video twice and then gave their feedback. Across the experiments, we collected 4,286 valid eye-tracking trials on 438 videos. Our analysis indicates that gaze patterns offer informative evaluation signals that align with explicit human feedback. Building on the findings, we propose the first computational method GazeEval to predict the perceived visual realism of AI-generated videos from eye-tracking data. It achieves 71.3% accuracy in prediction, providing evidence that eye tracking data carries informative signals about how people evaluate a video. We release the study data to support further research on eye tracking for evaluating AI-generated video.

Figures & tables

Figure 1. Our study investigates eye tracking as an evaluation signal for video generation. Left : The first frames and text descriptions of real videos are used for video generation with generative models, producing an AI-generated video set of physical scenes. Middle : Participants watch a mix of real and AI-generated videos while their eye movements are recorded under the Visual Turing Test (VTT): Participants are asked to judge whether each video is AI-generated and rate the visual realism after watching the video twice. Right : We analyze predicted saliency maps and eye-tracking data from the first and second watches. We train GazeEval that predicts percieved visual realism from eye tracking data. Our findings indicate that eye tracking data from two-pass VTT can serve as an effective signal for video evaluation.

Figure 2. Group comparisons for the two-pass Visual Turing Test. (a–c) report the four saliency evaluation metrics for the three attention-map comparisons; (d) report the four eye movement statistics for the second watch. Each panel carries two comparisons: AI vs. real clips and judged AI vs. judged real. Brackets mark Wilcoxon signed-rank tests: ${}^{*}\,p<.05$ , ${}^{**}\,p<.01$ . The full report can be found in Appendix A .

Figure 3. An illustration of five frames (one per second) from a video generated by the prompt “ A glass beverage dispenser pouring grapefruit juice into a glass that contains some water. ” The clip is decomposed second by second here to make the temporal pattern visible. The top row shows the bottom-up dynamic video saliency predicted by SalFoM, focusing on the motion area. The other two rows shows the aggregated attention maps from the first and second watch. The human attention map during the first watch aligns more closely with the bottom-up saliency map, while the attention map from the second watch shows greater divergence, as viewers attempt to identify anomalies. In this video, the error is that while the glass dispenser pours out water, the water level remains unchanged. During the second watch, viewers focused more on the water level.

Figure 4. Sample videos from five video generative models with human attention maps from our data collection. All the study videos and human attention maps can be found in the supplemental material.

Figure 5. The Spearman correlation was calculated between each saliency evaluation metric and viewers’ own five-point visual realism ratings across all (viewer, clip) pairs for the 300 AI-generated clips. The analysis includes three types of attention-map comparisons. Six metrics (CC, SIM, NSS, AUC, sAUC, IG) measure agreement, while two (KL and EMD) measure distance, leading to an expectation of opposite signs for their correlations. In all cases, a greater consistency in gaze patterns corresponds with videos that are perceived as less realistic.

Figure 6. From eye tracking data to visual realism prediction. Top: This presents a running example. Accompanying the stimulus keyframe are the attention maps from viewers’ first and second watches, each generated by pooling the entire clip. On the right, SalFoM’s prediction for the same clip is shown, derived from the raw pixels. The three comparisons form the basis for the model inputs: the first watch compared with the second watch, and each of the two watches compared against SalFoM’s prediction. Middle: Each comparison is evaluated using eight saliency metrics: CC, SIM, EMD, KL, NSS, AUC, sAUC, and IG, resulting in the 24 video-level features. Since SalFoM generates a density map without attached fixations, the directed metrics are applied in one direction only. The resulting vector feeds into a classifier from the six candidate families listed, utilizing cross-validated ROC-AUC within each training fold. Bottom: This histogram represents the held-out probabilities for all 300 clips, categorized by the ratings the clips actually received. The bars above the line indicate the 165 clips with a mean rating exceeding 3, while the bars below indicate the 135 clips that did not. Each clip is scored based on the fold that held it out within a single scene-grouped partition.

Method	#Features	Accuracy	Bal. Accuracy	ROC-AUC
Trivial baseline	–	$0.550$	$0.500$	$0.500$
Stimulus-only baseline (VBench features)	7	$0.600\pm 0.039$	$0.614\pm 0.044$	$0.660\pm 0.060$
Eye movement statistics	8	$0.503\pm 0.069$	$0.522\pm 0.065$	$0.496\pm 0.096$
Saliency evaluation metrics (1st vs. SalFoM)	8	$0.617\pm 0.073$	$0.618\pm 0.063$	$0.665\pm 0.076$
Saliency evaluation metrics (2nd vs. SalFoM)	8	$0.667\pm 0.059$	$0.665\pm 0.057$	$0.689\pm 0.039$
Saliency evaluation metrics (1st vs. 2nd)	8	$0.687\pm 0.034$	$0.690\pm 0.034$	$0.751\pm 0.046$

Table 1. Predicting perceived visual realism from gaze at video level. The input features are from video stimulus and eye tracking data. Scores are the mean

\pm

standard deviation over 5 held-out folds. #Features is the number of features. Within each metric, the best score is colored in dark blue and the second is in light blue.

Figure 7. Accuracy against the number of viewers per video for GazeEval with saliency evaluation metrics (1st vs. 2nd vs. SalFoM). Accuracy rises monotonically with the number of viewers $k$ from $56.0\%$ ( $k=1$ ) to $71.7\%$ ( $k=8$ ). In our dataset, some clips have fewer viewers than $k$ , so $k=7$ and $k=8$ average $6.88$ and $7.60$ viewers.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 8. Group comparisons for the two-pass Visual Turing Test. (a–c) report the eight saliency evaluation metrics for the three attention-map comparisons; (d–e) report the four eye movement statistics for each pass. Each panel carries two comparisons: AI vs. real clips and judged AI vs. judged real. Brackets mark Wilcoxon signed-rank tests: $*\,p<.05$ , $**\,p<.01$ .

Video Generation Evaluation

Video generation evaluation measures visual quality, prompt adherence, temporal consistency, physical plausibility, and task-specific behavior in synthesized videos.

See all 192 papers

HumanScore: Benchmarking Human Motions in Generated Videos
Apr 22, 2026Yusu Fang, Tiange Xiang, Tian Tan +4Physical Consistency in Video Generation Temporal Consistency in Video Generation
Stanford University · Peking University
RAVEN-Eval: Rubric-Guided Automatic Evaluation for AI Video Generation Models Based on LMM Preference Judgement
Aug 10, 2026Ziheng Jia, Jiaying Qian, Zicheng Zhang +3LLM-as-a-Judge Rubric-Based Evaluation
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
VGI-Bench: Probing Visual Intelligence in Video Generation Models
Aug 20, 2026Xuan He, Cong Wei, Yuhao Cheng +20Video Generation Evaluation
University of Illinois Urbana Champaign · University of Waterloo · Vector Institute +6

Apr 22, 2026J/K move · Enter open · S save

HumanScore: Benchmarking Human Motions in Generated Videos

Yusu Fang, Tiange Xiang, Tian Tan, Narayan Schuetz+3

Stanford University · Peking University

Physical Consistency in Video Generation Temporal Consistency in Video Generation Video Generation Evaluation

Open paper AskPDF arXiv

Recent advances in model architectures, compute, and data scale have driven rapid progress in video generation, producing increasingly realistic content. Yet, no prior method systematically measures how faithfully these systems render human bodies and motion dynamics. In this paper, we present HumanScore, a systematic framework to evaluate the quality of human motions in AI-generated videos. HumanScore defines six interpretable metrics spanning kinematic plausibility, temporal stability, and biomechanical consistency, enabling fine-grained diagnosis beyond visual realism alone. Through carefully designed prompts, we elicit a diverse set of movements at varying intensities and evaluate videos generated by thirteen state-of-the-art models. Our analysis reveals consistent gaps between perceptual plausibility and motion biomechanical fidelity, identifies recurrent failure modes (e.g., temporal jitter, anatomically implausible poses, and motion drift), and produces robust model rankings from quantitative and physically meaningful criteria.

Abstract

Video Generation Evaluation

Explore similar work

HumanScore: Benchmarking Human Motions in Generated Videos

RAVEN-Eval: Rubric-Guided Automatic Evaluation for AI Video Generation Models Based on LMM Preference Judgement

VGI-Bench: Probing Visual Intelligence in Video Generation Models