Predicting human gaze in video is fundamental to advancing scene understanding and multimodal interaction. While traditional saliency maps provide spatial probability distributions and scanpaths offer ordered fixations, both abstractions often collapse the fine-grained temporal dynamics of raw gaze. Furthermore, existing models are typically constrained to short-term windows (≈ 3-5s), failing to capture the long-range behavioral dependencies inherent in real-world content. We present a generative framework for infinite-horizon raw gaze prediction in videos of arbitrary length. By leveraging an autoregressive diffusion model, we synthesize gaze trajectories characterized by continuous spatial coordinates and high-resolution timestamps. Our model is conditioned on a saliency-aware visual latent space. Quantitative and qualitative evaluations demonstrate that our approach significantly outperforms existing approaches in long-range spatio-temporal accuracy and trajectory realism.
Figures & tables
Figure 1: Our training and inference pipelines. The input of each iteration consists of the n-dimensional input vector with gaze history (GT) and noise for future prediction, and video frames corresponding to the prediction window. During inference, the gaze history of this input vector is filled with former predictions in an autoregressive manner. Video latents are generated frame-by-frame with a modified version of UNISAL. The blue boxes indicate our modifications, while the orange boxes indicate the original architecture. The low-dimensional video latents are processed by the U-Net with cross-attention at each block.
Figure 2: Comparison of the trajectories of video gaze predictions with different methods. The image shows a representative frame selected from the video, but the path is generated for the entire duration of the video. Generated trajectories (color map indicates timestamps) are overlaid on the ground truth human gaze (black trajectories, which are also shown in the left column with the color map).
Figure 3: Temporal progression of video gaze predictions with different methods. Generated trajectories are overlaid on the ground-truth human gaze (black). Note the closer proximity between our method and the ground truth, indicating greater similarity compared with alternative approaches.
Dataset
Method
Levenshtein ( ×103 ) ↓
Disc. Fréchet ( ×102 ) ↓
DTW ( ×104 ) ↓
Max. Temp. Corr. ↑
Mean
Best
Mean
Best
Mean
Best
Mean
Best
DIEM
DeepGaze III [ 27 ]
4.36
4.26
4.78
4.12
20.07
18.02
0.149
0.230
DiffEye [ 25 ]
3.78
3.71
3.82
3.29
12.47
11.66
0.176
0.254
GazeFormer [ 34 ]
4.26
4.12
4.17
3.42
15.64
13.71
0.188
0.272
HAT [ 55 ]
4.45
4.32
5.16
4.28
23.44
20.42
0.131
0.223
Chen et al. [ 10 ]
4.23
4.09
4.36
3.35
15.97
13.51
0.175
0.270
Table 1: Scanpath evaluation on DIEM and DHF1K test sets. Lower is better for all metrics except Maximum Temporal Correlation. Metrics are scaled as indicated in the column headers. Bold values indicate the best scores, while underlined values denote the second-best scores.
Figure 4: User study results. Each row corresponds to one video. Our model (blue) receives consistently more votes than TPP-Gaze (green).
Method
Levenshtein ( ×103 ) ↓
Disc. Fréchet ( ×102 ) ↓
DTW ( ×104 ) ↓
Max. Temp. Corr. ↑
Mean
Best
Mean
Best
Mean
Best
Mean
Best
Ours regular
3.40
3.31
4.14
3.35
12.33
11.44
0.235
0.314
10-Prediction
3.97
3.10
3.67
2.90
17.31
6.88
0.048
0.126
90-Prediction
3.77
3.63
4.38
3.58
14.65
13.84
0.230
0.309
No initial history
3.41
3.31
4.13
3.34
12.39
11.57
0.235
0.314
MAGVIT2 Conditioning
3.82
3.76
7.97
6.96
54.92
48.15
0.054
0.082
Table 2: Ablation study results . Regular training and if not denoted otherwise is run with 90 history samples and 45 prediction samples on the DIEM dataset. Lower scores are better for all metrics except Maximum Temporal Correlation. Metrics are scaled as indicated in the column headers. Bold values indicate the best scores, while underlined values denote the second-best scores.
Egocentric gaze prediction enables many downstream applications but remains challenging, as human gaze is inherently stochastic. This stochasticity is constrained by structured temporal dynamics alternating between fixations and saccades, top-down influences from tasks, and bottom-up visual saliency. Based on this observation, we introduce GazeFlow, a framework that directly models gaze as a joint distribution of temporal gaze positions conditioned upon both top-down and bottom-up information. In particular, GazeFlow uses conditional flow matching (CFM): a learned velocity field iteratively transports a Gaussian noise sample into a plausible gaze trajectory drawn from this joint distribution. The velocity field is conditioned on bottom-up visual features extracted by a video encoder and on top-down task information obtained by globally querying these features. On standard datasets, GazeFlow achieves state-of-the-art performance on per-frame metrics, and the generated trajectories align better with human gaze temporal dynamics.
Sheng Zhao, Weikai Lin, Yuhao Zhu
Department of Computer Science University of Rochester
We study the problem of human gaze modeling, which aims to generate the gaze patterns a viewer produces while observing a visual stimulus. Gaze is primarily captured through two modalities: continuous eye-tracking trajectories, which describe fine-grained motion dynamics, and discrete scanpaths, which describe high-level fixation structure. Because gaze varies substantially across viewers and trials, we treat this variability as a defining property rather than noise and model gaze as a stochastic generative process. Existing generative gaze models supervise on only one of these two representations in isolation. We hypothesize that trajectories and scanpaths describe gaze at complementary scales and are jointly informative during training, and test this hypothesis through ST-DiffEye, a joint trajectory-scanpath diffusion framework that couples both modalities by concatenating them as an additional raw input channel, requiring no architectural overhead beyond an input and output channel expansion. We further introduce a principled evaluation framework based on the Continuous Ranked Probability Score (CRPS), which generalizes any existing sequence similarity metric into a proper scoring rule that jointly assesses the accuracy and diversity of generated gaze. Experiments on task-driven visual search, covering both target-present and target-absent scenarios, and on free-viewing benchmarks demonstrate state-of-the-art performance. These results, along with detailed ablations, confirm the benefit of joint modeling and the value of distribution-aware evaluation in capturing the intrinsic variability of human gaze. Project webpage: https://st-diffeye.github.io/
Understanding human visual attention on a scene over time has applications in domains such as interface design and inferring cognitive states. Modeling visual scanpaths has historically relied on specialized architectures with hand-crafted priors. While these architectures can model fixation sequences, their rigid structural biases restrict easy extendability and flexible conditioning. For instance, integrating task-specific instructions or adapting to distinct viewer identities requires custom, disjoint architectural additions. We frame scanpath prediction purely as a discrete sequence modeling task. By mapping coordinates into a text vocabulary, we leverage the pretrained representations of Vision-Language Models. This framing absorbs diverse factors of variation: simple prompting allows for global conditioning, such as providing viewer identities to capture personalized biases, or task-specific objectives like visual search. The framework can also integrate per-fixation attributes, such as individual fixation durations, alongside spatial locations. The autoregressive alignment enables the scalable, exact computation of per-fixation log-likelihoods, directly equivalent to the commonly used Information Gain (IG) metric. Our model, DeepGaze3.5-VL, establishes a new state-of-the-art across multiple datasets, achieving 2.18 bits of IG on MIT1003, a 46% improvement over DeepGaze III. This advantage persists even when baselines use identical high-capacity vision encoders. Beyond predictive performance, our generative framework serves as a powerful computational tool for direct behavioral interventions, allowing for controlled in-silico simulations that would be experimentally difficult or impossible to conduct in vivo. We demonstrate this ability by performing controlled interventions on the durations of pre-saccadic fixations, recovering known oculomotor phenomena purely from data.