Infinite Gaze Generation for Videos with Autoregressive Diffusion

Jenna Kang; Colin Groth; Tong Wu; Finley Torrens; Patsorn Sangkloy; Gordon Wetzstein; Qi Sun

cs.CVMar 26, 2026

Infinite Gaze Generation for Videos with Autoregressive Diffusion

Authors: Jenna Kang, Colin Groth, Tong Wu, Finley Torrens, Patsorn Sangkloy, Gordon Wetzstein, Qi Sun

Organizations: New York University · Stanford University

Abstract

Predicting human gaze in video is fundamental to advancing scene understanding and multimodal interaction. While traditional saliency maps provide spatial probability distributions and scanpaths offer ordered fixations, both abstractions often collapse the fine-grained temporal dynamics of raw gaze. Furthermore, existing models are typically constrained to short-term windows ( $\approx$ 3-5s), failing to capture the long-range behavioral dependencies inherent in real-world content. We present a generative framework for infinite-horizon raw gaze prediction in videos of arbitrary length. By leveraging an autoregressive diffusion model, we synthesize gaze trajectories characterized by continuous spatial coordinates and high-resolution timestamps. Our model is conditioned on a saliency-aware visual latent space. Quantitative and qualitative evaluations demonstrate that our approach significantly outperforms existing approaches in long-range spatio-temporal accuracy and trajectory realism.

Figures & tables

Panel of Figure 1 — Figure 1: Our training and inference pipelines. The input of each iteration consists of the n-dimensional input vector with gaze history (GT) and noise for future prediction, and video frames corresponding to the prediction window. During inference, the gaze history of this input vector is filled with former predictions in an autoregressive manner. Video latents are generated frame-by-frame with a modified version of UNISAL. The blue boxes indicate our modifications, while the orange boxes indicate the original architecture. The low-dimensional video latents are processed by the U-Net with cross-attention at each block.

Figure 2: Comparison of the trajectories of video gaze predictions with different methods. The image shows a representative frame selected from the video, but the path is generated for the entire duration of the video. Generated trajectories (color map indicates timestamps) are overlaid on the ground truth human gaze (black trajectories, which are also shown in the left column with the color map).

Figure 3: Temporal progression of video gaze predictions with different methods. Generated trajectories are overlaid on the ground-truth human gaze (black). Note the closer proximity between our method and the ground truth, indicating greater similarity compared with alternative approaches.

Dataset	Method	Levenshtein ( $\times 10^{3}$ ) $\downarrow$		Disc. Fréchet ( $\times 10^{2}$ ) $\downarrow$		DTW ( $\times 10^{4}$ ) $\downarrow$		Max. Temp. Corr. $\uparrow$
Dataset	Method	Mean	Best	Mean	Best	Mean	Best	Mean	Best
DIEM	DeepGaze III [ 27 ]	4.36	4.26	4.78	4.12	20.07	18.02	0.149	0.230
	DiffEye [ 25 ]	3.78	3.71	3.82	3.29	12.47	11.66	0.176	0.254
	GazeFormer [ 34 ]	4.26	4.12	4.17	3.42	15.64	13.71	0.188	0.272
	HAT [ 55 ]	4.45	4.32	5.16	4.28	23.44	20.42	0.131	0.223
	Chen et al. [ 10 ]	4.23	4.09	4.36	3.35	15.97	13.51	0.175	0.270

Table 1: Scanpath evaluation on DIEM and DHF1K test sets. Lower is better for all metrics except Maximum Temporal Correlation. Metrics are scaled as indicated in the column headers. Bold values indicate the best scores, while underlined values denote the second-best scores.

Figure 4: User study results. Each row corresponds to one video. Our model (blue) receives consistently more votes than TPP-Gaze (green).

Method	Levenshtein ( $\times 10^{3}$ ) $\downarrow$		Disc. Fréchet ( $\times 10^{2}$ ) $\downarrow$		DTW ( $\times 10^{4}$ ) $\downarrow$		Max. Temp. Corr. $\uparrow$
Method	Mean	Best	Mean	Best	Mean	Best	Mean	Best
Ours regular	3.40	3.31	4.14	3.35	12.33	11.44	0.235	0.314
10-Prediction	3.97	3.10	3.67	2.90	17.31	6.88	0.048	0.126
90-Prediction	3.77	3.63	4.38	3.58	14.65	13.84	0.230	0.309
No initial history	3.41	3.31	4.13	3.34	12.39	11.57	0.235	0.314
MAGVIT2 Conditioning	3.82	3.76	7.97	6.96	54.92	48.15	0.054	0.082

Table 2: Ablation study results . Regular training and if not denoted otherwise is run with 90 history samples and 45 prediction samples on the DIEM dataset. Lower scores are better for all metrics except Maximum Temporal Correlation. Metrics are scaled as indicated in the column headers. Bold values indicate the best scores, while underlined values denote the second-best scores.

Gaze Forecasting

See all 11 papers

GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction
Sep 29, 2026Sheng Zhao, Weikai Lin, Yuhao ZhuGaze Estimation Egocentric Video Understanding
Department of Computer Science University of Rochester
ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling
Jun 13, 2026Brian Nlong Zhao, Ozgur Kara, Junho Kim +1Eye Tracking Conditional Generative Modeling
University of Illinois Urbana-Champaign
DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction
Jul 2, 2026Susmit Agrawal, Matthias Bethge, Matthias KümmererVisual Attention Scanpath Prediction
University of Tübingen · Tübingen AI Center · IMPRS-IS

Sep 29, 2026J/K move · Enter open · S save

GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction

Sheng Zhao, Weikai Lin, Yuhao Zhu

Department of Computer Science University of Rochester

Gaze Estimation Egocentric Video Understanding Gaze Forecasting

Open paper AskPDF arXiv

Egocentric gaze prediction enables many downstream applications but remains challenging, as human gaze is inherently stochastic. This stochasticity is constrained by structured temporal dynamics alternating between fixations and saccades, top-down influences from tasks, and bottom-up visual saliency. Based on this observation, we introduce GazeFlow, a framework that directly models gaze as a joint distribution of temporal gaze positions conditioned upon both top-down and bottom-up information. In particular, GazeFlow uses conditional flow matching (CFM): a learned velocity field iteratively transports a Gaussian noise sample into a plausible gaze trajectory drawn from this joint distribution. The velocity field is conditioned on bottom-up visual features extracted by a video encoder and on top-down task information obtained by globally querying these features. On standard datasets, GazeFlow achieves state-of-the-art performance on per-frame metrics, and the generated trajectories align better with human gaze temporal dynamics.

Abstract

Gaze Forecasting

Explore similar work

GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction

ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling

DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction