EyeTAG: Eye Trajectory-Aware Gaze Estimation

Jungmin Lee; Niamat Ullah; Yoseob Han

cs.CVOct 1, 2026

EyeTAG: Eye Trajectory-Aware Gaze Estimation

Authors: Jungmin Lee, Niamat Ullah, Yoseob Han

Organizations: Department of Information and Telecommunication Engineering Soongsil University Seoul, Republic of Korea · Department of Electronic Engineering Soongsil University Seoul, Republic of Korea

Abstract

Gaze estimation under natural head-eye motion underpins applications from driver monitoring to human-computer interaction. Single-frame methods predict each frame independently, so consecutive outputs fluctuate as jitter. Multi-frame methods reduce this, but they learn motion implicitly inside appearance features, so the gaze trajectory is never an explicit variable. We propose EyeTAG (Eye Trajectory-Aware Gaze Estimation), a causal multi-frame framework built around an explicit first-order gaze prior: at each step it differentiates its own recent predictions and feeds the resulting trajectory back as a compact kinematic token. Because differencing is translation-invariant in gaze space, this token carries subject-invariant motion rather than personal gaze offsets. Face and eye streams supply visual evidence, fused by cross-attention and a causal Transformer decoder. EyeTAG reduces the mean angular error by about 1.0 $^\circ$ on Gaze360 and performs on par with the strongest baseline on EVE (2.56 $^\circ$ vs. 2.58 $^\circ$ ). Within-model ablations, which keep the encoder and the rest of the architecture fixed and vary only the gaze history, show that the differential formulation, rather than temporal context alone, removes the systematic saccade bias that persists even with an absolute gaze-history prior. Our code is available at https://github.com/peter8366/EyeTAG.

Figures & tables

Figure 1: Visualization of gaze direction $\mathbf{g}_{t}$ and its underlying motion kinematics. (a) Definition of the 3D coordinate system. (b) Four consecutive frames illustrating the gaze vector evolving over time. (c) Overlaid 3D visualization, where $\Delta\mathbf{g}_{t}$ denotes the differential change (i.e., velocity) between consecutive gaze directions.

Figure 2: Overview of the EyeTAG framework and its autoregressive components. (a) Main architecture. The DSVE ( $\mathcal{F},\mathcal{E}$ ) extracts face and eye embeddings from multi-frame inputs, while the TAKE ( $\mathcal{T}$ ) encodes temporal gaze-motion priors from previous predictions. These features are fused by the Cross-Modal Causal Attention Fusion (CCAF; $\mathcal{C}$ ) module with a causal Transformer decoder, followed by the Gaze Vector Predictor (GVP; $\mathcal{G}$ ) to estimate the final 3D gaze vector $\hat{\mathbf{g}}_{t}$ . (b) Autoregressive update loop. (c) Legend for the modules used in the diagram.

Figure 3: Detailed architecture of the EyeTAG framework. (a),(b) DSVE ( $\mathcal{F}$ , $\mathcal{E}$ ) and (c) TAKE ( $\mathcal{T}$ ) modules generate visual features (global face, local eyes) and a kinematic prior (from $\Delta\hat{\mathbf{G}}$ ), respectively. These are fused by (d) CCAF ( $\mathcal{C}$ ) and regressed by (e) GVP ( $\mathcal{G}$ ) to produce the final gaze $\hat{\mathbf{g}}_{t}$ . (f) Notation for operations used in the diagram.

Split	(a) Gaze360 Dataset [ Kellnhofer et al.(2019)Kellnhofer, Recasens, Stent, Matusik, and Torralba ]			(b) EVE Dataset [ Park et al.(2020)Park, Aksan, Zhang, and Hilliges ]
Split	# Clips	# Frames	Avg Length (s)	# Clips	# Frames	Avg Length (s)
Train	321	84,902	8.8	3,015	1,286,850	13.2
Validation	85	11,318	4.4	336	155,640	13.8
Test	68	16,031	7.9	387	169,860	14.0

Table 1: Summary statistics of the (a) Gaze360 and (b) EVE datasets used in our experiments. For each split we report the number of clips (# Clips), the number of annotated frames (# Frames), and the average clip length in seconds (Avg Length (s)).

Type	Method	(a) Gaze360 Dataset [ Kellnhofer et al.(2019)Kellnhofer, Recasens, Stent, Matusik, and Torralba ]				(b) EVE Dataset [ Park et al.(2020)Park, Aksan, Zhang, and Hilliges ]	(c) Complexity
Type	Method	All	Semi-Front	Front	Mean	All	Time (ms)	GFLOPs
SF	L2CS-Net [ Abdelrahman et al.(2023)Abdelrahman, Hempel, Khalifa, Al-Hamadi, and Dinges ]	11.10 $\pm$ 9.82	10.77 $\pm$ 8.30	9.92 $\pm$ 7.97	10.60 $\pm$ 8.70	5.06 $\pm$ 6.12	5.42	16.57
	GazeTR-Hybrid [ Cheng and Lu(2022) ]	10.56 $\pm$ 9.24	10.30 $\pm$ 8.05	9.63 $\pm$ 7.90	10.16 $\pm$ 8.40	3.63 $\pm$ 3.14	5.35	1.84
	CrossGaze [ Catruna et al.(2024)Catruna, Cosma, and Radoi ]	10.47 $\pm$ 8.45	10.29 $\pm$ 7.79	9.23 $\pm$ 7.31	10.00 $\pm$ 7.85	3.30 $\pm$ 2.68	14.31	1.43
MF	Gaze360 [ Kellnhofer et al.(2019)Kellnhofer, Recasens, Stent, Matusik, and Torralba ]	10.46 $\pm$ 8.92	10.18 $\pm$ 7.69	9.32 $\pm$ 7.58	9.99 $\pm$ 8.06	2.73 $\pm$ 1.77	2.83	12.79
	STAGE [ Jindal et al.(2024)Jindal, Yadav, and Manduchi ]	12.08 $\pm$ 10.82	11.63 $\pm$ 8.53	9.43 $\pm$ 6.72	11.05 $\pm$ 8.69	2.58 $\pm$ 2.11	10.26	39.55

Table 2: Quantitative comparison of 3D gaze estimation on the (a) Gaze360 and (b) EVE datasets, together with (c) the model complexity. We report the Mean Angular Error (MAE, °; lower is better), where the Mean column is the average over the All, Semi-Front, and Front subsets. The (c) Complexity columns report the inference time (in milliseconds) and the computational cost (GFLOPs). SF and MF denote single-frame and multi-frame methods. The best and second-best results in each column are marked in bold and underline.

Type	Method	(a) Fixation Jitter	(b) Saccade Bias	(c) Absolute Mean
SF	L2CS-Net [ Abdelrahman et al.(2023)Abdelrahman, Hempel, Khalifa, Al-Hamadi, and Dinges ]	5.60 $\pm$ 0.16	0.02 $\pm$ 0.17	2.81 $\pm$ 0.17
	GazeTR-Hybrid [ Cheng and Lu(2022) ]	5.48 $\pm$ 0.16	0.84 $\pm$ 0.18	3.16 $\pm$ 0.17
	CrossGaze [ Catruna et al.(2024)Catruna, Cosma, and Radoi ]	5.06 $\pm$ 0.14	0.23 $\pm$ 0.15	2.65 $\pm$ 0.15
MF	Gaze360 [ Kellnhofer et al.(2019)Kellnhofer, Recasens, Stent, Matusik, and Torralba ]	1.71 $\pm$ 0.10	-4.24 $\pm$ 0.17	2.98 $\pm$ 0.14
	STAGE [ Jindal et al.(2024)Jindal, Yadav, and Manduchi ]	3.81 $\pm$ 0.11	-0.89 $\pm$ 0.26	2.35 $\pm$ 0.19
	EyeTAG (Visual-only)	3.51 $\pm$ 0.12	-0.46 $\pm$ 0.16	1.99 $\pm$ 0.14

Table 3: Temporal behavior analysis (all values in °/frame). (a) Fixation Jitter

J

: mean predicted velocity under near-static gaze (

v^{\mathrm{GT}}<1^{\circ}

/frame,

N{=}1{,}003

frame pairs); lower is better. (b) Saccade Bias

B

: mean signed velocity difference

v^{\mathrm{pred}}-v^{\mathrm{GT}}

under rapid gaze transitions (

v^{\mathrm{GT}}>5^{\circ}

/frame,

N{=}3{,}306

frame pairs); closer to zero is better. (c) Absolute Mean

\tfrac{1}{2}(J{+}|B|)

; lower is better. SF and MF denote single-frame and multi-frame methods, respectively. The EyeTAG variants keep the encoder and remaining architecture fixed and differ only in the gaze-history component. The best and second-best results in each column are marked in bold and underline, respectively.

Figure 4: Per-frame Mean Angular Error (MAE, °) on Gaze360 for the (a) All, (b) Semi-Front, and (c) Front head-pose subsets (lower is better).

	Fusion Strategy	All	Semi-Front	Front	Mean
(a)	Self-Attn + Concat (face $\parallel$ eye)	9.80 $\pm$ 8.14	9.60 $\pm$ 7.45	8.71 $\pm$ 7.43	9.37 $\pm$ 7.67
(b)	Bi. Cross-Attn (face $\leftrightarrow$ eye)	9.66 $\pm$ 8.16	9.49 $\pm$ 7.55	8.62 $\pm$ 7.49	9.26 $\pm$ 7.73
(Ours)	Uni. Cross-Attn (face $\rightarrow$ eye)	9.29 $\pm$ 7.78	9.12 $\pm$ 7.23	8.08 $\pm$ 7.23	8.83 $\pm$ 7.41

Table 4: Effect of fusion strategy. MAE (°); lower is better. The best and second-best results in each column are marked in bold and underline, respectively.

	Gaze History	All	Semi-Front	Front	Mean
(a)	No Gaze History (visual-only)	9.72 $\pm$ 8.15	9.55 $\pm$ 7.76	8.30 $\pm$ 7.53	9.19 $\pm$ 7.81
(b)	Normal Gaze Prior ( $w/~\hat{G}$ )	9.63 $\pm$ 7.94	9.46 $\pm$ 7.38	8.22 $\pm$ 7.07	9.10 $\pm$ 7.46
(Ours)	Kinematic Gaze Prior ( $w/~\Delta\hat{G}$ )	9.29 $\pm$ 7.78	9.12 $\pm$ 7.23	8.08 $\pm$ 7.23	8.83 $\pm$ 7.41

Table 5: Ablation study on the gaze history representation. MAE (°); lower is better. The proposed kinematic prior

\Delta\hat{\mathbf{G}}

achieves the lowest error across all subsets. The best and second-best results in each column are marked in bold and underline, respectively.

Figure 5: Distribution of normal versus differential gaze yaw on Gaze360. The $y$ -axis is on a log scale.

$T$	4	8	16	24	32	48 ∗	64
Mean	9.87 $\pm$ 7.57	9.49 $\pm$ 7.36	9.47 $\pm$ 7.41	9.35 $\pm$ 7.56	8.91 $\pm$ 7.39	8.83 $\pm$ 7.41	8.82 $\pm$ 7.40
Time (ms)	10.78	11.12	11.38	11.41	13.91	19.97	25.23
GFLOPs	10.13	20.26	40.53	60.79	81.05	121.57	162.09

Table 6: Effect of temporal window size

T

. For each

T

we report the Mean Angular Error (MAE, °), the average inference time (Time, in milliseconds) per input window on a single NVIDIA RTX A6000 (FP32, batch size 1), and the computational cost (GFLOPs); ∗ denotes the adopted setting. Accuracy improves up to

T=48

and then saturates, while the inference time and computational cost keep increasing. The best and second-best Mean values are marked in bold and underline, respectively.

Figure S1: Gaze360 evaluation subsets on the unit sphere: Front (green, $|\mathrm{yaw}|\leq 20^{\circ}$ ), Semi-Front (blue, $|\mathrm{yaw}|\leq 90^{\circ}$ ), and All (red, $|\mathrm{yaw}|\leq 180^{\circ}$ ).

	Pretraining	All	Semi-Front	Front	Mean
(a)	From scratch	20.41 $\pm$ 16.39	18.03 $\pm$ 15.54	16.05 $\pm$ 14.50	18.16 $\pm$ 15.47
(b)	ImageNet [ Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei ]	11.69 $\pm$ 10.22	11.34 $\pm$ 8.64	10.15 $\pm$ 8.50	11.06 $\pm$ 9.12
( Ours )	VGGFace2 [ Cao et al.(2018)Cao, Shen, Xie, Parkhi, and Zisserman ]	9.29 $\pm$ 7.78	9.12 $\pm$ 7.23	8.08 $\pm$ 7.23	8.83 $\pm$ 7.41

Table S1: Effect of the face encoder

\mathcal{F}

pretraining strategy on Gaze360 [ Kellnhofer et al.(2019)Kellnhofer, Recasens, Stent, Matusik, and Torralba ] . The face encoder architecture (ResNet-50) and the eye encoder (ResNet-18) are held fixed. Only the pretraining of the face encoder is varied across three settings: from scratch, generic ImageNet pretraining, and face-domain VGGFace2 pretraining. MAE (°); lower is better. The best and second-best results in each column are marked in bold and underline, respectively.

Eye Backbone	Type	All	Semi-Front	Front	Mean
EfficientNet-B0 [ Tan and Le(2019) ]	2D, no temporal	10.77 $\pm$ 10.80	10.37 $\pm$ 8.83	8.26 $\pm$ 6.56	9.80 $\pm$ 8.73
TSM [ Lin et al.(2019)Lin, Gan, and Han ]	2D + temporal	9.47 $\pm$ 7.89	9.30 $\pm$ 7.30	8.31 $\pm$ 7.35	9.03 $\pm$ 7.51
3D CNN [ Tran et al.(2015)Tran, Bourdev, Fergus, Torresani, and Paluri ] (scratch)	3D, temporal	9.41 $\pm$ 7.90	9.22 $\pm$ 7.27	8.17 $\pm$ 7.06	8.93 $\pm$ 7.41
( Ours ) ResNet-18 (shared)	2D, no temporal	9.29 $\pm$ 7.78	9.12 $\pm$ 7.23	8.08 $\pm$ 7.23	8.83 $\pm$ 7.41

Table S2: Effect of the eye-encoder backbone on Gaze360 [ Kellnhofer et al.(2019)Kellnhofer, Recasens, Stent, Matusik, and Torralba ] . MAE (°); lower is better. The best and second-best results in each column are marked in bold and underline, respectively.

Input:

\mathbf{X}_{t,T}=\{\mathbf{x}_{t-T+1},~\cdots,~\mathbf{x}_{t}\}

, window size

T

Output:

\hat{\mathbf{g}}_{t}

— Training (per epoch) —

1. for each frame

t

\mathbf{e}_{t}^{\mathrm{face}},~\mathbf{e}_{t}^{\mathrm{eye}}\leftarrow\mathrm{DSVE}(\mathbf{X}_{t,T})

b. Construct gaze history:

sample

\hat{\mathbf{G}}_{t-1,T-1}

from

\mathbf{G}^{\mathrm{GT}}_{t-1,T-1}

\mathcal{C}

with ratio

\alpha

\triangleright

\alpha:1.0\rightarrow 0.0

over ep. 1–5

Algorithm S1 Training and inference procedure of EyeTAG.

	Gaze History Schedule	All	Semi-Front	Front	Mean
(a)	Teacher forcing only ( $\alpha=1$ )	39.53 $\pm$ 32.45	38.98 $\pm$ 31.81	31.64 $\pm$ 22.37	36.71 $\pm$ 28.87
(b)	Fully autoregressive ( $\alpha=0$ )	10.13 $\pm$ 7.88	9.96 $\pm$ 7.93	8.81 $\pm$ 7.54	9.63 $\pm$ 7.78
( Ours )	Scheduled sampling ( $\alpha:1\!\rightarrow\!0$ )	9.63 $\pm$ 7.94	9.46 $\pm$ 7.38	8.22 $\pm$ 7.07	9.10 $\pm$ 7.46

Table S3: Effect of the scheduled sampling strategy used to construct the gaze history during training, evaluated on Gaze360 [ Kellnhofer et al.(2019)Kellnhofer, Recasens, Stent, Matusik, and Torralba ] with the normal gaze prior ( w /

\hat{\mathbf{G}}

) at

T{=}32

. MAE (°); lower is better. The best and second-best results in each column are marked in bold and underline, respectively.

$T$	All	Semi-Front	Front	Mean	Time (ms)	GFLOPs
4	10.96 $\pm$ 8.28	10.80 $\pm$ 7.74	9.58 $\pm$ 7.42	10.45 $\pm$ 7.81	9.87	10.13
8	10.17 $\pm$ 7.81	10.01 $\pm$ 7.28	9.23 $\pm$ 7.06	9.80 $\pm$ 7.38	9.97	20.26
16	10.26 $\pm$ 7.98	10.08 $\pm$ 7.42	9.15 $\pm$ 7.29	9.83 $\pm$ 7.56	10.23	40.53
24	9.96 $\pm$ 7.97	9.80 $\pm$ 7.36	8.65 $\pm$ 7.36	9.47 $\pm$ 7.56	11.45	60.79
32 ∗	9.63 $\pm$ 7.94	9.46 $\pm$ 7.38	8.22 $\pm$ 7.07	9.10 $\pm$ 7.46	13.95	81.05
48	9.94 $\pm$ 7.94	9.65 $\pm$ 7.47	8.61 $\pm$ 7.65	9.40 $\pm$ 7.69	19.98	121.57

Table S4: Effect of temporal window length

T

on Gaze360 [ Kellnhofer et al.(2019)Kellnhofer, Recasens, Stent, Matusik, and Torralba ] under the normal gaze prior (

\textbf{w/}~\hat{\mathbf{G}}

). MAE (°); lower is better. The best and second-best results in each column are marked in bold and underline, respectively.

Type	Method	All	Semi-Front	Front	Mean
SF	L2CS-Net [ Abdelrahman et al.(2023)Abdelrahman, Hempel, Khalifa, Al-Hamadi, and Dinges ]	10.07 $\pm$ 7.66	9.85 $\pm$ 6.40	9.90 $\pm$ 6.37	9.94 $\pm$ 6.81
	GazeTR-Hybrid [ Cheng and Lu(2022) ]	9.42 $\pm$ 7.27	9.26 $\pm$ 6.50	9.46 $\pm$ 6.31	9.38 $\pm$ 6.69
	CrossGaze [ Catruna et al.(2024)Catruna, Cosma, and Radoi ]	9.22 $\pm$ 6.84	9.09 $\pm$ 6.05	8.55 $\pm$ 5.90	8.95 $\pm$ 6.26
MF	Gaze360 [ Kellnhofer et al.(2019)Kellnhofer, Recasens, Stent, Matusik, and Torralba ]	9.70 $\pm$ 7.02	9.47 $\pm$ 6.00	9.41 $\pm$ 5.40	9.53 $\pm$ 6.14
	STAGE [ Jindal et al.(2024)Jindal, Yadav, and Manduchi ]	8.87 $\pm$ 5.84	8.87 $\pm$ 5.84	8.69 $\pm$ 5.77	8.81 $\pm$ 5.82
	EyeTAG ( $\textbf{w/}~\hat{\mathbf{G}}$ )	8.80 $\pm$ 6.41	8.68 $\pm$ 6.08	7.95 $\pm$ 6.22	8.48 $\pm$ 6.23

Table S5: Robustness to an incomplete gaze history on Gaze360 [ Kellnhofer et al.(2019)Kellnhofer, Recasens, Stent, Matusik, and Torralba ] . We evaluate only the early frames of each test clip (

0\leq t<T

T=48

), where the autoregressive gaze history is only partially available and is zero-padded. MAE (°); lower is better. EyeTAG (

\textbf{w/}~\hat{\mathbf{G}}

) uses the normal gaze prior

\hat{\mathbf{G}}

, and EyeTAG (Ours) uses the kinematic prior

\Delta\hat{\mathbf{G}}

. The best and second-best results in each column are marked in bold and underline, respectively.

Explore similar work

PAGE: Towards Practical Human-level Gaze Target Estimation
Jul 6, 2026Zhoutong Ye, Chengwen Zhang, Zhaibin Cui +10Gaze Target Estimation Scene Reasoning
1Tsinghua University · 2Jinan University
GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformer
May 1, 2026Xinyuan Zhao, Yihang Wu, Ahmad Chaddad +2Gaze Target Estimation Detection Transformer
School of Artificial Intelligence, Guilin University of Electronic Technology, Guilin, China · The Laboratory for Imagery, Vision and Artificial Intelligence, École de Technologie Supérieure, Montreal, Canada · Information Technology Department, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia +2
PicoEyes: Unified Gaze Estimation Framework for Mixed Reality with a Large-Scale Multi-View Dataset
May 8, 2026Fuxin Duan, Hui WangGaze Target Estimation Camera Intrinsics
Pico, Bytedance

Jul 6, 2026, cs.CVJ/K move · Enter open · S save

PAGE: Towards Practical Human-level Gaze Target Estimation

Zhoutong Ye, Chengwen Zhang, Zhaibin Cui, Mingze Sun+9

1Tsinghua University · 2Jinan University

Gaze Target Estimation Scene Reasoning

Open paper AskPDF arXiv

Gaze target estimation, the task of predicting where a person is looking in a scene, is crucial to understanding human attention and intent. It is a challenging task that combines high-level understanding of global scene semantics and precise spatial reasoning using human appearance (e.g. pose, eye orientation). As a result, human-level performance remains elusive for existing models, limiting their practical application. To this end, we propose PaGE (Practical Gaze Estimator), a gaze estimation model that explicitly models the complex interaction between scene and head features. Using a PaGE model with a large ViT-H+ backbone as the teacher, we further distill student models with lighter backbones on a much larger and more diverse unlabeled dataset. The architectural improvements and novel training recipe allow PaGE to achieve state-of-the-art performance on several gaze estimation tasks, outperforming humans in 7 out of 9 metrics while reducing the human-AI gap by at least 60% in the remaining 2. The distilled student models retain most of the teacher's performance while being lightweight enough for practical deployment on robots and consumer devices. The code and model checkpoints are available at our project page.

Abstract

Explore similar work

PAGE: Towards Practical Human-level Gaze Target Estimation

GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformer

PicoEyes: Unified Gaze Estimation Framework for Mixed Reality with a Large-Scale Multi-View Dataset