Real Eyes Realize Faster: Gaze Stability and Pupil Novelty for Efficient Egocentric Learning

Ajan Subramanian; Sumukh Bettadapura; Rohan Sathish

cs.CVMar 4, 2026

Real Eyes Realize Faster: Gaze Stability and Pupil Novelty for Efficient Egocentric Learning

Authors: Ajan Subramanian, Sumukh Bettadapura, Rohan Sathish

Organizations: Kubo Technologies

Abstract

Always-on egocentric cameras are increasingly used as demonstrations for embodied robotics, imitation learning, and assistive AR, but the resulting video streams are dominated by redundant and low-quality frames. Under the storage and battery constraints of wearable devices, choosing which frames to keep is as important as how to learn from them. We observe that modern eye-tracking headsets provide a continuous, training-free side channel that decomposes into two complementary axes: gaze fixation captures visual stability (quality), while pupil response captures arousal-linked moments (novelty). We operationalize this insight as a Dual-Criterion Frame Curator that first gates frames by gaze quality and then ranks the survivors by pupil-derived novelty. On the Visual Experience Dataset (VEDB), curated frames at 10% budget match the classification performance of the full stream, and naive signal fusion consistently destroys both contributions. The benefit is task-dependent: pupil ranking improves activity recognition, while gaze-only selection already dominates for scene recognition, confirming that the two signals serve genuinely different roles. Our method requires no model inference and operates at capture time, offering a path toward efficient, always-on egocentric data curation.

Figures & tables

Figure 1: Quality–Novelty Decomposition. (a) Gaze confidence (x) captures stability; pupil response (y) captures novelty. Random includes junk; gaze-only yields clean but redundant frames; dual targets high stability and novelty. (b) Two-stage pipeline: gaze gate (top 75%)

\rightarrow

pupil ranking within budget.

Strategy	Stage 1 (Gate)	Stage 2 (Rank)	Role
Random	—	Uniform random	Baseline
Gaze-only	—	Rank by $g(t)$	Quality baseline
Pupil-abs	—	Rank by $\|p(t)\|$	Novelty baseline
Combined	—	$0.5\,g+0.5\,\|p\|$	Naive fusion
Dual	Top 75% by $g(t)$	Rank by $\|p(t)\|$	Proposed
Gate+Random	Top 75% by $g(t)$	Uniform random	Ablation

Table 1: Frame selection strategies. Gate+Random is the ablation control that isolates the contribution of pupil ranking.

Figure 2: Correlation between physiological signals and DINOv2 feature change. (a) Pupil derivative

|dp/dt|

is positively correlated with feature change at all lags (mean

\rho=+0.038

). (b) Gaze quality

g(t)

is negatively correlated (

\rho=-0.037

), confirming it tracks stability. Error bars:

\pm 1

s.d. across sessions.

Figure 3: Learning curves: activity (left), scene (right). Dual at 10% budget matches the performance achieved using all frames. Shaded:

\pm 1

s.d. (10 seeds).

Strategy	AULC	95% CI	vs. Random	$p$
Dual	0.223	$[0.223,0.223]$	$+0.025$	$<0.001$
Gaze-only	0.218	$[0.218,0.218]$	$+0.020$	$<0.001$
Pupil-abs	0.215	$[0.215,0.215]$	$+0.018$	$<0.001$
Gate+Random †	0.205	$[0.196,0.215]$	$+0.007$	$0.210$
Naive fusion ‡	0.198	$[0.198,0.198]$	$+0.000$	$1.000$
Random	0.197	$[0.189,0.205]$	—	—

Table 2: AULC (mean macro

F1

across budgets 5%–100%) for activity recognition, 10 seeds. † Gate+Rand: gaze gate, random within pool. ‡ 3 seeds; deterministic ranking (zero variance).

Budget	Dual	Gate+Rand	$\Delta F1$	$p$	Cohen’s $d$
5%	0.205	$0.179\pm 0.031$	$+0.026$	$0.030$	0.86
10%	0.228	$0.188\pm 0.037$	$+0.040$	$0.009$	1.10
25%	0.220	$0.190\pm 0.020$	$+0.030$	$0.001$	1.54
50%	0.225	$0.215\pm 0.022$	$+0.010$	$0.230$	0.43
75%	0.228	$0.229\pm 0.001$	$-0.000$	$0.207$	—

Table 3: Ablation: dual vs. gate+random for activity recognition. Pupil ranking wins at 5–25% budgets.

Gate %	$b{=}10\%$	$b{=}25\%$	$b{=}50\%$	Mean
25%	0.176	0.229	0.229	0.211
50%	0.225	0.237	0.205	0.222
75%	0.228	0.220	0.225	0.224
90%	0.149	0.246	0.232	0.209
100% (no gate)	0.161	0.257	0.259	0.225

Table 4: Gate threshold ablation for activity recognition. 75% is optimal at 10% budget.

Task	Best Strategy	Dual vs. Random	$p$	Pupil Helps?
Activity (12-class)	Dual	$+0.025$ AULC	$<0.001$	Yes
Scene (16-class)	Gaze-only	$-0.012$ AULC	$1.000$	No

Table 5: Task-dependent pupil effect. Helps activity (

p<0.001

); not scene.

Figure 4: Task-Dependent Performance. (a) Activity: pupil ranking improves over gaze-only and random. (b) Scene: gaze-only dominates; pupil adds no benefit.

Strategy	AULC	95% CI	vs. Random	$p$
Gaze-only	0.280	$[0.280,0.280]$	$+0.015$	$<0.001$
Random	0.265	$[0.261,0.270]$	—	—
Gate+Random	0.262	$[0.258,0.266]$	$-0.003$	$1.000$
Naive fusion ‡	0.256	$[0.256,0.256]$	$-0.009$	$1.000$
Pupil-abs	0.255	$[0.255,0.255]$	$-0.011$	$1.000$
Dual	0.253	$[0.253,0.253]$	$-0.012$	$1.000$

Table 6: AULC (mean macro

F1

across budgets 5%–100%) for scene recognition, 10 seeds. ‡ 3 seeds; deterministic ranking (zero variance).

Task	Delayed Wins	No-Delay Wins	Interpretation
Activity	10 / 12	2 / 12	Delay captures transition arousal
Scene	4 / 12	8 / 12	Scene is frame-local

Table 7: Temporal alignment. Delayed wins for activity; no-delay for scene.

Figure 5: Qualitative Comparison. Top: dual-criterion (high gaze, high pupil). Bottom: random baseline, including blur and low-information content.

Task	Sessions	Best $F1$	Chance	Verdict
Phone vs. Computer	31	0.507	0.50	At chance
Cog. vs. Phys. Games	15	0.912	0.50	Visually separable

Table 8: Fine-grained classification. No benefit at representation floor (phone vs. computer) or for trivially separable categories.

Strategy	Linear	MLP	GBT
Dual (gaze75 + pupil)	0.223	0.220	0.133
Gaze-only	0.218	0.225	0.135
Pupil-abs	0.215	0.208	0.124
Gate + random	0.205	0.238	0.144
Naive fusion	0.198	0.219	0.132
Random	0.197	0.224	0.145

Table 9: AULC (mean macro

F1

across 6 budgets, 10 seeds) per strategy and classifier for activity recognition . Naive fusion ranks near the bottom across all three classifiers.

Strategy	Linear	MLP	GBT
Gaze-only	0.280	0.294	0.245
Random	0.265	0.282	0.276
Gate + random	0.262	0.280	0.270
Naive fusion	0.256	0.266	0.233
Pupil-abs	0.255	0.272	0.230
Dual (gaze75 + pupil)	0.253	0.265	0.240

Table 10: AULC (mean macro

F1

across 6 budgets, 10 seeds) per strategy and classifier for scene recognition . Gaze-only leads under Linear and MLP; rankings are more variable under GBT.

	Activity		Scene
Strategy	Default	Balanced	Default	Balanced
Dual (gaze75 + pupil)	0.223	0.219	0.253	0.247
Gaze-only	0.218	0.214	0.280	0.269
Pupil-abs	0.215	0.222	0.255	0.249
Gate + random	0.205	0.197	0.262	0.256
Naive fusion ‡	0.198	0.205	0.256	0.254

Table 11: AULC under default vs. balanced class weights (linear probe). Strategy rankings are preserved. ‡ 3 seeds.

Task	Strategy	Global $F1$	Stratified $F1$	$\Delta$
Activity	Dual (gaze75 + pupil)	0.228	0.248	$+0.020$
	Gaze-only	0.213	0.224	$+0.011$
	Random	0.184	0.193	$+0.008$
	Pupil-abs	0.161	0.188	$+0.027$
	Naive fusion ‡	0.164	0.198	$+0.034$
Scene	Dual (gaze75 + pupil)	0.270	0.268	$-0.001$

Table 12: Global vs. stratified at 10% budget. Stratified helps activity; mixed for scene. ‡ 3 seeds.

Gate %	5%	10%	25%	50%	75%	100%	Mean
25%	0.196	0.176	0.229	0.229	0.229	0.229	0.214
50%	0.132	0.225	0.237	0.205	0.205	0.205	0.201
75%	0.205	0.228	0.220	0.225	0.228	0.228	0.223
90%	0.167	0.149	0.246	0.232	0.185	0.216	0.199
100% (no gate)	0.173	0.161	0.257	0.259	0.218	0.224	0.215

Table 13: Gate threshold ablation (activity, 10 seeds, 6 budgets). 75% achieves best F1 at 10% budget.

Strategy	5%	10%	25%	50%	75%	100%	AULC
Dual (75% + pupil)	0.205	0.228	0.220	0.225	0.228	0.228	0.223
Gaze-only	0.207	0.213	0.230	0.204	0.227	0.225	0.218
Pupil-abs	0.173	0.161	0.257	0.259	0.218	0.224	0.215
Gate + random (75%)	0.179	0.188	0.190	0.215	0.229	0.229	0.205
Dual (50% + pupil)	0.132	0.225	0.237	0.205	0.205	0.205	0.201
Naive fusion ‡	0.177	0.164	0.190	0.208	0.223	0.225	0.198

Table 14: Per-budget macro

F1

and AULC for activity recognition , ranked by AULC. ‡ 3 seeds.

Strategy	5%	10%	25%	50%	75%	100%	AULC
Gaze-only	0.342	0.321	0.256	0.249	0.251	0.260	0.280
Random	0.294	0.270	0.257	0.256	0.256	0.259	0.265
Gate + random (75%)	0.283	0.274	0.257	0.256	0.251	0.251	0.262
Gate + random (50%)	0.289	0.271	0.252	0.249	0.249	0.249	0.260
Dual (50% + pupil)	0.259	0.277	0.271	0.249	0.249	0.249	0.259
Naive fusion ‡	0.261	0.269	0.238	0.252	0.259	0.260	0.256

Table 15: Per-budget macro

F1

and AULC for scene recognition , ranked by AULC. ‡ 3 seeds.

Statistic	Value
Total sessions	717
Sessions with eye tracking	136
Sessions with synchronized gaze + pupil	119
Unique observers	56
Age range	7–46 years
Total frames (1 fps)	154,819

Table 16: VEDB dataset statistics after quality filtering.

Video Frame Selection

Video frame selection chooses a compact set of frames that preserves important events, context, or task-specific evidence while reducing processing costs.

See all 65 papers

EyeTAG: Eye Trajectory-Aware Gaze Estimation
Oct 1, 2026Jungmin Lee, Niamat Ullah, Yoseob HanGaze Estimation
Department of Information and Telecommunication Engineering Soongsil University Seoul, Republic of Korea · Department of Electronic Engineering Soongsil University Seoul, Republic of Korea
EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras
Oct 2, 2026Kush Hari, Justin Kerr, Nidhya Shivakumar +7Active Vision for Robot Manipulation Active Vision
UC Berkeley · Amazon FAR · Stanford University
GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction
May 21, 2026Corentin Dumery, David Colmenares, Alexander Fix +3Gaze Estimation Synthetic Data Generation
Meta Reality Labs, Redmond, USA · EPFL, Lausanne, Switzerland · Work done during an internship at Meta

Oct 1, 2026J/K move · Enter open · S save

EyeTAG: Eye Trajectory-Aware Gaze Estimation

Jungmin Lee, Niamat Ullah, Yoseob Han

Department of Information and Telecommunication Engineering Soongsil University Seoul, Republic of Korea · Department of Electronic Engineering Soongsil University Seoul, Republic of Korea

Gaze Estimation

Open paper AskPDF arXiv

Gaze estimation under natural head-eye motion underpins applications from driver monitoring to human-computer interaction. Single-frame methods predict each frame independently, so consecutive outputs fluctuate as jitter. Multi-frame methods reduce this, but they learn motion implicitly inside appearance features, so the gaze trajectory is never an explicit variable. We propose EyeTAG (Eye Trajectory-Aware Gaze Estimation), a causal multi-frame framework built around an explicit first-order gaze prior: at each step it differentiates its own recent predictions and feeds the resulting trajectory back as a compact kinematic token. Because differencing is translation-invariant in gaze space, this token carries subject-invariant motion rather than personal gaze offsets. Face and eye streams supply visual evidence, fused by cross-attention and a causal Transformer decoder. EyeTAG reduces the mean angular error by about 1.0

^\circ

on Gaze360 and performs on par with the strongest baseline on EVE (2.56

^\circ

vs. 2.58

^\circ

). Within-model ablations, which keep the encoder and the rest of the architecture fixed and vary only the gaze history, show that the differential formulation, rather than temporal context alone, removes the systematic saccade bias that persists even with an absolute gaze-history prior. Our code is available at https://github.com/peter8366/EyeTAG.

Abstract

Video Frame Selection

Explore similar work

EyeTAG: Eye Trajectory-Aware Gaze Estimation

EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras

GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction