EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input

Matteo Stoiber; Niels Buus Lassen

cs.CVAug 16, 2026

EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input

Authors: Matteo Stoiber, Niels Buus Lassen

Organizations: Copenhagen Business School, Department of Digitalization, Copenhagen, Denmark

Abstract

The use of multimodal LLMs (MLLMs) for egocentric video understanding with wearable devices is constrained by the token budget. Memory and compute cost scale with the number of visual tokens, and high-resolution video quickly becomes expensive to transmit and process at scale. Prior work (GazeLLM) addresses this by cropping the video around the camera wearer's gaze. This reduces the number of visual tokens by about tenfold while maintaining or improving the quality of full-resolution descriptions. However, this compression strategy depends on dedicated eye-tracking hardware, which is unavailable on consumer smart glasses. Building a software-only substitute poses a joint constraint: the predictor must be accurate enough to preserve downstream description quality, yet light enough to run on-device, within the power and compute budget of a smartphone. We address this with EgoGazeLite, a lightweight dual-process gaze predictor for egocentric video. Across two MLLMs, three automated metrics, and two LLM judges, predicted-gaze crops show no significant difference from ground-truth-gaze crops. Equivalence is confirmed in all ten cases. EgoGazeLite achieves this at 15.7M parameters, 6.71 GFLOPs, and runs the full gaze-and-crop pipeline end-to-end in real time (21.6 ms/frame) on consumer accelerator hardware. Together, these results remove the need for eye-tracking hardware for token-efficient, gaze-conditioned egocentric video understanding with MLLMs.

Figures & tables

Panel of Figure 1 — Figure 1: EgoGazeLite removes GazeLLM’s eye-tracking dependency, enabling gaze-guided video cropping on edge devices while reducing MLLM input by 90%.

Figure 2: EgoGazeLite architecture. Current and previous frames pass through a shared EfficientNet-Lite4 backbone. The saliency prediction module computes a feature-level temporal difference and decodes it into a bottom-up saliency heatmap $G_{t}^{s}$ . The attention transition module extracts channel weights at the previous gaze location, propagates them through a gated LSTM (gated by an online I-DT fixation/saccade decision), and applies them to the current frame’s features to produce a top-down attention heatmap $G_{t}^{a}$ . A residual fusion module combines both heatmaps into the final prediction heatmap $G_{t}$ , from which the gaze coordinate ( $g_{t}$ ) is extracted via argmax.

Figure 3: Example crop conditions for a single frame (Sushi domain). Center, ground-truth, and predicted crop windows are shown on the full frame (left) and as extracted 448 $\times$ 448 crops (right). In this frame, the predicted and ground-truth crops overlap substantially (pred $\leftrightarrow$ GT distance: 94 px).

Domain	AUC	AAE (°)	Pixel distance
Cooking	0.9705	7.58	24.46
Soccer	0.9702	7.67	23.68
Health	0.9684	7.73	25.51
Bike Repair	0.9668	8.03	25.76
Dance	0.9587	9.34	28.97
Bouldering	0.9566	9.64	30.16

Table 1: In-distribution gaze prediction accuracy of the deployed EgoGazeLite checkpoint (trained on Ego-Exo4D Split A, Section 3 ) across seven training domains. Average Angular Error (AAE) uses the argmax convention (Section 3 ).

Figure 4: Predicted (orange) versus ground-truth (blue) gaze locations across four Ego-Exo4D domains, overlaid on EgoGazeLite’s predicted heatmap. Predicted and ground-truth points align closely in Cooking, Soccer, and Bike Repair; Health shows a larger deviation, illustrating a case where prediction is less precise.

Model	Parameters	FLOPs
EgoGazeLite	15.7M	6.71 G
Huang et al. [ 10 ]	51.0M	57.76 G
Lai et al. [ 13 ]	70.2M	94.22 G

Table 2: Model capacity comparison. EgoGazeLite uses substantially fewer parameters and FLOPs than prior egocentric gaze predictors.

MLLM	Condition	BLEU	ROUGE-L	SBERT	Claude	GPT-4o
Gemini 2.5 Flash	Center	$0.213{\scriptstyle\pm 0.009}$	$0.334{\scriptstyle\pm 0.008}$	$0.868{\scriptstyle\pm 0.010}$	$54.25{\scriptstyle\pm 2.24}$	$65.63{\scriptstyle\pm 1.96}$
	Ground-truth	$\mathbf{0.248}{\scriptstyle\pm 0.010}$	$\mathbf{0.376}{\scriptstyle\pm 0.009}$	$\mathbf{0.912}{\scriptstyle\pm 0.005}$	$\mathbf{68.04}{\scriptstyle\pm 2.14}$	$75.19{\scriptstyle\pm 1.71}$
	Predicted	$0.234{\scriptstyle\pm 0.009}$	$0.361{\scriptstyle\pm 0.009}$	$0.904{\scriptstyle\pm 0.005}$	$64.28{\scriptstyle\pm 2.16}$	$\mathbf{75.52}{\scriptstyle\pm 1.67}$
Gemini 2.5 Pro	Center	$0.244{\scriptstyle\pm 0.007}$	$0.398{\scriptstyle\pm 0.007}$	$0.876{\scriptstyle\pm 0.010}$	$53.79{\scriptstyle\pm 2.24}$	$68.04{\scriptstyle\pm 2.28}$
	Ground-truth	$\mathbf{0.300}{\scriptstyle\pm 0.008}$	$0.457{\scriptstyle\pm 0.008}$	$\mathbf{0.911}{\scriptstyle\pm 0.007}$	$68.15{\scriptstyle\pm 2.32}$	$79.42{\scriptstyle\pm 1.94}$
	Predicted	$\mathbf{0.300}{\scriptstyle\pm 0.008}$	$\mathbf{0.459}{\scriptstyle\pm 0.008}$	$0.905{\scriptstyle\pm 0.007}$	$\mathbf{71.42}{\scriptstyle\pm 2.12}$	$\mathbf{80.54}{\scriptstyle\pm 1.89}$

Table 3: Description quality by crop condition, MLLM, and metric (mean

\pm

SE). Automated metrics are on a 0–1 scale; LLM judges on a 0–100 scale. Bold indicates the best-performing condition per column.

Figure 5: Per-task description quality by crop condition, across the two MLLMs (rows: Gemini 2.5 Flash, Gemini 2.5 Pro) and two LLM judges (columns: Claude Sonnet 4.6, GPT-4o). Predicted-gaze (orange) and ground-truth-gaze (blue) crops track each other on every task, while center crops (green) score consistently lower. The rightmost ’ALL’ bar in each panel is the mean across all tasks. Error bars are $\pm 1$ SE. Soccer shows the smallest gap between gaze-conditioned and center crops.

MLLM	Metric	Mean diff	$d_{z}$	$p_{\mathrm{Holm}}$	TOST $p_{\mathrm{Holm}}$	Primary	Sensitivity
		Difference			Equivalence (TOST)
Flash	BLEU	$-0.014$	$-0.12$	$0.163$	$<0.001$	✓	$\times$
	ROUGE-L	$-0.016$	$-0.17$	$0.054$	$<0.001$	✓	$\times$
	SBERT	$-0.008$	$-0.15$	$0.085$	$<0.001$	✓	$\times$
	Claude	$-3.76$	$-0.15$	$0.076$	$<0.001$	✓	$\times$
	GPT-4o	$+0.34$	$+0.02$	$0.795$	$<0.001$	✓	✓

Table 4: Predicted vs. ground-truth gaze: difference and equivalence tests. Difference columns report the mean paired difference, Cohen’s

d_{z}

(mean paired difference divided by SD of paired differences), and the Holm-corrected paired-

t

-test

p

-value. The TOST

p_{\mathrm{Holm}}

column reports the Holm-corrected equivalence test result; standardized effect sizes and 90% confidence intervals for these tests are shown in Figure 6 . Margins are set at

0.5\times

(primary) and

0.2\times

(sensitivity) the pooled SD of the ground-truth and predicted score distributions per metric, following Cohen’s medium/small effect-size convention [ 4 ] .

Figure 6: Equivalence testing (TOST) of predicted-vs-ground-truth gaze crops, Holm-corrected across all ten cells. Points show the standardized mean paired difference (Cohen’s $d_{z}$ ), bars show 90% confidence intervals. The shaded band is an approximate $d_{z}$ -scale mapping of the TOST margins ( $0.5\times$ the pooled SD of the ground-truth and predicted score distributions per metric, Table 4 ).

Explore similar work

Mar 26, 2026

GazeQwen: Lightweight Gaze-Conditioned LLM Modulation for Streaming Video Understanding

Current multimodal large language models (MLLMs) cannot effectively utilize eye-gaze information for video understanding, even when gaze cues are supplied via visual overlays or text descriptions. We introduce GazeQwen, a parameter efficient approach that equips an open-source MLLM with gaze awareness through hidden-state modulation. At its core is a compact gaze resampler (~1-5 M trainable parameters) that encodes V-JEPA 2.1 video features together with fixation-derived positional encodings and produces additive residuals injected into selected LLM decoder layers via forward hooks. An optional second training stage adds low-rank adapters (LoRA) to the LLM for tighter integration. Evaluated on all 10 tasks of the StreamGaze benchmark, GazeQwen reaches 63.9% accuracy, a +16.1 point gain over the same Qwen2.5-VL-7B backbone with gaze as visual prompts and +10.5 points over GPT-4o, the highest score among all open-source and proprietary models tested. These results suggest that learning where to inject gaze within an LLM is more effective than scaling model size or engineering better prompts. All code and checkpoints are available at https://github.com/phamtrongthang123/gazeqwen .

Trong Thang Pham, Hien Nguyen, Ngan Le

University of Arkansas · University of Houston

Multimodal Large Language Models Streaming Video Understanding Parameter-Efficient Fine-Tuning

Ask PDF

Mar 12, 2026

Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing

Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: https://autogaze.github.io/.

Baifeng Shi, Stephanie Fu, Long Lian +10

UC Berkeley · NVIDIA · Clarifai +1

Efficient VLM Inference Visual Token Pruning Temporal Video Understanding Video QA

Ask PDF

Apr 2, 2026

Ego-Grounding for Personalized Question-Answering in Egocentric Videos

We present the first systematic analysis of multimodal large language models (MLLMs) in personalized question-answering requiring ego-grounding - the ability to understand the camera-wearer in egocentric videos. To this end, we introduce MyEgo, the first egocentric VideoQA dataset designed to evaluate MLLMs' ability to understand, remember, and reason about the camera wearer. MyEgo comprises 541 long videos and 5K personalized questions asking about "my things", "my activities", and "my past". Benchmarking reveals that competitive MLLMs across variants, including open-source vs. proprietary, thinking vs. non-thinking, small vs. large scales all struggle on MyEgo. Top closed- and open-source models (e.g., GPT-5 and Qwen3-VL) achieve only~46% and 36% accuracy, trailing human performance by near 40% and 50% respectively. Surprisingly, neither explicit reasoning nor model scaling yield consistent improvements. Models improve when relevant evidence is explicitly provided, but gains drop over time, indicating limitations in tracking and remembering "me" and "my past". These findings collectively highlight the crucial role of ego-grounding and long-range memory in enabling personalized QA in egocentric videos. We hope MyEgo and our analyses catalyze further progress in these areas for egocentric personalized assistance. Data and code are available at https://github.com/Ryougetsu3606/MyEgo

Junbin Xiao, Shenglang Zhang, Pengxiang Zhu +1

University of Science and Technology of China · National University of Singapore

Egocentric Video QA Question Answering Multimodal Large Language Models Memory-Augmented Video Understanding Long-Video QA

Ask PDF