GazeLens for Efficient Egocentric Video Question Answering with Simple Gaze-Driven Adaptive Token Reduction

Trong-Thang Pham; Anh Nguyen; Hien Van Nguyen; Ngan Le

cs.CVOct 20, 2026

GazeLens for Efficient Egocentric Video Question Answering with Simple Gaze-Driven Adaptive Token Reduction

Authors: Trong-Thang Pham, Anh Nguyen, Hien Van Nguyen, Ngan Le

Abstract

Smart glasses can help users understand their activities and surroundings, but small vision-language models must preserve useful evidence within limited computation. This is difficult because redundant frames consume the visual budget, while aggressive spatial compression can discard relevant details. Although gaze identifies the wearer's focus, the question may require evidence from the surrounding scene. To address these constraints, we propose GazeLens, which combines frame selection with gaze- and question-guided token reduction. First, HOG transitions and image sharpness select frames before visual encoding. For each selected frame, Gaussian pooling then forms gaze-focused and complementary summaries. Because their relevance depends on the question, a lightweight gate blends them into a primary token. Together with a global scene token, this retains both focused and contextual summaries within two tokens per frame. On StreamGaze, GazeLens achieves higher macro and micro accuracy than the evaluated full-grid and token-reduction baselines, with lower input-forward FLOPs than the shared-backbone token-reduction baselines.

Figures & tables

Figure 1: Teaser of our gaze-driven adaptive token reduction framework for egocentric video QA. In the left panel, egocentric video and gaze data are inherently noisy and redundant, with dense frames and unstable gaze signals. In the middle panel, existing VLM approaches suffer from inefficient frame selection (too dense or too sparse) and often ignore gaze or rely on computationally expensive adaptive attention mechanisms. In the right panel, our approach combines (1) Dynamic Frame Selection , (2) Focus-Aware Gaze Aggregation , and (3) Question-Conditioned Semantic Gating . Together, these modules produce two visual tokens per frame. This design reduces the visual token budget and model computation while improving performance under strict token budgets.

Method	Adaptive frames	Gaze guidance	Peripheral context	Question guidance
GazeLLM ( Rekimoto, 2025 )	$\times$	$\checkmark$	$\times$	$\times$
LLaMA-VID ( Li et al., 2024 )	$\times$	$\times$	$\checkmark$	$\checkmark$
Voila-A ( Yan et al., 2024 )	$\times$	$\checkmark$	$\checkmark$	$\times$
GazeVLM ( Yoon et al., 2025 )	$\times$	$\checkmark$	$\checkmark$	$\times$
GazeLens (Ours)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Coverage of the design requirements. Adaptive frames denotes content-dependent selection before learned visual encoding. The other columns concern spatial compression. A check marks explicit support within the reduction mechanism, including whether compression is conditioned on the question.

Figure 2: Overview of the proposed focus-aware, question-conditioned token reduction framework. (1) Dynamic Frame Selection. A continuous video stream is filtered using HOG transition and sharpness cues to retain informative frames while discarding small-change and blurry frames. This filtering process produces a dynamic input sequence $\mathcal{V}$ . (2) Focus-Aware Gaze Aggregation. Each selected frame is encoded into spatial features and combined with gaze $(x_{t},y_{t})$ to produce a focus token $\mathbf{g}_{t}$ , an unfocus token $\mathbf{c}_{t}$ , and a global token $\mathbf{m}_{t}$ . (3) Question-Conditioned Semantic Gating. Question token embeddings are mean-pooled and L2-normalized to $\mathbf{t}$ . A two-layer MLP and sigmoid then yield a scalar gate $s\in[0,1]$ . The gate modulates gaze-focused versus contextual information as $\mathbf{p}_{t}=s\cdot\mathbf{g}_{t}+(1-s)\cdot\mathbf{c}_{t}$ . (4) Token Assembly and Training Objective. The per-frame tokens $\mathbf{p}_{t}$ and $\mathbf{m}_{t}$ are linearly projected, concatenated with the question, and fed into a language model to predict the answer, optimized with a cross-entropy (CE) loss.

Method	Params (M) $\downarrow$	TFLOPs $\downarrow$	Fr.	Past $\uparrow$	Present $\uparrow$	Proactive $\uparrow$	Macro $\uparrow$	Micro $\uparrow$
Below 300M parameters
SVLM-256M	256.48	3.966	16	0.328	0.283	0.537	0.383	0.480
Ours-256M	258.24	2.857	Adap.	0.351	0.532	0.643	0.509	0.617
300M+ parameters
SVLM-500M	507.48	4.742	16	0.350	0.449	0.578	0.459	0.582
GazeLLM	508.60	3.791	16	0.309	0.233	0.636	0.393	0.502

Table 2: StreamGaze accuracy and model cost, grouped by total parameter count. Params counts unique parameters, including vision encoders and added modules. TFLOPs are per input forward (Appendix E.2 ). Macro equally averages Past, Present, and Proactive, while Micro pools test examples. Both Ours variants report three-seed means. Across-seed variability is reported separately in Table B.1 . Bold/italic mark best/second-best accuracy. Adap. denotes dynamic selection, SVLM denotes SmolVLM2, and Mob-VGPT denotes Mobile-VideoGPT.

Panel of Figure 3 — Figure 3: Accuracy versus input-forward computation at each model’s native image resolution. Muted colors identify baselines, while green and red highlight Ours-256M and Ours-500M. Linear marker size scales with the square root of total parameters. Higher and further left is better. The horizontal axis is logarithmic. FLOPs exclude frame selection, preprocessing, and answer generation (Appendix E.2 ).

Panel of Figure 4 — Figure 4: EgoGazeVQA transfer on 1,265 held-out examples. $T$ denotes uniform sampling, whereas ours is dynamic. Higher and further left is better.

Figure 5: Adaptive versus uniform frame selection for example video sequences. Red boxes mark redundant or motion-blurred frames in the uniform selections. Frame IDs run from 1 to 8.

Table 8

Panel of Figure 6 — Figure 6: Gate distributions from one evaluated checkpoint. Boxes show quartiles and medians, while whiskers span min–max. Groups pool evaluation rows, including repeated question templates.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Evaluation set	Selector	Examples	Frames	Accuracy
Combined	Uniform T16	1265	15.72	56.84
Combined	Uniform T7	1265	7.00	58.10
Combined	Uniform T8	1265	8.00	58.02
Combined	GazeLens selector	1265	5.74	57.71
Ego4D	Uniform T16	577	15.90	51.30
Ego4D	Uniform T7	577	7.00	51.99

Appendix

Table 5: Frame-selection comparison on EgoGazeVQA. Each row reports a selector on the indicated evaluation set. Examples is the number of evaluation items, and Frames is the mean selected frames per item. Accuracy is in percent.

Noise $\sigma$	0.00	0.01	0.02	0.05	0.10	0.20
Macro	0.539 $\pm 0.015$	0.537	0.534	0.526	0.514	0.498
Micro	0.640 $\pm 0.005$	0.638	0.635	0.627	0.615	0.599

Appendix

Table 11: Model performance under synthetic spatial gaze noise at test time.

Egocentric Video QA

Egocentric video question answering uses first-person camera footage to answer grounded questions about activities, objects, space, and time.

See all 46 papers

EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input
Aug 16, 2026Matteo Stoiber, Niels Buus LassenGaze Estimation Efficient Multimodal Inference
Copenhagen Business School, Department of Digitalization, Copenhagen, Denmark
GazeQwen: Lightweight Gaze-Conditioned LLM Modulation for Streaming Video Understanding
Mar 26, 2026Trong Thang Pham, Hien Nguyen, Ngan LeMultimodal Large Language Models Streaming Video Understanding
University of Arkansas · University of Houston
GleanVID: Complementary Token Selection for Efficient Video Large Language Models
Sep 29, 2026Shuo Yang, Changbai Li, Rui Tang +3Efficient VLM Inference Video-Language Models
Beihang University · Communication University of China

Aug 16, 2026J/K move · Enter open · S save

EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input

Matteo Stoiber, Niels Buus Lassen

Copenhagen Business School, Department of Digitalization, Copenhagen, Denmark

Gaze Estimation Efficient Multimodal Inference Egocentric Video Understanding Edge ML

Open paper AskPDF arXiv

The use of multimodal LLMs (MLLMs) for egocentric video understanding with wearable devices is constrained by the token budget. Memory and compute cost scale with the number of visual tokens, and high-resolution video quickly becomes expensive to transmit and process at scale. Prior work (GazeLLM) addresses this by cropping the video around the camera wearer's gaze. This reduces the number of visual tokens by about tenfold while maintaining or improving the quality of full-resolution descriptions. However, this compression strategy depends on dedicated eye-tracking hardware, which is unavailable on consumer smart glasses. Building a software-only substitute poses a joint constraint: the predictor must be accurate enough to preserve downstream description quality, yet light enough to run on-device, within the power and compute budget of a smartphone. We address this with EgoGazeLite, a lightweight dual-process gaze predictor for egocentric video. Across two MLLMs, three automated metrics, and two LLM judges, predicted-gaze crops show no significant difference from ground-truth-gaze crops. Equivalence is confirmed in all ten cases. EgoGazeLite achieves this at 15.7M parameters, 6.71 GFLOPs, and runs the full gaze-and-crop pipeline end-to-end in real time (21.6 ms/frame) on consumer accelerator hardware. Together, these results remove the need for eye-tracking hardware for token-efficient, gaze-conditioned egocentric video understanding with MLLMs.

Abstract

Egocentric Video QA

Explore similar work

EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input

GazeQwen: Lightweight Gaze-Conditioned LLM Modulation for Streaming Video Understanding

GleanVID: Complementary Token Selection for Efficient Video Large Language Models