Smart glasses can help users understand their activities and surroundings, but small vision-language models must preserve useful evidence within limited computation. This is difficult because redundant frames consume the visual budget, while aggressive spatial compression can discard relevant details. Although gaze identifies the wearer's focus, the question may require evidence from the surrounding scene. To address these constraints, we propose GazeLens, which combines frame selection with gaze- and question-guided token reduction. First, HOG transitions and image sharpness select frames before visual encoding. For each selected frame, Gaussian pooling then forms gaze-focused and complementary summaries. Because their relevance depends on the question, a lightweight gate blends them into a primary token. Together with a global scene token, this retains both focused and contextual summaries within two tokens per frame. On StreamGaze, GazeLens achieves higher macro and micro accuracy than the evaluated full-grid and token-reduction baselines, with lower input-forward FLOPs than the shared-backbone token-reduction baselines.
Figures & tables
Figure 1: Teaser of our gaze-driven adaptive token reduction framework for egocentric video QA. In the left panel, egocentric video and gaze data are inherently noisy and redundant, with dense frames and unstable gaze signals. In the middle panel, existing VLM approaches suffer from inefficient frame selection (too dense or too sparse) and often ignore gaze or rely on computationally expensive adaptive attention mechanisms. In the right panel, our approach combines (1) Dynamic Frame Selection , (2) Focus-Aware Gaze Aggregation , and (3) Question-Conditioned Semantic Gating . Together, these modules produce two visual tokens per frame. This design reduces the visual token budget and model computation while improving performance under strict token budgets.
Method
Adaptive frames
Gaze guidance
Peripheral context
Question guidance
GazeLLM ( Rekimoto, 2025 )
×
✓
×
×
LLaMA-VID ( Li et al., 2024 )
×
×
✓
✓
Voila-A ( Yan et al., 2024 )
×
✓
✓
×
GazeVLM ( Yoon et al., 2025 )
×
✓
✓
×
GazeLens (Ours)
✓
✓
✓
✓
Table 1: Coverage of the design requirements. Adaptive frames denotes content-dependent selection before learned visual encoding. The other columns concern spatial compression. A check marks explicit support within the reduction mechanism, including whether compression is conditioned on the question.
Figure 2: Overview of the proposed focus-aware, question-conditioned token reduction framework. (1) Dynamic Frame Selection. A continuous video stream is filtered using HOG transition and sharpness cues to retain informative frames while discarding small-change and blurry frames. This filtering process produces a dynamic input sequence V . (2) Focus-Aware Gaze Aggregation. Each selected frame is encoded into spatial features and combined with gaze (xt,yt) to produce a focus token gt , an unfocus token ct , and a global token mt . (3) Question-Conditioned Semantic Gating. Question token embeddings are mean-pooled and L2-normalized to t . A two-layer MLP and sigmoid then yield a scalar gate s∈[0,1] . The gate modulates gaze-focused versus contextual information as pt=s⋅gt+(1−s)⋅ct . (4) Token Assembly and Training Objective. The per-frame tokens pt and mt are linearly projected, concatenated with the question, and fed into a language model to predict the answer, optimized with a cross-entropy (CE) loss.
Method
Params (M) ↓
TFLOPs ↓
Fr.
Past ↑
Present ↑
Proactive ↑
Macro ↑
Micro ↑
Below 300M parameters
SVLM-256M
256.48
3.966
16
0.328
0.283
0.537
0.383
0.480
Ours-256M
258.24
2.857
Adap.
0.351
0.532
0.643
0.509
0.617
300M+ parameters
SVLM-500M
507.48
4.742
16
0.350
0.449
0.578
0.459
0.582
GazeLLM
508.60
3.791
16
0.309
0.233
0.636
0.393
0.502
Table 2: StreamGaze accuracy and model cost, grouped by total parameter count. Params counts unique parameters, including vision encoders and added modules. TFLOPs are per input forward (Appendix E.2 ). Macro equally averages Past, Present, and Proactive, while Micro pools test examples. Both Ours variants report three-seed means. Across-seed variability is reported separately in Table B.1 . Bold/italic mark best/second-best accuracy. Adap. denotes dynamic selection, SVLM denotes SmolVLM2, and Mob-VGPT denotes Mobile-VideoGPT.
Figure 3: Accuracy versus input-forward computation at each model’s native image resolution. Muted colors identify baselines, while green and red highlight Ours-256M and Ours-500M. Linear marker size scales with the square root of total parameters. Higher and further left is better. The horizontal axis is logarithmic. FLOPs exclude frame selection, preprocessing, and answer generation (Appendix E.2 ).
Figure 4: EgoGazeVQA transfer on 1,265 held-out examples. T denotes uniform sampling, whereas ours is dynamic. Higher and further left is better.
Figure 5: Adaptive versus uniform frame selection for example video sequences. Red boxes mark redundant or motion-blurred frames in the uniform selections. Frame IDs run from 1 to 8.
Table 8
Figure 6: Gate distributions from one evaluated checkpoint. Boxes show quartiles and medians, while whiskers span min–max. Groups pool evaluation rows, including repeated question templates.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Evaluation set
Selector
Examples
Frames
Accuracy
Combined
Uniform T16
1265
15.72
56.84
Combined
Uniform T7
1265
7.00
58.10
Combined
Uniform T8
1265
8.00
58.02
Combined
GazeLens selector
1265
5.74
57.71
Ego4D
Uniform T16
577
15.90
51.30
Ego4D
Uniform T7
577
7.00
51.99
Appendix
Table 5: Frame-selection comparison on EgoGazeVQA. Each row reports a selector on the indicated evaluation set. Examples is the number of evaluation items, and Frames is the mean selected frames per item. Accuracy is in percent.
Noise σ
0.00
0.01
0.02
0.05
0.10
0.20
Macro
0.539 ±0.015
0.537
0.534
0.526
0.514
0.498
Micro
0.640 ±0.005
0.638
0.635
0.627
0.615
0.599
Appendix
Table 11: Model performance under synthetic spatial gaze noise at test time.
The use of multimodal LLMs (MLLMs) for egocentric video understanding with wearable devices is constrained by the token budget. Memory and compute cost scale with the number of visual tokens, and high-resolution video quickly becomes expensive to transmit and process at scale. Prior work (GazeLLM) addresses this by cropping the video around the camera wearer's gaze. This reduces the number of visual tokens by about tenfold while maintaining or improving the quality of full-resolution descriptions. However, this compression strategy depends on dedicated eye-tracking hardware, which is unavailable on consumer smart glasses. Building a software-only substitute poses a joint constraint: the predictor must be accurate enough to preserve downstream description quality, yet light enough to run on-device, within the power and compute budget of a smartphone. We address this with EgoGazeLite, a lightweight dual-process gaze predictor for egocentric video. Across two MLLMs, three automated metrics, and two LLM judges, predicted-gaze crops show no significant difference from ground-truth-gaze crops. Equivalence is confirmed in all ten cases. EgoGazeLite achieves this at 15.7M parameters, 6.71 GFLOPs, and runs the full gaze-and-crop pipeline end-to-end in real time (21.6 ms/frame) on consumer accelerator hardware. Together, these results remove the need for eye-tracking hardware for token-efficient, gaze-conditioned egocentric video understanding with MLLMs.
Matteo Stoiber, Niels Buus Lassen
Copenhagen Business School, Department of Digitalization, Copenhagen, Denmark
Current multimodal large language models (MLLMs) cannot effectively utilize eye-gaze information for video understanding, even when gaze cues are supplied via visual overlays or text descriptions. We introduce GazeQwen, a parameter efficient approach that equips an open-source MLLM with gaze awareness through hidden-state modulation. At its core is a compact gaze resampler (~1-5 M trainable parameters) that encodes V-JEPA 2.1 video features together with fixation-derived positional encodings and produces additive residuals injected into selected LLM decoder layers via forward hooks. An optional second training stage adds low-rank adapters (LoRA) to the LLM for tighter integration. Evaluated on all 10 tasks of the StreamGaze benchmark, GazeQwen reaches 63.9% accuracy, a +16.1 point gain over the same Qwen2.5-VL-7B backbone with gaze as visual prompts and +10.5 points over GPT-4o, the highest score among all open-source and proprietary models tested. These results suggest that learning where to inject gaze within an LLM is more effective than scaling model size or engineering better prompts. All code and checkpoints are available at https://github.com/phamtrongthang123/gazeqwen .
Video Large Language Models (VideoLLMs) have achieved strong video understanding capabilities but incur substantial inference overhead due to the large number of visual tokens. Existing VideoLLM token compression methods largely rely on selection-independent scoring, overlooking cross-frame complementarity and consequently retaining redundant evidence across frames. Instead, we view video token selection as a progressive evidence accumulation process. It aims to retain visual evidence that is individually informative and collectively complementary under a limited token budget. Building on this insight, we introduce GleanVID, a training-free inference acceleration framework for VideoLLMs. Specifically, GleanVID first allocates the global token budget across frames according to temporal novelty and then selects tokens by jointly considering local representativeness and subspace complementarity, thereby preserving richer and less redundant visual evidence. Extensive experiments across diverse VideoLLMs and benchmarks demonstrate that GleanVID consistently achieves state-of-the-art performance. Notably, with only 25% of visual tokens, GleanVID preserves 98.6% of Qwen3-VL's original performance while reducing its prefill latency by 44.7%. On LLaVA-OV-7B, GleanVID at a 25% retention ratio even slightly surpasses the original model.
Shuo Yang, Changbai Li, Rui Tang +3
Beihang University · Communication University of China