Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intent for realistic applications such as Augmented Reality (AR) glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether Multimodal Large Language Models (MLLMs) can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs utilize gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively assess streaming video understanding. Past and present tasks require immediate responses based on observations available at query time, whereas proactive tasks require models to continuously process incoming frames and respond at the appropriate future moment. To build StreamGaze, we develop a gaze-video Question Answering (QA) generation pipeline that aligns egocentric videos with raw gaze trajectories through fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, highlighting key limitations in gaze-based temporal reasoning, intent modeling, and proactive prediction. We further provide detailed analyses of gaze prompting strategies, reasoning behaviors, and task-specific failure modes, offering insights into current limitations and directions for future research. All data and code are publicly available to support research in gaze-guided streaming video understanding.
Figures & tables
Figure 1: StreamGaze ’s task taxonomy. We introduce StreamGaze , the first benchmark designed to evaluate how effectively MLLMs use gaze for temporal and proactive reasoning in streaming videos. We introduce gaze-guided past , present , and proactive streaming video understanding tasks.
Figure 2
Figure 3: Gaze-guided streaming data construction pipeline. Given egocentric video and raw gaze projections ( Section 3.1 ), we first extract fixation moments across the entire video ( Section 3.2 ). Next, we divide each frame into FOV and out-of-FOV regions and extract objects within the gaze area ( Section 3.3 ). Finally, we construct scanpaths and generate streaming QA pairs ( Section 4 ).
Method
Params
Frames
Past
Present
Proactive
Overall
NFI
OTP
SR
GSM
OI (E)
OI (H)
OAR
FAP
GTA
OAA
Human
-
-
0.700
0.889
0.903
0.707
0.960
0.920
0.800
0.840
0.765
0.780
0.827
Closed-Source MLLMs
Claude Opus 4.8 PBC (2025)
-
16
0.556
0.429
0.616
0.694
0.720
0.713
0.611
0.424
0.643
0.573
0.598
GPT-4o OpenAI (2024)
-
16
0.601
0.449
0.535
0.580
0.729
0.730
0.596
0.370
0.597
0.149
0.535
Claude Opus 4.6 PBC (2025)
-
16
0.533
0.425
0.573
0.640
0.479
0.525
0.343
0.413
0.646
0.662
0.524
Table 2: Comparison of various MLLMs across StreamGaze tasks. We evaluate four categories of models, reporting accuracy for past, present and proactive tasks. Each section is sorted by overall performance, and the best per task across all models (excluding Human) is bolded .
Past
Present
Proactive
Avg.
Qwen2.5-VL (w/o gaze)
0.423
0.500
0.384
0.446
+ text prompt
0.403
0.499
0.341
0.429
+ visual prompt
0.398
0.503
0.342
0.429
+ salience map
0.394
0.546
0.386
0.454
Table 3: Ablation of gaze input prompting. We report accuracy across different gaze input strategies.
Threshold
Value
# Fix.
# QA
ρ
0.10
10.3
257
0.972
Radius ( rthresh )
0.15
17.6
432
1.000
0.30
32.0
442
0.996
1.0 s
93.3
449
0.983
Duration ( τdur )
2.0 s
17.6
432
1.000
3.0 s
5.1
205
0.933
Table 4: Sensitivity analysis of fixation extraction thresholds. We report the average number of extracted fixations per video (# Fix.), the total number of generated QA pairs (# QA), and model ranking stability (Spearman’s ρ ) across 13 MLLMs relative to the default configuration. Light blue rows are the default threshold settings.
Figure 7
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Data statistics of StreamGaze . We report domain proportion, video length, word cloud for FOV extracted objects.
Figure 9
Task
Answer Correctness (%)
Distractor Quality (%)
Representative Failure Mode
NFI
100
98
Incorrect never-fixated label
OTP
96
100
Ambiguous object transition
SR
100
98
Background object misannotation
GSM
96
100
Ambiguous fixation ordering
OI
100
98
Confusable object distractor
OAR
94
96
Ambiguous object state
Appendix
Table 7: QA-level annotation quality audit. We manually evaluate 50 QA pairs per task for answer correctness and distractor quality. Representative failure modes are reported for each task. N/A indicates tasks without distractor options.
Fine-tuning datasets
Past
Present
Proactive
Overall
External
EgoExoLearn
Ego4D-Gaze
NFI
OTP
SR
GSM
OI (E)
OI (H)
OAR
FAP
GTA
OAA
0.526
0.486
0.341
0.418
0.635
0.477
0.413
0.489
0.504
0.502
0.479
✓
0.467
0.477
0.300
0.347
0.368
0.562
0.420
0.392
0.147
0.381
0.386
✓
✓
0.592
0.563
0.469
0.385
0.622
0.616
0.544
0.359
0.474
0.792
0.542
✓
✓
0.517
0.545
0.495
0.373
0.540
0.452
0.496
0.351
0.487
0.737
0.500
✓
✓
✓
0.583
0.540
0.495
0.388
0.601
0.641
0.575
0.331
0.409
0.651
0.521
Appendix
Table 8: Fine-tuning results on ViSpeak Fu et al. [2025b] . We construct four variants of training data for our fine-tuning experiments, drawing from both external resources Perrett et al. [2025] , Peng et al. [2025] and our own automatically generated data based on Ego4D-Gaze Grauman et al. [2022] and EgoExoLearn Huang et al. [2024] .
Figure 7: Example of negative sampling strategy for proactive task fine-tuning. We apply negative sampling to fine-tune ViSpeak.
Figure 8: Gaze prompting strategies for StreamGaze We adopt three strategies for inputting gaze information to MLLMs.
Figure 9: Task-wise effect of gaze input strategies on StreamGaze . Different tasks respond unevenly to textual, visual, and salience-map prompts, revealing strong task-specific behavior in streaming gaze understanding.
Method
Params
Frames
Past
Present
Proactive
Overall
NFI
OTP
SR
GSM
OI (E)
OI (H)
OAR
FAP
GTA
OAA
GPT-4o OpenAI [2024]
-
16
0.601
0.459
0.507
0.607
0.725
0.731
0.594
0.375
0.608
0.148
0.536
+ visual prompt
-
16
0.601
0.449
0.535
0.580
0.729
0.730
0.596
0.370
0.597
0.149
0.535
InternVL3.5 Wang et al. [2025a]
8B
Adaptive
0.469
0.370
0.526
0.510
0.626
0.637
0.463
0.382
0.376
0.048
0.441
+ visual prompt
8B
Adaptive
0.490
0.311
0.573
0.548
0.627
0.628
0.466
0.372
0.373
0.051
0.444
ViSpeak Fu et al. [2025b]
7B
1 fps
0.493
0.374
0.417
0.521
0.591
0.560
0.438
0.336
0.625
0.334
0.469
Appendix
Table 9: Detailed performance of gaze input prompting across each StreamGaze task. We ablate effect of gaze input prompting on GPT-4o OpenAI [2024] , InternVL3.5 Wang et al. [2025a] and ViSpeak Fu et al. [2025b] .
Figure 16
Method
Frames
Past
Present
Proactive
Overall
NFI
OTP
SR
GSM
OI (E)
OI (H)
OAR
FAP
GTA
OAA
InternVL3.5 [ Wang et al., 2025a ]
32
0.511
0.407
0.578
0.511
0.660
0.668
0.489
0.380
0.628
0.111
0.494
EgoGPT [ Yang et al., 2025b ]
32
0.521
0.393
0.453
0.550
0.690
0.620
0.304
0.368
0.250
0.194
0.436
Qwen2.5-VL [ Wang et al., 2024 ]
32
0.169
0.445
0.246
0.538
0.561
0.671
0.568
0.394
0.463
0.163
0.422
MiniCPM-V [ Yao et al., 2024 ]
32
0.430
0.374
0.354
0.296
0.334
0.379
0.438
0.345
0.480
0.216
0.365
VITA 1.5 [ Fu et al., 2025a ]
32
0.226
0.211
0.280
0.210
0.275
0.149
0.040
0.250
0.076
0.037
0.175
Appendix
Table 11: Comparison of open-source MLLMs under a unified 32-frame budget. We evaluate all models using 32 input frames to control for differences in frame budgets. Performance may differ from the main results due to changes in model-specific inference configurations. The best performance in each task is bolded .
Task
Model
±10 s
±15 s
±20 s
GTA
Qwen3-VL-8B [ Yang et al., 2025a ]
0.4733 (1)
0.4915 (1)
0.5096 (1)
TimeChat-Online [ Yao et al., 2025 ]
0.2448 (2)
0.2834 (2)
0.3227 (2)
VITA-1.5 [ Fu et al., 2025a ]
0.0564 (3)
0.0668 (3)
0.0784 (3)
OAA
Qwen3-VL-8B [ Yang et al., 2025a ]
0.6321 (1)
0.6352 (1)
0.6379 (1)
TimeChat-Online [ Yao et al., 2025 ]
0.1966 (2)
0.2264 (2)
0.2575 (2)
VITA-1.5 [ Fu et al., 2025a ]
0.0511 (3)
0.0585 (3)
0.0646 (3)
Appendix
Table 12: Sensitivity to proactive evaluation tolerance. Accuracy under different temporal tolerance windows for GTA and OAA. Model rankings are shown in parentheses.
Table 19
Figure 11: Qualitative comparison with text reasoning and gaze-based reasoning. We visualize examples from the scene reconstruction task comparing (a) text and (b) gaze-based reasoning.
Figure 12: Qualitative comparison with text reasoning and visual-based reasoning. We visualize examples from the object identification task comparing (a) text and (b) visual-based reasoning.
Figure 13: StreamGaze data example for OTP and NFI task.
Figure 14: StreamGaze data example for SR and GSM task.
Figure 15: StreamGaze data example for OI (Easy/Hard) task.
Figure 16: StreamGaze data example for OAR and FAP task.
Figure 17: StreamGaze data example for GTA and OAA task.
Figure 18: Prompt for FOV region object extraction.
Figure 19: Prompt for out-of-FOV region object extraction.
Figure 20: Prompt for text-based reasoning.
Figure 21: Prompt for gaze-based reasoning.
Figure 22: Prompt for visual-based reasoning.
Figure 23: HTML for human verification of StreamGaze data construction.
Figure 24: HTML for human oracle evaluation of StreamGaze .
Figure 25: Instructions provided to human annotators.
Current multimodal large language models (MLLMs) cannot effectively utilize eye-gaze information for video understanding, even when gaze cues are supplied via visual overlays or text descriptions. We introduce GazeQwen, a parameter efficient approach that equips an open-source MLLM with gaze awareness through hidden-state modulation. At its core is a compact gaze resampler (~1-5 M trainable parameters) that encodes V-JEPA 2.1 video features together with fixation-derived positional encodings and produces additive residuals injected into selected LLM decoder layers via forward hooks. An optional second training stage adds low-rank adapters (LoRA) to the LLM for tighter integration. Evaluated on all 10 tasks of the StreamGaze benchmark, GazeQwen reaches 63.9% accuracy, a +16.1 point gain over the same Qwen2.5-VL-7B backbone with gaze as visual prompts and +10.5 points over GPT-4o, the highest score among all open-source and proprietary models tested. These results suggest that learning where to inject gaze within an LLM is more effective than scaling model size or engineering better prompts. All code and checkpoints are available at https://github.com/phamtrongthang123/gazeqwen .
Streaming video understanding requires multimodal large language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term memory that filters temporal redundancy before visual encoding with a latent long-term memory that consolidates historical video content into visual latents accessible to subsequent reasoning. During generation, an attention-guided retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-video benchmarks. Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient reasoning.
Muxin Fu, Yifan Zhang, Wentao Zhang +5
Tongji University · Nanyang Technological University · University of Michigan +2
We introduce EgoSAT, the first comprehensive benchmark for egocentric video reasoning in streaming settings, designed to evaluate the capabilities of modern vision-language models (VLMs). The benchmark targets streaming interaction understanding, where video frames arrive sequentially and models must continuously interpret evolving visual context. EgoSAT unifies several previously distinct tasks within a single streaming framework. In this formulation, queries about completed events correspond to retrospective reasoning, queries about ongoing activities require online understanding, and queries about future actions involve prospective anticipation. This unified setting requires models to reason about the past, present, and future while operating under the constraint that only previously observed frames are available. EgoSAT contains 1,997 unique videos spanning 165 hours of egocentric footage and around 4,800 high-quality question-answer pairs, carefully designed to probe reasoning across varying temporal contexts. Using this benchmark, we evaluate a diverse set of both open-weight and closed-weight VLMs, providing a systematic assessment of their ability for streaming interaction understanding. By distinguishing answerability and conducting diagnostics on confidence of models, we find existing models not only struggle with prospective and retrospective modeling, but also exhibit severe mis-calibration: confidence often fails to track inherent answerability, leading to dangerous "confidently wrong" behaviors. Project page: https://leiyj23.github.io/EgoSAT/
Yijia Lei, Jinzhao Li, Yichi Zhang +3
College of AI, Tsinghua University · University of Wisconsin–Madison