Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intent for realistic applications such as Augmented Reality (AR) glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether Multimodal Large Language Models (MLLMs) can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs utilize gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively assess streaming video understanding. Past and present tasks require immediate responses based on observations available at query time, whereas proactive tasks require models to continuously process incoming frames and respond at the appropriate future moment. To build StreamGaze, we develop a gaze-video Question Answering (QA) generation pipeline that aligns egocentric videos with raw gaze trajectories through fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, highlighting key limitations in gaze-based temporal reasoning, intent modeling, and proactive prediction. We further provide detailed analyses of gaze prompting strategies, reasoning behaviors, and task-specific failure modes, offering insights into current limitations and directions for future research. All data and code are publicly available to support research in gaze-guided streaming video understanding.
Figures & tables
Figure 1: StreamGaze ’s task taxonomy. We introduce StreamGaze , the first benchmark designed to evaluate how effectively MLLMs use gaze for temporal and proactive reasoning in streaming videos. We introduce gaze-guided past , present , and proactive streaming video understanding tasks.
Figure 2
Figure 3: Gaze-guided streaming data construction pipeline. Given egocentric video and raw gaze projections ( Section 3.1 ), we first extract fixation moments across the entire video ( Section 3.2 ). Next, we divide each frame into FOV and out-of-FOV regions and extract objects within the gaze area ( Section 3.3 ). Finally, we construct scanpaths and generate streaming QA pairs ( Section 4 ).
Method
Params
Frames
Past
Present
Proactive
Overall
NFI
OTP
SR
GSM
OI (E)
OI (H)
OAR
FAP
GTA
OAA
Human
-
-
0.700
0.889
0.903
0.707
0.960
0.920
0.800
0.840
0.765
0.780
0.827
Closed-Source MLLMs
Claude Opus 4.8 PBC (2025)
-
16
0.556
0.429
0.616
0.694
0.720
0.713
0.611
0.424
0.643
0.573
0.598
GPT-4o OpenAI (2024)
-
16
0.601
0.449
0.535
0.580
0.729
0.730
0.596
0.370
0.597
0.149
0.535
Claude Opus 4.6 PBC (2025)
-
16
0.533
0.425
0.573
0.640
0.479
0.525
0.343
0.413
0.646
0.662
0.524
Table 2: Comparison of various MLLMs across StreamGaze tasks. We evaluate four categories of models, reporting accuracy for past, present and proactive tasks. Each section is sorted by overall performance, and the best per task across all models (excluding Human) is bolded .
Past
Present
Proactive
Avg.
Qwen2.5-VL (w/o gaze)
0.423
0.500
0.384
0.446
+ text prompt
0.403
0.499
0.341
0.429
+ visual prompt
0.398
0.503
0.342
0.429
+ salience map
0.394
0.546
0.386
0.454
Table 3: Ablation of gaze input prompting. We report accuracy across different gaze input strategies.
Threshold
Value
# Fix.
# QA
ρ
0.10
10.3
257
0.972
Radius ( rthresh )
0.15
17.6
432
1.000
0.30
32.0
442
0.996
1.0 s
93.3
449
0.983
Duration ( τdur )
2.0 s
17.6
432
1.000
3.0 s
5.1
205
0.933
Table 4: Sensitivity analysis of fixation extraction thresholds. We report the average number of extracted fixations per video (# Fix.), the total number of generated QA pairs (# QA), and model ranking stability (Spearman’s ρ ) across 13 MLLMs relative to the default configuration. Light blue rows are the default threshold settings.
Figure 7
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Data statistics of StreamGaze . We report domain proportion, video length, word cloud for FOV extracted objects.
Figure 9
Task
Answer Correctness (%)
Distractor Quality (%)
Representative Failure Mode
NFI
100
98
Incorrect never-fixated label
OTP
96
100
Ambiguous object transition
SR
100
98
Background object misannotation
GSM
96
100
Ambiguous fixation ordering
OI
100
98
Confusable object distractor
OAR
94
96
Ambiguous object state
Appendix
Table 7: QA-level annotation quality audit. We manually evaluate 50 QA pairs per task for answer correctness and distractor quality. Representative failure modes are reported for each task. N/A indicates tasks without distractor options.
Fine-tuning datasets
Past
Present
Proactive
Overall
External
EgoExoLearn
Ego4D-Gaze
NFI
OTP
SR
GSM
OI (E)
OI (H)
OAR
FAP
GTA
OAA
0.526
0.486
0.341
0.418
0.635
0.477
0.413
0.489
0.504
0.502
0.479
✓
0.467
0.477
0.300
0.347
0.368
0.562
0.420
0.392
0.147
0.381
0.386
✓
✓
0.592
0.563
0.469
0.385
0.622
0.616
0.544
0.359
0.474
0.792
0.542
✓
✓
0.517
0.545
0.495
0.373
0.540
0.452
0.496
0.351
0.487
0.737
0.500
✓
✓
✓
0.583
0.540
0.495
0.388
0.601
0.641
0.575
0.331
0.409
0.651
0.521
Appendix
Table 8: Fine-tuning results on ViSpeak Fu et al. [2025b] . We construct four variants of training data for our fine-tuning experiments, drawing from both external resources Perrett et al. [2025] , Peng et al. [2025] and our own automatically generated data based on Ego4D-Gaze Grauman et al. [2022] and EgoExoLearn Huang et al. [2024] .
Figure 7: Example of negative sampling strategy for proactive task fine-tuning. We apply negative sampling to fine-tune ViSpeak.
Figure 8: Gaze prompting strategies for StreamGaze We adopt three strategies for inputting gaze information to MLLMs.
Figure 9: Task-wise effect of gaze input strategies on StreamGaze . Different tasks respond unevenly to textual, visual, and salience-map prompts, revealing strong task-specific behavior in streaming gaze understanding.
Method
Params
Frames
Past
Present
Proactive
Overall
NFI
OTP
SR
GSM
OI (E)
OI (H)
OAR
FAP
GTA
OAA
GPT-4o OpenAI [2024]
-
16
0.601
0.459
0.507
0.607
0.725
0.731
0.594
0.375
0.608
0.148
0.536
+ visual prompt
-
16
0.601
0.449
0.535
0.580
0.729
0.730
0.596
0.370
0.597
0.149
0.535
InternVL3.5 Wang et al. [2025a]
8B
Adaptive
0.469
0.370
0.526
0.510
0.626
0.637
0.463
0.382
0.376
0.048
0.441
+ visual prompt
8B
Adaptive
0.490
0.311
0.573
0.548
0.627
0.628
0.466
0.372
0.373
0.051
0.444
ViSpeak Fu et al. [2025b]
7B
1 fps
0.493
0.374
0.417
0.521
0.591
0.560
0.438
0.336
0.625
0.334
0.469
Appendix
Table 9: Detailed performance of gaze input prompting across each StreamGaze task. We ablate effect of gaze input prompting on GPT-4o OpenAI [2024] , InternVL3.5 Wang et al. [2025a] and ViSpeak Fu et al. [2025b] .
Figure 16
Method
Frames
Past
Present
Proactive
Overall
NFI
OTP
SR
GSM
OI (E)
OI (H)
OAR
FAP
GTA
OAA
InternVL3.5 [ Wang et al., 2025a ]
32
0.511
0.407
0.578
0.511
0.660
0.668
0.489
0.380
0.628
0.111
0.494
EgoGPT [ Yang et al., 2025b ]
32
0.521
0.393
0.453
0.550
0.690
0.620
0.304
0.368
0.250
0.194
0.436
Qwen2.5-VL [ Wang et al., 2024 ]
32
0.169
0.445
0.246
0.538
0.561
0.671
0.568
0.394
0.463
0.163
0.422
MiniCPM-V [ Yao et al., 2024 ]
32
0.430
0.374
0.354
0.296
0.334
0.379
0.438
0.345
0.480
0.216
0.365
VITA 1.5 [ Fu et al., 2025a ]
32
0.226
0.211
0.280
0.210
0.275
0.149
0.040
0.250
0.076
0.037
0.175
Appendix
Table 11: Comparison of open-source MLLMs under a unified 32-frame budget. We evaluate all models using 32 input frames to control for differences in frame budgets. Performance may differ from the main results due to changes in model-specific inference configurations. The best performance in each task is bolded .
Task
Model
±10 s
±15 s
±20 s
GTA
Qwen3-VL-8B [ Yang et al., 2025a ]
0.4733 (1)
0.4915 (1)
0.5096 (1)
TimeChat-Online [ Yao et al., 2025 ]
0.2448 (2)
0.2834 (2)
0.3227 (2)
VITA-1.5 [ Fu et al., 2025a ]
0.0564 (3)
0.0668 (3)
0.0784 (3)
OAA
Qwen3-VL-8B [ Yang et al., 2025a ]
0.6321 (1)
0.6352 (1)
0.6379 (1)
TimeChat-Online [ Yao et al., 2025 ]
0.1966 (2)
0.2264 (2)
0.2575 (2)
VITA-1.5 [ Fu et al., 2025a ]
0.0511 (3)
0.0585 (3)
0.0646 (3)
Appendix
Table 12: Sensitivity to proactive evaluation tolerance. Accuracy under different temporal tolerance windows for GTA and OAA. Model rankings are shown in parentheses.
Table 19
Figure 11: Qualitative comparison with text reasoning and gaze-based reasoning. We visualize examples from the scene reconstruction task comparing (a) text and (b) gaze-based reasoning.
Figure 12: Qualitative comparison with text reasoning and visual-based reasoning. We visualize examples from the object identification task comparing (a) text and (b) visual-based reasoning.
Figure 13: StreamGaze data example for OTP and NFI task.
Figure 14: StreamGaze data example for SR and GSM task.
Figure 15: StreamGaze data example for OI (Easy/Hard) task.
Figure 16: StreamGaze data example for OAR and FAP task.
Figure 17: StreamGaze data example for GTA and OAA task.
Figure 18: Prompt for FOV region object extraction.
Figure 19: Prompt for out-of-FOV region object extraction.
Figure 20: Prompt for text-based reasoning.
Figure 21: Prompt for gaze-based reasoning.
Figure 22: Prompt for visual-based reasoning.
Figure 23: HTML for human verification of StreamGaze data construction.
Figure 24: HTML for human oracle evaluation of StreamGaze .
Figure 25: Instructions provided to human annotators.