AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding
Authors: Lu Wang, Zhuoran Jin, Longxiang Wang, Junhong Chen, Yupu Hao, Yao Xu, Daojian Zeng, Yubo Chen, Kang Liu, Guang Liu, Jun Zhao
Organizations: The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China · Hunan Normal University, China · Minzu University of China, Beijing, China
Recent advances in multimodal large language models have increased interest in interaction paradigms for video understanding. The evolution from offline to streaming and proactive interaction has gradually shifted models from passive responders toward more proactive agents that can decide when to answer. However, existing proactive paradigms typically assume that the information required to answer a query is complete, leaving models to only wait or answer. In the real world, however, information is often incomplete due to uncertainty from either the video or the user. In proactive paradigms, such uncertainty can cause the model to wait forever or generate highly hallucinatory responses. We therefore argue that proactive interaction should be redefined with a third action: ask, allowing models to actively ask for missing information. To study when and what models should ask before answering, we introduce AskStream, the first multi-turn benchmark spanning video and user uncertainty for streaming video understanding, with four task categories and eight subtasks. It contains 438 videos with 2,714 manually annotated frame-level time windows across three types: wait, ask, and answer. Across both multiple-choice and open-ended settings, even the strongest model attains only 52.18% on our proposed Ask-F1 metric, highlighting substantial limitations in proactive information seeking under uncertainty.
Figures & tables
Figure 1: Evolution of interaction paradigms. AskStream extends the conventional action space of wait and answer by introducing ask , enabling models to proactively seek missing information from the video or user before answering.
Figure 2: Analysis of model uncertainty. (a) Across interaction turns, correct evidence consistently lowers normalized answer entropy, unlike mismatched evidence. (b) Relevant evidence shifts probability toward the correct answer. (c) Accuracy improves from 42.3% (blurred) to 62.3% after asking for key evidence, nearing 63.1% with fully clear input.
Benchmark
Eval.
Opt.
Task
Dur.
Timing
FL
Ask
MT
Pro.
(A) Offline video benchmarks
Video-MME ( Fu et al., 2025a )
MC
4
12
17.0
–
✗
✗
✗
✗
LongVideoBench ( Wu et al., 2024 )
MC
4.73 †
17
7.9
–
✗
✗
✗
✗
(B) Streaming video benchmarks
StreamingBench ( Lin et al., 2026b )
MC
4
18
4.1
TS
✗
✗
✓
✓
RIVER-Bench ( Shi et al., 2026 )
ALL
4
3
24.1 †
TS
✗
✗
✗
✓
Table 1: Comparison of video understanding benchmarks. MC & OE denote multiple-choice & open-ended evaluation. Opt., Task, & Dur. denote average options, task number, and average video duration. TS & Win. denote timestamps and time windows. FL, MT, & Pro. denote frame-level annotation, multi-turn interaction, and proactive tasks. † denotes values measured by us.
Figure 3: AskStream task taxonomy and benchmark construction. Left: eight subtasks spanning video and user uncertainty, with examples of wait , ask , and answer interactions. Right: the five-stage construction pipeline.
Figure 4: Statistics of AskStream. (a) Task taxonomy, data distribution, and sources. (b) Total effective durations of wait , ask , and answer across subtasks. (c) Video durations and sample counts across domains. The benchmark averages 3.2 interaction turns per video and contains 2714 manually annotated timestamps in total.
Figure 5: Error-type distributions of offline models in the multiple-choice setting. Each vertex shows the percentage of the corresponding error type.
Video
User
Explicit
Implicit
Intent
Preference
Metric
Method
Backbone
KVEM
VIM
ICM
PACM
AUQ
WUQ
UPM
UPC
Sask
Sans
Ask-F1
Offline closed-source models
GPT-5.6-Sol
-
18.10
2.93
3.05
0
15.69
17.50
18.60
26.55
18.11
23.62
20.50
[0.35pt/1.25pt] Claude Sonnet 5
-
21.48
13.01
7.60
9.70
13.28
29.42
7.93
29.53
20.87
24.37
22.48
Gemini-3.8 Flash
-
37.17
17.46
33.26
17.73
31.13
43.10
64.03
81.36
44.63
62.80
52.18
Table 2: Main results on AskStream in the multiple-choice setting. Scores reported for individual subtasks are Ask-F1 scores. Bold denotes the best performance on each subtask.
Figure 6: Oracle ablation of decision stages. Results on KVEM using Gemini-3.8 Flash in the multiple-choice setting. (a) Effect of different oracle signals. (b) ASK-F1 under different amounts and combinations of oracle information.
Figure 7: Analysis Experiments. (a) Impact of contextual memory. We use Gemini-3.8 Flash under the multiple-choice setting on four subtasks, KVEM, PACM, AUQ, and UPM, to analyze the effect of cross-turn contextual memory. (b) Impact of ASK budget. We evaluate Gemini-3.8 Flash on KVEM under the open-ended setting to analyze the effect of different ASK budgets.
Method
Logarithmic
Linear
Exponential
Offline closed-source
GPT-5.6-Sol
22.04
20.50
16.65
Claude Sonnet 5
24.16
22.48
18.27
Gemini-3.8 Flash
56.11
52.18
42.34
Offline open-source
MiMo-V2.5
15.81
14.66
11.82
Table 3: Sensitivity of Ask-F1 to alternative temporal weighting functions. Results are reported for the seven models under the multiple-choice setting. Linear is the default metric, while logarithmic and exponential replace its within-window temporal weighting function. Spearman ρ w.r.t. Linear measures the rank correlation of model rankings relative to the default Linear metric, where ρ=1.00 indicates identical rankings.
Comparison
ASK
ANSWER
Overall
Agreement (%)
Agreement (%)
Agreement (%)
Human&Human
95.33
97.00
96.17
Gemini Judge vs. Human Consensus
All outputs
92.67
94.33
93.50
Gemini-3.8 Flash outputs
93.06
94.44
93.75
Non Gemini outputs
92.31
94.23
93.27
Table 4: Validation of Gemini-3.8 Flash as the open-ended judge on 300 sampled turns. Human&Human reports the agreement between two independent human annotators and serves as a reference for annotation consistency. ASK Agreement and ANSWER Agreement denote the percentage of identical correctness judgments for ASK and ANSWER responses, respectively. Overall Agreement is computed over both ASK and ANSWER judgments. For the Gemini Judge vs. Human Consensus rows, human consensus is used as the reference judgment. Results are further separated into Gemini-3.8 Flash and non Gemini outputs to examine potential self-preference.
Context Strategy
Sask
Sans
Ask-F1
StreamHarness
44.63
62.80
52.18
Sliding Window
35.92
49.71
41.70
Uniform Sampling (128)
22.41
33.08
26.72
Uniform Sampling (64)
13.86
21.94
16.99
Table 5: Ablation of StreamHarness using Gemini-3.8 Flash under the multiple-choice setting. Uniform sampling ensures that each valid temporal window contains at least one sampled frame.
Video
User
Explicit
Implicit
Intent
Preference
Metric
Method
Backbone
KVEM
VIM
ICM
PACM
AUQ
WUQ
UPM
UPC
Sask
Sans
Ask-F1
Offline closed-source models
Claude Sonnet 5
-
13.59
17.62
0
4.78
12.69
12.28
0
19.73
15.88
10.99
12.99
[0.35pt/1.25pt] GPT-5.6-Sol
-
6.30
3.90
2.80
0
16.20
9.96
7.63
16.52
12.05
18.80
14.69
Gemini 3.8 Flash
-
27.61
29.40
35.79
0
35.87
32.12
14.41
18.43
23.34
38.97
29.19
Table 6: Main results on AskStream in the Open-ended setting.
Figure 8: Web interface for KVEM annotation. Annotators review each event together with its tracked entities and keyframes, refine entity bounding boxes when necessary, select the gold entity, and annotate the temporal decision windows for streaming interaction. The interface also supports event-level query annotation and visualization of the corresponding timeline, facilitating consistent construction of KVEM instances.
Figure 9: Web interface for KVEM human verification. The fully masked video and the revealed video, in which only the gold entity is visible, are presented side by side for comparison. Annotators jointly inspect the query, gold entity, and ASK candidates to verify that the masked video contains no information leakage, that revealing the requested gold entity is sufficient to uniquely answer the query, and that the correct and incorrect ASK options correspond to appropriate and inappropriate information requests, respectively.
Figure 10: Multi-view annotation interface for the VIM task. The interface presents temporally synchronized egocentric and multiple exocentric video streams from Ego-Exo4D, together with the key-step-based turn structure and temporal decision windows. Annotators jointly inspect all viewpoints, designate the initial viewpoint, identify alternative viewpoints that reveal the missing visual evidence, and construct viewpoint-dependent queries and ASK candidates.
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.
Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed visual content. In real-world scenarios, assistants must additionally anticipate what assistance is needed and decide when to provide it. To address this gap, we introduce \textsc{P3IBench}, a benchmark for proactive video understanding in real-life streaming scenarios. It evaluates three complementary capabilities: daily perception, risk prediction and intervention. We benchmark 12 representative open-source and proprietary video-language models, including streaming and adapted non-streaming systems. Our evaluation shows that current models struggle to make well-calibrated response decisions. Finally, we construct additional proactive training data and train \textsc{P3I-Assistant-4B}, achieving improvements on several tasks, though gains remain uneven across capabilities. Together, these findings show that effective proactive assistance requires not only visual understanding, but also precise response timing and appropriate restraint.
Boyu Cui, Junhao Luo, Zhihao Wang +5
Fudan University · The Chinese University of Hong Kong · Shanghai Innovation Institute
Recent advances in Streaming Video Understanding has enabled a new interaction paradigm where models respond proactively to user queries. Current proactive VideoLLMs rely on per-frame triggering decision making, which suffers from an efficiency-accuracy dilemma. We propose Em-Garde, a novel framework that decouples semantic understanding from streaming perception. At query time, the Instruction-Guided Proposal Parser transforms user queries into structured, perceptually grounded visual proposals; during streaming, a Lightweight Proposal Matching Module performs efficient embedding-based matching to trigger responses. Experiments on StreamingBench and OVO-Bench demonstrate consistent improvements over prior models in proactive response accuracy and efficiency, validating an effective solution for proactive video understanding under strict computational constraints.
Yikai Zheng, Xin Ding, Yifan Yang +6
Institute for AI Industry Research (AIR), Tsinghua University · University of Science and Technology of China · 4Microsoft Research Asia +1
Streaming video understanding requires models to process continuous multimodal input while maintaining temporal context. Existing evaluations are predominantly reactive: they query a model at a selected timestamp and therefore do not assess when it should respond. Proactive interaction instead requires monitoring a standing request, responding within an appropriate interval after the target event, and otherwise remaining silent. We introduce LiveProBench, which evaluates models at one-second stream intervals without an explicit response cue. Its six subtasks vary trigger ambiguity and timing tolerance. Event Sensitivity geometrically combines response and silence rates on the same recording; four window-based subtasks distinguish early, in-window, and missed responses; and Duplicate Counting penalizes omissions and repetitions. Premature responses outnumber missed responses for half of the evaluated models, revealing a substantial gap in the temporal decision-making required for human-like interaction.
Kaixuan Du, Xin Wan, Hang Zhang +4
School of Automation Science and Electrical Engineering, Beihang University · Qwen Business Unit of Alibaba