StreamClarify: Proactive Clarification for Grounded Streaming Video Question Answering

Zhuchen Shao; Bo Chen; Klara Nahrstedt

cs.CVOct 31, 2026

StreamClarify: Proactive Clarification for Grounded Streaming Video Question Answering

Authors: Zhuchen Shao, Bo Chen, Klara Nahrstedt

Organizations: University of Illinois Urbana-Champaign

Abstract

Streaming video question answering (QA) requires real-time reasoning over continuous video streams under strict latency and active-context budgets. Current streaming vision-language models (VLMs) often give confident yet ungrounded answers when queries are ambiguous or have false premises, or when relevant visual evidence is no longer in the active context. Proactive clarification can help, but must satisfy the same latency and memory constraints. We present StreamClarify, a training-free framework for grounded streaming QA that combines a pre-answer uncertainty check with hierarchical memory. The check uses the entropy of a binary pre-answer judgment as a lightweight uncertainty score to decide whether to answer directly or request clarification. To support re-answering, a temporal-pyramid KV cache maintains a bounded active visual context, while a CPU-side semantic retrieval archive provides access to historical frames. We further construct StreamClarify-QA, a diagnostic benchmark of 1,837 QA pairs from StreamBench and OVO-Bench, spanning standard, referential ambiguity, and unanswerable scenarios. Across three VLM backbones, StreamClarify improves Macro accuracy by up to 8.2 points over backbone-matched baselines. At comparable trigger rates, entropy-based selection achieves more than 18 points higher precision than random selection in identifying initially incorrect answers. Under oracle user guidance, clarification improves Macro accuracy by up to 3.5 points. We also report a controlled, oracle-assisted Qwen3-VL simulation on Ambiguous QA with generated questions and referent-only user replies, where the entropy-gated condition improves accuracy by 8.3 points.

Figures & tables

Figure 1: Overview of StreamClarify. The framework augments a frozen streaming VLM with a pre-answer uncertainty check in a two-stage generation loop and hierarchical memory. (a) When pre-answer entropy does not exceed the trigger threshold, the system answers directly using the active temporal-pyramid KV (TP-KV) cache. (b) TP-KV keeps recent frames at full resolution and compresses older observed frames within a fixed active budget. (c) When entropy exceeds the threshold, the system requests clarification and converts the resulting user guidance into a retrieval query. (d) Semantic search retrieves relevant past clips from the archive and combines them with the active TP-KV context for answering after clarification.

Figure 2: StreamClarify-QA construction pipeline. (a) Dense captions are aligned to each query timestamp. (b) Eligible QA pairs are transformed into Referential Ambiguity or Unanswerable variants, followed by type-specific validation. OVO-Bench HLD questions are retained unchanged.

Dataset	#Vid	#QA	Q/Vid	Dur.(s)
StreamBench	260	1,556	6.0	${\sim}$ 253
OVO-Bench	95	281	${\sim}$ 3.0	${\sim}$ 383

Table 1: Statistics of StreamClarify-QA. (a) Video and QA overview by source. (b) Scenario distribution across both datasets. Abbreviations: Vid = videos, Dur. = duration, Ref. Ambig. = referential ambiguity, and Unans. = unanswerable.

		StreamBench							OVO-Bench
		Per-Cat. Acc.			Overall Acc.		System		Per-Cat. Acc.			Overall Acc.		System
		Std.	Amb.	UQ	Micro	Macro	TTFT $\downarrow$	ProcFPS $\uparrow$	Std.	Amb.	UQ	Micro	Macro	TTFT $\downarrow$	ProcFPS $\uparrow$
Closed	GPT-5-mini	73.4	16.9	8.0	47.8	32.8	–	–	58.5	54.2	25.2	39.9	46.0	3.829	–
Open	VideoLLM-Online	49.5	11.8	0.5	31.1	20.6	0.064	9.43	13.4	22.9	0.7	8.2	12.3	0.066	11.44
	Flash-VStream	62.1	11.4	17.2	42.6	30.2	0.149	1.97	45.1	47.9	16.6	30.2	36.5	0.125	1.91
	ReKV	67.1	16.5	6.2	43.6	29.9	0.181	4.53	58.5	47.9	30.5	41.6	45.6	0.053	6.76

Table 2: Per-category streaming video QA results on StreamBench and OVO-Bench in the no-clarification setting using each method’s standard configuration. Shaded columns denote the overall accuracy and system-efficiency columns. Greyed text reports the per-category breakdown (Std.

=

Standard, Amb.

=

Ambiguous, UQ

=

Unanswerable QA). Arrows indicate the preferred direction. Accuracy values are percentages, and TTFT is reported in seconds.

LLaVA-OV-1.5
	Overall Performance							Clarification
Method	Std.	Amb.	UQ	Micro	Macro	TTFT $\downarrow$	ProcFPS $\uparrow$	Trig.	Prec.	Rec.
w/o Clar.	64.5	21.5	51.7	53.7	45.9	0.316	5.14	0.0	–	–
w/ Random	65.6	24.8	49.3	54.1	46.6	1.145	4.05	27.8	44.5	26.7
w/ Entropy	66.9	31.1	49.8	56.0	49.3	1.166	4.04	28.4	63.0	38.5
InternVL-3.5

Table 3: Oracle-guided analysis on StreamBench and OVO-Bench pooled at comparable trigger rates near 30%. Guidance is provided only after a trigger. Grey text shows per-category accuracy, while shaded columns report overall accuracy and efficiency. Trig. is the trigger rate; Prec. and Rec. treat initially incorrect answers as positives. TTFT is averaged over all 1,837 queries. ProcFPS is the unweighted mean of per-video processing throughput across 355 videos.

		Overall Accuracy		Paired Change
Condition	Selection	Correct	Micro (%)	W $\rightarrow$ C	C $\rightarrow$ W	$\Delta$ Correct
w/o Clar.	None	68	22.5	–	–	–
Generated Clar.	Random-gated	87.8	29.1	29.5	9.7	+19.8
Generated Clar.	Entropy-gated	93	30.8	34	9	+25
Direct Retrieval	All-query	69	22.8	20	19	+1
Generated Clar.	All-query	105	34.8	55	18	+37

Table 4: Generated-question analysis on 302 Ambiguous QA pairs with Qwen3-VL. Random-gated entries, including paired transition counts, are means over

1{,}000

selections of 162 queries. W

\rightarrow

C and C

\rightarrow

W denote paired correctness transitions.

Method	Macro (%)	TTFT (s)
Direct Answer	49.3	0.085
Probability	50.3	0.365
V/K Markers	51.3	0.210
Ours (Single)	51.0	0.100
Ours (Two-stage)	53.7	0.260

Table 5: Comparison of pre-answer scoring strategies on OVO-Bench with Qwen3-VL without clarification or retrieval.

Figure 3: Trigger Precision and Recall on OVO-Bench with Qwen3-VL. Trigger rates are computed over all 281 questions. Gray bands span the methods’ expected random Precision (a: 42.35–48.04%, b: 44.13–46.26%). They are not confidence intervals. Gray lines show expected Recall under uniform random selection, equal to the trigger rate.

Figure 4: TP-KV hyperparameter ablations on OVO-Bench with Qwen3-VL in the no-clarification setting. Bars show ProcFPS and lines show Macro Accuracy.

	StreamBench				OVO-Bench
Method	Micro	Macro	TTFT $\downarrow$	ProcFPS $\uparrow$	Micro	Macro	TTFT $\downarrow$	ProcFPS $\uparrow$
StreamChat	52.8	45.4	0.314	3.42	50.5	46.2	0.198	3.68
StreamBridge	57.7	45.7	1.170	24.51	47.7	48.8	1.197	44.86
StreamingVLM	56.7	47.8	0.064	8.86	47.0	46.3	0.071	9.59
Ours	58.0	51.3	0.232	8.46	56.9	52.0	0.252	9.23

Table 6: Matched-6K Qwen3-VL results without clarification. Accuracy is reported in percent, TTFT in seconds, and ProcFPS as online processing throughput.

Figure 5: Effect of (a) retrieval depth and (b) clarification rate on OVO-Bench with Qwen3-VL under oracle user guidance. Bars show ProcFPS and lines show Macro Accuracy.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Analysis	Evaluation population	Backbone	Clarification setting
Main comparison	StreamBench ( $1{,}556$ QA) and OVO-Bench ( $281$ QA), reported separately	LLaVA-OV-1.5, InternVL-3.5, Qwen3-VL	Disabled
Oracle-guided analysis	StreamBench and OVO-Bench pooled ( $1{,}837$ QA)	LLaVA-OV-1.5, InternVL-3.5, Qwen3-VL	Oracle guidance after triggering
Generated-question analysis	Ambiguous QA from StreamBench and OVO-Bench, pooled ( $302$ QA)	Qwen3-VL	No clarification, Generated Clarification (random-gated, entropy-gated, and all-query), Direct Retrieval, and Oracle Guidance
Pre-answer scoring	OVO-Bench ( $281$ QA)	Qwen3-VL	Disabled
TP-KV hyperparameters	OVO-Bench ( $281$ QA)	Qwen3-VL	Disabled
Matched-6K comparison	StreamBench ( $1{,}556$ QA) and OVO-Bench ( $281$ QA), reported separately	Qwen3-VL	Disabled

Appendix

Table 8: Dataset, backbone, and clarification scope of the main experiments and ablations.

Dataset	Backbone	Selected baseline	Difference	Paired 95% CI
StreamBench	LLaVA-OV-1.5	StreamChat	$+0.2$	$[-2.4,+2.7]$
StreamBench	InternVL-3.5	StreamBridge	$+1.1$	$[-1.4,+3.6]$
StreamBench	Qwen3-VL	StreamingVLM	$+3.3$	$[+1.0,+5.7]$
OVO-Bench	LLaVA-OV-1.5	StreamBridge	$+8.2$	$[+2.2,+13.8]$
OVO-Bench	InternVL-3.5	StreamBridge	$+5.7$	$[-1.8,+13.2]$
OVO-Bench	Qwen3-VL	StreamBridge	$+7.8$	$[+2.4,+12.9]$

Appendix

Table 9: Video-level paired Macro Accuracy intervals for the main-table configurations. Differences are StreamClarify minus the selected baseline and follow the one-decimal accuracies reported in Table 2 . Differences and interval endpoints are percentage points. Intervals are computed from paired question-level correctness and 100,000 whole-video bootstrap replicates.

Judge	w/o	Direct	Gen. all	Gen. gated	Oracle	Gated gain (95% CI)
Qwen3-14B	22.5	22.8	34.8	30.8	38.4	$+8.3\ [4.2,12.5]$
GPT-5	20.5	22.2	37.4	31.1	42.4	$+10.6\ [6.5,14.9]$
Gemini 3.6 Flash	29.1	26.5	43.4	37.1	50.7	$+8.0\ [3.3,12.8]$

Appendix

Table 10: Evaluator sensitivity in the generated-question analysis. Accuracies are percentages. Gains and interval endpoints are percentage points. Gain compares the entropy-gated condition with w/o Clarification. Intervals use paired whole-video resampling.

Method	Main-comparison setting	Matched-6K setting
StreamChat	No persistent visual KV cache. At each query, its native retrieval selects frames from a CPU-side archive to form a fresh visual context.	The archive and retrieval procedure are unchanged, while the retrieved visual context supplied to the backbone is limited to 6K.
StreamBridge	Visual embeddings are accumulated in a memory buffer and progressively compressed. When a question arrives, the current buffer is used to prefill the backbone under its native 16K context limit.	The same query-time prefilling procedure, with the embedding buffer compressed to fit the 6K active-context limit.
StreamingVLM	A sliding window retaining 512 attention-sink tokens and up to 6,192 recent visual tokens, for 6.7K tokens in total.	The same sliding-window design with 512 attention-sink tokens and up to 5,632 recent visual tokens, for 6K tokens in total.
StreamClarify	TP-KV with a 10K-token budget, a 20 s full-granularity recent window, and compression ratio $\rho=0.25$ .	The same TP-KV policy under a 6K budget, preserving the 20 s recent window and the compression ratio $\rho=0.25$ .

Appendix

Table 13: Qwen3-VL memory settings in the main comparison and the matched-6K comparison. Active context denotes the visual context available to the backbone for answering. The 6K limit does not include CPU archive storage.

Streaming VQA

See all 8 papers

Oct 15, 2026

ProV-Mem: Proactive Memory for Streaming Video Question Answering

Proactive streaming video question answering requires a model to continuously monitor an incoming stream and autonomously decide when to speak, what to say, and whether to emit a response. Existing streaming systems face a fundamental difficulty: retaining long visual histories is costly, whereas discarding them can lead to missed events and repeated responses. We propose ProV-Mem, a training-free proactive memory framework built around a frozen VLM backbone. ProV-Mem decomposes the proactive decision into three lightweight memory tiers: a temporal-trigger memory controls when to query the backbone and recovers from prolonged silence, an instructional memory guides what to say through a fixed prompting prior for concise and visually grounded generation, and an episodic memory determines whether to emit by filtering near-duplicate responses. Under an idealized conditional-sufficiency assumption, we derive a stagewise approximation bound relating the decomposed process to a full-state joint policy. We further introduce Cost-Aware Area Under the Curve (CAUC), which extends PAUC by explicitly discounting redundant, overly frequent, and unnecessarily long emissions. Experiments on the four scenarios of ProactiveVideoQA show that ProV-Mem achieves the best CAUC across all scenarios and the best PAUC on WEB, EGO, and TV, while maintaining low per-response inference latency.

Bingbing Wang, Jing Li, Han Gao +3

Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China · Research Institute for Generative AI, The Hong Kong Polytechnic University, Hong Kong, China · Tencent Inc., China +1

Streaming VQA Video Question Answering Memory-Augmented VLMs

Ask PDF

Oct 20, 2026

PACE: Plan, Attend, Check, and Emit for Training-Free Response Timing in Streaming Video Understanding

Streaming video understanding requires a model to decide not only what to answer but also when to answer. Evidence arrives progressively, and recent benchmarks identify premature responses, issued before the queried event has concluded, as a recurring failure mode. Existing approaches often learn response timing through additional training, while training-free alternatives typically rely on scalar gating signals that do not explicitly represent when an answer becomes final. We organize chunk-wise streaming errors into three patterns: sampling variation can make an answer unstable, a stable answer may lack support in the observed video, or an answer may be supported by the current evidence while the event remains unfinished. We propose PACE (Plan, Attend, Check, and Emit), a training-free framework designed around these three failure modes. Given a question, PACE drafts a plan specifying what to observe and what will signal completion. As the stream unfolds, it invokes an off-the-shelf video LLM at informative moments, evaluates each draft for stability, grounding in its cited frames, and completion, and records verified observations in an event ledger. Experiments on ProactiveVideoQA and OVO-Bench Forward show that PACE improves over the official protocols by up to 5.9 average PAUC points and 6.2 average accuracy points across proprietary and open-source backbones, yielding state-of-the-art performance.

Siqi Guo, Yuxia Chen, Wenbo Zhu +4

Nanyang Technological University · University of Electronic Science and Technology of China · University of California, Berkeley +3

Streaming VQA Streaming Video Understanding Video Question Answering

Ask PDF

Sep 29, 2026

Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning

Streaming video assistance requires models to answer asynchronous questions from an observed prefix under a fixed context budget. Existing approaches model response timing or compress history, but an online state formed before future questions are known can omit visual details before later questions reveal their relevance; the retained state alone cannot recover them. We introduce Watch-Think-Interact (WTI), a closed-loop framework for multi-question streaming video reasoning. WTI maintains compact natural-language memory entries tagged with source-video time ranges; these entries support direct reasoning when sufficient and otherwise anchor selective recall of finer visual evidence. For each question, WTI answers when current context and memory suffice, continues watching when required evidence has not appeared, or recalls a relevant past interval and decides again after incorporating the returned chunks, without replaying the full observed history. To train this behavior, we construct WTI-82K, comprising 82,335 timed questions across 4,812 causally aligned trajectories, and develop Stream-GDPO to optimize complete multi-question streaming rollouts using trajectory-level feedback for response timing, source-video recall, and memory updates. WTI achieves state-of-the-art aggregate performance among the compared open-source streaming baselines, reaching 83.3% on StreamingBench and 73.6% weighted overall accuracy on OVO-Bench.

Ziheng Huang, Yicheng Bao, Xueheng Li +8

East China Normal University · University of Science and Technology of China · Xiamen University

Streaming Video Understanding Video Question Answering Video Reasoning Memory-Augmented Video Understanding Long-Video QA RL for Video Reasoning

Ask PDF

Abstract

Streaming VQA

Explore similar work

ProV-Mem: Proactive Memory for Streaming Video Question Answering

PACE: Plan, Attend, Check, and Emit for Training-Free Response Timing in Streaming Video Understanding

Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning