When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs

Xiang Hu; Jiazuo Yu; Lu Zhang; Yunzhi Zhuge; Huchuan Lu

cs.CVSep 29, 2026

When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs

Authors: Xiang Hu, Jiazuo Yu, Lu Zhang, Yunzhi Zhuge, Huchuan Lu

Organizations: IIAU Lab, Dalian University of Technology

Abstract

Streaming video understanding requires Video Large Language Models (Video-LLMs) to reason over continuous visual streams under causal constraints. As the visual history grows, a bounded visual?processing budget requires evidence selection that balances temporal recency with query relevance. Recent-only selection excludes potentially relevant historical evidence, whereas Semantic-only retrieval can displace useful recent context when relevance scores are ambiguous. We introduce WRWS (When to Retrieve, When to Stay), a training-free framework for uncertainty-adaptive evidence allocation. A lightweight external vision-language encoder scores query relevance across the observed history, while an adaptive allocation module uses the normalized entropy of the similarity distribution as a proxy for retrieval uncertainty. WRWS favors semantic retrieval when relevance cues are reliable and strengthens the recency prior under uncertainty. Following a retrieve-first, encode-later pipeline, WRWS selects evidence before target-model visual encoding, such that only the selected observations are processed by the costly target Video-LLM. Experiments across four Video-LLM families and multiple model scales demonstrate competitive accuracy on StreamingBench and OVO-Bench. In our efficiency evaluation, WRWS reduces average vision-to-answer time to 47.93% of the state-of-the-art method. Code will be released.

Figures & tables

Figure 1 : Motivation and key idea of WRWS. Under a bounded target-model visual-processing budget, (a) Recent-only selection may lose useful earlier evidence, while (b) Semantic-only retrieval may displace useful recent context when similarity is ambiguous. (c) WRWS adapts the semantic–recency trade-off to retrieval uncertainty and follows retrieve first, encode later , selecting evidence with lightweight representations before expensive target-model visual encoding.

Figure 3 : Overview of WRWS. WRWS uses lightweight representations to estimate query–observation relevance and retrieval uncertainty over the causally available history. The uncertainty adaptively balances semantic relevance and temporal recency to rank observations under a fixed Top- $K$ budget. Only the selected observations are then encoded by the target Video-LLM, realizing our retrieve-first, encode-later strategy. The lower-right panel illustrates how peaked and diffuse relevance distributions shift evidence allocation toward semantics and recency, respectively.

Model	#Frames	StreamingBench	OVO-Bench
Model	#Frames	Real-Time	Backward	Real-Time	Forward	Avg.
Human	–	91.46	92.33	93.20	92.90	92.81
Proprietary MLLMs
Gemini 1.5 Pro ( Team et al., 2024 )	1 fps	75.69	62.54	69.32	57.15	63.00
GPT-4o ( Hurst et al., 2024 )	64	73.28	60.75	64.46	53.40	59.54
Claude 3.5 Sonnet ( Anthropic, 2024 )	20	72.44	–	–	–	–

Table 2 : Performance comparison (%) on StreamingBench and OVO-Bench. Methods prefixed with “+” are applied to the corresponding base model.

\dagger

denotes results reproduced following the official benchmark protocol. Gray rows denote WRWS. Best results are shown in bold .

Figure 5 : Understanding adaptive evidence allocation. (a) The relative utility of Recent-only and Semantic-only varies across temporal tasks. (b) When WRWS departs from Recent-only, lower retrieval uncertainty is associated with larger adjustments and a higher semantic share among the introduced evidence. Error bars denote 95% within-task stratified bootstrap confidence intervals.

Allocation	Qwen2.5	Qwen3
$\alpha=0.00$ (Semantic-only)	48.46	51.76
$\alpha=0.50$	52.84	56.15
$\alpha=1.00$ (Recent-only)	55.73	58.10
Adaptive †	55.98	58.59

Table 4 : Ablation studies of WRWS.

\dagger

denotes the default setting. Best results are shown in bold .

Metric	Target-VLM visual encoding	Other pipeline overhead	Inference TTFT	Total VTAT
Time (s)	7.596	4.342	0.243	12.181
Fraction (%)	62.36	35.65	1.99	100.00

Table 5 : Breakdown of VTAT for Qwen2.5-VL-7B under the encode-first baseline. Target-VLM visual encoding accounts for 62.36% of total VTAT. Other pipeline overhead is the residual after subtracting target-VLM visual encoding and inference TTFT from VTAT, which includes processing not captured by those two timers.

Model	Method	Streaming	Backward	Real-Time	Forward	B+R	All
	HERMES	81.32	46.78	73.21	–	60.00	–
Qwen3-VL-8B	WRWS	80.39	50.76	80.60	44.17	65.68	58.51
	HERMES	78.40	54.00	71.90	–	62.95	–
Qwen3-VL-4B	WRWS	80.07	52.24	78.64	41.26	65.44	57.38

Table 8 : Comparison with HERMES under the 2-fps online evaluation setting. HERMES uses a 6K visual-token budget. WRWS selects

K=6

observations under its default Qwen3-VL streaming configuration. “B+R” denotes the average over the Backward and Real-Time groups. “–” indicates results not reported by HERMES. Gray rows denote WRWS.

Model	Method	$K$	Streaming	Backward	Real-Time	Forward	B+R	All
	WRWS	2	78.83	60.46	83.03	42.91	71.74	62.13
	WRWS	4	83.51	62.17	85.33	45.26	73.75	64.25
	WRWS	6	83.71	62.35	84.94	45.77	73.59	64.35
	WRWS	8	83.67	62.35	84.62	47.33	73.49	64.77
30B-A3B	WRWS	16	84.31	64.53	81.29	50.41	72.91	65.41

Table 10 : Performance under different target-model visual-processing budgets. We vary the number of selected observations

K

for Qwen3-VL-30B-A3B. “B+R” denotes the average over the Backward and Real-Time groups. The gray row denotes the default setting (

K=6

Model	Method	$K$	Streaming	Backward	Real-Time	Forward	B+R	All
Qwen2.5-VL
	SimpleStream	4	77.83	48.50	74.64	43.19	61.57	55.44
3B	WRWS	4	78.07	47.27	74.56	45.21	60.91	55.68
	SimpleStream	4	80.31	49.03	80.58	42.05	64.81	57.22
32B	WRWS	4	80.23	49.28	80.13	42.25	64.65	57.22
	SimpleStream	16	84.31	61.46	79.52	42.07	70.49	61.02

Table 12 : Comparison with SimpleStream across model scales. For each target model, we compare against the best reported SimpleStream configuration, which may use a model-specific evidence budget

K

. WRWS keeps

K=4

for all Qwen2.5-VL models and

K=6

for all Qwen3-VL models. “B+R” denotes the average over the Backward and Real-Time groups. Gray rows denote WRWS.

Model	Allocation	Backward	Real-Time	Forward	B+R	All
	$\alpha=0.00$ (Semantic-only)	47.12	59.08	39.17	53.10	48.46
	$\alpha=0.25$	47.97	64.46	40.72	56.21	51.05
	$\alpha=0.50$	50.22	68.34	39.96	59.28	52.84
	$\alpha=0.75$	49.97	73.95	38.58	61.96	54.16
	$\alpha=1.00$ (Recent-only)	50.43	77.20	39.56	63.82	55.73
Qwen2.5-VL-7B	Adaptive	50.59	77.13	40.23	63.86	55.98

Table 14 : Comparison of adaptive and fixed evidence allocation on OVO-Bench. Fixed

\alpha

values interpolate between Semantic-only (

\alpha=0

) and Recent-only (

\alpha=1

). Adaptive denotes the query-adaptive allocation used by WRWS. “B+R” denotes the average over the Backward and Real-Time groups. Gray rows denote Adaptive.

Group	# Samples	VTAT (s)			Inference TTFT (s)			GPU Memory (GB)
Group	# Samples	WRWS	SimpleStream	Ratio	WRWS	SimpleStream	Ratio	WRWS	SimpleStream
Overall	3035	5.84	12.18	47.93%	0.2431	0.2429	100.08%	20.76	17.28
Backward	631	6.37	15.56	40.92%	0.2788	0.2793	99.82%
Real-Time	837	7.17	16.50	43.44%	0.2863	0.2866	99.91%
Forward	1567	4.92	8.51	57.74%	0.2057	0.2050	100.35%

Table 17 : Detailed efficiency comparison with SimpleStream on OVO-Bench. VTAT measures the elapsed time from the start of processing the first video observation to the generation of the first answer token, while inference TTFT measures the elapsed time from the start of target Video-LLM inference for answering to the generation of the first answer token. “Ratio” reports WRWS relative to SimpleStream, where a lower VTAT ratio indicates larger latency savings. GPU memory is measured over the full evaluation.

Panel of Figure 6 — Figure 6 : Qualitative examples of adaptive evidence allocation. (a) When earlier observations are more relevant to the query, WRWS retrieves historical evidence while keeping recent context. (b) When the relevant action occurs near the query time, WRWS shifts the evidence budget toward recent observations. WRWS answers both queries correctly.

Panel of Figure 7 — Figure 7 : Qualitative failure cases. (a) WRWS selects both earlier and recent observations in which the banana is visible, yet the target Video-LLM predicts “apple” instead of “banana”. This suggests that evidence selection alone does not eliminate downstream visual recognition or reasoning errors. (b) When WRWS concentrates its budget on recent observations, the earlier evidence needed to recover the towel’s previous location is not selected, leading to an incorrect answer.

Explore similar work

What Should a Streaming Video Model Remember?
Jun 15, 2026Haonan Ge, Yiwei Wang, Hang Wu +1Streaming Video Understanding Streaming
University of California, Santa Barbara · University of California, Merced · The University of Queensland
StreamFlow: Dynamic Memory Flows for Streaming Video Understanding
Aug 11, 2026Muxin Fu, Yifan Zhang, Wentao Zhang +5Streaming Video Understanding Long-Video Benchmarks
Tongji University · Nanyang Technological University · University of Michigan +2
Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding
Sep 3, 2026Hongyu Qu, Guangming Yao, Ling Xing +7Streaming Video Understanding Modern Data-Streaming Systems
Nanjing University of Science and Technology · Ant Group · National University of Singapore +1

Jun 15, 2026, cs.CVJ/K move · Enter open · S save

What Should a Streaming Video Model Remember?

Haonan Ge, Yiwei Wang, Hang Wu, Yujun Cai

University of California, Santa Barbara · University of California, Merced · The University of Queensland

Streaming Video Understanding Streaming Long-Term Memory

Open paper AskPDF arXiv

Streaming video understanding models must answer queries at any moment during an ongoing stream, using only what they have observed so far and under fixed memory and computation budgets. Existing methods address this by adding memory banks, retrieval modules, or visual token compression to preserve long-range history. However, strong recent-window baselines show that indiscriminate history injection can dilute current-scene perception, suggesting that the key challenge is not whether to use memory, but how to allocate it selectively. We formulate this as budgeted online latent evidence allocation and propose \textbf{SelectStream}, a selective latent-memory framework that keeps the current observation directly visible to a frozen VLM while exposing historical information only through a compact, query-conditioned evidence budget. Three coordinated mechanisms govern when to write, what to preserve, and how to retrieve: surprise-driven adaptive windowing, priority-preserving consolidation, and query-conditioned graph reasoning over a fixed-capacity latent memory graph. Retrieved evidence is calibrated and injected as latent tokens for answer generation, without replaying frames or growing the context with stream length. Experimental results show that SelectStream achieves strong online streaming performance and preserves general video understanding, reaching 82.67% on StreamingBench, 67.03% on OVO-Bench, and 74.4% average accuracy on offline video benchmarks, while outperforming strong recent-window baselines and prior streaming memory methods.

Abstract

Explore similar work

What Should a Streaming Video Model Remember?

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding