StreamReady: Learning What to Answer and When in Long Streaming Videos

Shehreen Azad; Vibhav Vineet; Yogesh Singh Rawat

cs.CVMar 9, 2026

StreamReady: Learning What to Answer and When in Long Streaming Videos

Authors: Shehreen Azad, Vibhav Vineet, Yogesh Singh Rawat

Organizations: Center for Research in Computer Vision, University of Central Florida · Microsoft Research

Abstract

Streaming video understanding often involves time-sensitive scenarios where models need to answer exactly when the supporting visual evidence appears: answering before the evidence reflects speculation, answering after it has passed reduces real-time utility. To capture this behavior, we introduce a readiness-aware formulation of streaming video understanding with the Answer Readiness Score (ARS), a timing-aware objective with asymmetric early and late penalties. When combined with correctness, ARS defines an effective accuracy that measures not just whether a model is right, but whether it answers at the appropriate moment. Building on this formulation, we introduce StreamReady, a framework to unify temporal reasoning with on-time answering through a lightweight readiness mechanism that decides if sufficient evidence has been observed before responding. To evaluate this capability, we further introduce ProReady-QA, a benchmark with annotated answer evidence windows and proactive multi-turn questions across local and global contexts. StreamReady achieves superior performance on ProReady-QA, and consistently outperforms prior methods across eight additional streaming and offline long-video benchmarks, demonstrating robust and broadly generalizable video understanding capability.

Figures & tables

Figure 2: Framework Overview. StreamReady encodes streaming videos into a visual memory tree and reasons through short and long-term branches. A learnable <RDY> token, guided by a readiness head, gates the reasoning output until sufficient evidence is observed. Once ready, the long-term representation, enriched with contextual information from past QA pairs, is sent to the LLM for answering, enabling readiness-aware streaming behavior.

Method	Size	SSR		CRR		REC		GSD		CTD		Average
Method	Size	Acc.	ARS	Acc.	ARS	Acc.	ARS	Acc.	ARS	Acc.	ARS	Acc.	ARS	Acc e
Offline Video MLLMs
InternVL2 [ 8 ]	8B	57.2	0.17	51.7	0.39	24.3	0.43	29.4	0.39	31.8	0.49	38.9	0.37	0.20
LLaVA-OneVision [ 22 ]	7B	71.8	0.48	58.2	0.53	22.3	0.45	39.4	0.24	34.8	0.18	45.3	0.38	0.29
Qwen-2-VL [ 41 ]	7B	67.9	0.32	53.1	0.39	20.7	0.31	35.1	0.52	30.3	0.28	41.4	0.34	0.20
LLaVA-NeXT-Video [ 60 ]	7B	67.4	0.47	58.3	0.41	24.3	0.42	40.3	0.31	21.8	0.38	42.4	0.40	0.31

Table 2: Performance comparison of readiness-aware streaming understanding on ProReady-QA. †: Qwen-2-VL backbone. Metrics include accuracy (Acc.), Answer Readiness Score (ARS), effective accuracy (Acc e ). Best and second-best performances are highlighted.

Table 3: Performance comparison of streaming video understanding. †: Qwen-2-VL backbone. Accuracy is the reported metric.

Open-source Offline Video MLLMs
Method	Size	VidMME	MLVU	MVB	EgoSch
VideoChat-GPT [ 29 ]	7B	-	31.3	32.7	49.6
LLaMA-VID [ 26 ]	7B	33.2	41.9	38.5	38.5
InternVL2 [ 8 ]	8B	54.0	64.0	65.8	55.0
LongVA [ 59 ]	7B	52.6	56.3	51.3	46.7
LLaVA-OneVision [ 22 ]	7B	58.2	64.7	56.7	60.1

Table 4: Performance comparison of offline long-video understanding. †: Qwen-2-VL backbone. Accuracy is reported.

Panel of Table 5 — Table 5: Ablation studies for each component.

Method	ProReady-QA						StreamingBench		Vid-MME
	REC		GSD		CTD		Real	Context.	Long
	Acc.	ARS	Acc.	ARS	Acc.	ARS	Acc.	Acc.	Acc.
Contribution of Each Memory Bank
Short-Term Memory ( $\mathcal{M}_{\text{V1}}$ )	12.7	0.51	24.1	0.56	31.7	0.49	41.3	12.4	36.7
+ Long-Term Memory ( $\mathcal{M}_{\text{V2}},\mathcal{M}_{\text{V3}}$ )	37.6	0.66	55.3	0.68	40.7	0.58	69.4	32.1	62.6
+ Contextual Memory ( $\mathbf{\mathcal{M}_{\text{C}}}$ )	39.6	0.68	61.2	0.68	43.5	0.59	78.3	48.2	N/A

Table 6: Additional architectural ablation studies on ProReady-QA, StreamingBench, and VideoMME long.

Figure 7: Effect of level capacity and depth of visual memory tree on overall performance. When varying the capacity of a specific level, the capacities of all other levels are held fixed. For the 2-level variant, the centroid and prototype capacities are merged into a single long-term level. For the 4-level variant, the prototype capacity is split across two abstraction layers to create two prototype levels. The selected capacity/depth is highlighted.

Figure 8: Effect of number of query-aware retrieval slots on ProReady-QA.

Method	Token	Acc.	Lat.
Full memory	108	56.4	8.7s
Single-stage	32	40.7	3.2s
Hierarchical	32	56.6	3.1s

Table 7: Token retrieval & reasoning strategies under identical budget.

Method	ProReady-QA		StrmB	Vid-MME
Method	Acc.	ARS	Acc.	Acc.
Oryx-1.5-7B	53.9 ( $\uparrow 14.4$ )	0.64 ( $\uparrow 0.38$ )	63.6 ( $\uparrow 15.2$ )	64.3 ( $\uparrow 3.8$ )
LLaVA-OV 7B	55.2 ( $\uparrow 9.8$ )	0.65 ( $\uparrow 0.27$ )	60.3 ( $\uparrow 12.9$ )	63.8 ( $\uparrow 5.6$ )
Qwen-2-VL 7B	56.4 ( $\uparrow 15$ )	0.69 ( $\uparrow 0.35$ )	63.4 ( $\uparrow 18.2$ )	65.8 ( $\uparrow 2.5$ )

Table 8: Effect of changing backbone of StreamReady.

Methods and Timing Supervision	Acc.	ARS
StreamBridge [ 40 ] w/ BCE ‡	53.1	0.60
StreamBridge [ 40 ] w/ Contrastive	53.2	0.63
StreamReady w/o timing sup.	56.3	0.58
StreamReady w/ BCE	56.4	0.65
StreamReady w/ Contrastive ‡	56.4	0.69

Table 9: Effect of Readiness Supervision using ProReady-QA. ‡ denotes original setup

Figure 9: Temporal characteristics of ARS in relation to PAUC [ 44 ] .

Figure 10: Qualitative analysis of readiness-aware streaming understanding on ProReady-QA. StreamReady shows superior performance by consistently giving accurate and on-time answers.

Method	DVC	SLC	TVG
Qwen2-VL	22.6	13.2	25.3
Dispider	33.8	18.8	36.1
StreamBridge	38.3	22.6	34.3
StreamReady	43.4	24.1	36.8

Table 10: Dense task performance (F1-score) comparison on ET-Bench.

Figure 11: Effect of readiness threshold (ProReady-QA).

Figure 12: Readiness failure in counting task.

Task	Dataset	#QA Pairs
VQA	MSRVTT-QA [ 49 ]	10k
	MSVD-QA [ 49 ]	10k
	ActivityNet-QA [ 54 ]	32k
	MovieChat-1k [ 38 ]	10k

Table 11: Dataset Statistics for training the reasoning module and readiness mechanism

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 124 papers

Don't Pause! Every prediction matters in a streaming video
Apr 27, 2026Dibyadip Chatterjee, Zhanzhong Pang, Fadime Sener +2Streaming Video Understanding Efficient VLM Inference
National University of Singapore · Google Inc.
Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning
Sep 29, 2026Ziheng Huang, Yicheng Bao, Xueheng Li +8Streaming Video Understanding Video QA
East China Normal University · University of Science and Technology of China · Xiamen University
Have I Seen Enough? Frozen Video-Language Models Encode Evidence Readiness
Oct 6, 2026Dan Ben-Ami, Kobi Cohen, Chaim BaskinStreaming Video Understanding Video-Language Model Evaluation
INSIGHT Lab, Ben-Gurion University of the Negev, Israel · Ben-Gurion University of the Negev, Israel

Apr 27, 2026J/K move · Enter open · S save

Don't Pause! Every prediction matters in a streaming video

Dibyadip Chatterjee, Zhanzhong Pang, Fadime Sener, Yale Song+1

National University of Singapore · Google Inc.

Streaming Video Understanding Efficient VLM Inference Video QA Adaptive Computation Streaming Inference

Open paper AskPDF arXiv

Streaming video models should respond the moment an event unfolds, not after the moment has passed. Yet existing online VideoQA benchmarks remain largely retrospective. They pause the video at fixed timestamps, pose questions about current or past events, and score models only at those moments. This protocol leaves streaming predictions untested. To close this gap, we introduce SPOT-Bench, featuring multi-turn proactive queries that evaluate general streaming perception and assistive capabilities required by an always-on, real-time assistant. SPOT-Bench comes with Timeliness-F1, a consolidated metric that measures streaming predictions by their temporal precision and balanced coverage across the entire video. Our benchmark reveals: (i) offline models detect events reliably but spam predictions unprompted; (ii) post-training for silence reduces spamming but induces unresponsiveness; (iii) half of the streaming video expects no response, which we term dead-time - compute spent here does not affect response latency. These findings motivate AsynKV, a training-free streaming adaptation of offline models, that retains their event perception while improving their streaming behavior. AsynKV features a long-short term memory, utilized efficiently by scaling compute during dead-time. It serves as a strong baseline on SPOT-Bench, outperforming existing streaming models, and achieves state-of-the-art on retrospective benchmarks.

Abstract

Streaming Video Understanding

Explore similar work

Don't Pause! Every prediction matters in a streaming video

Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning

Have I Seen Enough? Frozen Video-Language Models Encode Evidence Readiness