STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding

Junho Kim; Hosu Lee; James M. Rehg; Minsu Kim; Yong Man Ro

cs.CVMar 29, 2026

STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding

Authors: Junho Kim, Hosu Lee, James M. Rehg, Minsu Kim, Yong Man Ro

Organizations: UIUC · KAIST · Google DeepMind

Abstract

Recent progress in video large language models (Video-LLMs) has enabled strong offline reasoning over long and complex videos. However, real-world deployments increasingly require streaming perception and proactive interaction, where video frames arrive online and the system must decide not only what to respond, but also when to respond. In this work, we revisit proactive activation in streaming video as a structured sequence modeling problem, motivated by the observation that temporal transitions in streaming video naturally form span-structured activation patterns. To capture this span-level structure, we model activation signals jointly over a sliding temporal window and update them iteratively as new frames arrive. We propose STRIDE (Structured Temporal Refinement with Iterative DEnoising), which employs a lightweight masked diffusion module at the activation interface to jointly predict and progressively refine activation signals across the window. Extensive experiments on diverse streaming benchmarks and downstream models demonstrate that STRIDE shows more reliable and temporally coherent proactive responses, significantly improving when-to-speak decision quality in online streaming scenarios.

Figures & tables

Figure 1: Overview of STRIDE , which operates in a streaming setting where frames arrive online. A lightweight activation model based on masked diffusion maintains an activation region over a sliding temporal window and iteratively denoises masked activation states to predict a coherent trigger segment. A trigger is issued only if an active span is sustained for a predefined span ratio. When activation is triggered, the accumulated frame context is forwarded to a downstream Video-LLM to generate the response.

Figure 2: Activation modeling and inference stage of STRIDE . Training applies sequence duplication and three masking strategies (boundary-anchored masking, span unmasking, full masking). During inference, the activation window slides with incoming frames, retaining confident past decisions while selectively re-masking and progressively denoising uncertain positions.

Method	# of Frames	Real-Time Visual Perception							Backward Tracing				Forward Act. Responding				Overall
Method	# of Frames	OCR	ACR	ATR	STU	FPD	OJR	Avg.	EPM	ASI	HLD	Avg.	REC	SSR	CRR	Avg.	Avg.
Human
Human	-	93.96	92.57	94.83	92.70	91.09	94.02	93.20	92.59	93.02	91.37	92.33	95.48	89.67	93.56	92.90	92.81
Proprietary Models (Offline), Single-Turn Evaluation
Gemini 1.5 Pro Reid et al. (2024)	1 FPS	85.91	66.97	79.31	58.43	63.37	61.96	69.32	58.59	76.35	52.64	62.54	35.53	74.24	61.67	57.15	63.00
GPT-4o OpenAI (2024)	64	69.80	64.22	71.55	51.12	70.30	59.78	64.46	57.91	75.68	48.66	60.75	27.58	73.21	59.40	53.40	59.54

Table 1: Evaluation results on OVO-Bench Niu et al. (2025) . Baseline-AR uses autoregressive binary prediction. Offline models follow the original single-turn protocol with segmented clips, whereas streaming methods process frames sequentially.

Method	# of Frames	Real-Time Visual Understanding											Omni-Source Understanding					Contextual Understanding					Overall
Method	# of Frames	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT	Avg.	ER	SCU	SD	MA	Avg.	ACU	MCU	SQA	PO	Avg.	Avg.
Human
Human	-	89.47	92.00	93.60	91.47	95.65	92.52	88.00	88.75	89.74	91.30	91.46	88.00	88.24	93.60	90.27	90.26	88.80	90.40	95.00	100	93.55	91.66
Proprietary Models (Offline)
Gemini 1.5 pro Reid et al. (2024)	1 FPS	79.02	80.47	83.54	79.67	80.00	84.74	77.78	64.23	71.95	48.70	75.69	46.80	39.60	74.90	80.00	60.22	51.41	40.73	54.80	45.10	48.73	67.07
GPT-4o OpenAI (2024)	64	77.11	80.47	83.91	76.47	70.19	83.80	66.67	62.19	69.12	49.22	73.28	41.20	37.20	43.60	56.00	44.50	41.20	38.40	32.80	56.86	38.70	60.15

Table 2: Evaluation results on StreamingBench Lin et al. (2024b) . Baseline-AR uses autoregressive binary prediction. Offline models follow the original single-turn protocol with segmented clips, whereas streaming methods process frames sequentially.

Table 5

Figure 3: Activation transition frequency results around event boundaries on ET-Bench TVG. Baseline-AR model shows frequent oscillations near boundaries, whereas STRIDE produces more robust activation spans.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Videos	Items	Single	Multi	Segs
ActivityNet-Captions Caba Heilbron et al. [2015]	10,009	37,421	37,421	–	37,421
LITA Huang et al. [2024b]	10,000	32,489	32,489	–	32,489
YouCook2 Zhou et al. [2018]	1,333	11,480	10,337	1,143 ( $\times$ 7.7)	19,174
ET-Instruct Liu et al. [2024b]	91,121	136,072	71,966	64,106 ( $\times$ 5.1)	398,609
Charades Sigurdsson et al. [2016]	7,811	48,684	48,067	617 ( $\times$ 2.1)	49,374
CharadesEgo Sigurdsson et al. [2018]	6,158	61,575	57,828	3,747 ( $\times$ 2.0)	65,488

Appendix

Table 6: Training dataset statistics. Videos : number of source videos; Items : total annotation entries; Single : single-event entries; Multi : multi-event entries (average event count in parentheses); Segs : total training segments.

Config	Value
Input frames	1 FPS
LR scheduler	Linear warm-up with cosine decay
Warmup steps	512
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.999$ )
Global Batch size	256
Learning rate (lang. head)	$3\times 10^{-5}$

Appendix

Table 7: Detailed training hyperparameters for STRIDE .

Figure 5: Sensitivity of STRIDE to the retention constant $\tau$ across five temporal understanding tasks (TVG, EPM, TAL, DVC, SLC) in ET-Bench Liu et al. [2024b] . The $y$ -axis shows the score difference relative to each task’s average. Task-wise average scores are shown in the legend.

Method	Real-Time Visual Perception							Backward Tracing				Fwd. Act. Responding				Overall
Method	OCR	ACR	ATR	STU	FPD	OJR	Avg.	EPM	ASI	HLD	Avg.	REC	SSR	CRR	Avg.	Avg.
Gemma3-4B Team et al. [2025]	65.8	48.6	56.0	36.0	66.3	50.0	53.78	44.4	41.9	3.2	29.83	14.4	61.4	52.5	42.77	42.13
+ STRIDE -2B	73.2	60.6	64.7	39.3	71.3	56.5	60.93	47.8	52.0	4.8	34.87	42.6	64.6	60.0	55.73	50.51
+ STRIDE -4B	75.2	56.9	67.2	40.4	67.3	56.5	60.58	51.5	48.0	4.3	34.60	46.5	66.2	60.0	57.57	50.92
InternVL3-8B Wang et al. [2025b]	65.8	52.3	68.1	51.1	71.3	62.0	61.77	58.9	66.9	9.7	45.17	36.6	64.1	43.3	48.00	51.64
+ STRIDE -2B	75.8	54.1	80.2	56.7	74.3	65.2	67.72	58.9	65.5	11.3	45.23	40.1	67.7	66.2	58.00	56.98

Appendix

Table 8: Scalability analysis on activation backbone scale ( STRIDE -2B vs. 4B) on OVO-Bench Niu et al. [2025] across multiple downstream Video-LLMs.

Method	Real-Time Visual Understanding											Omni-Source Understanding					Contextual Understanding					Overall
Method	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT	Avg.	ER	SCU	SD	MA	Avg.	ACU	MCU	SQA	PO	Avg.	Avg.
Gemma3-4B Team et al. [2025]	63.8	69.5	68.8	54.4	60.2	65.1	59.3	40.7	62.7	21.8	57.49	28.8	31.2	30.4	46.4	34.20	34.4	31.2	38.8	12.4	29.20	40.30
+ STRIDE -2B	66.8	71.9	66.6	57.2	66.5	70.7	60.2	43.1	65.0	23.8	60.00	35.6	31.6	36.0	44.0	36.80	33.6	35.2	44.8	41.6	38.80	45.20
+ STRIDE -4B	66.2	63.3	68.1	58.1	67.1	71.7	63.0	41.5	66.7	21.2	59.93	32.8	30.4	36.0	46.4	36.40	35.2	36.8	48.0	44.0	41.00	45.78
InternVL3-8B Wang et al. [2025b]	74.9	82.0	75.7	61.2	72.0	67.6	74.1	66.7	78.1	34.2	68.71	40.4	27.6	38.8	45.6	38.10	38.0	26.0	36.8	31.2	33.00	46.60
+ STRIDE -2B	74.9	78.9	76.7	68.6	77.0	77.3	77.8	71.5	83.0	33.2	72.45	39.6	22.4	44.0	50.8	39.20	34.0	35.2	43.2	42.8	38.80	50.15

Appendix

Table 9: Scalability analysis on activation backbone scale ( STRIDE -2B vs. 4B) on StreamingBench Lin et al. [2024b] across multiple downstream Video-LLMs.

Panel of Figure 6 — Figure 6: Qualitative example from ET-Bench (TVG).

Panel of Figure 8 — Figure 8: Qualitative example from ET-Bench (EPM).

Panel of Figure 9 — Figure 9: Qualitative example from ET-Bench (TAL).

Panel of Figure 10 — Figure 10: Qualitative example from ET-Bench (DVC).

Panel of Figure 11 — Figure 11: Qualitative example from ET-Bench (SLC).

Panel of Figure 12 — Figure 12: Qualitative example from OVO-Bench (EPM).

Panel of Figure 13 — Figure 13: Qualitative example from OVO-Bench (OJR).

Panel of Figure 14 — Figure 14: Qualitative example from OVO-Bench (OJR).

Panel of Figure 15 — Figure 15: Qualitative example from StreamingBench (Attribute Recognition).

Panel of Figure 16 — Figure 16: Qualitative example from StreamingBench (Object Recognition).

Panel of Figure 17 — Figure 17: Qualitative example from StreamingBench (Text-Rich Understanding).

Panel of Figure 18 — Figure 18: Failure Case from StreamingBench (Proactive Output).

Panel of Figure 19 — Figure 19: Failure Case from StreamingBench (Scene Understanding).

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 124 papers

Don't Pause! Every prediction matters in a streaming video
Apr 27, 2026Dibyadip Chatterjee, Zhanzhong Pang, Fadime Sener +2Streaming Video Understanding Efficient VLM Inference
National University of Singapore · Google Inc.
Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding
Mar 19, 2026Yikai Zheng, Xin Ding, Yifan Yang +6Streaming Video Understanding Efficient VLM Inference
Institute for AI Industry Research (AIR), Tsinghua University · University of Science and Technology of China · 4Microsoft Research Asia +1
LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams
Jun 16, 2026Zhenyu Yang, Kairui Zhang, Bing Wang +2Streaming Video Understanding Long-Video Understanding

Apr 27, 2026J/K move · Enter open · S save

Don't Pause! Every prediction matters in a streaming video

Dibyadip Chatterjee, Zhanzhong Pang, Fadime Sener, Yale Song+1

National University of Singapore · Google Inc.

Streaming Video Understanding Efficient VLM Inference Video QA Adaptive Computation Streaming Inference

Open paper AskPDF arXiv

Streaming video models should respond the moment an event unfolds, not after the moment has passed. Yet existing online VideoQA benchmarks remain largely retrospective. They pause the video at fixed timestamps, pose questions about current or past events, and score models only at those moments. This protocol leaves streaming predictions untested. To close this gap, we introduce SPOT-Bench, featuring multi-turn proactive queries that evaluate general streaming perception and assistive capabilities required by an always-on, real-time assistant. SPOT-Bench comes with Timeliness-F1, a consolidated metric that measures streaming predictions by their temporal precision and balanced coverage across the entire video. Our benchmark reveals: (i) offline models detect events reliably but spam predictions unprompted; (ii) post-training for silence reduces spamming but induces unresponsiveness; (iii) half of the streaming video expects no response, which we term dead-time - compute spent here does not affect response latency. These findings motivate AsynKV, a training-free streaming adaptation of offline models, that retains their event perception while improving their streaming behavior. AsynKV features a long-short term memory, utilized efficiently by scaling compute during dead-time. It serves as a strong baseline on SPOT-Bench, outperforming existing streaming models, and achieves state-of-the-art on retrospective benchmarks.

Abstract

Streaming Video Understanding

Explore similar work

Don't Pause! Every prediction matters in a streaming video

Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding

LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams