DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding

Boyuan Sun; Deshui Miao; Wenzhao Gao; Shaoyong Jia; Shaohui Jiao; Chi Liu; Yan Shu; Derek Li

cs.CVOct 14, 2026

DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding

Authors: Boyuan Sun, Deshui Miao, Wenzhao Gao, Shaoyong Jia, Shaohui Jiao, Chi Liu, Yan Shu, Derek Li, +3 more

Organizations: VCIP, Nankai University · ByteDance Inc. · IQuest Research · Institute of Automation, Chinese Academy of Sciences

Abstract

We propose DeltaStreamer, a novel agentic framework for streaming video understanding inspired by how humans observe, remember, and investigate. Streaming video understanding requires a system to retain evidence before knowing which details future queries will need. Existing approaches often obscure event transitions and omit relevant details when compressing video history, and cannot recover missing visual evidence. DeltaStreamer addresses these limitations by combining change-aware memory with query-driven planning and targeted evidence gathering with replay. Before a query arrives, it continuously observes the stream and explicitly records visual changes in structured memory, preserving how events unfold rather than only what is present. When a query arrives, a query-driven planner identifies the required evidence and guides memory retrieval and targeted replay. By revisiting historical frames, the agent can recover visual details omitted during observation and ground its answer in evidence beyond the retained descriptions. This connects continuous observation with active investigation tailored to each question. Experiments on OVO-Bench and StreamingBench demonstrate improvements over representative streaming video understanding methods, with overall scores of 66.07% and 59.53%, respectively. Project page: https://bbbbchan.github.io/DeltaStreamer/

Figures & tables

Fig. 1 : Comparison of streaming video understanding paradigms. Compression or truncation in KV-cache reuse and multi-turn dialogue can obscure transitions and discard details needed by later queries. DeltaStreamer records explicit changes in structured memory and uses query-driven planning and targeted replay to recover missing visual evidence.

Fig. 2 : Overview of DeltaStreamer. Before the query, adjacent-frame comparison produces delta-augmented observations stored in hierarchical memory. After the query, the planner guides evidence retrieval and tool use, while targeted replay recovers visual details for verification.

Model	Real-Time Visual Perception							Backward Tracing				RT+BT Avg.	Forward Active Responding				Overall
Model	OCR	ACR	ATR	STU	FPD	OJR	Avg.	EPM	ASI	HLD	Avg.	RT+BT Avg.	REC	SSR	CRR	Avg.	Overall
Human
Human Agents	93.96	92.57	94.83	92.70	91.09	94.02	93.20	92.59	93.02	91.37	92.33	92.77	95.48	89.67	93.56	92.90	92.81
Proprietary MLLMs
Gemini 1.5 Pro	85.91	66.97	79.31	58.43	63.37	61.96	69.32	58.59	76.35	52.64	62.54	65.93	35.53	74.24	61.67	57.15	63.00
Gemini 3 Flash	86.58	68.81	79.31	58.99	70.30	70.65	72.44	58.92	76.35	52.15	62.47	67.46	35.53	74.24	61.67	57.15	64.02

Tab. 1 : The experimental results on the OVO-Bench benchmark. RT+BT Avg. is the average over the Real-Time Visual Perception and Backward Tracing groups.

Method	Real-Time Visual Understanding											Omni-Source Understanding					Contextual Understanding					Overall
Method	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT	Avg.	ER	SCU	SD	MA	Avg.	ACU	MCU	SQA	PO	Avg.	Overall
Human
Human	89.47	92.00	93.60	91.47	95.65	92.52	88.00	88.75	89.74	91.30	91.46	88.00	88.24	93.60	90.27	90.26	88.80	90.40	95.00	100.00	93.55	91.66
Proprietary MLLMs
Gemini 1.5 Pro	79.02	80.47	83.54	79.67	80.00	84.74	77.78	64.23	71.95	48.70	75.69	46.80	39.60	74.90	80.00	60.22	51.41	40.73	54.80	45.10	48.73	67.07
GPT-4o	77.11	80.47	83.91	76.47	70.19	83.80	66.67	62.19	69.12	49.22	73.28	41.20	37.20	43.60	56.00	44.50	41.20	38.40	32.80	56.86	38.70	60.15

Tab. 2 : Evaluation results on StreamingBench [ 28 ] .

Setting	Input Frames	Delta Representation	OVO-Bench
1F-NoDelta	$I_{t}$	No Delta	58.87
1F-TextDelta	$I_{t}$	Text-Memory comparison	63.19
2F-NoDelta	$(I_{t-1},I_{t})$	No Delta	61.12
2F-VisualDelta (Ours)	$(I_{t-1},I_{t})$	Visual comparison	66.07

Tab. 3: Ablation of delta-augmented observation. 2F-NoDelta receives the same two-frame input as our method but does not explicitly generate or store a delta field. Comparing the two-frame variants tests explicit change recording at the same visual-input count.

Tab. 4 : Ablation of memory components. All variants use identical observations, tools, planner, and answer decoding; only the component shown in each mini-table is changed.

Panel of Table 8 — Tab. 5 : Planner and component ablations on OVO-Bench. Left: cumulative planner settings, with persistence retaining the plan across post-query rounds. Right: combinations of delta (D), hierarchical memory (M), and planner (P).

Tab. 6 : Ablation of memory rendering and core visual tools. All variants keep the model, observation protocol, the same planner, and answer decoding fixed.

Fig. 4 : Qualitative examples of DeltaStreamer. The upper case recalls the action following the first block placement. The lower case recovers the missing wall letters from a historical crop.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Tool	Description
Memory	memory_search	Cross-layer keyword retrieval over all memory DAG layers
	memory_read	Retrieve specific events by query with time-range filtering
	memory_list	List events filtered by kind and time range, chronologically
	memory_write	Write a model-selected fact into memory
Track	track_init	Initialize a SAM2 object tracker on a box or prompt
	track_query	Query the current position and confidence of a tracked object

Appendix

Tab. 7: Tool suite of DeltaStreamer, grouped by function.

Fig. 5 : Visible memory (a) and tool use (b) on OVO-Bench and StreamingBench.

Metric	OVO-Bench	StreamingBench
Observation before query
Visual observations per sample	70.85	118.31
Observation length (chars.)	1,279.15	1,588.48
Delta length (chars.)	155.54	172.98
Memory at completion
Stored working records	131.91	261.71

Appendix

Tab. 8: Observation, memory, and planning statistics. All values are means.

Fig. 12 : OVO-Bench examples of temporal reasoning, attribute recall, and checking query premises.

Fig. 13 : StreamingBench examples of action sequencing, scene changes, and visual reasoning.

Fig. 14 : Some examples of using replay or historical crops.

Explore similar work

StreamScout: Learning When to Look Deeper for Streaming Video Understanding
Aug 31, 2026Ce Zhang, Jing Bi, Jinxi He +9Adaptive Inference Streaming Video Understanding
Carnegie Mellon University · TikTok · University of Rochester +2
An Efficient Streaming Video Understanding Framework with Agentic Control
May 18, 2026Jinming Liu, Jianguo Huang, Zhaoyang Jia +7Streaming Video Understanding Adaptive Model Routing
1Shanghai Jiao Tong University · 2Eastern Institute of Technology, Ningbo, China · 3Microsoft Research Asia
Thinking in Streaming Video
Mar 13, 2026Zikang Liu, Longteng Guo, Handong Li +7Streaming Video Understanding Memory-Augmented Video Understanding
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.

Aug 31, 2026J/K move · Enter open · S save

StreamScout: Learning When to Look Deeper for Streaming Video Understanding

Ce Zhang, Jing Bi, Jinxi He, Jianshu Zhang+8

Carnegie Mellon University · TikTok · University of Rochester+2

Adaptive Inference Streaming Video Understanding Efficient VLM Inference Video QA Memory-Augmented Video Understanding

Open paper AskPDF arXiv

Streaming video understanding requires answering questions that arrive at arbitrary moments over an unbounded video stream. Existing systems primarily focus on what to retain in a bounded memory, yet access that memory using the same fixed-cost procedure for every query, despite substantial variation in the evidence required. We argue that deciding how deeply to access memory for each query is as important as deciding what the memory should store. To this end, we introduce StreamScout, an adaptive inference framework that maintains only a lightweight textual timeline in context as the stream unfolds. At query time, StreamScout progressively augments the timeline with up to three increasingly informative visual views: a glance at recent frames, a uniform look-back over the past stream, and query-salient retrieval. At each stage, the model answers immediately if the available evidence is sufficient; otherwise, it escalates to the next view. To improve this stop-or-escalate policy, we probe the cascade on an auxiliary set and distill the model's empirical competence boundary into supervision for a lightweight LoRA adaptation, yielding StreamScout-S. We further refine the policy through reinforcement learning, allowing the model to explore stopping behaviors beyond imitation of the distilled decisions, yielding StreamScout-R. Across three backbones and three streaming benchmarks, StreamScout and its variants consistently outperform prior streaming methods while substantially reducing inference cost and token consumption; on OVO-Bench, for instance, StreamScout-S improves Qwen3-VL-8B by 14.65 points while using 59% fewer tokens than uniform sampling and answering in 1.04 s on average.

Abstract

Explore similar work

StreamScout: Learning When to Look Deeper for Streaming Video Understanding

An Efficient Streaming Video Understanding Framework with Agentic Control

Thinking in Streaming Video