We propose DeltaStreamer, a novel agentic framework for streaming video understanding inspired by how humans observe, remember, and investigate. Streaming video understanding requires a system to retain evidence before knowing which details future queries will need. Existing approaches often obscure event transitions and omit relevant details when compressing video history, and cannot recover missing visual evidence. DeltaStreamer addresses these limitations by combining change-aware memory with query-driven planning and targeted evidence gathering with replay. Before a query arrives, it continuously observes the stream and explicitly records visual changes in structured memory, preserving how events unfold rather than only what is present. When a query arrives, a query-driven planner identifies the required evidence and guides memory retrieval and targeted replay. By revisiting historical frames, the agent can recover visual details omitted during observation and ground its answer in evidence beyond the retained descriptions. This connects continuous observation with active investigation tailored to each question. Experiments on OVO-Bench and StreamingBench demonstrate improvements over representative streaming video understanding methods, with overall scores of 66.07% and 59.53%, respectively. Project page: https://bbbbchan.github.io/DeltaStreamer/
Figures & tables
Fig. 1 : Comparison of streaming video understanding paradigms. Compression or truncation in KV-cache reuse and multi-turn dialogue can obscure transitions and discard details needed by later queries. DeltaStreamer records explicit changes in structured memory and uses query-driven planning and targeted replay to recover missing visual evidence.
Fig. 2 : Overview of DeltaStreamer. Before the query, adjacent-frame comparison produces delta-augmented observations stored in hierarchical memory. After the query, the planner guides evidence retrieval and tool use, while targeted replay recovers visual details for verification.
Model
Real-Time Visual Perception
Backward Tracing
RT+BT Avg.
Forward Active Responding
Overall
OCR
ACR
ATR
STU
FPD
OJR
Avg.
EPM
ASI
HLD
Avg.
REC
SSR
CRR
Avg.
Human
Human Agents
93.96
92.57
94.83
92.70
91.09
94.02
93.20
92.59
93.02
91.37
92.33
92.77
95.48
89.67
93.56
92.90
92.81
Proprietary MLLMs
Gemini 1.5 Pro
85.91
66.97
79.31
58.43
63.37
61.96
69.32
58.59
76.35
52.64
62.54
65.93
35.53
74.24
61.67
57.15
63.00
Gemini 3 Flash
86.58
68.81
79.31
58.99
70.30
70.65
72.44
58.92
76.35
52.15
62.47
67.46
35.53
74.24
61.67
57.15
64.02
Tab. 1 : The experimental results on the OVO-Bench benchmark. RT+BT Avg. is the average over the Real-Time Visual Perception and Backward Tracing groups.
Tab. 3: Ablation of delta-augmented observation. 2F-NoDelta receives the same two-frame input as our method but does not explicitly generate or store a delta field. Comparing the two-frame variants tests explicit change recording at the same visual-input count.
Tab. 4 : Ablation of memory components. All variants use identical observations, tools, planner, and answer decoding; only the component shown in each mini-table is changed.
Fig. 3 : Illustrative delta comparison.
Tab. 5 : Planner and component ablations on OVO-Bench. Left: cumulative planner settings, with persistence retaining the plan across post-query rounds. Right: combinations of delta (D), hierarchical memory (M), and planner (P).
Tab. 6 : Ablation of memory rendering and core visual tools. All variants keep the model, observation protocol, the same planner, and answer decoding fixed.
Fig. 4 : Qualitative examples of DeltaStreamer. The upper case recalls the action following the first block placement. The lower case recovers the missing wall letters from a historical crop.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Group
Tool
Description
Memory
memory_search
Cross-layer keyword retrieval over all memory DAG layers
memory_read
Retrieve specific events by query with time-range filtering
memory_list
List events filtered by kind and time range, chronologically
memory_write
Write a model-selected fact into memory
Track
track_init
Initialize a SAM2 object tracker on a box or prompt
track_query
Query the current position and confidence of a tracked object
Appendix
Tab. 7: Tool suite of DeltaStreamer, grouped by function.
Fig. 5 : Visible memory (a) and tool use (b) on OVO-Bench and StreamingBench.
Metric
OVO-Bench
StreamingBench
Observation before query
Visual observations per sample
70.85
118.31
Observation length (chars.)
1,279.15
1,588.48
Delta length (chars.)
155.54
172.98
Memory at completion
Stored working records
131.91
261.71
Appendix
Tab. 8: Observation, memory, and planning statistics. All values are means.
Fig. 12 : OVO-Bench examples of temporal reasoning, attribute recall, and checking query premises.
Fig. 13 : StreamingBench examples of action sequencing, scene changes, and visual reasoning.
Fig. 14 : Some examples of using replay or historical crops.
Streaming video understanding requires answering questions that arrive at arbitrary moments over an unbounded video stream. Existing systems primarily focus on what to retain in a bounded memory, yet access that memory using the same fixed-cost procedure for every query, despite substantial variation in the evidence required. We argue that deciding how deeply to access memory for each query is as important as deciding what the memory should store. To this end, we introduce StreamScout, an adaptive inference framework that maintains only a lightweight textual timeline in context as the stream unfolds. At query time, StreamScout progressively augments the timeline with up to three increasingly informative visual views: a glance at recent frames, a uniform look-back over the past stream, and query-salient retrieval. At each stage, the model answers immediately if the available evidence is sufficient; otherwise, it escalates to the next view. To improve this stop-or-escalate policy, we probe the cascade on an auxiliary set and distill the model's empirical competence boundary into supervision for a lightweight LoRA adaptation, yielding StreamScout-S. We further refine the policy through reinforcement learning, allowing the model to explore stopping behaviors beyond imitation of the distilled decisions, yielding StreamScout-R. Across three backbones and three streaming benchmarks, StreamScout and its variants consistently outperform prior streaming methods while substantially reducing inference cost and token consumption; on OVO-Bench, for instance, StreamScout-S improves Qwen3-VL-8B by 14.65 points while using 59% fewer tokens than uniform sampling and answering in 1.04 s on average.
Ce Zhang, Jing Bi, Jinxi He +9
Carnegie Mellon University · TikTok · University of Rochester +2
Streaming video requires handling dynamic information density under strict latency budgets. Yet, existing methods typically employ static strategies, such as fixed memory compression or reliance on a single model, forcing a trade-off: fast models fail on complex queries, while always-on heavy models violate real-time constraints and overcomplicate simple queries. Rather than fixing these decisions upfront, we propose R3-Streaming (Remember, Respond, Reason), which formulates streaming video understanding as a cascaded control problem: for each query, the system compresses memory, judges response readiness, and routes computation sequentially, so that each downstream decision builds on progressively refined information states. To optimize this pipeline, we introduce an age-aware forgetting policy for memory compression, as aggressively compressing historical frames can yield substantial performance gains. For compute routing, we propose TB-GRPO, a target-balanced reinforcement learning objective that routes hard queries to a stronger model while preventing mode collapse. Extensive evaluations demonstrate that R3-Streaming achieves state-of-the-art results among streaming MLLMs, reaching 57.92 on OVO-Bench and 76.36 on StreamingBench, while reducing visual token usage by 95 to 96 percent.
Jinming Liu, Jianguo Huang, Zhaoyang Jia +7
1Shanghai Jiao Tong University · 2Eastern Institute of Technology, Ningbo, China · 3Microsoft Research Asia
Real-time understanding of continuous video streams is essential for interactive assistants and multimodal agents operating in dynamic environments. However, most existing video reasoning approaches follow a batch paradigm that defers reasoning until the full video context is observed, resulting in high latency and growing computational cost that are incompatible with streaming scenarios. In this paper, we introduce ThinkStream, a framework for streaming video reasoning based on a Watch--Think--Speak paradigm that enables models to incrementally update their understanding as new video observations arrive. At each step, the model performs a short reasoning update and decides whether sufficient evidence has accumulated to produce a response. To support long-horizon streaming, we propose Reasoning-Compressed Streaming Memory (RCSM), which treats intermediate reasoning traces as compact semantic memory that replaces outdated visual tokens while preserving essential context. We further train the model using a Streaming Reinforcement Learning with Verifiable Rewards scheme that aligns incremental reasoning and response timing with the requirements of streaming interaction. Experiments on multiple streaming video benchmarks show that ThinkStream significantly outperforms existing online video models while maintaining low latency and memory usage. Code, models and data will be released at https://github.com/johncaged/ThinkStream
Zikang Liu, Longteng Guo, Handong Li +7
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.