Streaming video understanding (SVU) requires models to answer user queries over continuously evolving video streams. Unlike offline video QA, SVU must handle queries whose evidence may lie in the past, appear in the current scene, or emerge only after future observations. Existing streaming video systems often rely on compact visual memories or fixed evidence-access schemes, which can lose fine-grained details and fail to adapt the answering process to the temporal intent of each query. In this paper, we propose TimeTraveler, a streaming VQA framework that performs temporal strategy planning with a Time Dictionary. Inspired by the need to search across different temporal regions of a stream, TimeTraveler stores each observed moment as a timestamp-indexed structured caption and uses this dictionary as an explicit source of queryable evidence. Given a query, TimeTraveler determines whether to recall past evidence, read the present context, or wait for future observations, and then applies a strategy-specific evidence acquisition process. Comparative experiments on streaming video QA benchmarks show that TimeTraveler improves streaming question answering by preserving fine-grained temporal evidence and selecting the appropriate temporal strategy for each query. The project page is available at https://geppa.github.io/TimeTraveler.
Figures & tables
Figure 1: Conceptual comparison between prior memory-based SVU and TimeTraveler. Prior methods preserve streaming history as agentic memory Xiong et al. (2025) , pruned tokens Zhang et al. (2026) , or KV-memory recall Yang et al. (2025a) , but rely on a single evidence-access path and do not ensure that retained evidence remains query-recoverable. In contrast, TimeTraveler plans whether each query should be grounded in the past, current scene, or future, and accesses a structured Time Dictionary accordingly.
Figure 2: Overview of TimeTraveler. Incoming video frames are encoded into structured captions and stored in a timestamp-indexed Time Dictionary. Given a query, a Temporal Strategy Planner acquires evidence following temporal intent: past queries retrieve relevant captions through similarity-based lookback, current queries access local captions, and future queries continue observation until evidence appears. Then, the retrieved evidence is used to produce the final answer.
Method
#Frames
Real-Time Visual Perception
Backward Tracing
Forward Act. Responding
Overall
OCR
ACR
ATR
STU
FPD
OJR
Avg.
EPM
ASI
HLD
Avg.
REC
SSR
CRR
Avg.
Avg.
Human
Human
-
93.96
92.57
94.83
92.70
91.09
94.02
93.20
92.59
93.02
91.37
92.33
95.48
89.67
93.56
92.90
92.81
Proprietary Models (Offline)
Gemini 1.5 Pro ( Team et al., 2024 )
1 fps
85.91
66.97
79.31
58.43
63.37
61.96
69.32
58.59
76.35
52.64
62.54
35.53
74.24
61.67
57.15
63.00
GPT-4o ( Hurst et al., 2024 )
64
69.80
64.22
71.55
51.12
70.30
59.78
64.46
57.91
75.68
48.66
60.75
27.58
73.21
59.40
53.40
59.54
Table 1: Evaluation results on OVO-Bench. Overall accuracy is calculated as the macro average across taxonomies. Best results among open-source models are boldfaced and runner-up results are underlined . Proprietary model results are shown with gray background for reference.
Model
#Frames
Real-Time Visual Understanding
Omni-Source Understanding
Contextual Understanding
Overall
OP
CR
CS
ATP
EU
TR
PR
SU
ACP
CT
All
ER
SCU
SD
MA
All
ACU
MCU
SQA
PO
All
Human
Human ‡
-
89.47
92.00
93.60
91.47
95.65
92.52
88.00
88.75
89.74
91.30
91.46
88.00
88.24
93.60
90.27
90.26
88.80
90.40
95.00
100.00
93.55
91.66
Proprietary MLLMs
Gemini 1.5 pro ( Team et al., 2024 )
1 fps
79.02
80.47
83.54
79.67
80.00
84.74
77.78
64.23
71.95
48.70
75.69
46.80
39.60
74.90
80.00
60.22
51.41
40.73
54.80
45.10
48.73
67.07
GPT-4o ( Hurst et al., 2024 )
64
77.11
80.47
83.91
76.47
70.19
83.80
66.67
62.19
69.12
49.22
73.28
41.20
37.20
43.60
56.00
44.50
41.20
38.40
32.80
56.86
38.70
60.15
Table 2: Performance comparison on StreamingBench. Best results among open-source models are boldfaced and runner-up results are underlined .
Memory Type
#Tokens
Real-Time Acc.
Backward Acc.
Forward Acc.
Overall Acc.
No Memory
–
63.24
43.44
48.42
51.70
Visual Prototype Memory
203.11
66.19
44.16
51.36
53.90
Flat Text Memory
236.60
70.98
48.57
47.83
55.79
Structured Dictionary (Ours)
245.23
73.28
51.37
49.41
58.02
Table 3: Comparison of memory representations under an equal token budget on OVO-Bench. #Tokens denotes the average number of memory tokens injected into the answer model at inference time. Qwen3-VL-8B is used as the backbone.
Table 6Table 7
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Stream Processing (always-on)
Stage
Latency
TFLOPs
Video decode
15.72 ms
–
Captioner
605.82 ms
1.53
Embedder
15.97 ms
0.20
Total (per 1 s)
637.51 ms
1.73
Appendix
Table 8: Asynchronous latency breakdown of TimeTraveler. The two blocks run independently: stream processing executes continuously, while query processing is triggered on demand. Stream latency is normalized to one second of input at 1 fps. For the past path, the QA cost is linearly interpolated between the measured 1- and 8-caption settings at the average number of retrieved captions ( Kˉ⋆=2.68 over past-routed OVO-Bench queries). Qwen3-VL-8B is used as the backbone.
Memory Design
Per-frame
1 min
5 min
10 min
30 min
1 hour
Visual Full KV Cache
33.75 MB
1.98 GB
9.89 GB
19.78 GB
59.33 GB
118.65 GB
Visual Hidden State
1.88 MB
112.5 MB
562.5 MB
1.10 GB
3.30 GB
6.59 GB
Visual Prototype ( K=32 )
–
1.6 MB
1.6 MB
1.6 MB
1.6 MB
1.6 MB
Time Dictionary (Ours)
Caption Text
0.78 KB
46 KB
234 KB
469 KB
1.4 MB
2.8 MB
Retrieval Embedding
2.00 KB
120 KB
600 KB
1.2 MB
1.2 MB
1.2 MB
Appendix
Table 9: Memory footprint comparison across memory designs on Qwen3-VL-8B ( Bai et al., 2025a ) . Visual tokens per frame are measured at 240 for inputs resized to max_pixels=401408 . Visual prototype uses K=32 slots. Our structured dictionary stores 800-byte UTF-8 caption text and a 1024-dim FP16 embedding per frame. Retrieval embeddings are maintained in a finite queue covering the most recent 600 s, so their cost is capped beyond 10 minutes.
Captioner
Frames/call
Throughput
RT
BW
FWD
OVO
SB
No memory (baseline)
–
–
63.24
43.44
48.42
51.70
59.33
Qwen3-VL-2B-FP8 (Ours)
1
1.65
73.28
51.37
49.41
58.02
64.51
Qwen3.5-0.8B
1
3.52
72.46
50.21
50.94
57.87
62.92
Qwen3-VL-2B-FP8
8
8.97
69.31
48.97
50.91
56.40
61.55
Appendix
Table 10: Accuracy–throughput trade-off of the captioning stage. Captioning throughput is measured in frames per second; all other components are identical to our default setting with Qwen3-VL-8B as the answer model. OVO-Bench reports the three category averages and the overall score; SB denotes StreamingBench overall.
Method
Visual memory
Text memory
Memory budget
Memory footprint
Answer latency
Answer-model memory
OVO-BW
SB-RT
InfiniPot-V
4K compressed KV
–
3.83K
220 MB
53.75 ms †
16.80 GB
45.60
76.76
TimeTraveler
Current frame
K⋆≈2.5
0.70K
39 MB
58.79 ms
16.62 GB
57.14
69.96
Current frame
Top-20
∼ 3.8K
214 MB
72.02 ms
16.79 GB
56.37
70.88
Uniformly selected frames
K⋆≈2.5
∼ 3.8K
214 MB
91.81 ms
16.79 GB
53.39
77.72
Appendix
Table 11: Accuracy–efficiency comparison with InfiniPot-V ( Kim et al., 2025b ) . Both methods use the same backbone (Qwen2.5-VL-7B), environment, and evaluation protocol, and InfiniPot-V is run with its official implementation. Memory budget counts the answer-time tokens provided to the answer model. † InfiniPot-V reuses its compressed KV states built during streaming, so its answer latency mainly consists of question processing and answer generation, whereas TimeTraveler prefills its visual and textual evidence at query time.
Method
Real-Time
Backward
Forward
Overall
QueryStream-7B ∗ ( Zhang et al., 2026 )
61.40
42.10
39.03
47.51
StreamAgent-7B ∗ ( Yang et al., 2025a )
61.30
41.70
45.40
49.40
Qwen2.5-VL-7B ( Bai et al., 2025b )
63.46
44.21
39.40
49.02
+ TimeTraveler
71.71
48.96
36.90
52.52
Appendix
Table 12: Same-backbone comparison on OVO-Bench with Qwen2.5-VL-7B as the answer model. ∗ Results reported in the original papers, which do not report backbone-only OVO-Bench scores; the backbone-only and TimeTraveler rows are evaluated in our environment with the captioner, planner, embedder, and all hyperparameters unchanged.
Backbone
Method
Acc.
Qwen3-VL-8B
Baseline
52.40
+ Time Dictionary w/o gating
45.20
+ Time Dictionary w/ gating (Ours)
62.90
InternVL3.5-8B
Baseline
40.00
+ Time Dictionary w/o gating
41.20
+ Time Dictionary w/ gating (Ours)
53.23
Appendix
Table 13: Effect of the future gating on StreamingBench Proactive Output.
Policy
RT
BW
FWD
Overall
Ours (512 frames for all queries)
73.28
51.37
49.41
58.02
Past → 512, Current → 1
78.38
51.51
46.29
58.73
Past → 0, Current → 1
76.34
54.99
38.92
56.75
Appendix
Table 14: Routing-based visual budget allocation on OVO-Bench. Qwen3-VL-8B is used as the answer model, and the number of visual frames it receives is assigned per temporal strategy using the planner output.
Selection Strategy
Backward Avg.
Top- K Selection ( K=4 )
49.64
Online Probabilistic Selection (Ours)
51.37
Appendix
Table 15: Effect of adaptive past-evidence selection on OVO Backward Tracing. Since this comparison isolates the past-lookback module, we report overall backward accuracy only. Results are obtained by applying our method on top of Qwen3-VL-8B ( Bai et al., 2025a ) . We compare the proposed online probabilistic lookback with a fixed top- K variant under the same routing setup. Replacing adaptive selection with fixed top- K retrieval degrades performance.
Real-Time Visual Perception
Backward Tracing
Forward Active Responding
OCR
ACR
ATR
STU
FPD
OJR
EPM
ASI
HLD
REC
SSR
CRR
Mean K⋆
2.09
2.27
2.24
2.20
1.43
2.61
2.54
2.80
2.52
2.77
3.21
2.68
Median K⋆
2
2
2
2
1
2
2
2
2
2
3
2
Max K⋆
7
6
6
7
2
5
7
10
9
13
10
19
Appendix
Table 16: Number of captions selected by past lookback ( K⋆ ) across OVO-Bench question types. Statistics are computed over the 1,473 query instances routed to past.
Tpast
Backward Avg.
0.01
50.39
0.02
51.37
0.05
50.34
0.10
50.16
Appendix
Table 17
Answer model
0.01
0.02 (Ours)
0.05
0.10
Range
Qwen3-VL-8B
50.39
51.37
50.34
50.16
1.21
InternVL3.5-8B
34.43
34.72
35.57
34.47
1.14
Appendix
Table 20: Sensitivity to Tpast across answer backbones on OVO-Bench Backward Tracing (average accuracy). All other settings are fixed. The no-memory baseline is 43.44 for Qwen3-VL-8B and 32.25 for InternVL3.5-8B.
Figure 3: Textual input sensitivity on OVO-Bench. We replace a fraction ρ of the Time Dictionary captions with irrelevant captions sampled from unrelated videos and measure the resulting Overall accuracy. Dashed lines indicate the baseline (no memory) and our full method (clean captions).
Figure 4: Qualitative example of lookback retrieval on OVO-Bench. Given a query at time tq , the similarity score over past captions (orange curve) peaks at the frame most relevant to the question. The retrieved Time Dictionary caption preserves fine-grained details such as on-screen text ( “FRONT RAISE” ), spatial layout, and highlighted muscle groups, which are essential for answering correctly.
We propose DeltaStreamer, a novel agentic framework for streaming video understanding inspired by how humans observe, remember, and investigate. Streaming video understanding requires a system to retain evidence before knowing which details future queries will need. Existing approaches often obscure event transitions and omit relevant details when compressing video history, and cannot recover missing visual evidence. DeltaStreamer addresses these limitations by combining change-aware memory with query-driven planning and targeted evidence gathering with replay. Before a query arrives, it continuously observes the stream and explicitly records visual changes in structured memory, preserving how events unfold rather than only what is present. When a query arrives, a query-driven planner identifies the required evidence and guides memory retrieval and targeted replay. By revisiting historical frames, the agent can recover visual details omitted during observation and ground its answer in evidence beyond the retained descriptions. This connects continuous observation with active investigation tailored to each question. Experiments on OVO-Bench and StreamingBench demonstrate improvements over representative streaming video understanding methods, with overall scores of 66.07% and 59.53%, respectively. Project page: https://bbbbchan.github.io/DeltaStreamer/
Boyuan Sun, Deshui Miao, Wenzhao Gao +8
VCIP, Nankai University · ByteDance Inc. · IQuest Research +1
Streaming video understanding requires answering questions that arrive at arbitrary moments over an unbounded video stream. Existing systems primarily focus on what to retain in a bounded memory, yet access that memory using the same fixed-cost procedure for every query, despite substantial variation in the evidence required. We argue that deciding how deeply to access memory for each query is as important as deciding what the memory should store. To this end, we introduce StreamScout, an adaptive inference framework that maintains only a lightweight textual timeline in context as the stream unfolds. At query time, StreamScout progressively augments the timeline with up to three increasingly informative visual views: a glance at recent frames, a uniform look-back over the past stream, and query-salient retrieval. At each stage, the model answers immediately if the available evidence is sufficient; otherwise, it escalates to the next view. To improve this stop-or-escalate policy, we probe the cascade on an auxiliary set and distill the model's empirical competence boundary into supervision for a lightweight LoRA adaptation, yielding StreamScout-S. We further refine the policy through reinforcement learning, allowing the model to explore stopping behaviors beyond imitation of the distilled decisions, yielding StreamScout-R. Across three backbones and three streaming benchmarks, StreamScout and its variants consistently outperform prior streaming methods while substantially reducing inference cost and token consumption; on OVO-Bench, for instance, StreamScout-S improves Qwen3-VL-8B by 14.65 points while using 59% fewer tokens than uniform sampling and answering in 1.04 s on average.
Ce Zhang, Jing Bi, Jinxi He +9
Carnegie Mellon University · TikTok · University of Rochester +2
Streaming video understanding often involves time-sensitive scenarios where models need to answer exactly when the supporting visual evidence appears: answering before the evidence reflects speculation, answering after it has passed reduces real-time utility. To capture this behavior, we introduce a readiness-aware formulation of streaming video understanding with the Answer Readiness Score (ARS), a timing-aware objective with asymmetric early and late penalties. When combined with correctness, ARS defines an effective accuracy that measures not just whether a model is right, but whether it answers at the appropriate moment. Building on this formulation, we introduce StreamReady, a framework to unify temporal reasoning with on-time answering through a lightweight readiness mechanism that decides if sufficient evidence has been observed before responding. To evaluate this capability, we further introduce ProReady-QA, a benchmark with annotated answer evidence windows and proactive multi-turn questions across local and global contexts. StreamReady achieves superior performance on ProReady-QA, and consistently outperforms prior methods across eight additional streaming and offline long-video benchmarks, demonstrating robust and broadly generalizable video understanding capability.
Shehreen Azad, Vibhav Vineet, Yogesh Singh Rawat
Center for Research in Computer Vision, University of Central Florida · Microsoft Research