Streaming video understanding (SVU) requires models to answer user queries over continuously evolving video streams. Unlike offline video QA, SVU must handle queries whose evidence may lie in the past, appear in the current scene, or emerge only after future observations. Existing streaming video systems often rely on compact visual memories or fixed evidence-access schemes, which can lose fine-grained details and fail to adapt the answering process to the temporal intent of each query. In this paper, we propose TimeTraveler, a streaming VQA framework that performs temporal strategy planning with a Time Dictionary. Inspired by the need to search across different temporal regions of a stream, TimeTraveler stores each observed moment as a timestamp-indexed structured caption and uses this dictionary as an explicit source of queryable evidence. Given a query, TimeTraveler determines whether to recall past evidence, read the present context, or wait for future observations, and then applies a strategy-specific evidence acquisition process. Comparative experiments on streaming video QA benchmarks show that TimeTraveler improves streaming question answering by preserving fine-grained temporal evidence and selecting the appropriate temporal strategy for each query. The project page is available at https://geppa.github.io/TimeTraveler.
Figures & tables
Figure 1: Conceptual comparison between prior memory-based SVU and TimeTraveler. Prior methods preserve streaming history as agentic memory Xiong et al. (2025) , pruned tokens Zhang et al. (2026) , or KV-memory recall Yang et al. (2025a) , but rely on a single evidence-access path and do not ensure that retained evidence remains query-recoverable. In contrast, TimeTraveler plans whether each query should be grounded in the past, current scene, or future, and accesses a structured Time Dictionary accordingly.
Figure 2: Overview of TimeTraveler. Incoming video frames are encoded into structured captions and stored in a timestamp-indexed Time Dictionary. Given a query, a Temporal Strategy Planner acquires evidence following temporal intent: past queries retrieve relevant captions through similarity-based lookback, current queries access local captions, and future queries continue observation until evidence appears. Then, the retrieved evidence is used to produce the final answer.
Method
#Frames
Real-Time Visual Perception
Backward Tracing
Forward Act. Responding
Overall
OCR
ACR
ATR
STU
FPD
OJR
Avg.
EPM
ASI
HLD
Avg.
REC
SSR
CRR
Avg.
Avg.
Human
Human
-
93.96
92.57
94.83
92.70
91.09
94.02
93.20
92.59
93.02
91.37
92.33
95.48
89.67
93.56
92.90
92.81
Proprietary Models (Offline)
Gemini 1.5 Pro ( Team et al., 2024 )
1 fps
85.91
66.97
79.31
58.43
63.37
61.96
69.32
58.59
76.35
52.64
62.54
35.53
74.24
61.67
57.15
63.00
GPT-4o ( Hurst et al., 2024 )
64
69.80
64.22
71.55
51.12
70.30
59.78
64.46
57.91
75.68
48.66
60.75
27.58
73.21
59.40
53.40
59.54
Table 1: Evaluation results on OVO-Bench. Overall accuracy is calculated as the macro average across taxonomies. Best results among open-source models are boldfaced and runner-up results are underlined . Proprietary model results are shown with gray background for reference.
Model
#Frames
Real-Time Visual Understanding
Omni-Source Understanding
Contextual Understanding
Overall
OP
CR
CS
ATP
EU
TR
PR
SU
ACP
CT
All
ER
SCU
SD
MA
All
ACU
MCU
SQA
PO
All
Human
Human ‡
-
89.47
92.00
93.60
91.47
95.65
92.52
88.00
88.75
89.74
91.30
91.46
88.00
88.24
93.60
90.27
90.26
88.80
90.40
95.00
100.00
93.55
91.66
Proprietary MLLMs
Gemini 1.5 pro ( Team et al., 2024 )
1 fps
79.02
80.47
83.54
79.67
80.00
84.74
77.78
64.23
71.95
48.70
75.69
46.80
39.60
74.90
80.00
60.22
51.41
40.73
54.80
45.10
48.73
67.07
GPT-4o ( Hurst et al., 2024 )
64
77.11
80.47
83.91
76.47
70.19
83.80
66.67
62.19
69.12
49.22
73.28
41.20
37.20
43.60
56.00
44.50
41.20
38.40
32.80
56.86
38.70
60.15
Table 2: Performance comparison on StreamingBench. Best results among open-source models are boldfaced and runner-up results are underlined .
Memory Type
#Tokens
Real-Time Acc.
Backward Acc.
Forward Acc.
Overall Acc.
No Memory
–
63.24
43.44
48.42
51.70
Visual Prototype Memory
203.11
66.19
44.16
51.36
53.90
Flat Text Memory
236.60
70.98
48.57
47.83
55.79
Structured Dictionary (Ours)
245.23
73.28
51.37
49.41
58.02
Table 3: Comparison of memory representations under an equal token budget on OVO-Bench. #Tokens denotes the average number of memory tokens injected into the answer model at inference time. Qwen3-VL-8B is used as the backbone.
Table 6Table 7
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Stream Processing (always-on)
Stage
Latency
TFLOPs
Video decode
15.72 ms
–
Captioner
605.82 ms
1.53
Embedder
15.97 ms
0.20
Total (per 1 s)
637.51 ms
1.73
Appendix
Table 8: Asynchronous latency breakdown of TimeTraveler. The two blocks run independently: stream processing executes continuously, while query processing is triggered on demand. Stream latency is normalized to one second of input at 1 fps. For the past path, the QA cost is linearly interpolated between the measured 1- and 8-caption settings at the average number of retrieved captions ( Kˉ⋆=2.68 over past-routed OVO-Bench queries). Qwen3-VL-8B is used as the backbone.
Memory Design
Per-frame
1 min
5 min
10 min
30 min
1 hour
Visual Full KV Cache
33.75 MB
1.98 GB
9.89 GB
19.78 GB
59.33 GB
118.65 GB
Visual Hidden State
1.88 MB
112.5 MB
562.5 MB
1.10 GB
3.30 GB
6.59 GB
Visual Prototype ( K=32 )
–
1.6 MB
1.6 MB
1.6 MB
1.6 MB
1.6 MB
Time Dictionary (Ours)
Caption Text
0.78 KB
46 KB
234 KB
469 KB
1.4 MB
2.8 MB
Retrieval Embedding
2.00 KB
120 KB
600 KB
1.2 MB
1.2 MB
1.2 MB
Appendix
Table 9: Memory footprint comparison across memory designs on Qwen3-VL-8B ( Bai et al., 2025a ) . Visual tokens per frame are measured at 240 for inputs resized to max_pixels=401408 . Visual prototype uses K=32 slots. Our structured dictionary stores 800-byte UTF-8 caption text and a 1024-dim FP16 embedding per frame. Retrieval embeddings are maintained in a finite queue covering the most recent 600 s, so their cost is capped beyond 10 minutes.
Captioner
Frames/call
Throughput
RT
BW
FWD
OVO
SB
No memory (baseline)
–
–
63.24
43.44
48.42
51.70
59.33
Qwen3-VL-2B-FP8 (Ours)
1
1.65
73.28
51.37
49.41
58.02
64.51
Qwen3.5-0.8B
1
3.52
72.46
50.21
50.94
57.87
62.92
Qwen3-VL-2B-FP8
8
8.97
69.31
48.97
50.91
56.40
61.55
Appendix
Table 10: Accuracy–throughput trade-off of the captioning stage. Captioning throughput is measured in frames per second; all other components are identical to our default setting with Qwen3-VL-8B as the answer model. OVO-Bench reports the three category averages and the overall score; SB denotes StreamingBench overall.
Method
Visual memory
Text memory
Memory budget
Memory footprint
Answer latency
Answer-model memory
OVO-BW
SB-RT
InfiniPot-V
4K compressed KV
–
3.83K
220 MB
53.75 ms †
16.80 GB
45.60
76.76
TimeTraveler
Current frame
K⋆≈2.5
0.70K
39 MB
58.79 ms
16.62 GB
57.14
69.96
Current frame
Top-20
∼ 3.8K
214 MB
72.02 ms
16.79 GB
56.37
70.88
Uniformly selected frames
K⋆≈2.5
∼ 3.8K
214 MB
91.81 ms
16.79 GB
53.39
77.72
Appendix
Table 11: Accuracy–efficiency comparison with InfiniPot-V ( Kim et al., 2025b ) . Both methods use the same backbone (Qwen2.5-VL-7B), environment, and evaluation protocol, and InfiniPot-V is run with its official implementation. Memory budget counts the answer-time tokens provided to the answer model. † InfiniPot-V reuses its compressed KV states built during streaming, so its answer latency mainly consists of question processing and answer generation, whereas TimeTraveler prefills its visual and textual evidence at query time.
Method
Real-Time
Backward
Forward
Overall
QueryStream-7B ∗ ( Zhang et al., 2026 )
61.40
42.10
39.03
47.51
StreamAgent-7B ∗ ( Yang et al., 2025a )
61.30
41.70
45.40
49.40
Qwen2.5-VL-7B ( Bai et al., 2025b )
63.46
44.21
39.40
49.02
+ TimeTraveler
71.71
48.96
36.90
52.52
Appendix
Table 12: Same-backbone comparison on OVO-Bench with Qwen2.5-VL-7B as the answer model. ∗ Results reported in the original papers, which do not report backbone-only OVO-Bench scores; the backbone-only and TimeTraveler rows are evaluated in our environment with the captioner, planner, embedder, and all hyperparameters unchanged.
Backbone
Method
Acc.
Qwen3-VL-8B
Baseline
52.40
+ Time Dictionary w/o gating
45.20
+ Time Dictionary w/ gating (Ours)
62.90
InternVL3.5-8B
Baseline
40.00
+ Time Dictionary w/o gating
41.20
+ Time Dictionary w/ gating (Ours)
53.23
Appendix
Table 13: Effect of the future gating on StreamingBench Proactive Output.
Policy
RT
BW
FWD
Overall
Ours (512 frames for all queries)
73.28
51.37
49.41
58.02
Past → 512, Current → 1
78.38
51.51
46.29
58.73
Past → 0, Current → 1
76.34
54.99
38.92
56.75
Appendix
Table 14: Routing-based visual budget allocation on OVO-Bench. Qwen3-VL-8B is used as the answer model, and the number of visual frames it receives is assigned per temporal strategy using the planner output.
Selection Strategy
Backward Avg.
Top- K Selection ( K=4 )
49.64
Online Probabilistic Selection (Ours)
51.37
Appendix
Table 15: Effect of adaptive past-evidence selection on OVO Backward Tracing. Since this comparison isolates the past-lookback module, we report overall backward accuracy only. Results are obtained by applying our method on top of Qwen3-VL-8B ( Bai et al., 2025a ) . We compare the proposed online probabilistic lookback with a fixed top- K variant under the same routing setup. Replacing adaptive selection with fixed top- K retrieval degrades performance.
Real-Time Visual Perception
Backward Tracing
Forward Active Responding
OCR
ACR
ATR
STU
FPD
OJR
EPM
ASI
HLD
REC
SSR
CRR
Mean K⋆
2.09
2.27
2.24
2.20
1.43
2.61
2.54
2.80
2.52
2.77
3.21
2.68
Median K⋆
2
2
2
2
1
2
2
2
2
2
3
2
Max K⋆
7
6
6
7
2
5
7
10
9
13
10
19
Appendix
Table 16: Number of captions selected by past lookback ( K⋆ ) across OVO-Bench question types. Statistics are computed over the 1,473 query instances routed to past.
Tpast
Backward Avg.
0.01
50.39
0.02
51.37
0.05
50.34
0.10
50.16
Appendix
Table 17
Answer model
0.01
0.02 (Ours)
0.05
0.10
Range
Qwen3-VL-8B
50.39
51.37
50.34
50.16
1.21
InternVL3.5-8B
34.43
34.72
35.57
34.47
1.14
Appendix
Table 20: Sensitivity to Tpast across answer backbones on OVO-Bench Backward Tracing (average accuracy). All other settings are fixed. The no-memory baseline is 43.44 for Qwen3-VL-8B and 32.25 for InternVL3.5-8B.
Figure 3: Textual input sensitivity on OVO-Bench. We replace a fraction ρ of the Time Dictionary captions with irrelevant captions sampled from unrelated videos and measure the resulting Overall accuracy. Dashed lines indicate the baseline (no memory) and our full method (clean captions).
Figure 4: Qualitative example of lookback retrieval on OVO-Bench. Given a query at time tq , the similarity score over past captions (orange curve) peaks at the frame most relevant to the question. The retrieved Time Dictionary caption preserves fine-grained details such as on-screen text ( “FRONT RAISE” ), spatial layout, and highlighted muscle groups, which are essential for answering correctly.