Recent streaming video understanding methods increasingly rely on complex memory mechanisms to handle long video streams. We challenge this trend with a simple finding: a sliding-window baseline that feeds only the most recent N frames to an off-the-shelf VLM already matches or surpasses published streaming models. We formalize this baseline as SimpleStream and evaluate it against 13 major offline and online video LLM baselines on OVO-Bench and StreamingBench. Despite its simplicity, SimpleStream delivers consistently strong performance. With only 4 recent frames, it reaches 67.7% average accuracy on OVO-Bench and 80.59% on StreamingBench. Controlled ablations further show that the value of longer context is backbone-dependent rather than uniformly increasing with model scale, and reveal a consistent perception-memory trade-off: adding more historical context can improve recall, but often weakens real-time perception. This suggests that stronger memory, retrieval, or compression modules should not be taken as evidence of progress unless they clearly outperform SimpleStream under the same protocol. We therefore argue that future streaming benchmarks should separate recent-scene perception from long-range memory, so that performance improvements from added complexity can be evaluated more clearly.
Figures & tables
Figure 1: A Strong Simple Baseline for Streaming Video Understanding. (a) Overview of SimpleStream : given a streaming video and a query, only the most recent N frames are fed to an off-the-shelf VLM. (b) Perception-memory comparison on OVO-Bench, where SimpleStream consistently lies on the upper-right frontier across backbone families and window sizes.
Figure 2: A landscape of streaming video understanding methods. Most existing approaches differ mainly in how they preserve and reuse historical information under bounded budgets, while SimpleStream keeps only a small recent frame window.
Model
#Frames
Streaming Bench
OVO-Bench
Real-Time Visual Perception
Backward Tracing
Avg.
OCR
ACR
ATR
STU
FPD
OJR
Avg.
EPM
ASI
HLD
Avg.
Human
–
91.46
94.0
92.6
94.8
92.7
91.1
94.0
93.2
92.6
93.0
91.4
92.3
92.77
Offline Video LLMs
Qwen2.5-VL-7B ( Bai et al., 2025b )
1 fps
73.31
67.8
55.1
67.2
42.1
66.3
60.9
59.9
51.5
58.8
23.7
44.7
52.28
LLaVA-OneVision-7B ( Li et al., 2025a )
32
71.12
66.4
57.8
73.3
53.4
71.3
62.0
64.0
54.2
55.4
21.5
43.7
53.85
Table 1: Main results on OVO-Bench ( Li et al., 2025b ) and StreamingBench ( Lin et al., 2024 ) . Per-task accuracy on OVO-Bench under its Real-Time Visual Perception and Backward Tracing tracks. “StreamingBench” denotes RTVU accuracy. “–” indicates unreported results. † : Qwen2.5-VL-7B + HERMES (4K tokens). Task abbreviations follow OVO-Bench: OCR, ACR, ATR, STU, FPD, and OJR denote Optical Character Recognition, Action Recognition, Attribute Recognition, Spatial Understanding, Future Prediction, and Object Recognition; EPM, ASI, and HLD denote Episodic Memory, Action Sequence Identification, and Hallucination Detection. “Avg.” is the mean of Real-Time and Backward category averages. For a fair comparison with most baselines, we sample at 1 fps in SimpleStream and cap maximum frames at 2, 4, or 8.
Model
2 Frames
4 Frames
8 Frames
16 Frames
Bwd.
Real-Time
Avg.
Bwd.
Real-Time
Avg.
Bwd.
Real-Time
Avg.
Bwd.
Real-Time
Avg.
Qwen2.5-VL
Qwen2.5-VL-3B
42.09
70.87
56.48
42.89
73.47
58.18
40.98
72.07
56.52
45.75
70.60
58.17
Qwen2.5-VL-7B
49.50
75.00
62.22
51.90
78.40
65.13
50.80
76.60
63.70
48.87
74.45
61.66
Qwen2.5-VL-32B
45.87
78.80
62.33
48.64
83.03
65.84
49.43
81.70
65.56
49.97
80.69
65.33
Qwen2.5-VL-72B
57.65
77.63
67.64
58.71
80.36
69.53
56.75
81.25
69.00
60.61
80.91
70.76
Table 2: Model scale effects under a fixed recent-window evaluation protocol on OVO-Bench. We vary model scale within each backbone family while keeping all other evaluation settings unchanged. For each recent window, we report Backward Tracing accuracy, Real-Time Visual Perception accuracy, and their mean (“Avg.”). The 7B and 8B rows reuse the corresponding 2-frame, 4-frame, and 8-frame runs from the main experiment under the same protocol. “–” indicates unavailable or incomplete runs.
Figure 5Figure 6
Figure 5: Model-scaling ablation on OVO-Bench. Average accuracy versus recent-window size for Qwen2.5-VL (left) and Qwen3-VL (right) checkpoints. Stars mark the best window for each checkpoint. Many checkpoints peak at 4f, but several prefer longer windows, including Qwen3-VL-4B at 16f and larger checkpoints at 8f or 16f.
Figure 6: Perception cost ( ΔP ) and Memory Gain ( ΔM ) relative to the SimpleStream Qwen2.5-VL + 2f configuration. Green bars show changes in OVO-Bench Real-Time average. Blue bars show changes on the mean of EPM and ASI. Relative to this short-window reference, many methods improve memory, but external baselines still incur a substantial perception cost.
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.
Streaming Video Large Language Models (VLMs) enable continuous video understanding, yet existing benchmarks focus on low-dynamic scenarios. Under bounded context budgets, models must balance temporal history, spatial resolution, and temporal granularity; sparse sampling at 1--2 FPS misses fast events. We introduce FastBench to evaluate high-dynamic perception in real-world video streams. Its trajectory-grounded pipeline combines QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification using SAM3 and CoTracker3 trajectories, and three rounds of human inspection. FastBench contains 306 QA pairs across eight domains, six capabilities, and forward, instant, and backward temporal scopes, with human-annotated evidence intervals. We also present ProactiveFrame, a training-free baseline that adjusts incoming frame rates through text tokens. A dual-tier sliding window retains recent high-FPS observations while downsampling older ones into sparse history. Experiments reveal substantial limitations: the strongest model, Gemini-3.5-Flash, scores only 50.7%. Denser sampling improves Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains saturate as history is compressed. ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points, yet remains well below oracle-guided focusing, showing that current VLMs struggle to determine from the stream alone when finer temporal perception is needed. FastBench provides a testbed for high-dynamic streaming video understanding. Code and data: https://github.com/Ashone3/FastBench.
Humans effortlessly perceive the present while remembering the past, yet streaming VLMs often trade off real-time perception against long-term memory. Prior work shows that shortening the context can sharpen current-scene perception at the expense of long-range recall. To reconcile these abilities, we introduce StreamTTT, which writes long-range history into online-updated fast weights outside the attention context. This leaves a short sliding key-value cache dedicated to recent evidence, mitigating attention dilution. We train StreamTTT jointly on offline long-video QA and a newly constructed real-time QA corpus. On OVO-Bench, under each model's reported input protocol, StreamTTT-4B outperforms the same-scale SimpleStream-4B by 0.6 points in real-time perception and 5.3 points in backward tracing. It also surpasses the larger SimpleStream-8B by 0.73 points on StreamingBench's Real-Time Visual Understanding (RTVU) subset. Our code is publicly available at https://github.com/zeyun-zhong/StreamTTT.
Joya Chen, Zeyun Zhong, Mike Zheng Shou
National University of Singapore · Karlsruhe Institute of Technology
Streaming video understanding requires multimodal large language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term memory that filters temporal redundancy before visual encoding with a latent long-term memory that consolidates historical video content into visual latents accessible to subsequent reasoning. During generation, an attention-guided retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-video benchmarks. Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient reasoning.
Muxin Fu, Yifan Zhang, Wentao Zhang +5
Tongji University · Nanyang Technological University · University of Michigan +2