Streaming video understanding requires models to robustly encode, store, and retrieve information from a continuous video stream to support accurate video question answering (VQA). Existing state-of-the-art approaches rely on key-value caching to accumulate frame-level information over time, but use a limited number of tokens per frame, leading to the loss of fine-grained visual details. In this work, we propose scaling the token budget to enable more granular spatiotemporal understanding and reasoning. First, we find that current methods are ill-equipped to handle dense streams: their feature encoding causes query-frame similarity scores to increase over time, biasing retrieval toward later frames. To address this, we introduce an adaptive selection strategy that reduces token redundancy while preserving local spatiotemporal information. We further propose a training-free retrieval mixture-of-experts that leverages external models to better identify relevant frames. Our method, MemStream, achieves +8.0% on CG-Bench, +8.5% on LVBench, and +2.4% on VideoMME (Long) over ReKV with Qwen2.5-VL-7B.
Figures & tables
Figure 1: (a) We propose constructing the key–value cache with sparse sliding-window attention and design an adaptive key selection (AKS) strategy to sparsify the sliding window. (b) During question-answering, we merge complementary retrieval signals from external models via a training-free mixture-of-experts.
Figure 2: Increasing per-frame token budget leads to substantial declines in average layer-wise recall across a variety of different questions.
Figure 3: In (a), we identify a systematic trend where query-frame similarity scores progressively increase across the video. In (b), we observe that self-similarity maps of the key representations become more redundant as we increase tokens per frame.
Figure 4: We compute the normalized entropy for 677 sliding windows under 2 separate token budgets. At the higher token budget, the sliding window attention tends to exhibit higher entropy. Rather than increased informativeness, this suggests a struggle to focus on relevant frames at higher token budgets.
Method
Num Tokens
CG-Bench
LVBench
Qwen2.5-VL-7B
64
34.27
37.25
+ ReKV ( 0.5 FPS→128 )
64
35.23
37.06
Qwen2.5-VL-7B
128
37.02
39.19
+ ReKV ( 0.5 FPS→128 )
128
37.00
36.60
Qwen2.5-VL-7B
256
38.43
41.51
+ ReKV ( 0.5 FPS→128 )
256
37.45
37.96
Table 1: We show how question-answering performance with KV-cache memory (using ReKV as a reference implementation) degrades when passed in frames at higher token budgets. Here, ( 0.5 FPS→128 ) denotes that the video is sampled at 0.5 FPS with 128 frames used for question-answering.
Figure 5: We measure the recall for which, given a question, each layer retrieves the features for the CG-Bench “clue” frames. There is a massive variance in recall scores, but in general, they tend to be quite low.
Figure 6: (a) Our model processes the actual video stream a single time only (instead of once per question, as in offline VQA systems) to encode video features. We perform adaptive key selection to construct a sparse KV-cache. (b) MemStream uses matching with the question tokens to retrieve the relevant K/V features from the cache. Since we select features on a frame-wise basis, we can perform this retrieval with arbitrary vision-language models. We leverage this to construct a training-free mixture-of-experts for the retrieval.
Settings
Benchmarks
Model
Ntokens
Nframes
Encoding
Retrieval
Train?
CG-Bench
LVBench
VideoMME
Offline Video Question Answering
Qwen2.5-VL-7B
+ Uniform Sample
17K
( 0.5 FPS→128 )
N/A
N/A
✗
38.43
41.51
55.67
+ Uniform Sample
68K
( 0.5 FPS→512 )
N/A
N/A
✗
41.48
40.03
52.11
Online Video Question Answering
Table 2: Offline VQA Results. Effectiveness of MemStream for long video question answering. We highlight the best performance in bold . For VideoMME, we use the “Long” subset only.
Settings
RVS-Ego
RVS-Movie
Running Speed
Memory Usage
Model
Encoding
Retrieval
Acc
Score
Acc
Score
Video Enc.
Latency
GPU
KV-Cache
Qwen2.5-VL-7B
ReKV ( Di et al., 2025 )
Full
Internal
64.2
4.00
61.6
3.65
8.47 FPS
2.8s
29 GB
11.1 GB/h
MemStream (Ours)
AKS
Internal
67.8
4.01
59.1
3.60
8.50 FPS
2.6s
29 GB
11.1 GB/h
MemStream (Ours)
AKS
MoE
67.4
4.01
59.7
3.60
8.68 FPS
2.6s
32 GB
11.1 GB/h
Table 3: StreamingVQA Results. Evaluation of MemStream on RVS-Ego and RVS-Movie, measuring answer accuracy and quality, as well as runtime and memory usage. Following ReKV, we use GPT-3.5-Turbo as our LLM judge.
Encoding Type
Strategy
Comp. Rate
CG-Bench
LVBench
VideoMME
Full
∼1×
36.17
39.64
51.78
Patch-wise
A.1
∼4×
40.02
40.41
51.78
A.2
∼16×
41.15
42.93
52.56
B.1
∼12×
42.18
43.06
52.89
AKS
∼16×
41.63
43.77
54.56
Frame-wise
A.3
∼8×
41.63
42.35
53.78
Table 4: Encoding Strategy Ablations. Comparison of patch-wise and frame-wise encoding strategies across benchmarks.
Retrieval Strategy
External Model
CG-Bench
LVBench
VideoMME
Internal Only
–
41.63
43.77
54.56
External Only
CLIP
42.39
47.45
53.67
External Only
PECore
43.21
47.19
52.33
MoE (Ours)
CLIP
43.84
47.39
55.22
MoE (Ours)
PECore
44.19
48.10
54.22
Table 5: Retrieval Strategy Ablation. Comparison of internal, external, and MoE retrieval. We use the AKS encoding strategy and RRF fusion, with equal weights for the internal and external rankings.
Fusion Method
CG-Bench
LVBench
VideoMME
L2-Concat
43.57
48.03
52.89
RRF (Ours)
44.19
48.10
54.22
Table 6: Fusion Strategy Ablation. Comparison of early- and late-fusion methods for combining internal and external retrieval signals. We use PE-Core as the external encoder and the AKS encoding strategy.
Figure 7: Qualitative Results. We compare ReKV’s (red) and MemStream’s (blue) respective retrievals and answers. The ground-truth segment is shown in green.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
# Videos
Avg. Length
# QA Pairs
CG-Bench ( Chen et al., 2024 )
1,219
27 min.
12,129
LVBench ( Wang et al., 2025 )
103
68 min.
1,549
VideoMME (Long) ( Fu et al., 2025 )
300
41 min.
900
RVS-Ego ( Zhang et al., 2024 )
10
60 min.
1,465
RVS-Movie ( Zhang et al., 2024 )
22
30 min.
1,905
Appendix
Table 1: Dataset statistics. We detail the number of videos, questions, and average length for each benchmark.
Figure 8: More Qualitative Examples of Retrievals. We select a representative subset of retrievals for each method.
Encoding
Comp. Rate
CG-Bench
LVBench
VideoMME
Full
1×
36.17
39.64
51.78
Pooled (A.1)
∼4×
40.02
40.41
51.78
∼8×
39.81
40.35
50.33
∼16×
39.54
39.57
50.22
Dilated (A.2)
∼4×
40.17
41.12
51.89
∼8×
41.69
42.67
51.78
Appendix
Table 2: Effect of Sliding-Window Compression Rate. We evaluate performance of static attention patterns at varying sliding-window compression rates.
Settings
CG-Bench Average Recall@64 Scores
Model
Method
Encoding
Retrieval
Entity Cognition
Entity Perception
Event Cognition
Event Perception
Text Perception
Mean
Internal Retrieval
Uniform
Uniform
Internal
20.57
22.20
21.98
22.29
26.12
22.24
ReKV
Full
Internal
14.49
15.59
16.03
15.22
19.63
16.27
MemStream
A.2 ( ×4 )
Internal
19.97
21.42
20.86
20.23
27.31
21.59
MemStream
A.2 ( ×16 )
Internal
24.89
27.17
26.08
26.94
35.31
27.41
Appendix
Table 3: CG-Bench Recall. We show how frequently the top 64 retrieves features overlap the features from the ground truth clue frames in CG-Bench. ReKV’s struggles to retrieve the features corresponding to the most relevant frames help explain its degradation in question-answering performance.
Settings
CG-Bench Accuracy
Model
Method
Encoding
Retrieval
Entity Cognition
Entity Perception
Event Cognition
Event Perception
Text Perception
Mean
Internal Retrieval
Uniform
Uniform
Internal
34.03
37.06
36.76
35.35
49.52
38.43
ReKV
Full
Internal
33.12
36.35
34.73
31.36
45.72
36.17
MemStream
A.2 ( ×4 )
Internal
35.70
39.08
39.26
36.02
52.42
40.17
MemStream
A.2 ( ×16 )
Internal
36.61
40.14
39.26
37.99
54.36
41.15
Appendix
Table 4: CG-Bench Breakdown. We breakdown CG-Bench performance for a subset of question-types. We bold the best result and underline the second-best.
Settings
LVBench Accuracy
Model
Method
Encoding
Retrieval
Entity Recog.
Event Understand.
Key Info. Retrieval
Reasoning
Summarizing
Temporal Ground.
Mean
Internal Retrieval
Uniform
Uniform
Internal
41.65
39.72
42.96
42.29
43.10
39.09
41.51
ReKV
Full
Internal
38.55
39.57
43.30
40.30
29.31
35.00
39.64
MemStream
A.2 ( ×4 )
Internal
40.33
41.11
44.33
41.29
31.03
36.82
41.12
MemStream
A.2 ( ×16 )
Internal
41.80
42.35
45.02
41.79
34.48
39.09
42.93
Appendix
Table 5: LVBench Breakdown. We breakdown LVBench performance across all question-types. We bold the best result and underline the second-best.
Figure 9: We plot layer-wise average recall scores for full sliding-window attention, AKS, and AKS combined with retrieval mixture-of-experts with PECore as the external retriever.
Streaming video understanding requires multimodal large language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term memory that filters temporal redundancy before visual encoding with a latent long-term memory that consolidates historical video content into visual latents accessible to subsequent reasoning. During generation, an attention-guided retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-video benchmarks. Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient reasoning.
Muxin Fu, Yifan Zhang, Wentao Zhang +5
Tongji University · Nanyang Technological University · University of Michigan +2
Streaming video understanding requires processing unbounded video streams with limited memory and computation, posing two key challenges. First, continuously constructing new and evicting old key-value(KV) caches is required for unbounded streams. Secondly, due to the high cost of collecting and training on unbounded streams, models must learn from short sequences while generalizing to long streams. Existing streaming VideoVLLMs fail to scale to unbounded video streams or focus on cache reuse strategies, leaving the impact of cache construction underexplored. In this paper, we propose Decoupled Streaming Cache(DSCache), a training-free cache construction mechanism that adapts pretrained offline models to streaming settings. DSCache maintains a cumulative past KV cache while constructing a separate instant cache on-demand, decoupled from past caches to preserve the informativeness of recent inputs. To enable position extrapolation beyond the training length, DSCache further incorporates a position-agnostic encoding strategy, ensuring KV caches to support unseen positions and preventing position overflow. Experiments on Streaming Video QA benchmarks demonstrate DSCache's state-of-the-art performance, with an average 2.5% accuracy gains over prior methods.
Long streaming video QA remains challenging due to growing visual tokens and limited reasoning length of large language models (LLMs). KV-caching stores the Key-Value (KV) of the historical tokens via LLM prefill and enables more efficient streaming QA. However, existing methods cache every one or two frames, causing redundant memory usage and losing fine-grained spatial details within frame or temporal contexts across frames. This paper proposes MuKV, a method that features a multi-grained KV cache compression module and a semi-hierarchical retrieval approach to improve both efficiency and accuracy for long streaming VideoQA. For the offline KV cache, MuKV extracts visual representations at patch-, frame-, and segment-levels. The multiple levels of granularity preserve both local cues and global temporal context, while maintaining efficiency with a dual signal token compression mechanism guided by self-attention and frequency. For online QA, MuKV designs a semi-hierarchical retrieval method to retrieve relevant KV caches for answer generation. Experiments on long-streaming VideoQA benchmarks show that MuKV significantly improves answer accuracy, without sacrificing memory and online QA efficiency. Moreover, our compression mechanism alone brings consistent benefits across answer accuracy, memory, and QA efficiency over baselines, showcasing highly effective contribution.
Junbin Xiao, Jiajun Chen, Tianxiang Sun +2
University of Science and Technology of China · National University of Singapore