Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory

Vatsal Agarwal; Saksham Suri; Matthew Gwilliam; Pulkit Kumar; Abhinav Shrivastava

cs.CVFeb 20, 2026

Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory

Authors: Vatsal Agarwal, Saksham Suri, Matthew Gwilliam, Pulkit Kumar, Abhinav Shrivastava

Organizations: University of Maryland, College Park · TikTok

Abstract

Streaming video understanding requires models to robustly encode, store, and retrieve information from a continuous video stream to support accurate video question answering (VQA). Existing state-of-the-art approaches rely on key-value caching to accumulate frame-level information over time, but use a limited number of tokens per frame, leading to the loss of fine-grained visual details. In this work, we propose scaling the token budget to enable more granular spatiotemporal understanding and reasoning. First, we find that current methods are ill-equipped to handle dense streams: their feature encoding causes query-frame similarity scores to increase over time, biasing retrieval toward later frames. To address this, we introduce an adaptive selection strategy that reduces token redundancy while preserving local spatiotemporal information. We further propose a training-free retrieval mixture-of-experts that leverages external models to better identify relevant frames. Our method, MemStream, achieves +8.0% on CG-Bench, +8.5% on LVBench, and +2.4% on VideoMME (Long) over ReKV with Qwen2.5-VL-7B.

Figures & tables

Figure 1: (a) We propose constructing the key–value cache with sparse sliding-window attention and design an adaptive key selection (AKS) strategy to sparsify the sliding window. (b) During question-answering, we merge complementary retrieval signals from external models via a training-free mixture-of-experts.

Figure 2: Increasing per-frame token budget leads to substantial declines in average layer-wise recall across a variety of different questions.

Figure 3: In (a), we identify a systematic trend where query-frame similarity scores progressively increase across the video. In (b), we observe that self-similarity maps of the key representations become more redundant as we increase tokens per frame.

Figure 4: We compute the normalized entropy for 677 sliding windows under 2 separate token budgets. At the higher token budget, the sliding window attention tends to exhibit higher entropy. Rather than increased informativeness, this suggests a struggle to focus on relevant frames at higher token budgets.

Method	Num Tokens	CG-Bench	LVBench
Qwen2.5-VL-7B	64	34.27	37.25
+ ReKV ( $\small\text{0.5 FPS}\rightarrow\text{128}$ )	64	35.23	37.06
Qwen2.5-VL-7B	128	37.02	39.19
+ ReKV ( $\small\text{0.5 FPS}\rightarrow\text{128}$ )	128	37.00	36.60
Qwen2.5-VL-7B	256	38.43	41.51
+ ReKV ( $\small\text{0.5 FPS}\rightarrow\text{128}$ )	256	37.45	37.96

Table 1: We show how question-answering performance with KV-cache memory (using ReKV as a reference implementation) degrades when passed in frames at higher token budgets. Here, (

\text{0.5 FPS}\rightarrow\text{128}

) denotes that the video is sampled at 0.5 FPS with 128 frames used for question-answering.

Figure 5: We measure the recall for which, given a question, each layer retrieves the features for the CG-Bench “clue” frames. There is a massive variance in recall scores, but in general, they tend to be quite low.

Figure 6: (a) Our model processes the actual video stream a single time only (instead of once per question, as in offline VQA systems) to encode video features. We perform adaptive key selection to construct a sparse KV-cache. (b) MemStream uses matching with the question tokens to retrieve the relevant K/V features from the cache. Since we select features on a frame-wise basis, we can perform this retrieval with arbitrary vision-language models. We leverage this to construct a training-free mixture-of-experts for the retrieval.

Settings						Benchmarks
Model	$N_{\text{tokens}}$	$N_{\text{frames}}$	Encoding	Retrieval	Train?	CG-Bench	LVBench	VideoMME
Offline Video Question Answering
Qwen2.5-VL-7B
+ Uniform Sample	17K	( $\small\text{0.5 FPS}\to 128$ )	N/A	N/A	✗	38.43	41.51	55.67
+ Uniform Sample	68K	( $\small\text{0.5 FPS}\to 512$ )	N/A	N/A	✗	41.48	40.03	52.11
Online Video Question Answering

Table 2: Offline VQA Results. Effectiveness of MemStream for long video question answering. We highlight the best performance in bold . For VideoMME, we use the “Long” subset only.

Settings			RVS-Ego		RVS-Movie		Running Speed		Memory Usage
Model	Encoding	Retrieval	Acc	Score	Acc	Score	Video Enc.	Latency	GPU	KV-Cache
Qwen2.5-VL-7B
ReKV ( Di et al., 2025 )	Full	Internal	64.2	4.00	61.6	3.65	8.47 FPS	2.8s	29 GB	11.1 GB/h
MemStream (Ours)	AKS	Internal	67.8	4.01	59.1	3.60	8.50 FPS	2.6s	29 GB	11.1 GB/h
MemStream (Ours)	AKS	MoE	67.4	4.01	59.7	3.60	8.68 FPS	2.6s	32 GB	11.1 GB/h

Table 3: StreamingVQA Results. Evaluation of MemStream on RVS-Ego and RVS-Movie, measuring answer accuracy and quality, as well as runtime and memory usage. Following ReKV, we use GPT-3.5-Turbo as our LLM judge.

Encoding Type	Strategy	Comp. Rate	CG-Bench	LVBench	VideoMME
	Full	${\sim}1\times$	36.17	39.64	51.78
Patch-wise	A.1	${\sim}4\times$	40.02	40.41	51.78
	A.2	${\sim}16\times$	41.15	42.93	52.56
	B.1	${\sim}12\times$	42.18	43.06	52.89
	AKS	${\sim}16\times$	41.63	43.77	54.56
Frame-wise	A.3	${\sim}8\times$	41.63	42.35	53.78

Table 4: Encoding Strategy Ablations. Comparison of patch-wise and frame-wise encoding strategies across benchmarks.

Retrieval Strategy	External Model	CG-Bench	LVBench	VideoMME
Internal Only	–	41.63	43.77	54.56
External Only	CLIP	42.39	47.45	53.67
External Only	PECore	43.21	47.19	52.33
MoE (Ours)	CLIP	43.84	47.39	55.22
MoE (Ours)	PECore	44.19	48.10	54.22

Table 5: Retrieval Strategy Ablation. Comparison of internal, external, and MoE retrieval. We use the AKS encoding strategy and RRF fusion, with equal weights for the internal and external rankings.

Fusion Method	CG-Bench	LVBench	VideoMME
L2-Concat	43.57	48.03	52.89
RRF (Ours)	44.19	48.10	54.22

Table 6: Fusion Strategy Ablation. Comparison of early- and late-fusion methods for combining internal and external retrieval signals. We use PE-Core as the external encoder and the AKS encoding strategy.

Figure 7: Qualitative Results. We compare ReKV’s (red) and MemStream’s (blue) respective retrievals and answers. The ground-truth segment is shown in green.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	# Videos	Avg. Length	# QA Pairs
CG-Bench ( Chen et al., 2024 )	1,219	27 min.	12,129
LVBench ( Wang et al., 2025 )	103	68 min.	1,549
VideoMME (Long) ( Fu et al., 2025 )	300	41 min.	900
RVS-Ego ( Zhang et al., 2024 )	10	60 min.	1,465
RVS-Movie ( Zhang et al., 2024 )	22	30 min.	1,905

Appendix

Table 1: Dataset statistics. We detail the number of videos, questions, and average length for each benchmark.

Figure 8: More Qualitative Examples of Retrievals. We select a representative subset of retrievals for each method.

Encoding	Comp. Rate	CG-Bench	LVBench	VideoMME
Full	$1\times$	36.17	39.64	51.78
Pooled (A.1)	${\sim}4\times$	40.02	40.41	51.78
	${\sim}8\times$	39.81	40.35	50.33
	${\sim}16\times$	39.54	39.57	50.22
Dilated (A.2)	${\sim}4\times$	40.17	41.12	51.89
	${\sim}8\times$	41.69	42.67	51.78

Appendix

Table 2: Effect of Sliding-Window Compression Rate. We evaluate performance of static attention patterns at varying sliding-window compression rates.

Settings				CG-Bench Average Recall@64 Scores
Model	Method	Encoding	Retrieval	Entity Cognition	Entity Perception	Event Cognition	Event Perception	Text Perception	Mean
Internal Retrieval
	Uniform	Uniform	Internal	20.57	22.20	21.98	22.29	26.12	22.24
	ReKV	Full	Internal	14.49	15.59	16.03	15.22	19.63	16.27
	MemStream	A.2 ( $\times 4$ )	Internal	19.97	21.42	20.86	20.23	27.31	21.59
	MemStream	A.2 ( $\times 16$ )	Internal	24.89	27.17	26.08	26.94	35.31	27.41

Appendix

Table 3: CG-Bench Recall. We show how frequently the top 64 retrieves features overlap the features from the ground truth clue frames in CG-Bench. ReKV’s struggles to retrieve the features corresponding to the most relevant frames help explain its degradation in question-answering performance.

Settings				CG-Bench Accuracy
Model	Method	Encoding	Retrieval	Entity Cognition	Entity Perception	Event Cognition	Event Perception	Text Perception	Mean
Internal Retrieval
	Uniform	Uniform	Internal	34.03	37.06	36.76	35.35	49.52	38.43
	ReKV	Full	Internal	33.12	36.35	34.73	31.36	45.72	36.17
	MemStream	A.2 ( $\times 4$ )	Internal	35.70	39.08	39.26	36.02	52.42	40.17
	MemStream	A.2 ( $\times 16$ )	Internal	36.61	40.14	39.26	37.99	54.36	41.15

Appendix

Table 4: CG-Bench Breakdown. We breakdown CG-Bench performance for a subset of question-types. We bold the best result and underline the second-best.

Settings				LVBench Accuracy
Model	Method	Encoding	Retrieval	Entity Recog.	Event Understand.	Key Info. Retrieval	Reasoning	Summarizing	Temporal Ground.	Mean
Internal Retrieval
	Uniform	Uniform	Internal	41.65	39.72	42.96	42.29	43.10	39.09	41.51
	ReKV	Full	Internal	38.55	39.57	43.30	40.30	29.31	35.00	39.64
	MemStream	A.2 ( $\times 4$ )	Internal	40.33	41.11	44.33	41.29	31.03	36.82	41.12
	MemStream	A.2 ( $\times 16$ )	Internal	41.80	42.35	45.02	41.79	34.48	39.09	42.93

Appendix

Table 5: LVBench Breakdown. We breakdown LVBench performance across all question-types. We bold the best result and underline the second-best.

Figure 9: We plot layer-wise average recall scores for full sliding-window attention, AKS, and AKS combined with retrieval mixture-of-experts with PECore as the external retriever.

Figure 10: Layer-Wise Query-Frame Scores 64 Tokens Per Frame @ 241 Frames

Figure 11: Layer-Wise Query-Frame Scores 256 Tokens Per Frame @ 241 Frames

Figure 12: Layer-Wise Self-Similarity Matrices at 64 Tokens Per Frame @ 241 Frames

Figure 13: Layer-Wise Self-Similarity Matrices at 256 Tokens Per Frame @ 241 Frames

Explore similar work

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding
Aug 11, 2026Muxin Fu, Yifan Zhang, Wentao Zhang +5Memory-Augmented VLMs Streaming Video Understanding
Tongji University · Nanyang Technological University · University of Michigan +2
Decouple and Cache: KV Cache Construction for Streaming Video Understanding
May 3, 2026Zhanzhong Pang, Dibyadip Chatterjee, Fadime Sener +1Streaming Video Understanding KV Caching
National University of Singapore.
MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering
May 21, 2026Junbin Xiao, Jiajun Chen, Tianxiang Sun +2KV Caching Video QA
University of Science and Technology of China · National University of Singapore

Aug 11, 2026J/K move · Enter open · S save

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

Muxin Fu, Yifan Zhang, Wentao Zhang, Fangming Guo+4

Tongji University · Nanyang Technological University · University of Michigan+2

Memory-Augmented VLMs Streaming Video Understanding Efficient VLM Inference Memory-Augmented Video Understanding

Open paper AskPDF arXiv

Streaming video understanding requires multimodal large language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term memory that filters temporal redundancy before visual encoding with a latent long-term memory that consolidates historical video content into visual latents accessible to subsequent reasoning. During generation, an attention-guided retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-video benchmarks. Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient reasoning.

Abstract

Explore similar work

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

Decouple and Cache: KV Cache Construction for Streaming Video Understanding

MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering