KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

Zihan Chen; Xuejian Rong; Xiaojuan Wang; Boqing Gong; Adi Zicher; Yael Pritch; Nikhil Karnad

cs.CVSep 26, 2026

KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

Authors: Zihan Chen, Xuejian Rong, Xiaojuan Wang, Boqing Gong, Adi Zicher, Yael Pritch, Nikhil Karnad

Organizations: University of Virginia · Google

Abstract

Vision-language models are increasingly used to understand long videos and continuous streams. However, dense visual tokens accumulate with video duration, making long-context inference prohibitively expensive. Existing training-free visual-token selection methods reduce this cost by retaining informative tokens, but may lose coherent event evidence and fail to distinguish detailed recent observations from long-range history. We propose KeyRec, a training-free framework for constructing bounded visual memory. During query-agnostic writing, KeyRec preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank. Candidate events are proposed according to their novelty relative to previously stored events and maintained through an online add--merge--evict update. When a question arrives, a text-only router adaptively allocates a fixed readout budget between recent and event memory, without reprocessing historical frames. KeyRec operates on model-facing visual embeddings and supports both modular encoder--projector VLMs and the encoder- and projector-free NEO-ov architecture. Across four streaming and long-video benchmarks and three VLM backbones, KeyRec achieves the best compressed performance in 13 of 15 settings using only 10% of the dense decoder-facing visual-token budget. It outperforms the strongest compressed baseline by 2.21--18.37 points on real-time questions, achieves the best compressed result in five of six long-video settings, and performs best in every NEO-ov 2B setting.

Figures & tables

Figure 1: Overview of KeyRec. During query-agnostic writing, incoming frames update a recent cache and a bounded key-event memory through event insertion, consolidation, and eviction. At query time, a text-only router allocates a fixed readout budget between recent and historical evidence for query-adaptive inference without replaying previous frames.

Benchmark	Model	Subset	Vanilla	STC-Pruner	STOM-CTR	CausalMem	KeyRec
OVO-Bench	Gemma 4 E2B	Backward	43.22	41.20	42.30	45.31	43.83
OVO-Bench	Gemma 4 E2B	Realtime	50.72	48.02	47.23	55.49	57.70
OVO-Bench	Gemma 4 E4B	Backward	53.86	53.01	50.49	54.45	54.45
OVO-Bench	Gemma 4 E4B	Realtime	59.83	51.67	53.26	62.51	64.81
OVO-Bench	NEO-ov 2B	Backward	38.66	35.22	–	39.09	41.90
OVO-Bench	NEO-ov 2B	Realtime	58.07	36.84	–	44.41	62.78

Table 1: Main results on streaming and long-video understanding benchmarks. Vanilla retains all visual tokens, whereas compressed methods use a matched decoder-facing budget of approximately

10\%

of the dense visual input. Bold indicates the best compressed result in each row; “–” denotes an unsupported architecture (STOM-CTR: StreamingTOM-CTR).

Figure 2: Readout-budget scaling on StreamingBench. Results are shown for Gemma 4 E2B (left) and Gemma 4 E4B (right). Dashed horizontal lines denote uncompressed Vanilla performance. The horizontal axis reports the decoder-facing visual-token ratio. KeyRec selects each readout from a visual-token memory approximately twice as large, whereas each baseline decodes its complete retained token buffer (STOM-CTR: StreamingTOM-CTR).

Figure 3: Scaling with the number of sampled frames on LongVideoBench’s (900s, 3600s] duration group. All compressed methods use a

10\%

decoder-facing ratio. Missing Vanilla results indicate memory limitations; StreamingTOM-CTR (STOM-CTR) does not support NEO-ov.

Figure 4: Effect of query-adaptive memory allocation on Gemma 4 E4B. All policies use the same constructed memory and decoder-facing budget; only the allocation between recent and event memory differs.

Frames	Method	E2E (s)	Peak Mem (GiB)
64	Vanilla	5.73	17.09
	STOM-CTR	5.92	15.05
	KeyRec	7.08	15.06
256	Vanilla	26.55	39.23
	STOM-CTR	20.04	15.47
	KeyRec	22.81	16.17

Table 2: Latency and memory scaling on LongVideoBench with Gemma E4B. Compressed methods use a

10\%

decoder-facing ratio.

OVO-Bench
Subset	Method	E2B	E4B
Backward	CausalMem	46.70	56.58
	KeyRec	48.61	56.15
Realtime	CausalMem	57.12	62.96
	KeyRec	57.49	65.71
StreamingBench

Table 3: Fixed-budget 1-FPS evaluation on streaming benchmarks with Gemma 4. Both methods use a fixed decoder-facing budget, independent of the observed prefix length. Bold indicates the better result.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Memory-serialization templates. Default preserves memory boundaries using neutral segment labels; Naive removes these boundaries; and Detailed additionally exposes memory types and temporal ranges.

Model	Template	OVO-B	OVO-R	LVB
Gemma 4 E4B	naive	56.40	63.72	41.21
	detailed	53.10	63.78	44.43
	default	54.45	64.81	45.77
NEO-ov 2B	naive	38.78	62.01	48.84
	detailed	39.86	61.90	48.99
	default	41.90	62.78	49.29

Appendix

Table 4: Ablation of memory serialization. All variants use identical selected visual evidence and differ only in their textual scaffold. OVO-B and OVO-R denote the Backward and Realtime subsets of OVO-Bench, and LVB denotes LongVideoBench. Bold indicates the best result for each model and benchmark.

$\boldsymbol{m}$	$\boldsymbol{K_{\mathrm{OVO}}}$	OVO Backward	$\boldsymbol{K_{\mathrm{LVB}}}$	LongVideoBench
8	32	53.31	64	44.88
16	16	54.45	32	45.77
32	8	50.43	16	44.88

Appendix

Table 5: Event granularity under fixed storage with Gemma 4 E4B. We keep the total event capacity

Km

and the recent/event readout allocation fixed while varying the number and visual token size of stored events.

$\boldsymbol{\gamma\backslash\delta}$	$\boldsymbol{1\,\mathrm{s}}$	$\boldsymbol{2\,\mathrm{s}}$	$\boldsymbol{4\,\mathrm{s}}$
0.65	54.48	54.67	54.27
0.75	54.48	$54.45^{\dagger}$	53.64
0.85	54.36	53.48	52.47

Appendix

Table 6: Sensitivity to event-consolidation thresholds on OVO-Bench Backward with Gemma 4 E4B.

\gamma

controls route similarity and

\delta

controls temporal locality.

\dagger

denotes the default configuration used in the main experiments.

Figure 6: Router-state distributions across backbones on OVO-Bench. Each stacked bar shows the percentage of questions assigned to each state. Across backbones, realtime questions favor recent memory, while backward questions shift toward balanced or event-oriented ones.

Parameter	Value	Role
$\gamma$	$0.75$	Route-similarity threshold
$\delta$	$2.0\,\mathrm{s}$	Maximum temporal merge gap
$\beta$	$0.2$	Route update weight
$\lambda$	$0.2$ / $0.0$	Temporal-coverage weight

Appendix

Table 7: Event-memory hyperparameters. The parameters are shared across settings and backbones. The two values of

\lambda

correspond to the streaming and long-video profiles, respectively.

Router state	Streaming	Long video
	Fraction ( $k_{q}$ )	Fraction ( $k_{q}$ )
AllRecent	$0\;(0)$	$0\;(0)$
HeavyRecent	$1/7\;(2)$	$2/7\;(8)$
Balanced	$2/7\;(4)$	$1/2\;(14)$
HeavyEvent	$6/7\;(12)$	$6/7\;(24)$
AllEvent	$1\;(14)$	$1\;(28)$

Appendix

Table 8: Router allocation codebooks. Each entry reports the effective fraction of the readout assigned to event memory and the corresponding number of complete events

k_{q}

in parentheses. The same profile is used across backbones.

Panel of Figure 7 — Figure 7: Text-only router prompts. The streaming router memory prompt interprets recent memory as evidence immediately preceding the query, whereas the offline long video memory router prompt treats it as evidence from the ending portion of the complete video.

Figure 8: KeyRec case study on StreamingBench with NEO-ov 2B. KeyRec constructs query-agnostic key events from selected visual patches, routes the history-dependent question to coverage-oriented retrieval, and produces a fixed-budget readout that supports the correct answer. The red border indicates the selected events after determining the budget allocation by the router.

Model	Method	Backward				Realtime
Model	Method	EPM	ASI	HLD	Avg.	STU	OJR	ATR	ACR	OCR	FPD	Avg.
Gemma 4 E2B	Vanilla	41.41	39.86	48.39	43.22	36.52	44.02	58.62	44.95	63.76	56.44	50.72
	STC-Pruner	36.70	35.81	51.08	41.20	41.57	42.93	56.03	35.78	56.38	55.45	48.02
	StreamingTOM-CTR	36.36	40.54	50.00	42.30	35.96	42.39	47.41	43.12	57.05	57.43	47.23
	CausalMem	40.40	41.22	54.30	45.31	45.51	52.72	62.93	42.20	71.14	58.42	55.49
	KeyRec	37.71	40.54	53.23	43.83	46.63	54.89	62.07	50.46	67.79	64.36	57.70

Appendix

Table 9: Per-task accuracy on the nine OVO-Bench tasks . Bold and underline indicate the best and second-best compressed methods, respectively; Vanilla is shown as the uncompressed reference. Backward contains EPM, ASI, and HLD; Realtime contains STU, OJR, ATR, ACR, OCR, and FPD.

Backbone	Method	CS	OR	AtR	PR	AcR	SU	EU	Cnt	TRU	CR	Overall
Gemma 4 E2B	Vanilla	59.31	54.77	60.00	60.19	41.64	45.53	43.75	43.01	57.32	37.50	51.28
	STC-Pruner	63.41	56.87	56.39	50.93	48.73	42.04	53.75	41.97	53.75	54.69	52.91
	StreamingTOM-CTR	62.46	55.31	56.07	40.74	47.88	46.34	51.88	32.12	53.27	53.12	51.36
	CausalMem	67.82	62.67	68.52	53.70	53.82	56.10	57.50	31.09	58.88	53.12	58.01
	KeyRec	72.24	65.94	74.43	65.74	56.66	57.72	64.38	17.62	64.80	58.59	61.29
Gemma 4 E4B	Vanilla	71.92	65.67	68.20	67.59	59.49	54.47	60.62	54.92	71.03	52.34	63.73

Appendix

Table 10: Per-task accuracy on the ten StreamingBench Real-Time tasks . Bold and underline indicate the best and second-best compressed methods, respectively; Vanilla is shown as the uncompressed reference.

Backbone	Method	Unnorm. Acc.	Level 1	Level 2	Level 3	Relevance	Rel.-Linear	Logic	Total
Gemma 4 E2B	Vanilla	18.47	10.19	6.01	4.84	7.36	17.29	5.09	6.56
	STC-Pruner	14.16	4.69	4.03	3.54	4.35	12.96	3.32	3.99
	StreamingTOM-CTR	15.69	6.73	5.77	3.93	5.92	14.93	3.89	5.21
	CausalMem	15.31	5.37	4.51	4.49	5.24	13.82	3.76	4.72
	KeyRec	15.41	5.69	4.82	4.35	5.24	13.73	4.10	4.84
Gemma 4 E4B	Vanilla	20.09	10.44	7.56	6.03	8.25	18.55	6.47	7.62

Appendix

Table 11: Video-MME v2 official ratings and unnormalized accuracy . Bold and underline indicate the best and second-best compressed methods, respectively; Vanilla is shown as the uncompressed reference.

Explore similar work

Sep 20, 2026

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding

Long-video understanding must capture transient visual evidence under strict token budgets, yet existing methods compress frames, append memory tokens, or alter internal key-value (KV) caches. We introduce Prefix-Steered Recurrent Memory (PREM), a memory-token-free framework for frozen vision-language models (VLMs). PREM separates video ingestion from query answering: a recurrent writer distills visual streams into a compact 256 KiB multi-slot associative state, while a question-conditioned readout adds memory-derived key/value (K/V) steering modulations to existing non-visual prompt prefixes during prefill. This enables write-once, query-many inference without extra prompt tokens or decoding recurrence. Across six long-video benchmarks in offline and streaming end-of-stream settings, PREM consistently outperforms frozen baselines at every evaluated visual budget. Under a constrained budget of 16 frames, PREM improves macro-average accuracy by 3.06% on Qwen2.5-VL-3B, with gains of 11.0% on action antonym identification and 9.9% on localized needle retrieval. These gains require tuning 0.24% of backbone parameters at 0.03 GiB of peak GPU memory overhead.

Siru Zhong, Qiongyan Wang, Xiaohui Lv +5

The Hong Kong University of Science and Technology (Guangzhou) · Beta Infinity · Tsinghua University

Memory-Augmented VLMs Recurrent Neural Networks Long-Video Understanding Memory-Augmented Video Understanding Long-Context VLMs

Ask PDF

Feb 20, 2026

Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory

Streaming video understanding requires models to robustly encode, store, and retrieve information from a continuous video stream to support accurate video question answering (VQA). Existing state-of-the-art approaches rely on key-value caching to accumulate frame-level information over time, but use a limited number of tokens per frame, leading to the loss of fine-grained visual details. In this work, we propose scaling the token budget to enable more granular spatiotemporal understanding and reasoning. First, we find that current methods are ill-equipped to handle dense streams: their feature encoding causes query-frame similarity scores to increase over time, biasing retrieval toward later frames. To address this, we introduce an adaptive selection strategy that reduces token redundancy while preserving local spatiotemporal information. We further propose a training-free retrieval mixture-of-experts that leverages external models to better identify relevant frames. Our method, MemStream, achieves +8.0% on CG-Bench, +8.5% on LVBench, and +2.4% on VideoMME (Long) over ReKV with Qwen2.5-VL-7B.

Vatsal Agarwal, Saksham Suri, Matthew Gwilliam +2

University of Maryland, College Park · TikTok

Streaming Video Understanding Memory-Augmented VLMs Video QA KV-Cache Management Query-Aware Video Frame Selection Visual Token Compression

Ask PDF

Oct 12, 2026

Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding

Hour-long videos, streaming feeds, and movie-length content are pushing vision-language models (VLMs) past their breaking point -- the standard recipe of encoding each frame into visual tokens yields sequences that grow linearly with video length, causing prohibitive memory consumption and inference latency. Existing approaches, such as token reduction and keyframe selection, shorten the token sequence but leave the O(n) scaling intact, failing to resolve the fundamental bottleneck. We propose FramePack-U, a strikingly simple, training-free frame packing framework that compresses arbitrarily long videos into a fixed size visual representation at constant memory cost. FramePack-U requires no fine-tuning or architectural modification -- it plugs directly into off-the-shelf VLMs at inference time. Inspired by FramePack from the generation literature, we propose a content-aware compression strategy for visual understanding under a fixed token budget. Specifically, FramePack-U (1) scores each frame by its semantic similarity to the text prompt and ranks frames in a diversity-driven manner, and (2) packs the selected frames at varying resolutions into a fixed-length context window. This concentrates the memory budget on frames that are both instruction-relevant and visually diverse. While FramePack-U can in theory pack arbitrarily many frames into the fixed budget, we identify an empirical instantiation that generalizes robustly across settings. Extensive experiments on LongVideoBench, MLVU, and Video-MME show that FramePack-U consistently outperforms full-resolution baselines and prior token reduction and keyframe selection methods across diverse model families, parameter scales, and frame budgets. Notably, on Qwen3.5-35B-A3B with a 128-frame budget, FramePack-U yields gains of +12.9%, +10.6%, and +9.7% over the official model on the three benchmarks, respectively.

Shuo Gao, Chenhao Zheng, Zhongzheng Ren

Renmin University of China · University of Washington · UNC-Chapel Hill

Long-Video Understanding Efficient VLM Inference Video Token Compression Query-Aware Video Frame Selection

Ask PDF