KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

Zihan Chen; Xuejian Rong; Xiaojuan Wang; Boqing Gong; Adi Zicher; Yael Pritch; Nikhil Karnad

cs.CVSep 26, 2026

KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

Authors: Zihan Chen, Xuejian Rong, Xiaojuan Wang, Boqing Gong, Adi Zicher, Yael Pritch, Nikhil Karnad

Organizations: University of Virginia · Google

Abstract

Vision-language models are increasingly used to understand long videos and continuous streams. However, dense visual tokens accumulate with video duration, making long-context inference prohibitively expensive. Existing training-free visual-token selection methods reduce this cost by retaining informative tokens, but may lose coherent event evidence and fail to distinguish detailed recent observations from long-range history. We propose KeyRec, a training-free framework for constructing bounded visual memory. During query-agnostic writing, KeyRec preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank. Candidate events are proposed according to their novelty relative to previously stored events and maintained through an online add--merge--evict update. When a question arrives, a text-only router adaptively allocates a fixed readout budget between recent and event memory, without reprocessing historical frames. KeyRec operates on model-facing visual embeddings and supports both modular encoder--projector VLMs and the encoder- and projector-free NEO-ov architecture. Across four streaming and long-video benchmarks and three VLM backbones, KeyRec achieves the best compressed performance in 13 of 15 settings using only 10% of the dense decoder-facing visual-token budget. It outperforms the strongest compressed baseline by 2.21--18.37 points on real-time questions, achieves the best compressed result in five of six long-video settings, and performs best in every NEO-ov 2B setting.

Figures & tables

Figure 1: Overview of KeyRec. During query-agnostic writing, incoming frames update a recent cache and a bounded key-event memory through event insertion, consolidation, and eviction. At query time, a text-only router allocates a fixed readout budget between recent and historical evidence for query-adaptive inference without replaying previous frames.

Benchmark	Model	Subset	Vanilla	STC-Pruner	STOM-CTR	CausalMem	KeyRec
OVO-Bench	Gemma 4 E2B	Backward	43.22	41.20	42.30	45.31	43.83
OVO-Bench	Gemma 4 E2B	Realtime	50.72	48.02	47.23	55.49	57.70
OVO-Bench	Gemma 4 E4B	Backward	53.86	53.01	50.49	54.45	54.45
OVO-Bench	Gemma 4 E4B	Realtime	59.83	51.67	53.26	62.51	64.81
OVO-Bench	NEO-ov 2B	Backward	38.66	35.22	–	39.09	41.90
OVO-Bench	NEO-ov 2B	Realtime	58.07	36.84	–	44.41	62.78

Table 1: Main results on streaming and long-video understanding benchmarks. Vanilla retains all visual tokens, whereas compressed methods use a matched decoder-facing budget of approximately

10\%

of the dense visual input. Bold indicates the best compressed result in each row; “–” denotes an unsupported architecture (STOM-CTR: StreamingTOM-CTR).

Figure 2: Readout-budget scaling on StreamingBench. Results are shown for Gemma 4 E2B (left) and Gemma 4 E4B (right). Dashed horizontal lines denote uncompressed Vanilla performance. The horizontal axis reports the decoder-facing visual-token ratio. KeyRec selects each readout from a visual-token memory approximately twice as large, whereas each baseline decodes its complete retained token buffer (STOM-CTR: StreamingTOM-CTR).

Figure 3: Scaling with the number of sampled frames on LongVideoBench’s (900s, 3600s] duration group. All compressed methods use a

10\%

decoder-facing ratio. Missing Vanilla results indicate memory limitations; StreamingTOM-CTR (STOM-CTR) does not support NEO-ov.

Figure 4: Effect of query-adaptive memory allocation on Gemma 4 E4B. All policies use the same constructed memory and decoder-facing budget; only the allocation between recent and event memory differs.

Frames	Method	E2E (s)	Peak Mem (GiB)
64	Vanilla	5.73	17.09
	STOM-CTR	5.92	15.05
	KeyRec	7.08	15.06
256	Vanilla	26.55	39.23
	STOM-CTR	20.04	15.47
	KeyRec	22.81	16.17

Table 2: Latency and memory scaling on LongVideoBench with Gemma E4B. Compressed methods use a

10\%

decoder-facing ratio.

OVO-Bench
Subset	Method	E2B	E4B
Backward	CausalMem	46.70	56.58
	KeyRec	48.61	56.15
Realtime	CausalMem	57.12	62.96
	KeyRec	57.49	65.71
StreamingBench

Table 3: Fixed-budget 1-FPS evaluation on streaming benchmarks with Gemma 4. Both methods use a fixed decoder-facing budget, independent of the observed prefix length. Bold indicates the better result.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Memory-serialization templates. Default preserves memory boundaries using neutral segment labels; Naive removes these boundaries; and Detailed additionally exposes memory types and temporal ranges.

Model	Template	OVO-B	OVO-R	LVB
Gemma 4 E4B	naive	56.40	63.72	41.21
	detailed	53.10	63.78	44.43
	default	54.45	64.81	45.77
NEO-ov 2B	naive	38.78	62.01	48.84
	detailed	39.86	61.90	48.99
	default	41.90	62.78	49.29

Appendix

Table 4: Ablation of memory serialization. All variants use identical selected visual evidence and differ only in their textual scaffold. OVO-B and OVO-R denote the Backward and Realtime subsets of OVO-Bench, and LVB denotes LongVideoBench. Bold indicates the best result for each model and benchmark.

$\boldsymbol{m}$	$\boldsymbol{K_{\mathrm{OVO}}}$	OVO Backward	$\boldsymbol{K_{\mathrm{LVB}}}$	LongVideoBench
8	32	53.31	64	44.88
16	16	54.45	32	45.77
32	8	50.43	16	44.88

Appendix

Table 5: Event granularity under fixed storage with Gemma 4 E4B. We keep the total event capacity

Km

and the recent/event readout allocation fixed while varying the number and visual token size of stored events.

$\boldsymbol{\gamma\backslash\delta}$	$\boldsymbol{1\,\mathrm{s}}$	$\boldsymbol{2\,\mathrm{s}}$	$\boldsymbol{4\,\mathrm{s}}$
0.65	54.48	54.67	54.27
0.75	54.48	$54.45^{\dagger}$	53.64
0.85	54.36	53.48	52.47

Appendix

Table 6: Sensitivity to event-consolidation thresholds on OVO-Bench Backward with Gemma 4 E4B.

\gamma

controls route similarity and

\delta

controls temporal locality.

\dagger

denotes the default configuration used in the main experiments.

Figure 6: Router-state distributions across backbones on OVO-Bench. Each stacked bar shows the percentage of questions assigned to each state. Across backbones, realtime questions favor recent memory, while backward questions shift toward balanced or event-oriented ones.

Parameter	Value	Role
$\gamma$	$0.75$	Route-similarity threshold
$\delta$	$2.0\,\mathrm{s}$	Maximum temporal merge gap
$\beta$	$0.2$	Route update weight
$\lambda$	$0.2$ / $0.0$	Temporal-coverage weight

Appendix

Table 7: Event-memory hyperparameters. The parameters are shared across settings and backbones. The two values of

\lambda

correspond to the streaming and long-video profiles, respectively.

Router state	Streaming	Long video
	Fraction ( $k_{q}$ )	Fraction ( $k_{q}$ )
AllRecent	$0\;(0)$	$0\;(0)$
HeavyRecent	$1/7\;(2)$	$2/7\;(8)$
Balanced	$2/7\;(4)$	$1/2\;(14)$
HeavyEvent	$6/7\;(12)$	$6/7\;(24)$
AllEvent	$1\;(14)$	$1\;(28)$

Appendix

Table 8: Router allocation codebooks. Each entry reports the effective fraction of the readout assigned to event memory and the corresponding number of complete events

k_{q}

in parentheses. The same profile is used across backbones.

Panel of Figure 7 — Figure 7: Text-only router prompts. The streaming router memory prompt interprets recent memory as evidence immediately preceding the query, whereas the offline long video memory router prompt treats it as evidence from the ending portion of the complete video.

Figure 8: KeyRec case study on StreamingBench with NEO-ov 2B. KeyRec constructs query-agnostic key events from selected visual patches, routes the history-dependent question to coverage-oriented retrieval, and produces a fixed-budget readout that supports the correct answer. The red border indicates the selected events after determining the budget allocation by the router.

Model	Method	Backward				Realtime
Model	Method	EPM	ASI	HLD	Avg.	STU	OJR	ATR	ACR	OCR	FPD	Avg.
Gemma 4 E2B	Vanilla	41.41	39.86	48.39	43.22	36.52	44.02	58.62	44.95	63.76	56.44	50.72
	STC-Pruner	36.70	35.81	51.08	41.20	41.57	42.93	56.03	35.78	56.38	55.45	48.02
	StreamingTOM-CTR	36.36	40.54	50.00	42.30	35.96	42.39	47.41	43.12	57.05	57.43	47.23
	CausalMem	40.40	41.22	54.30	45.31	45.51	52.72	62.93	42.20	71.14	58.42	55.49
	KeyRec	37.71	40.54	53.23	43.83	46.63	54.89	62.07	50.46	67.79	64.36	57.70

Appendix

Table 9: Per-task accuracy on the nine OVO-Bench tasks . Bold and underline indicate the best and second-best compressed methods, respectively; Vanilla is shown as the uncompressed reference. Backward contains EPM, ASI, and HLD; Realtime contains STU, OJR, ATR, ACR, OCR, and FPD.

Backbone	Method	CS	OR	AtR	PR	AcR	SU	EU	Cnt	TRU	CR	Overall
Gemma 4 E2B	Vanilla	59.31	54.77	60.00	60.19	41.64	45.53	43.75	43.01	57.32	37.50	51.28
	STC-Pruner	63.41	56.87	56.39	50.93	48.73	42.04	53.75	41.97	53.75	54.69	52.91
	StreamingTOM-CTR	62.46	55.31	56.07	40.74	47.88	46.34	51.88	32.12	53.27	53.12	51.36
	CausalMem	67.82	62.67	68.52	53.70	53.82	56.10	57.50	31.09	58.88	53.12	58.01
	KeyRec	72.24	65.94	74.43	65.74	56.66	57.72	64.38	17.62	64.80	58.59	61.29
Gemma 4 E4B	Vanilla	71.92	65.67	68.20	67.59	59.49	54.47	60.62	54.92	71.03	52.34	63.73

Appendix

Table 10: Per-task accuracy on the ten StreamingBench Real-Time tasks . Bold and underline indicate the best and second-best compressed methods, respectively; Vanilla is shown as the uncompressed reference.

Backbone	Method	Unnorm. Acc.	Level 1	Level 2	Level 3	Relevance	Rel.-Linear	Logic	Total
Gemma 4 E2B	Vanilla	18.47	10.19	6.01	4.84	7.36	17.29	5.09	6.56
	STC-Pruner	14.16	4.69	4.03	3.54	4.35	12.96	3.32	3.99
	StreamingTOM-CTR	15.69	6.73	5.77	3.93	5.92	14.93	3.89	5.21
	CausalMem	15.31	5.37	4.51	4.49	5.24	13.82	3.76	4.72
	KeyRec	15.41	5.69	4.82	4.35	5.24	13.73	4.10	4.84
Gemma 4 E4B	Vanilla	20.09	10.44	7.56	6.03	8.25	18.55	6.47	7.62

Appendix

Table 11: Video-MME v2 official ratings and unnormalized accuracy . Bold and underline indicate the best and second-best compressed methods, respectively; Vanilla is shown as the uncompressed reference.

Explore similar work

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding
Sep 20, 2026Siru Zhong, Qiongyan Wang, Xiaohui Lv +5Memory-Augmented VLMs Recurrent Neural Networks
The Hong Kong University of Science and Technology (Guangzhou) · Beta Infinity · Tsinghua University
Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory
Feb 20, 2026Vatsal Agarwal, Saksham Suri, Matthew Gwilliam +2Streaming Video Understanding Memory-Augmented VLMs
University of Maryland, College Park · TikTok
Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding
Oct 12, 2026Shuo Gao, Chenhao Zheng, Zhongzheng RenLong-Video Understanding Efficient VLM Inference
Renmin University of China · University of Washington · UNC-Chapel Hill

Sep 20, 2026J/K move · Enter open · S save

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding

Siru Zhong, Qiongyan Wang, Xiaohui Lv, Yuzheng Zhuang+4

The Hong Kong University of Science and Technology (Guangzhou) · Beta Infinity · Tsinghua University

Memory-Augmented VLMs Recurrent Neural Networks Long-Video Understanding Memory-Augmented Video Understanding Long-Context VLMs

Open paper AskPDF arXiv

Long-video understanding must capture transient visual evidence under strict token budgets, yet existing methods compress frames, append memory tokens, or alter internal key-value (KV) caches. We introduce Prefix-Steered Recurrent Memory (PREM), a memory-token-free framework for frozen vision-language models (VLMs). PREM separates video ingestion from query answering: a recurrent writer distills visual streams into a compact 256 KiB multi-slot associative state, while a question-conditioned readout adds memory-derived key/value (K/V) steering modulations to existing non-visual prompt prefixes during prefill. This enables write-once, query-many inference without extra prompt tokens or decoding recurrence. Across six long-video benchmarks in offline and streaming end-of-stream settings, PREM consistently outperforms frozen baselines at every evaluated visual budget. Under a constrained budget of 16 frames, PREM improves macro-average accuracy by 3.06% on Qwen2.5-VL-3B, with gains of 11.0% on action antonym identification and 9.9% on localized needle retrieval. These gains require tuning 0.24% of backbone parameters at 0.03 GiB of peak GPU memory overhead.

Abstract

Explore similar work

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding

Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory

Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding