MemEvo: Automatic Discovery of Streaming Video Memory Mechanisms

Guohong Liu; Jialei Ye; Shanhui Zhao; Yunxin Liu; Yuanchun Li

cs.AISep 29, 2026

MemEvo: Automatic Discovery of Streaming Video Memory Mechanisms

Authors: Guohong Liu, Jialei Ye, Shanhui Zhao, Yunxin Liu, Yuanchun Li

Organizations: Institute for AI Industry Research, Tsinghua University · Peking University

Abstract

Query-agnostic streaming video understanding requires vision-language models to continuously compress an indefinitely growing visual stream into a bounded memory before future queries are known. The performance depends critically on the memory mechanism--what observations to preserve, how to represent and consolidate them, and what information to retrieve when a query eventually arrives. Rather than designing a single memory architecture by hand, we formulate memory design as a search problem over executable memory programs. We introduce a lightweight domain-specific language that expresses memory mechanisms through structured primitives for representation, admission, retention, consolidation, budgeting, and retrieval, while enforcing causal and bounded-memory constraints. Although structured, the derived program space remains large and contains heterogeneous, conditionally dependent design choices whose effects can only be assessed via downstream execution. We therefore propose MemEvo, an LLM-driven auto-research framework that uses pretrained LLM as a semantics-aware proposal model to iteratively generate and refine candidate memory programs based on accumulated experimental feedback. At runtime, a deterministic evaluation pipeline validates and evaluates each candidate, while the underlying vision-language model remains frozen throughout discovery. We finally produce a training-free, bounded-memory mechanism. Extensive experiments on StreamingBench and OVO-Bench demonstrate strong streaming video understanding performance together with substantial context and inference efficiency.

Figures & tables

Figure 1: Overview of MemEvo. Streaming memory mechanism is represented as a structured program over three functional memory roles: recent memory, episodic evidence, and gist memory. During the offline discovery stage, an LLM-based research agent proposes candidate programs based on accumulated experimental feedback. At deployment, the selected memory program is executed to causally maintain bounded memory and construct query-time context for the frozen MLLM.

Figure 3: Automated research process of MemEvo. The left panel visualizes the three stages of the search process. Each node represents a program explored during the search, with darker colors indicating better performance on validation set. The dash line connects key milestones, while detailed search paths are hidden. At the end of each stage, the corresponding component of the memory program evolves from the initial program shown at the top to the final one shown at the bottom. The major milestones are also marked at the corresponding positions in the evolution plot on the right.

Training-based Methods
			StreamingBench	OVO-Bench
Method	Size	#Frames	RTVU	RT	BT	Avg.
VideoLLM-online	8B	2 fps	36.0	20.8	17.7	19.3
Flash-VStream	7B	1 fps	23.2	28.4	27.4	27.9
Dispider	7B	1 fps	67.6	54.6	36.1	45.4
TimeChat-Online	7B	1 fps	75.4	61.9	41.7	51.8

Table 1: Experiment results on subsets of StreamingBench and OVO-Bench. The highest score in each column is marked with bold, and the second highest with underline.

Training-based Methods
Method	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT	Avg.
VideoLLM-Online	39.1	40.1	34.5	31.1	46.0	32.4	31.5	34.2	42.5	27.9	36.0
Flash-VStream	25.9	43.6	24.9	23.9	27.3	13.1	18.5	25.2	23.9	48.7	23.2
Dispider	74.9	75.5	74.1	73.1	74.4	59.9	76.1	62.9	62.2	45.8	67.6
TimeChat-Online	80.8	79.7	80.8	83.3	74.8	78.8	78.7	64.2	68.8	58.0	75.3
StreamForest	83.1	82.8	82.7	84.3	77.5	78.2	76.9	69.1	75.6	54.4	77.3

Table 2: Fine-grained subcategory results on StreamingBench Real-Time subset. Baseline scores are officially reported values. The abbr for subcategories stand for: Object Perception , Causal Reasoning , Clips Summarization , Attribute Perception , Event Understanding , Text-Rich Understanding , Prospective Reasoning , Spatial Understanding , Action Perception and Counting .

Panel of Figure 4 — Figure 4: Left figure denotes the marginal effects of MemEvo’s different roles on StreamingBench subsets. The red dashed circle denotes the baseline of only adding recent memory. The right upper denotes the trends of context tokens and memory payload as the frames accumulate; and the right lower denotes average token cost and latency on OVO-Bench compared with uniform sampling.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Search round / DSL edit	Trials	Best Score	Main outcome
$E$	E1: $\operatorname{Retain}(r,B_{E})$ : capacity $\times$ retention	11	0.748356	Explore capacity and FIFO/temporal/reservoir retention
	E2: $\operatorname{Admit}(a)$	12	0.752949	Compare interval, histogram, SSIM, and optical-flow triggers
	E3: admission gap constraints	5	0.753435	Refine histogram- $q50$ admission with minimum/maximum gaps
	E4: $\operatorname{Retain}(r,B_{E})$ : coverage-change retention	9	0.753435	Compare histogram/SSIM/flow coverage criteria and weights
	E5: local program recombination	3	0.740043	Recombine promising admission, gap, retention, and capacity choices
Retrieval	$\operatorname{Retrieve}(E,q;k_{E})$ , $\operatorname{Retrieve}(G,q;k_{G})$	15	0.771990	Select $(k_{E},k_{G})=(8,4)$

Appendix

Table 3: Round-by-round evolution of the automated search.

Training-based Methods
Method	Base Model	Method Source	Data Source
VideoLLM-Online	Llama-3-8B	CVPR 2024	StreamForest paper
Flash-VStream	Qwen2-VL-7B	ICCV 2025	OASIS paper
Dispider	Qwen2-7B	CVPR 2025	OASIS paper
TimeChat-Online	Qwen2.5-VL-7B	ACM MM 2025	HERMES paper
StreamForest	Qwen2-7B	NeurIPS 2025 Spotlight	Official

Appendix

Table 4: Base models, publication information, and data sources of compared methods. “Official” means the scores are from the method’s own paper; otherwise the source is stated explicitly.

# Frames	Retrieval mean (ms)	Packing mean/p95 (s)	Peak GPU Memory (GiB)
64	33.2	1.139 / 1.289	18.875
128	33.3	1.080 / 1.203	18.880
256	33.9	1.142 / 1.283	18.878
512	34.0	1.144 / 1.239	18.890
1024	32.7	1.164 / 1.247	18.893

Appendix

Table 5: Fine-grained reader-side profiling result on StreamingBench.

Metric	Mean	Median	p95
Online (ms/frame)	911.35	937.39	1061.25
Sustainable FPS	1.124	1.067	1.483
Streaming update (ms/frame)	571.07	568.52	640.43
Post-vision memory (ms/frame)	472.87	483.46	543.54

Appendix

Table 6: Writer-side profiling metrics on subset of StreamingBench.

Panel of Figure 5 — Figure 5: Prompt template used in MemEvo.

Figure 6: Case 1. The integration of R/E/G make the MLLM successfully get the correct answer.

Figure 7: Case 2. The MLLM is confused with the time order of events and make a wrong decision.

Explore similar work

Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding
Jun 24, 2026Baiyang Song, Yuli Lin, Qiong Wu +5Streaming Video Understanding Video Understanding
1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China
MEMO: Multi-Level Entity-Aware Memory for Streaming Video Understanding
Sep 30, 2026Yinying Li, Yuqian Fu, Yulin Dai +3Streaming Video Understanding Streaming Video
East China Normal University · King Abdullah University of Science and Technology
Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding
May 8, 2026Hang Wu, Sherin Mary Mathews, Yujun Cai +2Streaming Video Understanding Modern Data-Streaming Systems
University of California, Merced · US Bank · University of Queensland

Jun 24, 2026, cs.CVJ/K move · Enter open · S save

Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding

Baiyang Song, Yuli Lin, Qiong Wu, Tao Chen+4

1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China

Streaming Video Understanding Video Understanding Multimodal Large Language Models Peak Memory

Open paper AskPDF arXiv

Currently, streaming video understanding is still a daunting task for existing \emph{multimodal large language models} (MLLMs). Its difficulties not only lie in handling the ever-increasing video frames, but also in the unpredictability of future video content and input instructions. In this paper, we study this task from the perspective of constructing a dynamic but fixed-budget memory bank, and propose a novel and training-free approach termed \emph{\textbf{CausalMem}}. CausalMem is dedicated to constructing a dynamic visual memory update mechanism, thereby maximizing the amount of information in streaming video within a limited memory space, much like the human brain. In practice, CausalMem estimates the redundancy of visual tokens and updates the memory bank via an online semantic basis, which models the principal semantics of the observed video stream. To validate CausalMem, we apply it to two representative MLLMs, namely LLaVA-OneVision and Qwen2.5-VL respectively, and conduct extensive experiments on both streaming and offline video understanding benchmarks. The experimental results not only show the great advantages than existing methods under both streaming and offline settings, \emph{e.g.},

+3.2\%

and

+3.0\%

average accuracy gains respectively, but also witness the superior semantic preservation for streaming videos, \emph{e.g.}, using 12

k

token budgets to memorize hour-long streaming videos, which achieves more than \textbf{20

\times

} visual token compression ratio and only occupies about \textbf{82 MB} storage. \textbf{Our code} is given in \href{https://github.com/hktk07/CausalMem}{CausalMem}.

Abstract

Explore similar work

Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding

MEMO: Multi-Level Entity-Aware Memory for Streaming Video Understanding

Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding