Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism

Tao Chen; Kun Zhang; Qiong Wu; Xiao Chen; Chao Chang; Xiaoshuai Sun; Yiyi Zhou; Rongrong Ji

cs.CVMar 31, 2026

Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism

Authors: Tao Chen, Kun Zhang, Qiong Wu, Xiao Chen, Chao Chang, Xiaoshuai Sun, Yiyi Zhou, Rongrong Ji

Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · National University of Defense Technology, 230000, P.R. China.

Abstract

Long video understanding is a key challenge that plagues the advancement of Multimodal Large language Models (MLLMs). In this paper, we study this problem from the perspective of visual memory mechanism, and proposed a novel and training-free approach, termed Flexible Memory (FlexMem). In principle, FlexMem aims to mimic human behavior of video watching, i.e., continually watching video content and recalling the most relevant memory fragments to answer the question. In this way, FlexMem can help MLLMs achieve video understanding of infinite lengths, unlike previous methods that process all video information at once and have input upper-limit. Concretely, FlexMem first consider the visual KV caches as the memory sources, and realize the effective memory transfer and writing via a dual-pathway compression design. Afterwards, FlexMem also explores different memory reading strategies for the diverse video understanding tasks, including the popular streaming one. To validate FlexMem, we apply it to two popular video-MLLMs, and conduct extensive experiments on five long video and one streaming video task. The experimental results show that on a single 3090 GPU, our FlexMem can achieve obvious improvements than existing efficient video understanding methods and process more than 1k frames, which also helps the base MLLMs achieve comparable or even better performance than SOTA MLLMs on some benchmarks, e.g. , GPT-4o and Gemini-1.5 Pro.

Figures & tables

Figure 1: Comparison between FlexMem (ours) and existing efficient video understanding methods for MLLMs on five benchmarks. All methods are run on the same device of one 3090 GPU, and our FlexMem presents obvious performance gains.

Figure 2: Illustration of the proposed FlexMem method. (a) FlexMem is an iterative method, and it encodes two types of compressed memories for each video clip

V_{i}

, namely Context Memory

C_{i}

and Local Memory

M_{i}

, based on the metrics of aggregation score

S_{i}

and local saliency score

\hat{S}_{i}

, respectively.

M_{i}

is then stored in the visual memory bank

M_{bank}

, while the context memory

C

are used in the iterative encoding step for information propagation. Besides, we can also retrieval some stored

M_{l}

as the long-term memory for encoding, while it is optional as well as the text instruction

T_{q}

. (b) The stored memories

M_{a}

will be recalled from the memory bank for the decoding of answers

Y

. (c) One intuitive and effective indexing for FlexMem is the Encoding-based one, which uses the cross-attention during memory encoding with

T_{q}

(a) to reflect the relevance of memories. (d) We also investigate the other fast index method, termed MemIndex , based on the compact index tensors for both question and visual memories, of which process is independent to the encoding of memories. Its selection of cache layers and tokens stems from the fitting results of the encoding-based index.

Method	Sampled Frames	Input Tokens	TimeScope	LVBench	MLVU	Video-MME				LongVideoBench
Method	Sampled Frames	Input Tokens	Test	Val	M-avg	Short	Medium	Long	All	Short	Medium	Long	All
LLaVA-Video 7B	64frm	13k	65.0	42.6	71.2	76.1	61.0	52.4	63.2	71.5	60.7	52.1	60.0
AKS [ 42 ]	1fps	13k	85.4	47.4	72.0	77.2	64.8	53.9	65.3	72.3	62.1	57.4	62.7
Panels [ 10 ]	1fps	13k	79.2	-	-	-	62.2	54.0	64.4	-	-	-	-
DToMA [ 61 ]	-	12k	-	-	71.7	-	-	-	65.0	-	-	-	59.6
Video-RAG [ 29 ]	-	15k	-	-	72.4	-	-	-	-	-	-	-	58.7

Table 1: A comparison of FlexMem with SOTA methods based on two recent MLLMs across five long VideoQA benchmarks. Sampled Frames denote the number of frames sampled from the video used for compression or selection, and Input Tokens denote the number of tokens used for question answering. The best and second-best results are shown in bold and underlined respectively. ∗ Tested on one A800.

Method	Sampled Frames	Input Tokens	TimeScope	LVBench	MLVU	Video-MME				LongVideoBench
Method	Sampled Frames	Input Tokens	Test	Val	M-avg	Short	Medium	Long	All	Short	Medium	Long	All
LLaVA-Video 7B	32frm	7k	58.3	41.4	68.5	74.8	58.4	52.0	61.7	70.6	59.0	50.5	58.6
AKS	1fps	7k	84.6	46.6	70.8	74.7	61.9	51.7	62.8	68.1	60.0	54.1	59.7
AdaRETAKE	384frm	40k	78.2	46.8	71.7	74.7	61.8	54.3	63.6	69.0	60.4	53.5	59.8
FlexMem	512/1024frm	13k	85.6	50.2	72.3	76.2	62.7	55.0	64.6	71.5	63.6	57.2	63.0

Table 2: Performance comparison of FlexMem against representative video RAG method (AKS) and visual compression methods based on LLaVA-Video across five long VideoQA benchmarks. All methods runs on a single 3090 with the fully use of memory overhead.

Method	LLM	TimeScope	LVBench	MLVU	Video-MME				LongVideoBench
Method	LLM	Test	Val	M-avg	Short	Medium	Long	All	Medium	Long	All
GPT-5	-	-	-	77.3	-	-	-	81.8	-	-	72.6
GPT-4o	-	-	27.0	64.6	80.0	70.3	65.3	71.9	69.1	60.9	66.7
Gemini-1.5-Pro	-	-	33.1	-	81.7	74.3	67.4	75.0	65.3	58.6	64.0
Video-XL [ 39 ]	7B	-	-	64.9	62.0	53.2	49.2	55.5	49	45.2	50.5
mPLUG-Owl3 [ 60 ]	7B	-	43.5	63.7	70.0	57.7	50.1	59.3	-	-	52.1

Table 3: Comparison between SOTA Video-MLLMs and LLaVA-Video with FlexMem on five long VideoQA benchmarks.

Offline Models
Methods	LLM	# Frames	Backward Tracing
Methods	LLM	# Frames	EPM	ASI	HLD	Average
Gemini-1.5-Pro	-	-	58.6	76.4	52.6	62.5
InternVL-V2 [ 8 ]	8B	64	48.2	57.4	24.7	43.4
LongVU [ 36 ]	7B	1fps	40.7	59.5	4.8	35.0
Online Models

Table 4: Comparison of our method, LLaVA-Video integrated with FlexMem and MemIndex, with SOTA online and offline models on backward tracing task of OVOBench. EPM, ASI and HLD denote EPisodic Memory , Action Sequence Identification and HaLlucination Detection , respectively.

Compression Strategy
	LongVideoBench				LVBench
Choices	Short	Medium	Long	All	Val
Context Compression Only	70.1	64.3	56.2	62.5	49.9
Local Compression Only	70.9	64.6	55.9	62.6	49.7
Dual-Pathway ‡	71.5	65.5	57.3	63.6	51.0
Context during Prefill

Table 5: Ablation studies on different designs of FlexMem under the encoding-based reading setting across two benchmarks. Methods marked with

{\ddagger}

indicate our chosen settings.

Layers	Text	Vision	MLVU				LVBench
Layers	Text	Vision	Single	Multi	Holistic	M-avg	Val
Encoding-based Index			77.1	54.8	77.3	72.4	51.0
All	All	All	76.9	54.0	77.1	72.0	46.3
3	All	All	77.2	53.3	77.5	72.2	46.6
3	Last-Token	All	77.4	53.3	77.3	72.3	46.8
3	Last-Token	AttEnc	77.1	53.1	77.5	72.1	45.7

Table 6: Ablation studies on index token designs of FlexMem with MemIndex on two benchmarks. Single and Multi denote the Single-Detail and Multi-Detail tasks on MLVU, respectively. AttEnc means token selection based on local saliency score.

Figure 3: Qualitative evaluation of FlexMem. Input Video denote the sampled frames, and Key Fragments are the selected clips for answer generation via memory recall mechanism. These results demonstrate FlexMem’s capacity in comprehensive and fine-grained visual understanding.

Explore similar work

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding
Aug 11, 2026Muxin Fu, Yifan Zhang, Wentao Zhang +5Memory-Augmented VLMs Streaming Video Understanding
Tongji University · Nanyang Technological University · University of Michigan +2
FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding
Feb 2, 2026Kangcong Li, Peng Ye, Lin Zhang +4Streaming Video Understanding Memory-Augmented Video Understanding
College of Future Information Technology, Fudan University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China +2
Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding
Jun 24, 2026Baiyang Song, Yuli Lin, Qiong Wu +5Streaming Video Understanding Multimodal Large Language Models
1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China

Aug 11, 2026J/K move · Enter open · S save

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

Muxin Fu, Yifan Zhang, Wentao Zhang, Fangming Guo+4

Tongji University · Nanyang Technological University · University of Michigan+2

Memory-Augmented VLMs Streaming Video Understanding Efficient VLM Inference Memory-Augmented Video Understanding

Open paper AskPDF arXiv

Streaming video understanding requires multimodal large language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term memory that filters temporal redundancy before visual encoding with a latent long-term memory that consolidates historical video content into visual latents accessible to subsequent reasoning. During generation, an attention-guided retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-video benchmarks. Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient reasoning.

Abstract

Explore similar work

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding

Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding