While multimodal large language models have demonstrated impressive short-term reasoning, they struggle with long-horizon video understanding due to limited context windows and static memory mechanisms that fail to mirror human cognitive efficiency. Existing paradigms typically fall into two extremes: vision-centric methods that incur high latency and redundancy through dense visual accumulation, or text-centric approaches that suffer from detail loss and hallucination via aggressive captioning. To bridge this gap, we propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory. MM-Mem structures memory hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic Schema, enabling the progressive distillation of fine-grained perceptual traces (verbatim) into high-level semantic schemas (gist). Furthermore, to govern the dynamic construction of memory, we derive a Semantic Information Bottleneck objective and introduce SIB-GRPO to optimize the trade-off between memory compression and task-relevant information retention. In inference, we design an entropy-driven top-down memory retrieval strategy. Extensive experiments across 4 benchmarks confirm that MM-Mem achieves state-of-the-art performance on both offline and streaming tasks, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization. Code and associated configurations are publicly available at https://github.com/EliSpectre/MM-Mem.
Figures & tables
Figure 1: Existing memory paradigms (a-b), inspiration (c), and our insight (d). (a) Vision-centric methods incur redundancy and high latency due to dense visual memories. (b) Text-centric methods suffer from information loss during captioning, leading to hallucination and ambiguity. (c) The natural complementarity between vision and text neatly aligns with the distinction between verbatim and gist traces in Fuzzy-Trace Theory . (d) Our MM-Mem is a bottom-up multimodal memory pyramid from sensory buffer to symbolic schema.
Figure 2: Overview of MM-Mem : MM-Mem unifies visual and textual memory through (left) a bottom-up memory construction process, which transforms raw sensory frames into abstract symbolic schemas, and (right) a top-down retrieval process that supports query-adaptive reasoning.
Method
Video-MME
MLVU
Short
Medium
Long
Overall
M-Avg
w/o
w/
w/o
w/
w/o
w/
w/o
w/
Proprietary Models
Gemini 1.5 Pro [ 29 ]
81.7
84.5
74.3
81.0
67.4
77.4
75.0
81.3
-
GPT-4o [ 12 ]
80.0
82.8
70.3
76.6
65.3
72.1
71.9
77.2
64.6
Table 1: Comparison on two long-video understanding benchmarks: Video-MME and MLVU . For Video-MME , we report results under both settings (w/ = with subtitles; w/o = without subtitles). For MLVU , we report M-Avg. Most baseline results are taken from the official leaderboards (as of 2026-01-01) or the respective papers. Results not reported in the original sources are marked with “–“. For agent-based systems, we report the best-performing configuration with a comparable parameter scale.
Figure 3: Visualization of ablation results and memory representations.
Method
VStream-QA-Ego
Accuracy
Score
Video-ChatGPT [ 21 ]
51.7
3.7
MovieChat [ 28 ]
52.2
3.4
Chat-UniVi [ 13 ]
50.9
3.8
LLaMA-VID [ 16 ]
54.8
3.9
Flash-VStream [ 41 ]
59.0
3.9
Table 2: Test performance on VStream-QA-Ego .
Method
HD-EPIC++
Accuracy
Qwen3-VL-8B [ 2 ]
25.88
Qwen2.5-VL-7B [ 2 ]
24.37
LLaVA-Video-7B [ 2 ]
25.37
VideoLLaMA 3-7B [ 40 ]
20.36
Qwen3-VL-4B [ 2 ]
24.91
Table 3: Evaluation on the built HD-EPIC++ .
Method
VRAM(GB)
Con. Time(s)
Infer. Latency(s)
Proprietary MLLMs
VideoAgent
N/A
N/A
67.25
Open-Source MLLMs
Qwen3-VL-8B
22.8
N/A
6.47
Video-RAG
23.0
N/A
25.93
Vgent
18.7
20.18
7.38
Table 4: Efficiency comparison on Video-MME without subtitles. Con. Time denotes construction time, and Infer. Latency denotes inference latency. All timing metrics are normalized to the time required to process one minute of video.
Figure 4: A qualitative example of MM-Mem ’s coarse-to-fine retrieval across memory layers.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: HD-EPIC++ is an egocentric long-horizon kitchen video benchmark with highly detailed annotations, covering fine-grained action perception, temporal reasoning, 3D spatial understanding, object motion, gaze, and diverse VQA tasks (e.g., recipes and ingredients).
LoRA
lora_rank
64
lora_alpha
128
lora_dropout
0.05
SIB-GRPO
epoch
3
Appendix
Table 5: Training and inference hyperparameters.
Figure 6: Answer Agent prompt template instructing the model to select the best option for a video-based multiple-choice question and respond only with the corresponding letter (A–D).
Figure 7: Prompts used by VStream-QA evaluation agent (system and user prompts).
ID
Method
Video-MME
Short
Medium
Long
Overall
(I)
w/o SIB−GRPO
80.1
73.9
71.9
75.3
(II)
w/o VisualMemory
76.5
70.1
66.1
70.9
(III)
w/o TextMemory
80.5
74.2
72.8
75.8
(IV)
w/o Symbolic
81.2
74.6
73.5
76.4
(V)
w/o Episodic
80.2
73.6
71.2
75.0
Appendix
Table 6: Ablation studies of individual modules on Video-MME with subtitles.
Method
VS-Ego
VS-Movie
Acc.
Sco.
Acc.
Sco.
Video-ChatGPT
51.7
3.7
54.4
3.4
MovieChat
52.2
3.4
39.1
2.3
Chat-UniVi
50.9
3.8
54
3.4
LLaMA-VID
54.8
3.9
51.4
3.4
Flash-VStream
59
3.9
56.1
3.4
Appendix
Table 7: Performance comparison on the VStream-QA benchmark, evaluated on the VS-Ego and VS-Movie splits in terms of accuracy (Acc.) and score (Sco.).