While multimodal large language models have demonstrated impressive short-term reasoning, they struggle with long-horizon video understanding due to limited context windows and static memory mechanisms that fail to mirror human cognitive efficiency. Existing paradigms typically fall into two extremes: vision-centric methods that incur high latency and redundancy through dense visual accumulation, or text-centric approaches that suffer from detail loss and hallucination via aggressive captioning. To bridge this gap, we propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory. MM-Mem structures memory hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic Schema, enabling the progressive distillation of fine-grained perceptual traces (verbatim) into high-level semantic schemas (gist). Furthermore, to govern the dynamic construction of memory, we derive a Semantic Information Bottleneck objective and introduce SIB-GRPO to optimize the trade-off between memory compression and task-relevant information retention. In inference, we design an entropy-driven top-down memory retrieval strategy. Extensive experiments across 4 benchmarks confirm that MM-Mem achieves state-of-the-art performance on both offline and streaming tasks, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization. Code and associated configurations are publicly available at https://github.com/EliSpectre/MM-Mem.
Figures & tables
Figure 1: Existing memory paradigms (a-b), inspiration (c), and our insight (d). (a) Vision-centric methods incur redundancy and high latency due to dense visual memories. (b) Text-centric methods suffer from information loss during captioning, leading to hallucination and ambiguity. (c) The natural complementarity between vision and text neatly aligns with the distinction between verbatim and gist traces in Fuzzy-Trace Theory . (d) Our MM-Mem is a bottom-up multimodal memory pyramid from sensory buffer to symbolic schema.
Figure 2: Overview of MM-Mem : MM-Mem unifies visual and textual memory through (left) a bottom-up memory construction process, which transforms raw sensory frames into abstract symbolic schemas, and (right) a top-down retrieval process that supports query-adaptive reasoning.
Method
Video-MME
MLVU
Short
Medium
Long
Overall
M-Avg
w/o
w/
w/o
w/
w/o
w/
w/o
w/
Proprietary Models
Gemini 1.5 Pro [ 29 ]
81.7
84.5
74.3
81.0
67.4
77.4
75.0
81.3
-
GPT-4o [ 12 ]
80.0
82.8
70.3
76.6
65.3
72.1
71.9
77.2
64.6
Table 1: Comparison on two long-video understanding benchmarks: Video-MME and MLVU . For Video-MME , we report results under both settings (w/ = with subtitles; w/o = without subtitles). For MLVU , we report M-Avg. Most baseline results are taken from the official leaderboards (as of 2026-01-01) or the respective papers. Results not reported in the original sources are marked with “–“. For agent-based systems, we report the best-performing configuration with a comparable parameter scale.
Figure 3: Visualization of ablation results and memory representations.
Method
VStream-QA-Ego
Accuracy
Score
Video-ChatGPT [ 21 ]
51.7
3.7
MovieChat [ 28 ]
52.2
3.4
Chat-UniVi [ 13 ]
50.9
3.8
LLaMA-VID [ 16 ]
54.8
3.9
Flash-VStream [ 41 ]
59.0
3.9
Table 2: Test performance on VStream-QA-Ego .
Method
HD-EPIC++
Accuracy
Qwen3-VL-8B [ 2 ]
25.88
Qwen2.5-VL-7B [ 2 ]
24.37
LLaVA-Video-7B [ 2 ]
25.37
VideoLLaMA 3-7B [ 40 ]
20.36
Qwen3-VL-4B [ 2 ]
24.91
Table 3: Evaluation on the built HD-EPIC++ .
Method
VRAM(GB)
Con. Time(s)
Infer. Latency(s)
Proprietary MLLMs
VideoAgent
N/A
N/A
67.25
Open-Source MLLMs
Qwen3-VL-8B
22.8
N/A
6.47
Video-RAG
23.0
N/A
25.93
Vgent
18.7
20.18
7.38
Table 4: Efficiency comparison on Video-MME without subtitles. Con. Time denotes construction time, and Infer. Latency denotes inference latency. All timing metrics are normalized to the time required to process one minute of video.
Figure 4: A qualitative example of MM-Mem ’s coarse-to-fine retrieval across memory layers.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: HD-EPIC++ is an egocentric long-horizon kitchen video benchmark with highly detailed annotations, covering fine-grained action perception, temporal reasoning, 3D spatial understanding, object motion, gaze, and diverse VQA tasks (e.g., recipes and ingredients).
LoRA
lora_rank
64
lora_alpha
128
lora_dropout
0.05
SIB-GRPO
epoch
3
Appendix
Table 5: Training and inference hyperparameters.
Figure 6: Answer Agent prompt template instructing the model to select the best option for a video-based multiple-choice question and respond only with the corresponding letter (A–D).
Figure 7: Prompts used by VStream-QA evaluation agent (system and user prompts).
ID
Method
Video-MME
Short
Medium
Long
Overall
(I)
w/o SIB−GRPO
80.1
73.9
71.9
75.3
(II)
w/o VisualMemory
76.5
70.1
66.1
70.9
(III)
w/o TextMemory
80.5
74.2
72.8
75.8
(IV)
w/o Symbolic
81.2
74.6
73.5
76.4
(V)
w/o Episodic
80.2
73.6
71.2
75.0
Appendix
Table 6: Ablation studies of individual modules on Video-MME with subtitles.
Method
VS-Ego
VS-Movie
Acc.
Sco.
Acc.
Sco.
Video-ChatGPT
51.7
3.7
54.4
3.4
MovieChat
52.2
3.4
39.1
2.3
Chat-UniVi
50.9
3.8
54
3.4
LLaMA-VID
54.8
3.9
51.4
3.4
Flash-VStream
59
3.9
56.1
3.4
Appendix
Table 7: Performance comparison on the VStream-QA benchmark, evaluated on the VS-Ego and VS-Movie splits in terms of accuracy (Acc.) and score (Sco.).
Memory has become an increasingly important component of agentic systems, as these systems are expected to reason over long-term experience. However, prior work has largely focused on unimodal memory, leaving multimodal memory relatively underexplored despite its central role in real-world applications. Compared with unimodal settings, multimodal memory introduces additional challenges, including heterogeneous input integration, person-centric information alignment, and evidence aggregation across different granularities. We present PyraVid, a hierarchical multimodal memory framework inspired by Event Segmentation Theory from cognitive science. PyraVid organizes long videos into a coarse-to-fine pyramid structure, enabling structured memory access and effective evidence aggregation. It further supports structure-guided memory expansion with pruning, allowing the retrieval of related events with strong causal connectivity but low semantic similarity while reducing noise. Experiments on multiple long-video understanding benchmarks show that PyraVid consistently improves performance across datasets, model scales, and question types, highlighting the effectiveness of hierarchical multimodal memory for long-horizon reasoning.
Sikuan Yan, Sicheng Dong, Haotong Wang +8
1Ludwig Maximilian University of Munich · 4Technical University of Munich · 3Huawei Heisenberg Research Center (Munich) +1
Existing multimodal long-term memory agents use external memory to overcome the limited context available for long videos. However, most methods emphasize what to store rather than how stored memory should be retrieved. When retrieval becomes inaccurate or repeatedly fails to obtain useful evidence, existing agents lack mechanisms to diagnose failures from previous task trajectories and adapt future search strategies.We introduce Reflective Retrieval Memory (RRM), a reflective memory framework for long-horizon multimodal reasoning. RRM augments an entity-centric multimodal memory graph with reflective experience memory, which distills transferable procedural retrieval knowledge from historical task trajectories. Unlike episodic and semantic memories that preserve factual evidence from the current video, reflective experience memory captures reusable search strategies across tasks. RRM converts retrieved experiences into query-level guidance, while answer generation remains conditioned only on factual evidence newly retrieved from the current video. A lifecycle management mechanism further regulates experience memory through usage frequency, reuse feedback, and temporal decay, thereby reducing redundancy and noise. RRM consistently outperforms previous state-of-the-art approaches on M3-Bench-Robot, M3-Bench-Web, and Video-MME-Long, demonstrating the effectiveness of reflective retrieval memory for long-horizon multimodal reasoning.
Jingxiang Fan, Junbao Zhuo, Bochao Zou
University of Science and Technology Beijing 30 Xueyuan Road, Haidian District, Beijing 100083, China
Multimodal large language models excel on short clips but struggle on hour-long videos in an online setting, where frames are processed incrementally under limited memory. Existing online methods either retain compact visual representations that lack semantic structure, or build higher-level memory stores organized around temporal proximity rather than explicit causal links, leaving multi-hop narrative reasoning to be reconstructed by the LLM at every query. We bridge this gap with Homer, a Hierarchical Online Memory Exploration and Reasoning framework. Homer's memory mirrors the multi-scale structure of long videos, ranging from raw perception, to recurring entities, to events connected by explicit temporal and causal relations. Its agentic reasoner then explores this memory the way humans do, locating the relevant scene, looking up details, and composing the answer through multi-round memory retrieval, with a harness that verifies and corrects each step. Homer outperforms the previous best agent method by +5.5, +10.8, and +4.4 points on M3-Bench-robot, M3-Bench-web, and Video-MME-Long, and consistently lifts three various LLM backbones, indicating a model-agnostic structural capability for grounded retrieval over long videos.
Yixin Ji, Fanghua Ye, Juntao Li +5
1Soochow University · 2Tencent Hunyuan Multimodal Department