Deploying Multimodal Large Language Models as the brain of embodied agents remains challenging, particularly under long-horizon observations and limited context budgets. Existing memory assisted methods often rely on textual summaries, which discard rich visual and spatial details and remain brittle in non-stationary environments. In this work, we propose a non-parametric memory framework that explicitly disentangles episodic and semantic memory for embodied exploration and question answering. Our retrieval-first, reasoning-assisted paradigm recalls episodic experiences via semantic similarity and verifies them through visual reasoning, enabling robust reuse of past observations without rigid geometric alignment. In parallel, we introduce a program-style rule extraction mechanism that converts experiences into structured, reusable semantic memory, facilitating cross-environment generalization. Extensive experiments demonstrate state-of-the-art performance on embodied question answering and exploration benchmarks, yielding a 7.3% gain in LLM-Match and an 11.4% gain in LLM MatchXSPL on A-EQA, as well as +7.7% success rate and +6.8% SPL on GOAT-Bench. Analyses reveal that our episodic memory primarily improves exploration efficiency, while semantic memory strengthens complex reasoning of embodied agents.
Figures & tables
Fig. 1 : Overview of our proposed HIMM. Episodic memory and semantic memory guide agent’s efficient exploration and accurate reasoning, acrossing sequential cognitive states which governed by meta memory.
TABLE I: Performance comparison on A-EQA across different question categories. For each category, we report the LLM-Match score (M) and the LLM-Match × SPL score. “CG” denotes ConceptGraphs. Results marked with * are taken from 3D-Mem [ 3 ] .
Fig. 2 : Illustration of our semantic space for embodied agents.
Fig. 3 : Prompt templates for visual reasoning on episodic memory recall.
Fig. 4 : Qualitative case study on A-EQA. Compared with 3D-Mem, our method demonstrates accurate cross-episode recall and more efficient exploration.
Fig. 5 : Qualitative examples of how program-style enhanced rules for semantic memory impact the reasoning trace for answering questions. We highlight failed reasoning trajectory in red and successful ones in green.
Method
Success Rate (%) ↑
SPL (%) ↑
Traditional Methods
Modular GOAT [ 10 ] *
24.9
17.2
Modular CLIP on Wheels [ 10 ] *
16.1
10.4
SenseAct-NN Skill Chain [ 10 ] *
29.5
11.3
SenseAct-NN Monolithic [ 10 ] *
12.3
6.8
MLLM-based Exploration
TABLE II: Results on GOAT-Bench under the Val Unseen split. “CG” denotes ConceptGraphs [ 23 ] . Methods marked with * are reported from GOAT-Bench, while those marked with † use MLLM-based exploration.
Method
Episodic Memory
Semantic Memory
LLM-Match
LLM-Match × SPL
Ours (Qwen2.5-VL-7B-Instruct)
✓
✓
43.1
23.7
✓
41.6
23.2
✓
39.1
17.5
Ours (GPT-4o)
✓
✓
65.6
48.7
✓
62.3
47.2
✓
60.9
41.1
TABLE III : Ablation studies on our method by selectively disabling episodic or semantic memory. Results are reported with two backbone MLLMs.
Long-horizon embodied interaction requires agents to retain and continually update information about the environment as they observe, act, and encounter change. Yet current agents struggle to maintain such memory reliably. Our analysis traces this limitation to four key deficiencies: weak fine-grained visual memory, unreliable dynamic world-state tracking, failing to record world state revealed by interaction outcomes, and limited generalization from prior experience. However, existing benchmarks do not directly assess these memory capabilities during long-horizon embodied interaction. To address this gap, we introduce EmbodiedMemory-Bench (EMem-Bench), comprising 2,554 interactive episodes across four task families. EMem-Bench requires agents to build and update memory from interaction history, then use it to complete a later task by acting in the environment. We further present Embodied-Memorizer (EMem), an external memory system that organizes embodied experience into spatial, event, and scene memories. We also train EMem-8B, an 8B policy that manages and uses these memories. We evaluate a diverse range of open-source and proprietary MLLMs and representative multimodal memory systems. Results show that current models remain weak and uneven across the four challenges. Under matched backbones, EMem achieves the best overall performance among the evaluated memory systems and improves both open-source and proprietary models, while EMem-8B further improves over its backbone. Project page: https://zju-omniai.github.io/EmbodiedMemoryBench/
Lizhou Liang, Xinyu Zhong, Miao Pan +7
Zhejiang University · Central South University · Institute of Software, Chinese Academy of Sciences
Long-term memory is a critical challenge for Large Language Model agents, as fixed context windows cannot preserve coherence across extended interactions. Existing memory systems represent conversation history as unstructured embedding vectors, retrieving information through semantic similarity. This paradigm fails to capture the associative structure of human memory, wherein related experiences progressively strengthen interconnections through repeated co-activation. Inspired by cognitive neuroscience, we identify three mechanisms central to biological memory: association, consolidation, and spreading activation, which remain largely absent in current research. To bridge this gap, we propose HeLa-Mem, a bio-inspired memory architecture that models memory as a dynamic graph with Hebbian learning dynamics. HeLa-Mem employs a dual-level organization: (1) an episodic memory graph that evolves through co-activation patterns, and (2) a semantic memory store populated via Hebbian Distillation, wherein a Reflective Agent identifies densely connected memory hubs and distills them into structured, reusable semantic knowledge. This dual-path design leverages both semantic similarity and learned associations, mirroring the episodic-semantic distinction in human cognition. Experiments on LoCoMo demonstrate superior performance across four question categories while using significantly fewer context tokens. Code is available on GitHub: https://github.com/ReinerBRO/HeLa-Mem
Jinchang Zhu, Jindong Li, Cheng Zhang +2
The Hong Kong University of Science and Technology (Guangzhou) · Jilin University · The Chinese University of Hong Kong
Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests. For embodied agents that control robots, visual and spatial observations introduce a broader choice of memory modalities, affecting both the information available for subsequent tasks and the model inference costs of constructing and using memory. We introduce MeMoREval, a benchmark for studying this problem through a controlled experience-request protocol. Agents first experience a sequence of events without knowing the subsequent tasks, and subsequently they use memory to respond to information and action requests without receiving new observations. The benchmark comprises 53 tasks across four simulated indoor environments and measures task success alongside the inference costs of memory construction and use. Our reference evaluations reveal that storing frames outperforms storing generated descriptions at lower write-time inference cost for the primary model, while storing frames with scene graphs achieves the highest aggregate task success for every tested model. Extensions to accumulated experience, selective frame retention, and visual input to memory writers demonstrate how the benchmark accommodates further memory-design investigations. Two physical-robot demonstrations establish the feasibility of applying its protocol beyond simulation.
Junxiang Wang, Yuchen Liu, Yi Wu +6
Dexmate · Carnegie Mellon University · Purdue University