Organizations: Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence (BAAI)
Recent advances in generative foundation models have expanded the capabilities of embodied AI, enabling increasingly general navigation, manipulation, and long-horizon task execution. However, effective interaction with dynamic environments requires more than only strong perception and reasoning: agents must act with partial knowledge and retain, retrieve, and update information from past observations, actions, and experiences. This survey examines embodied memory as a mechanism for preserving decision-relevant information beyond the current observation. We organize existing approaches along two complementary dimensions: six fundamental memory requirements, spanning world-state persistence, task progress, belief maintenance, and experiential learning; and eight architectural categories, including recurrent states, fast weights, visual and symbolic representations, spatial memory, and experience-based systems. We analyze their capabilities and limitations, and examine how existing benchmarks evaluate memory-dependent behavior, distinguishing genuine use of historical information from general task performance. Finally, we identify open challenges in selecting relevant experiences, maintaining consistent beliefs in changing environments, and consolidating knowledge across episodes. Our analysis provides a framework for understanding memory as an essential component of reliable, adaptive, and lifelong-horizon embodied intelligence.
Figures & tables
Figure 1: Conceptual boundaries and implementations of embodied memory. The six requirements in Sec. 3 describe task demands; the architectural categories in Sec. 4 describe implementation choices.
Figure 2: Representative works organized by embodied memory requirement. Six timelines group studies by the historical information an agent must preserve: hidden world state, action-induced state change, task state, delayed-relevance evidence, active belief maintenance, and cross-episode experience. Each row identifies the corresponding computational challenge and arranges works by year. Solid boxes denote methods or systems; dashed boxes marked B denote benchmarks. Works may appear in multiple rows when they address several requirements. Timelines are independently spaced for readability, with no ordering implied among works from the same year. Dates follow the cited bibliography, using the conference year where available and the preprint or report year otherwise.
Memory Requirement
What Must Persist?
Core Computational Challenge
Memory Operation
Typical Failure
Hidden world state
Entity identity, location, and spatial relations
Temporal entity binding across occlusion and viewpoint changes
Retain + retrieve
Identity drift; forgotten location
Action-induced state change
Current world state and its temporal versions
State supersession after the agent changes the environment
Update + invalidate
Stale belief
Task state
Phase, progress, count, constraints, and failures
Latent task-state tracking across visually similar execution stages
Accumulate + update
Repeated action; skipped stage
Delayed-relevance evidence
Sparse historical cues with future utility
Future-utility estimation under delayed relevance
Selective write + recall
Missed critical event
Active belief maintenance
World belief and uncertainty
Belief calibration under uncertainty and environmental change
Revise + reobserve
Belief inertia
Cross-episode experience
Reusable skills, rules, and environmental knowledge
Experience abstraction across repeated episodes
Consolidate + transfer
Overgeneral-ization
Table 1: A computational taxonomy of non-Markovian embodied memory requirements. Each requirement is associated with a distinct latent state, computational challenge, memory operation, and characteristic failure mode.
Memory Architecture
Primary Sec. 3 Requirement
Main Strength
Main Limitation
A1. Recurrent state
Task-state tracking, continuous belief
Fast, bounded-size updates
Information bottleneck; weak random access
A2. Fast-weight / TTT
Continuous belief, spatial or interaction-state accumulation
Table 2: Capability envelopes of embodied-memory architectures. The requirement column is a diagnostic cross-reference to Sec. 3 ; A1–A7 are non-exclusive mechanism or organization families, while A8 denotes their composition. Episodic retrieval and skill consolidation are two modes within A7, not additional top-level categories.
VLM answer/plan quality and physical completion are different outcomes
Table 3: Evaluation interfaces, outcomes, and memory implications of general embodied benchmarks. Entries summarize the named benchmark protocols rather than prescribing one universal sensor or action configuration. A score should be interpreted using its own completion rule and observation privileges.
Benchmark
EB
SS
TS
FU
BC
EA
RMBench [ 19 ]
✓
✓
RoboMME [ 20 ]
✓
✓
✓
LIBERO-Mem [ 114 ]
✓
RoboMemArena [ 66 ]
✓
✓
✓
MemoryBench [ 93 ]
✓
RoboTwin-MeM [ 8 ]
△
Table 4: Coverage of the six memory requirements from Sec. 3 in memory-oriented benchmarks. EB (R1): Entity Binding; SS (R2): State Supersession; TS (R3): Task-State Tracking; FU (R4): Future-Utility Estimation; BC (R5): Belief Calibration; EA (R6): Experience Abstraction. These are the computational operations associated with the same six requirements, not six additional memory types. ✓denotes systematic task coverage, while △ indicates partial or indirect pressure without dedicated evaluation. The mapping is our interpretation of task protocols, not a claim of causal isolation; HuMeGo concerns pre-goal object history, with selection and state-change coverage treated as indirect; EMem-Bench’s EA mark concerns its controlled generalization family, while MemTransfer probes reuse under mismatch rather than learning an abstraction mechanism.
Long-horizon embodied interaction requires agents to retain and continually update information about the environment as they observe, act, and encounter change. Yet current agents struggle to maintain such memory reliably. Our analysis traces this limitation to four key deficiencies: weak fine-grained visual memory, unreliable dynamic world-state tracking, failing to record world state revealed by interaction outcomes, and limited generalization from prior experience. However, existing benchmarks do not directly assess these memory capabilities during long-horizon embodied interaction. To address this gap, we introduce EmbodiedMemory-Bench (EMem-Bench), comprising 2,554 interactive episodes across four task families. EMem-Bench requires agents to build and update memory from interaction history, then use it to complete a later task by acting in the environment. We further present Embodied-Memorizer (EMem), an external memory system that organizes embodied experience into spatial, event, and scene memories. We also train EMem-8B, an 8B policy that manages and uses these memories. We evaluate a diverse range of open-source and proprietary MLLMs and representative multimodal memory systems. Results show that current models remain weak and uneven across the four challenges. Under matched backbones, EMem achieves the best overall performance among the evaluated memory systems and improves both open-source and proprietary models, while EMem-8B further improves over its backbone. Project page: https://zju-omniai.github.io/EmbodiedMemoryBench/
Lizhou Liang, Xinyu Zhong, Miao Pan +7
Zhejiang University · Central South University · Institute of Software, Chinese Academy of Sciences
To assist humans over extended periods in real homes, embodied agents must remember user routines, world states, and past interactions. Existing long-term memory benchmarks mainly evaluate language-centric retrieval and question answering, while embodied benchmarks often focus on short-horizon task execution without testing long-term memory use in dynamic environments. We introduce WorldLines, a project-driven benchmark for long-horizon embodied household assistance. It constructs temporally extended household traces with dialogues, actions, execution feedback, object and device state changes, and converts them into evidence-linked samples for Memory QA and Embodied Task Planning. We further propose ObsMem, an observer-grounded memory framework that maintains visibility-aware memories and action-native state trails for state-aware decisions. Experiments reveal persistent challenges in partial observability, overwritten world states, and translating long-term memory into embodied plans, while ObsMem offers a stronger reference architecture for this setting.
Deploying Multimodal Large Language Models as the brain of embodied agents remains challenging, particularly under long-horizon observations and limited context budgets. Existing memory assisted methods often rely on textual summaries, which discard rich visual and spatial details and remain brittle in non-stationary environments. In this work, we propose a non-parametric memory framework that explicitly disentangles episodic and semantic memory for embodied exploration and question answering. Our retrieval-first, reasoning-assisted paradigm recalls episodic experiences via semantic similarity and verifies them through visual reasoning, enabling robust reuse of past observations without rigid geometric alignment. In parallel, we introduce a program-style rule extraction mechanism that converts experiences into structured, reusable semantic memory, facilitating cross-environment generalization. Extensive experiments demonstrate state-of-the-art performance on embodied question answering and exploration benchmarks, yielding a 7.3% gain in LLM-Match and an 11.4% gain in LLM MatchXSPL on A-EQA, as well as +7.7% success rate and +6.8% SPL on GOAT-Bench. Analyses reveal that our episodic memory primarily improves exploration efficiency, while semantic memory strengthens complex reasoning of embodied agents.
Ji Li, Bo Wang, Jing Xia +2
The University of Hong Kong, Hong Kong SAR. · Beijing Institute of Technology, Beijing, China.