When the Past Shapes Action: Understanding Embodied Memory

Siqi Li; Ziyi Bai; Xinrun Xu; Tinglei Huang; Börje F. Karlsson

cs.ROOct 30, 2026

When the Past Shapes Action: Understanding Embodied Memory

Authors: Siqi Li, Ziyi Bai, Xinrun Xu, Tinglei Huang, Börje F. Karlsson

Organizations: Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence (BAAI)

Abstract

Recent advances in generative foundation models have expanded the capabilities of embodied AI, enabling increasingly general navigation, manipulation, and long-horizon task execution. However, effective interaction with dynamic environments requires more than only strong perception and reasoning: agents must act with partial knowledge and retain, retrieve, and update information from past observations, actions, and experiences. This survey examines embodied memory as a mechanism for preserving decision-relevant information beyond the current observation. We organize existing approaches along two complementary dimensions: six fundamental memory requirements, spanning world-state persistence, task progress, belief maintenance, and experiential learning; and eight architectural categories, including recurrent states, fast weights, visual and symbolic representations, spatial memory, and experience-based systems. We analyze their capabilities and limitations, and examine how existing benchmarks evaluate memory-dependent behavior, distinguishing genuine use of historical information from general task performance. Finally, we identify open challenges in selecting relevant experiences, maintaining consistent beliefs in changing environments, and consolidating knowledge across episodes. Our analysis provides a framework for understanding memory as an essential component of reliable, adaptive, and lifelong-horizon embodied intelligence.

Figures & tables

Figure 1: Conceptual boundaries and implementations of embodied memory. The six requirements in Sec. 3 describe task demands; the architectural categories in Sec. 4 describe implementation choices.

Figure 2: Representative works organized by embodied memory requirement. Six timelines group studies by the historical information an agent must preserve: hidden world state, action-induced state change, task state, delayed-relevance evidence, active belief maintenance, and cross-episode experience. Each row identifies the corresponding computational challenge and arranges works by year. Solid boxes denote methods or systems; dashed boxes marked B denote benchmarks. Works may appear in multiple rows when they address several requirements. Timelines are independently spaced for readability, with no ordering implied among works from the same year. Dates follow the cited bibliography, using the conference year where available and the preprint or report year otherwise.

Memory Requirement	What Must Persist?	Core Computational Challenge	Memory Operation	Typical Failure
Hidden world state	Entity identity, location, and spatial relations	Temporal entity binding across occlusion and viewpoint changes	Retain + retrieve	Identity drift; forgotten location
Action-induced state change	Current world state and its temporal versions	State supersession after the agent changes the environment	Update + invalidate	Stale belief
Task state	Phase, progress, count, constraints, and failures	Latent task-state tracking across visually similar execution stages	Accumulate + update	Repeated action; skipped stage
Delayed-relevance evidence	Sparse historical cues with future utility	Future-utility estimation under delayed relevance	Selective write + recall	Missed critical event
Active belief maintenance	World belief and uncertainty	Belief calibration under uncertainty and environmental change	Revise + reobserve	Belief inertia
Cross-episode experience	Reusable skills, rules, and environmental knowledge	Experience abstraction across repeated episodes	Consolidate + transfer	Overgeneral-ization

Table 1: A computational taxonomy of non-Markovian embodied memory requirements. Each requirement is associated with a distinct latent state, computational challenge, memory operation, and characteristic failure mode.

Memory Architecture	Primary Sec. 3 Requirement	Main Strength	Main Limitation
A1. Recurrent state	Task-state tracking, continuous belief	Fast, bounded-size updates	Information bottleneck; weak random access
A2. Fast-weight / TTT	Continuous belief, spatial or interaction-state accumulation	Bounded adaptive state	Low explicit addressability; memory/adaptation ambiguity
A3. Persistent visual tokens	Entity binding, dense perceptual history	High visual fidelity	Growing context; stale evidence
A4. Event / keyframe	Future-utility estimation	Efficient long-horizon retention	Selection errors can be irreversible
A5. Text / symbolic	Task-state tracking, semantic state	High compression; interpretable	Loss of visual detail
A6. Graph / 3D	Entity binding, spatial belief	Explicit geometric grounding	Update and data-association errors

Table 2: Capability envelopes of embodied-memory architectures. The requirement column is a diagnostic cross-reference to Sec. 3 ; A1–A7 are non-exclusive mechanism or organization families, while A8 denotes their composition. Episodic retrieval and skill consolidation are two modes within A7, not additional top-level categories.

Benchmark	Policy interface	Reported outcome	What is jointly evaluated	Memory interpretation limit
LIBERO [ 107 ]	Language, visual/proprioceptive input; robot actions	Goal-completion success; sequential-learning studies also examine transfer and retention	Language grounding, manipulation, and knowledge transfer	Training-time retention and within-episode history use require separate evidence
CALVIN [ 111 ]	Successive language subtasks and sensor inputs; continuous actions	Success across 1–5 subtasks; average completed chain length	Compositional execution and generalization	Successive instructions supply part of task-progress state
RoboCasa [ 108 ]	Language and robot observations; manipulation rollout	Task success under kitchen goal conditions	Generalist manipulation across tasks and environments	A visible goal state may reduce the need for distant recall
RoboCasa365 [ 112 ]	Multi-task/foundation-model rollouts;	Task success; performance across sequential training phases	Atomic/composite generalization and learned competence retention	Continued optimization is distinct from deployment memory
RoboTwin 2.0 [ 109 ]	Language and multimodal robot observations; bimanual actions	Rollout task success under randomized conditions	Coordinated manipulation and domain robustness	Randomization alone does not isolate historical dependence
VLABench [ 110 ]	Language-conditioned robot control; separate VLM/workflow evaluations	Execution success; protocol-specific reasoning/plan metrics	Grounding, compositional reasoning, and execution	VLM answer/plan quality and physical completion are different outcomes

Table 3: Evaluation interfaces, outcomes, and memory implications of general embodied benchmarks. Entries summarize the named benchmark protocols rather than prescribing one universal sensor or action configuration. A score should be interpreted using its own completion rule and observation privileges.

Benchmark	EB	SS	TS	FU
RMBench [ 19 ]	✓		✓
RoboMME [ 20 ]	✓	✓	✓
LIBERO-Mem [ 114 ]	✓
RoboMemArena [ 66 ]	✓	✓	✓
MemoryBench [ 93 ]	✓
RoboTwin-MeM [ 8 ]				$\triangle$

Table 4: Coverage of the six memory requirements from Sec. 3 in memory-oriented benchmarks. EB (R1): Entity Binding; SS (R2): State Supersession; TS (R3): Task-State Tracking; FU (R4): Future-Utility Estimation; BC (R5): Belief Calibration; EA (R6): Experience Abstraction. These are the computational operations associated with the same six requirements, not six additional memory types. ✓denotes systematic task coverage, while

\triangle

indicates partial or indirect pressure without dedicated evaluation. The mapping is our interpretation of task protocols, not a claim of causal isolation; HuMeGo concerns pre-goal object history, with selection and state-change coverage treated as indirect; EMem-Bench’s EA mark concerns its controlled generalization family, while MemTransfer probes reuse under mismatch rather than learning an abstraction mechanism.

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 199 papers

EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks
Sep 23, 2026Lizhou Liang, Xinyu Zhong, Miao Pan +7Embodied Agent Benchmarks Long-Term Memory for Embodied Agents
Zhejiang University · Central South University · Institute of Software, Chinese Academy of Sciences
WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents
Jun 17, 2026Yehang Zhang, Jianchong Su, Haojian Huang +7Robot Task Planning Long-Horizon Agent Evaluation
HKUST(GZ) · Knowin · HKUST
HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering
Feb 17, 2026Ji Li, Bo Wang, Jing Xia +2Long-Term Memory for Embodied Agents Memory-Augmented Agents
The University of Hong Kong, Hong Kong SAR. · Beijing Institute of Technology, Beijing, China.

Sep 23, 2026J/K move · Enter open · S save

EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks

Lizhou Liang, Xinyu Zhong, Miao Pan, Xiaohe Zhou+6

Zhejiang University · Central South University · Institute of Software, Chinese Academy of Sciences

Embodied Agent Benchmarks Long-Term Memory for Embodied Agents Long-Horizon Agent Tasks Embodied AI Memory-Augmented Agents

Open paper AskPDF arXiv

Long-horizon embodied interaction requires agents to retain and continually update information about the environment as they observe, act, and encounter change. Yet current agents struggle to maintain such memory reliably. Our analysis traces this limitation to four key deficiencies: weak fine-grained visual memory, unreliable dynamic world-state tracking, failing to record world state revealed by interaction outcomes, and limited generalization from prior experience. However, existing benchmarks do not directly assess these memory capabilities during long-horizon embodied interaction. To address this gap, we introduce EmbodiedMemory-Bench (EMem-Bench), comprising 2,554 interactive episodes across four task families. EMem-Bench requires agents to build and update memory from interaction history, then use it to complete a later task by acting in the environment. We further present Embodied-Memorizer (EMem), an external memory system that organizes embodied experience into spatial, event, and scene memories. We also train EMem-8B, an 8B policy that manages and uses these memories. We evaluate a diverse range of open-source and proprietary MLLMs and representative multimodal memory systems. Results show that current models remain weak and uneven across the four challenges. Under matched backbones, EMem achieves the best overall performance among the evaluated memory systems and improves both open-source and proprietary models, while EMem-8B further improves over its backbone. Project page: https://zju-omniai.github.io/EmbodiedMemoryBench/

Abstract

Agent Memory

Explore similar work

EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks

WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents

HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering