Remember Where You Walk: Learning Spatial Memory for End-to-End Legged Robot Exploration

Leon Magnus; Nico Bohlinger; Jan Peters; Oleg Arenz

cs.ROOct 27, 2026

Remember Where You Walk: Learning Spatial Memory for End-to-End Legged Robot Exploration

Authors: Leon Magnus, Nico Bohlinger, Jan Peters, Oleg Arenz

Organizations: Technical University of Darmstadt, Germany. · Robotics Institute Germany (RIG). · German Research Center for AI (DFKI).

Abstract

Humanoids and other legged robots operating in complex environments must remember information that leaves their field of view and only becomes relevant much later. Reinforcement learning policies address this using memory networks such as LSTMs that are trained end-to-end with the policy to memorize such task-relevant aspects. Yet, such architectures often fail to preserve sufficient information over long horizons. In this work, we propose a network architecture based on Hadamard memory that specifically targets robot navigation. Our Local Spatial Matrix Memory (LSMM) enables RL agents to store generic task-relevant information at specific, position-based memory cells, enabling flexible and spatially grounded memorization. We evaluate LSMM on the task of autonomous maze exploration, where it matches the maze coverage of strong recurrent and finite-history baselines while being the only architecture that remembers the maze layout, reducing the reconstruction error threefold.

Figures & tables

Fig. 1 : (a) The Unitree Go2 autonomously explores a previously unseen $5\times 5\,\mathrm{m}$ outdoor maze. The red line shows the walked path. (b) The matrix memory at the end of the run. The policy has written into the cells along its path, shown with the three memory channels interpreted as RGB colors, while unwritten cells keep their initial value in gray. (c) and (d) Shows the robot navigating through the maze.

Fig. 2 : Actor architecture with the proposed LSMM memory module. Both branches share

\mathbf{z}_{t}^{\text{h}}

and

\mathbf{o}_{t}^{\text{p}}

, while the memory branch additionally uses noisy position and velocity estimates

\hat{\mathbf{p}}_{t}

and

\hat{\mathbf{v}}_{t}

to update the matrix memory

\mathbf{L}_{t}

. A CNN then extracts

\mathbf{z}_{t}^{\text{m}}

from

\mathbf{L}_{t}

and an auxiliary MLP predicts the maze wall layout

\hat{\mathbf{y}}_{t}

. The actor maps

\mathbf{z}_{t}^{\text{m}}

and its observation

\mathbf{o}_{t}

\boldsymbol{\mu}_{t}

and

\boldsymbol{\sigma}_{t}

, from which

\mathbf{a}_{t}

is sampled.

Fig. 3 : Percentage of explored cells at the end of an episode. Bars show the mean over 100 unseen evaluation mazes and three seeds per architecture, whiskers the standard deviation. All architectures reach a comparable coverage.

Fig. 4 : Average number of visits per maze cell across all evaluation episodes. $D_{\text{KL}}(p\,\|\,u)$ measures the divergence between the normalized visit distribution $p$ and a uniform distribution $u$ , where lower values mean more even exploration. LSMM spreads its visits most evenly over mazes.

Fig. 5 : Maze reconstruction error over the course of an episode. Lines show the mean over 100 unseen evaluation mazes and three seeds, shaded areas the standard deviation. LSMM is the only architecture that significantly reduces the error while exploring.

Fig. 6 : Predicted maze layout at three times during an evaluation episode. The trajectory walked up to each time is overlaid, colored from black to orange by trajectory time. The prediction of LSMM fills in wall by wall as the robot explores and closely matches the ground truth at the end of the episode, while the baselines struggle to remember the walls in the areas they have been before.

Fig. 7 : Maze reconstruction error of LSMM for memory sizes of

50\times 50\times C

with

C\in\{1,2,3,5,10\}

channels. Lines show the mean over 100 unseen evaluation mazes and three seeds, shaded areas the standard deviation. All memory sizes perform comparably.

Fig. 8 : Auxiliary prediction of spatially varying environment properties. Bars show the normalized mean absolute error over all visited cells at the final time step.

Fig. 9 : Time spent in each cell by our deployed LSMM policy. Compared to the simulation results, this deployment run distributes its visits less uniformly and follows a more repetitive pattern, which is also visible in the matrix memory in Figure 1 (b).

Fig. 10 : Maze reconstruction error over deployment time of our LSMM policy. The line shows the reconstruction error of the deployment run shown in Figure 1 . The policy reduces the error significantly within the first

30\,\mathrm{s}

Hyperparameter	Value
Training steps	$2.5\cdot 10^{9}$
Batch size (envs $\times$ steps per env)	$5\,000\times 20=100\,000$
Minibatch size / updates per batch	$25\,000$ / $5$
Discount factor $\gamma$ / GAE $\lambda$	$0.995$ / $0.9$
Clipping range	$0.2$
Entropy coef. (linearly annealed)	$4\cdot 10^{-4}\rightarrow 10^{-4}$

TABLE I : Hyperparameters used in all experiments.

Reward	Term	Coeff.
Task
Cell visit (E)	$\exp(-n_{\text{cell}}/200)$	$0.16$
Maze coverage (E)	fraction of visited cells	$0.1$
Goal direction (G)	$\tfrac{1}{2}\big(1{+}\cos(\mathbf{v}_{xy},\mathbf{g}_{t})\big)\min(\lVert\mathbf{v}_{xy}\rVert,1)$	$0.04$
Goal reached (G)	$\mathds{1}[d_{\text{goal}}{<}0.25]$	$0.0625$
Navigation shaping

TABLE II : Reward terms and coefficients, with

[x]_{+}=\max(x,0)

and

\overline{\,\cdot\,}

the mean over joints. E/G: exploration/goal episodes only;

n_{\text{cell}}

: steps in the current cell;

\delta_{i}

: offset from the cell center toward walled side

i

;

n^{\text{air}}_{\text{pairs}}

: leg pairs with both feet airborne;

t^{\text{air}}

: foot air time.

Robotic Spatial Memory

See all 30 papers

MEM: Multi-Scale Embodied Memory for Vision Language Action Models
Mar 4, 2026Marcel Torne, Karl Pertsch, Homer Walke +14Memory-Augmented Agents Long-Term Memory for Embodied Agents
Physical Intelligence · Stanford University · UC Berkeley +1
Spatial Memory by Agents, for Agents
Oct 15, 2026Qianwei Wang, Ce Zhang, Zhanpeng Luo +3Robotic Spatial Memory Agent Memory
Qianwei Wang is with Princeton University · Ce Zhang is with Carnegie Mellon University · Zhanpeng Luo is with Johns Hopkins University +2
Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs
Sep 28, 2026Tanguy Dieudonné, Jack B. Jedlicki, Heng YangMemory-Augmented VLMs Long-Horizon Robotic Manipulation
Harvard University · ETH Zürich

Mar 4, 2026J/K move · Enter open · S save

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder+13

Physical Intelligence · Stanford University · UC Berkeley+1

Memory-Augmented Agents Long-Term Memory for Embodied Agents Robotic Control Long-Horizon Robotic Manipulation Vision-Language-Action Models ICL for Robotics

Open paper AskPDF arXiv

Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.

Abstract

Robotic Spatial Memory

Explore similar work

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

Spatial Memory by Agents, for Agents

Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs