Memory before Goal: Role of Pre-History Memory for Humanoid Loco-Manipulation

Chenyuan Chen; Ran Ding; Lechen Zhang; Zewen He; Yoshihiko Nakamura

cs.ROOct 22, 2026

Memory before Goal: Role of Pre-History Memory for Humanoid Loco-Manipulation

Authors: Chenyuan Chen, Ran Ding, Lechen Zhang, Zewen He, Yoshihiko Nakamura

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE

Abstract

Humanoid loco-manipulation in dynamic scenes is non-Markovian: cues vanish as objects change or are hidden, or leave the view as the robot turns and walks, yet whole-body vision-language-action (VLA) policies see only the present. We present WholeBodyMem, an object-guided perceptual memory that keeps the pre-history of object events seen before the robot acts on its goal. SAM3 tracks the task objects, and their events causally select key frames within an adaptive budget, which the action expert reads by cross-attention together with object tokens encoding appearance, geometry and time. We also contribute HuMeGo, a humanoid memory benchmark with six simulated and six real-world Unitree G1 tasks. Keeping the right frames rather than more frames, WholeBodyMem surpasses memoryless and uniform-memory baselines, raising its base policy's success from 29.1% to 72.6% in simulation and from 36.7% to 88.3% on the real robot, while also improving the standard SIMPLE tasks (70.2% to 86.9%).

Figures & tables

Fig. 1: Memory in a real whole-body task. Top: the cola stops first on the middle seat (1.1 s), then a sprite and the cola stop on the other two; the robot turns away to fetch the cola and, when it returns at 30.0 s, the three seats look the same.

Fig. 2: Memory-dependent tasks in simulation, grouped by memory type. Each block shows one task and its memory suite (as in RoboMME [ 12 ] ; Sec. III-A ): the instruction, nine subtasks and the robot’s head-camera view at each of them. Bold text marks the cue and the answer; purple borders mark the cue frames, and a blue arrow points at the cue object.

Fig. 3: Overview of WholeBodyMem. $\times L$ : the $L{=}6$ DiT blocks of the action expert.

Task	InternVLA-M1 [ 32 ]	DP [ 33 ]	ACT [ 34 ]	EgoVLA [ 35 ]	$\pi_{0.5}$ [ 24 ]	GR00T [ 2 ]	$\Psi_{0}$ [ 1 ]	Uniform memory	WholeBodyMem (ours)
T1 XMovePick	0 / 0 / 0	5 / 3 / 3	7 / 6 / 3	3 / 3 / 1	8 / 6 / 3	6 / 6 / 4	9 / 10 / 6	10 / 10 / 5	10 / 10 / 7
T2 BendPickMP	4 / 5 / 4	10 / 9 / 10	9 / 9 / 9	6 / 9 / 6	8 / 8 / 6	10 / 10 / 9	10 / 10 / 9	10 / 10 / 10	10 / 10 / 10
T3 Handover	0 / 0 / 1	1 / 1 / 1	1 / 2 / 2	2 / 3 / 4	4 / 2 / 3	6 / 6 / 7	7 / 6 / 8	8 / 7 / 9	10 / 9 / 9
T4 TabletopGraspMP	6 / 7 / 9	4 / 4 / 1	10 / 10 / 8	6 / 5 / 4	10 / 10 / 9	10 / 10 / 8	10 / 10 / 7	9 / 10 / 8	10 / 10 / 9
T5 XMoveBendPick	5 / 4 / 4	0 / 0 / 0	2 / 2 / 3	3 / 6 / 6	0 / 0 / 0	6 / 2 / 4	9 / 4 / 4	8 / 6 / 10	9 / 9 / 8
T6 LocomotionPickBetweenTables	0 / 0 / 0	0 / 1 / 0	1 / 0 / 0	1 / 3 / 1	2 / 0 / 1	0 / 0 / 0	4 / 4 / 3	4 / 5 / 4	8 / 6 / 7

TABLE I: Results on the 14 SIMPLE Tasks.

Benchmark	Robot	Whole- body	Memory	Data	Tasks	Steps	Mem. (%)
MemoryBench [ 28 ]	Franka	✗	✓	Sim	3	–	–
LIBERO-Mem [ 31 ]	Franka	✗	✓	Sim	10	–	–
RoboMME [ 12 ]	Franka	✗	✓	Sim+Real	16+4	481	–
RoboMemArena [ 21 ]	Dual-arm	✗	✓	Sim+Real	26+5	1,076	–
HumanoidMimicGen [ 20 ]	G1	✓	✗	Sim	9	818	0
HumanoidArena [ 19 ]	G1	✓	✗	Sim	7	611	0

TABLE II: Comparison with Robot Memory and Humanoid Benchmarks.

	Counting (T)	Permanence (S)		Reference (O)
Method	FlashPad	HideMug	ShellGame	LitLamp	BlinkPad	TouchPad	Avg.
Success (L0 / L1 / L2)
$\Psi_{0}$ (no memory) [ 1 ]	7 / 7 / 5	11 / 14 / 12	7 / 11 / 7	11 / 6 / 4	11 / 11 / 11	3 / 11 / 8	8.3 / 10.0 / 7.8
Uniform memory	22 / 23 / 22	12 / 20 / 15	10 / 9 / 9	12 / 12 / 11	10 / 9 / 13	6 / 13 / 12	12.0 / 14.3 / 13.7
WholeBodyMem (ours)	26 / 27 / 23	21 / 20 / 20	21 / 20 / 19	18 / 15 / 12	26 / 22 / 25	26 / 25 / 26	23.0 / 21.5 / 20.8
Memory-decision accuracy (%)

TABLE III: Memory Task Results.

Fig. 4: Real-world memory tasks on the Unitree G1. Each panel shows the task instruction and four steps of a teleoperated demonstration; the line below gives the difficulty and the memory the decision needs. Blue arrows mark the cue and the answer (dashed: where the cue was, after it is gone); pink arrows show the robot’s motion in each step, straight for walking. S1–S3: the three seats of the café tasks, from front to back.

	Successes	Success
Method	L0 / L1 / L2	(%)
Uniform memory	12.0 / 14.3 / 13.7	44.4
WholeBodyMem w/o object tokens & head	18.0 / 19.0 / 17.3	60.4
WholeBodyMem w/o decision head	20.7 / 18.7 / 19.8	65.7
WholeBodyMem with memory as context	8.3 / 7.0 / 8.0	25.9
WholeBodyMem w/o adaptive memory	21.7 / 21.3 / 20.3	70.4

TABLE IV: Ablations on the Memory Tasks.

Task	Mem.	$\Psi_{0}$ [ 1 ]	WholeBodyMem
Trash bag	S	4/10	9/10
Remember-chip	O	3/10	10/10
Café A	T	4/10	9/10
Café B	O+S	3/10	9/10
Café C	O+S+T	3/10	8/10
Chip-cart	P	5/10	8/10

TABLE V: Real-World Results on the Unitree G1.

Humanoid Loco-Manipulation

Humanoid loco-manipulation studies how robots coordinate walking, balance, and whole-body interaction to move and handle objects.

See all 90 papers

Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs
Sep 28, 2026Tanguy Dieudonné, Jack B. Jedlicki, Heng YangMemory-Augmented VLMs Long-Horizon Robotic Manipulation
Harvard University · ETH Zürich
MEM: Multi-Scale Embodied Memory for Vision Language Action Models
Mar 4, 2026Marcel Torne, Karl Pertsch, Homer Walke +14Memory-Augmented Agents Long-Term Memory for Embodied Agents
Physical Intelligence · Stanford University · UC Berkeley +1
RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies
Mar 4, 2026Yinpei Dai, Hongze Fu, Jayjun Lee +6Memory-Augmented VLMs Robot Manipulation Benchmarks
University of Michigan · Stanford University · Figure AI

Sep 28, 2026J/K move · Enter open · S save

Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs

Tanguy Dieudonné, Jack B. Jedlicki, Heng Yang

Harvard University · ETH Zürich

Memory-Augmented VLMs Long-Horizon Robotic Manipulation Large Language Model-Based Robot Planning Robotic Manipulation Long-Term Memory for Embodied Agents Robotic Spatial Memory

Open paper AskPDF arXiv

Memory is essential for long-horizon, partially observed robotic manipulation: a robot must remember which object was placed in a drawer, whose cup it moved, or how many action cycles have elapsed. Recent vision-language-action (VLA) models embed memory directly inside the policy, but benchmarks show no single in-policy mechanism covers all spatio-temporal dimensions, trailing oracle methods by a wide margin. We argue that memory type dictates where memory should reside: short-term perceptual memory (repetition, timing, retracing) belongs inside the policy, while long-term object memory (persistent spatial state, containment, event history) belongs outside as an explicit, readable record. We present Ledger, a harness that realizes this split over a single fine-tuned

π_{0.5}

policy by pairing an in-policy frame-sampling memory with an external spatio-temporal object memory, the ledger, built from a SAM3 tracker and a VLM captioner of the demonstration and read by an LLM planner that decides at step boundaries. On RoboMME, Ledger reaches the highest four-suite average among the evaluated methods, 64.3% (vs. 45.9% for the strongest prior method under identical evaluation), leading object reference (60.7% vs. 40.3%) and object permanence (86.7% vs. 56.2%) using a single set of weights. Choosing the memory source at runtime, from the instruction and the record, removes the need for a task-level router.

Abstract

Humanoid Loco-Manipulation

Explore similar work

Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies