Long-horizon robotic manipulation fundamentally requires memory: agents must retain task-relevant information from past observations and interactions and use it to guide future actions. Existing approaches typically preserve raw historical observations, compressed visual tokens, or language summaries, but these representations either expose the action policy to increasingly long histories or risk discarding task-relevant temporal, semantic, and perceptual information. We introduce Mem4Act, a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control. Mem4Act addresses three coupled challenges: how to efficiently cover long histories, what information memory should retain, and how to ensure the policy actually relies on memory. To this end, it organizes history via multi-granularity temporal aggregation, preserves complementary semantic and perceptual information through question-answering supervision and key-frame DINO feature reconstruction, and employs memory-first two-stage training to encourage greater use of memory by the action policy. The resulting memory captures both high-level task state and fine-grained perceptual evidence while providing a bounded interface to the action policy. Across RoboMME, RMBench, and four real-world manipulation tasks, Mem4Act achieves state-of-the-art performance on memory-dependent control, with success rates of 49.88%, 61.0%, and 81.5%, respectively. Extensive ablations further demonstrate the importance of temporal aggregation, complementary memory supervision, and memory-first training.
Figures & tables
Figure 1: Motivation and overview of Mem4Act. (a) Mem4Act consolidates retained interaction history into a fixed-size latent memory, providing a bounded context to the action policy. (b) Mem4Act addresses coverage , content , and utilization problems through structured multi-granularity attention, semantic and perceptual supervision, and memory-first training. (c) Mem4Act improves memory-intensive control across simulation benchmarks and real-world robotic tasks.
Figure 2: Mem4Act structure. (a) Historical observations are encoded as compact visual tokens with state, phase, and time embeddings. (b) The latent memory tokens first attend within fine temporal windows, then are regrouped to attend broader windows. Global self-attention produces the final latent memory Mt . (c) Memory-related question answering and multi-keyframe DINO feature reconstruction supervise what this memory retains. (d) The resulting memory conditions action generation through memory modulation in the action expert. (e) Training first grounds the memory with retrospective feature supervision, then jointly adapts memory and policy for action generation.
Figure 3: Real-world manipulation tasks. We design four memory-intensive, long-horizon manipulation tasks for real-world evaluation. Task instructions are provided in Appendix Table S4 .
Model
Counting (Temporal)
Permanence (Spatial)
Reference (Object)
Imitation (Procedural)
Overall
DP
10.67
5.50
6.58
5.33
7.02
π0.5
28.78
17.00
17.16
8.78
17.93
SAM2ACT+
35.33
26.00
16.83
7.33
21.37
TTT+Expert
36.00
22.95
19.67
10.78
22.35
TokenDrop+Expert
59.44
23.61
25.33
31.06
34.86
FrameSamp+Expert
66.78
25.17
24.11
28.94
36.25
Table 1: RoboMME benchmark results. We report overall and category-wise scores across temporal, spatial, object, and procedural memory tasks; Exact task assignments are listed in Table S2 .
M(1)
M(n)
Method
Observe
Rearrange
Put Back
Swap
Swap T
Avg.
Battery
Blocks Rank.
Cover
Press
Avg.
Total
DP
1
0
0
11
20
6.4
10
10
0
0
5.0
5.8
ACT
1
29
0
2
2
6.8
19
0
0
0
4.8
5.9
π0.5
9
13
11
24
15
14.4
16
6
0
0
5.5
10.4
X-VLA
9
13
18
16
3
11.8
26
1
2
0
7.3
9.8
HAMLET †
12
52
44
22
31
32.2
15
5
30
0
12.5
23.4
Table 2: RMBench results. Mem4Act achieves the best overall performance with one policy trained across all tasks. † denotes reimplementation from open-source code; details in Section C.3 .
Figure 4: Mem4Act reliably leverages memory for long-horizon robotic manipulation. Across simulation and real-world tasks, it preserves historical semantic and perceptual information, tracks task progress, and supports correct actions in long-horizon tasks.
Method
Cover Blocks
Swap Blocks
Rearrange Blocks
Put Block Back
Overall
Mem-0
12
30
32
16
22.50
π0.5
10
18
76
8
28.00
MME-VLA
18
32
90
28
42.00
GPT-6 Astra (xhigh)
22
24
56
68
42.50
Mem4Act (ours)
78
70
100
78
81.50
Table 3: Real-world testing results. We report average success rates over 50 rollouts per task.
Configuration
Temporal windows
Tokens
Counting
Permanence
Reference
Imitation
Overall
Temporal structure (fixed 512 tokens)
No temporal masking
Full history
512
59.00
29.00
23.50
21.50
33.25
Single-scale
32
512
58.50
21.50
32.00
32.00
36.00
Repeated single-scale
32→32→32
512
57.50
27.00
28.00
35.00
36.88
Two-level hierarchy
32→8
512
70.50
25.00
31.00
46.00
43.13
Coarser hierarchy
16→8→4
512
71.50
22.50
37.50
49.00
45.13
Table 4: Memory capacity and temporal hierarchy ablations on RoboMME. Window sequences specify temporal partitions at successive aggregation levels.
Supervision
Sem.
Percep.
Both
Overall
Action (One-stage)
46.80
29.67
18.40
31.50
Action + QA
58.80
45.33
25.20
43.25
Action + DINO
55.60
50.67
28.80
45.38
Action + QA + DINO
61.60
50.33
37.60
49.88
Table 5: Memory ablations on RoboMME. (a) Effect of QA and DINO supervision across semantic, perceptual, and joint task groups. Action only uses one-stage training; all others use two-stage training. (b) Reliance on historical information. We replace only the past with that from another rollout of the same task. Implementation details are provided in Appendix Section C.5 .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Example of ground-truth QA data. The front-view observation and complete question prompt and answer sequence used for training. Movement directions in the answers are defined in the robot’s egocentric frame; the camera views the robot from the opposite side, so these directions appear reversed in the image.
Hyperparameter
Value
Memory-compressor attention heads
8
Memory-compressor MLP ratio
4
Action–memory attention heads
4
Flow-matching inference steps
10
Optimizer
AdamW
AdamW (β1,β2)
(0.9,0.95)
Appendix
Table S1: Additional architecture and optimization hyperparameters. Values separated by “/” denote Stages 1 and 2.
Table S3: RoboMME task groups for the auxiliary-supervision analysis.
Task
Language instruction
Rearrange Blocks
Move the block between the two mats onto the empty mat, press the button, then move the other block (the one that started on a mat) to the space between the two mats.
Swap Blocks
There are three bowls on the table, and two blocks are placed in two different bowls. You may move only one block at a time, and each bowl can hold at most one block. Swap the positions of the two blocks. Finally, press the button.
Put Back Block
There are four mats, one block, and a button on the table. One block is on one of the mats. First, put the block in the center, then press the button. Then, put the block back in its original position.
Cover Blocks
On the table, red, green, and blue blocks are arranged randomly along with three cups. From the current viewpoint, cover the blocks from left to right using the cups, and then uncover them again in the sequence red, green, and blue.
Appendix
Table S4: Language instructions for the real-world evaluation tasks.
Method
Parameters
FLOPs
Peak memory
Latency/cycle
Completion time
SR ↑
Mem4Act
3.548 B
489 G
7.2 GiB
335 ms
40 s
81.50
GPT-6 Astra
–
–
–
19239 ms
1038 s
42.50
MME-VLA
3.442 B
421 G
7.0 GiB
340 ms
49 s
42.00
π0.5
3.353 B
239 G
6.7 GiB
131 ms
47 s
28.00
Mem-0
2.358 B
4923 G
4.8 GiB
487 ms
66 s
22.50
Appendix
Table S5: Real-world inference efficiency. All local methods are evaluated with batch size 1 on the same GPU. FLOPs and latency include historical image encoding, memory processing, and action prediction. GPT-6 latency includes remote API inference and communication. Completion time is averaged over successful rollouts only. For Mem-0, parameters, FLOPs, peak memory, and cycle latency cover only the executor; the high-level planner is excluded.
Failure mode
Mem4Act
GPT-6 Astra
FrameSampling
π0.5
Mem-0
Incorrect behavior
34
10
89
70
49
Non-recoverable state
3
73
27
12
57
Stall
0
12
0
62
49
Timeout
–
20
–
–
–
Total failures ↓
37
115
116
144
155
Appendix
Table S6: Failure modes in the real-world evaluation. Each failed rollout is assigned one primary failure category. Entries report the number of failed rollouts ( ↓ ).
Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.
Marcel Torne, Karl Pertsch, Homer Walke +14
Physical Intelligence · Stanford University · UC Berkeley +1
Memory-augmented robotic policies are essential in handling memory-dependent tasks. However, existing approaches typically rely on simply extending the observation window, struggling to simultaneously achieve precise task-state tracking and robust long-horizon retention. To overcome these challenges, inspired by the Atkinson--Shiffrin memory model, we propose MemoAct, a hierarchical memory-augmented policy that leverages distinct memory tiers to tackle specific bottlenecks. Specifically, sensory memory filters immediate perceptual inputs, lossless short-term memory supports precise task-state tracking, and compressed long-term memory facilitates robust long-horizon retention. To enrich the evaluation landscape, we construct MemoryRTBench based on RoboTwin 2.0, comprising 6 manipulation tasks that systematically evaluate policy memory capabilities across three dimensions: sequential, spatial, and episodic memory. Extensive experiments across simulated and real-world scenarios demonstrate that MemoAct achieves superior performance compared to both existing Markovian baselines and history-aware policies. The project page is available at https://tlf-tlf.github.io/MemoActPage/.
World Action Models (WAMs) have emerged as a new powerful paradigm for embodied intelligence, learning action-relevant visual dynamics that significantly enhance generalization and robustness. However, existing WAMs still struggle with task-relevant memory in long-horizon robotic manipulation. To address this, we present HiMem-WAM, a Hierarchical Memory-Gated WAM that integrates motion-centric latent actions, high-level skill latents, and boundary-triggered memory updates. Specifically, we develop a hierarchical latent action framework that jointly learns low-level motion and high-level skill latents, providing structured temporal abstraction. Meanwhile, a boundary-aware memory gate writes compact task states at predicted skill transitions, enabling causal inference without test-time generation of future video or optical flow estimation. Evaluated on LIBERO, LIBERO-PLUS, RMBench and real-world tasks, HiMem-WAM shows that hierarchical latents improve robustness under deployment perturbations, and the memory module substantially benefits memory-dependent long-horizon manipulation.
Xiaoquan Sun, Ruijian Zhang, Chen Cao +12
3Huazhong University of Science and Technology · 1The University of Hong Kong · 2INFIFORCE +3