Long-horizon robotic manipulation fundamentally requires memory: agents must retain task-relevant information from past observations and interactions and use it to guide future actions. Existing approaches typically preserve raw historical observations, compressed visual tokens, or language summaries, but these representations either expose the action policy to increasingly long histories or risk discarding task-relevant temporal, semantic, and perceptual information. We introduce Mem4Act, a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control. Mem4Act addresses three coupled challenges: how to efficiently cover long histories, what information memory should retain, and how to ensure the policy actually relies on memory. To this end, it organizes history via multi-granularity temporal aggregation, preserves complementary semantic and perceptual information through question-answering supervision and key-frame DINO feature reconstruction, and employs memory-first two-stage training to encourage greater use of memory by the action policy. The resulting memory captures both high-level task state and fine-grained perceptual evidence while providing a bounded interface to the action policy. Across RoboMME, RMBench, and four real-world manipulation tasks, Mem4Act achieves state-of-the-art performance on memory-dependent control, with success rates of 49.88%, 61.0%, and 81.5%, respectively. Extensive ablations further demonstrate the importance of temporal aggregation, complementary memory supervision, and memory-first training.
Figures & tables
Figure 1: Motivation and overview of Mem4Act. (a) Mem4Act consolidates retained interaction history into a fixed-size latent memory, providing a bounded context to the action policy. (b) Mem4Act addresses coverage , content , and utilization problems through structured multi-granularity attention, semantic and perceptual supervision, and memory-first training. (c) Mem4Act improves memory-intensive control across simulation benchmarks and real-world robotic tasks.
Figure 2: Mem4Act structure. (a) Historical observations are encoded as compact visual tokens with state, phase, and time embeddings. (b) The latent memory tokens first attend within fine temporal windows, then are regrouped to attend broader windows. Global self-attention produces the final latent memory Mt . (c) Memory-related question answering and multi-keyframe DINO feature reconstruction supervise what this memory retains. (d) The resulting memory conditions action generation through memory modulation in the action expert. (e) Training first grounds the memory with retrospective feature supervision, then jointly adapts memory and policy for action generation.
Figure 3: Real-world manipulation tasks. We design four memory-intensive, long-horizon manipulation tasks for real-world evaluation. Task instructions are provided in Appendix Table S4 .
Model
Counting (Temporal)
Permanence (Spatial)
Reference (Object)
Imitation (Procedural)
Overall
DP
10.67
5.50
6.58
5.33
7.02
π0.5
28.78
17.00
17.16
8.78
17.93
SAM2ACT+
35.33
26.00
16.83
7.33
21.37
TTT+Expert
36.00
22.95
19.67
10.78
22.35
TokenDrop+Expert
59.44
23.61
25.33
31.06
34.86
FrameSamp+Expert
66.78
25.17
24.11
28.94
36.25
Table 1: RoboMME benchmark results. We report overall and category-wise scores across temporal, spatial, object, and procedural memory tasks; Exact task assignments are listed in Table S2 .
M(1)
M(n)
Method
Observe
Rearrange
Put Back
Swap
Swap T
Avg.
Battery
Blocks Rank.
Cover
Press
Avg.
Total
DP
1
0
0
11
20
6.4
10
10
0
0
5.0
5.8
ACT
1
29
0
2
2
6.8
19
0
0
0
4.8
5.9
π0.5
9
13
11
24
15
14.4
16
6
0
0
5.5
10.4
X-VLA
9
13
18
16
3
11.8
26
1
2
0
7.3
9.8
HAMLET †
12
52
44
22
31
32.2
15
5
30
0
12.5
23.4
Table 2: RMBench results. Mem4Act achieves the best overall performance with one policy trained across all tasks. † denotes reimplementation from open-source code; details in Section C.3 .
Figure 4: Mem4Act reliably leverages memory for long-horizon robotic manipulation. Across simulation and real-world tasks, it preserves historical semantic and perceptual information, tracks task progress, and supports correct actions in long-horizon tasks.
Method
Cover Blocks
Swap Blocks
Rearrange Blocks
Put Block Back
Overall
Mem-0
12
30
32
16
22.50
π0.5
10
18
76
8
28.00
MME-VLA
18
32
90
28
42.00
GPT-6 Astra (xhigh)
22
24
56
68
42.50
Mem4Act (ours)
78
70
100
78
81.50
Table 3: Real-world testing results. We report average success rates over 50 rollouts per task.
Configuration
Temporal windows
Tokens
Counting
Permanence
Reference
Imitation
Overall
Temporal structure (fixed 512 tokens)
No temporal masking
Full history
512
59.00
29.00
23.50
21.50
33.25
Single-scale
32
512
58.50
21.50
32.00
32.00
36.00
Repeated single-scale
32→32→32
512
57.50
27.00
28.00
35.00
36.88
Two-level hierarchy
32→8
512
70.50
25.00
31.00
46.00
43.13
Coarser hierarchy
16→8→4
512
71.50
22.50
37.50
49.00
45.13
Table 4: Memory capacity and temporal hierarchy ablations on RoboMME. Window sequences specify temporal partitions at successive aggregation levels.
Supervision
Sem.
Percep.
Both
Overall
Action (One-stage)
46.80
29.67
18.40
31.50
Action + QA
58.80
45.33
25.20
43.25
Action + DINO
55.60
50.67
28.80
45.38
Action + QA + DINO
61.60
50.33
37.60
49.88
Table 5: Memory ablations on RoboMME. (a) Effect of QA and DINO supervision across semantic, perceptual, and joint task groups. Action only uses one-stage training; all others use two-stage training. (b) Reliance on historical information. We replace only the past with that from another rollout of the same task. Implementation details are provided in Appendix Section C.5 .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Example of ground-truth QA data. The front-view observation and complete question prompt and answer sequence used for training. Movement directions in the answers are defined in the robot’s egocentric frame; the camera views the robot from the opposite side, so these directions appear reversed in the image.
Hyperparameter
Value
Memory-compressor attention heads
8
Memory-compressor MLP ratio
4
Action–memory attention heads
4
Flow-matching inference steps
10
Optimizer
AdamW
AdamW (β1,β2)
(0.9,0.95)
Appendix
Table S1: Additional architecture and optimization hyperparameters. Values separated by “/” denote Stages 1 and 2.
Table S3: RoboMME task groups for the auxiliary-supervision analysis.
Task
Language instruction
Rearrange Blocks
Move the block between the two mats onto the empty mat, press the button, then move the other block (the one that started on a mat) to the space between the two mats.
Swap Blocks
There are three bowls on the table, and two blocks are placed in two different bowls. You may move only one block at a time, and each bowl can hold at most one block. Swap the positions of the two blocks. Finally, press the button.
Put Back Block
There are four mats, one block, and a button on the table. One block is on one of the mats. First, put the block in the center, then press the button. Then, put the block back in its original position.
Cover Blocks
On the table, red, green, and blue blocks are arranged randomly along with three cups. From the current viewpoint, cover the blocks from left to right using the cups, and then uncover them again in the sequence red, green, and blue.
Appendix
Table S4: Language instructions for the real-world evaluation tasks.
Method
Parameters
FLOPs
Peak memory
Latency/cycle
Completion time
SR ↑
Mem4Act
3.548 B
489 G
7.2 GiB
335 ms
40 s
81.50
GPT-6 Astra
–
–
–
19239 ms
1038 s
42.50
MME-VLA
3.442 B
421 G
7.0 GiB
340 ms
49 s
42.00
π0.5
3.353 B
239 G
6.7 GiB
131 ms
47 s
28.00
Mem-0
2.358 B
4923 G
4.8 GiB
487 ms
66 s
22.50
Appendix
Table S5: Real-world inference efficiency. All local methods are evaluated with batch size 1 on the same GPU. FLOPs and latency include historical image encoding, memory processing, and action prediction. GPT-6 latency includes remote API inference and communication. Completion time is averaged over successful rollouts only. For Mem-0, parameters, FLOPs, peak memory, and cycle latency cover only the executor; the high-level planner is excluded.
Failure mode
Mem4Act
GPT-6 Astra
FrameSampling
π0.5
Mem-0
Incorrect behavior
34
10
89
70
49
Non-recoverable state
3
73
27
12
57
Stall
0
12
0
62
49
Timeout
–
20
–
–
–
Total failures ↓
37
115
116
144
155
Appendix
Table S6: Failure modes in the real-world evaluation. Each failed rollout is assigned one primary failure category. Entries report the number of failed rollouts ( ↓ ).