Memory-augmented robotic policies are essential in handling memory-dependent tasks. However, existing approaches typically rely on simply extending the observation window, struggling to simultaneously achieve precise task-state tracking and robust long-horizon retention. To overcome these challenges, inspired by the Atkinson--Shiffrin memory model, we propose MemoAct, a hierarchical memory-augmented policy that leverages distinct memory tiers to tackle specific bottlenecks. Specifically, sensory memory filters immediate perceptual inputs, lossless short-term memory supports precise task-state tracking, and compressed long-term memory facilitates robust long-horizon retention. To enrich the evaluation landscape, we construct MemoryRTBench based on RoboTwin 2.0, comprising 6 manipulation tasks that systematically evaluate policy memory capabilities across three dimensions: sequential, spatial, and episodic memory. Extensive experiments across simulated and real-world scenarios demonstrate that MemoAct achieves superior performance compared to both existing Markovian baselines and history-aware policies. The project page is available at https://tlf-tlf.github.io/MemoActPage/.
Figures & tables
Fig. 1 : (a) An example of a memory-dependent task. (b) Policies lacking historical awareness fail under identical observations, while existing representative memory mechanisms suffer from limited long-horizon retention and poor task-state tracking. (c) Inspired by the Atkinson–Shiffrin memory model, we propose MemoAct, which simultaneously enables precise task-state tracking and robust long-horizon retention. (d) Results on MemoryRTBench, RMBench, and real-world experiments demonstrate that MemoAct significantly outperforms baseline algorithms.
Fig. 2 : Overview of MemoAct Pipeline. First, the sensory distillation module encodes RGB images and proprioceptive states into high-fidelity sensory memory. Subsequently, the distilled sensory memory queries relevant historical context from the long short-term memory bank. Next, a gating network adaptively fuses the retrieved history with the current sensory memory to produce a condition embedding, which guides the action decoder to iteratively denoise noisy action trajectories into history-aware action trajectories. Finally, the long short-term memory consolidation module updates the memory bank after each forward pass.
Fig. 3 : Overview of MemoryRTBench and real-world tasks. The tasks are executed sequentially following the alphabetical order (i.e., A → B → C → …). Notably, identical observations encountered during the execution are highlighted in red, blue or gray.
Method
SHT
BPR
STR
SeqE
LlcE
SR
SpaE
LlcE
SR
SpaE
LlcE
SR
(a) Baselines
ACT [ 3 ]
-
-
17.3±5.7
-
-
2.3±2.1
-
-
1.0±1.0
DP [ 2 ]
-
-
13.7±4.2
-
-
1.7±1.5
-
-
0.0±0.0
SeedPolicy [ 24 ]
30.7±5.7
10.0±2.0
59.3±4.5
52.0±5.6
6.3±1.5
41.7±6.8
45.3±3.5
0.0±0.0
54.7±3.5
(b) Variants of the memory module
TABLE I : Performance comparison of MemoAct and baseline methods on MemoryRTBench across three different seeds (mean ± std, %). We only report success rates for memory-free baselines.
Tasks \ Methods
ACT [ 3 ]
DP [ 2 ]
SAMP [ 16 ]
MVMP [ 12 ]
SeedPolicy [ 24 ]
MemoAct (ours)
Put Back Block
0.0±0.0
0.0±0.0
23.3±4.0
34.0±4.6
35.0±4.0
41.0±4.0
Swap T
2.3±2.5
16.0±3.6
25.3±3.5
55.0±4.0
12.7±3.5
53.3±1.5
Rearrange Blocks
19.3±3.5
0.0±0.0
46.7±4.5
21.0±3.6
6.3±2.5
98.0±2.0
Observe and Pick Up
1.7±2.1
1.7±1.5
2.7±2.1
4.0±2.0
2.0±2.0
4.0±2.0
Average
5.8±2.0
4.4±1.3
24.5±3.5
28.5±3.6
14.0±3.0
49.1±2.4
TABLE II: Performance comparison results of MemoAct and baseline methods on the RMBench across three different seeds (mean ± std, %).
Tasks \ Methods
DP [ 2 ]
SAMP [ 16 ]
MVMP [ 12 ]
SeedPolicy [ 24 ]
MTIL [ 11 ]
MemoAct (ours)
CTBO
35
10
45
60
70
70
PBB
0
20
70
15
10
70
GRB
0
45
90
85
65
95
DSP
0
25
60
20
35
75
Average
8.75
25
66.25
45
45
77.5
TABLE III: Performance comparison results of MemoAct and baseline methods on real-world tasks (%).
Method \ Tasks
PBB
CTBO
In-Domain
Red-to-Yellow
Red-to-Green
In-Domain
Background-Shift
MemoAct
70
70
65 ( ↓ 5)
70
55 ( ↓ 15)
TABLE IV: Performance of MemoAct in generalization experiments (%).
Variants \ Tasks
SHT
BPR
STR
ITR
LBT
CBTCO
Average
DINOv2 [ 27 ] → ResNet18 [ 32 ]
79.7±4.5
62.0±6.6
79.3±4.7
90.3±3.5
83.0±6.6
59.7±7.0
75.7±5.5
w/o TPE
77.0±5.6
97.0±2.6
97.7±3.2
95.3±4.7
82.7±4.0
28.7±6.0
79.7±4.4
Consol. TransEnc. → Add
71.0±5.6
98.7±1.5
95.7±4.5
97.0±3.6
86.7±5.0
29.0±5.0
79.7±4.2
Gate → Add
81.7±4.2
99.0±1.0
96.3±3.1
97.7±2.5
88.7±4.5
25.0±6.6
81.4±3.7
MemoAct (ours)
84.7±4.2
100.0±0.0
100.0±0.0
98.7±2.3
90.7±4.2
92.7±4.2
94.5±2.5
TABLE V: Ablation experimental results evaluating different variants of MemoAct across three different seeds (mean ± std, %).
Fig. 4 : Performance comparison of MemoAct under different memory capacities (%), evaluated with seed 0.
Methods \ Tasks
SHT
BPR
STR
ITR
LBT
CBTCO
Average
DP3 [ 33 ]
8.3±3.5
46.7±6.0
50.3±4.5
2.0±2.0
16.3±4.5
9.0±4.6
22.1±1.0
DP3 + memory module (ours)
78.0±7.0
96.0±4.0
52.3±5.5
75.3±6.0
21.0±5.6
37.7±6.5
60.1±3.6
TABLE VI: Performance comparison between DP3 and DP3 + memory module (ours) on MemoryRTBench across three different seeds (mean ± std, %).