Robotic manipulation policies have made rapid progress in recent years, yet most existing approaches give limited consideration to memory capabilities. Consequently, they struggle to solve tasks that require reasoning over historical observations and maintaining task-relevant information over time, which are common requirements in real-world manipulation scenarios. Although several memory-aware policies have been proposed, systematic evaluation of memory-dependent manipulation remains underexplored, and the relationship between architectural design choices and memory performance is still not well understood. To address this gap, we introduce RMBench, a simulation benchmark comprising 9 manipulation tasks that span multiple levels of memory complexity, enabling systematic evaluation of policy memory capabilities. We further propose Mem-0, a modular manipulation policy with explicit memory components designed to support controlled ablation studies. Through extensive simulation and real-world experiments, we identify memory-related limitations in existing policies and provide empirical insights into how architectural design choices influence memory performance. The website is available at https://rmbench.github.io/.
Figures & tables
Figure 1: RMBench Tasks. We illustrate the nine memory-dependent tasks in RMBench along with their key execution steps. Tasks detailed description are shown in Appendix. A .
Figure 2: Mem-0 Pipeline . Mem-0 comprises a Planning Module and an Execution Module linked by a Subtask End Classifier. The Planning Module generates high-level subtasks from task instructions, observations, and key-frame memory, while the Execution Module produces low-level actions using the current observation, the subtask, and fused anchor and sliding memories in a diffusion-based policy. Upon subtask completion, a key frame is stored to enable iterative planning and execution until task completion.
Tasks
TMC
DP
ACT
Pi0.5
X-VLA
Mem-0 (ours)
Observe and Pick Up
M(1)
1%
1%
9%
9%
4%
Rearrange Blocks
M(1)
0%
29%
13%
13%
89%
Put Back Block
M(1)
0%
0%
11%
18%
90%
Swap Blocks
M(1)
11%
2%
24%
16%
67%
Swap T
M(1)
20%
2%
15%
3%
14%
Average
M(1)
6.4%
6.8%
14.4%
11.8%
52.8%( +38.4% )
Table 1: RMBench benchmark results. RMBench includes nine manipulation tasks across the M(1) and M(n) levels of Task Memory Complexity. We report success rates for five policies, each trained with 50 synthesized demonstrations and evaluated over 100 rollouts. ( Bold : best; Underlined : second-best; Green : relative improvement over the second-best).
Figure 3: Visualization of Baseline Typical Error. Because the baseline predicts the next action solely from the current observation, it struggles to perform reliably on non-Markovian tasks that require persistent memory over time.
M(1) Tasks
Observe and Pick Up
Rearrange Blocks
Put Back Block
Swap Blocks
Swap T
Average
Vanilla (ours)
4%
89%
90%
67%
14%
52.8%
w/o Anchor
4%
73%
35%
15%
7%
26.8%
w/o Sliding
3%
62%
78%
39%
20%
40.4%
Table 2: Ablation Studies. ( Bold : the best results; Underlined : the second-best results).
Tasks
ACT
Pi0.5
Mem-0 (ours)
Put Back Block
0.0%
10.0%
17.5%
Rearrange Blocks
0.0%
7.5%
37.5%
Cover Blocks
0.0%
0.0%
12.5%
Average
0.00%
5.83%
22.50%
Table 3: Real-world Experiment results.
Figure 4: Real-world Experiment Tasks. The real-world experimental setup is illustrated above.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Description
Observe and Pick Up
A reference object is placed on a shelf, and multiple objects are placed on the table. The robot first observes the reference object while remaining stationary. After the reference object is hidden, the robot must pick up the matching object from the table.
Rearrange Blocks
Two pads and a button are placed on the table. One block is positioned between the two pads, and another block is placed on one of the pads. The robot moves the middle block onto a pad, presses the button, and then moves the other block to the middle position.
Put Back Block
Four pads are arranged around a central position, with one block placed on one of the pads. The robot moves the block to the center, presses the button, and then returns the block to its original pad.
Swap Block
Three pads and a button are placed on the table, with two blocks placed on different pads. The robot uses the empty pad to swap the positions of the two blocks and then presses the button.
Swap T
Two T-shaped blocks with different colors are placed on the table. The robot picks up both blocks and swaps their positions and orientations.
Battery Try
Two batteries with random orientations and a dual-slot battery holder are placed on the table. The robot repeatedly attempts different insertion orders, placing both batteries into the holder with the correct orientations until the insertion succeeds.
Appendix
Table 4: Task descriptions of RMBench benchmark.
Configuration
Finetuning Type
LoRA Rank
Batch Size
Learning Rate
Epochs
LR Scheduler
Warmup Ratio
Dtype
Value
LoRA
8
16
1.0×10−4
25
Cosine
0.1
bf16
Appendix
Table 5: Hyperparameters for fine-tuning the Mem-0 Planning Module.
Configuration
Value
Batch Size
448
Iterations
30,000
Max Grad. Norm
2.5
LR Scheduler
Cosine
Warmup Ratio
0.05
Optimizer
AdamW
Appendix
Table 6: Hyperparameters for training the Mem-0 Execution Module.
Figure 11
Figure 7: Failure examples of Rearrange Blocks . Mem-0 redundantly presses the button, resulting in task failure.
Figure 8: Failure examples of Cover Blocks . The Classifier fails to accurately detect the completion of the Uncover xxx subtask, thereby preventing a subtask transition. As the instruction remains unchanged, the model is forced to operate under a wrong task context, leading to unintended and erratic behaviors.
Figure 9: Failure examples of Blocks Ranking Try . Upon pressing the button, the system is expected to transition to the next subtask to execute the swapping of designated blocks. However, the Classifier fails to trigger this transition promptly, causing the task to stall in the Press button state. This leads to a coordination conflict between the dual arms: the right hand attempts to initiate manipulation while the left hand remains tethered to the button-pressing instruction.
Figure 10: Failure examples of Press Button . (Top) Insufficient presses: the Classifier issues a false positive termination signal even when the button-press is unsuccessful. (Bottom) Excessive presses: the Classifier fails to recognize a successful subtask completion, leading to redundant execution of the same subtask.
Figure 11: Failure examples of Battery Try . (Top) For the horizontally oriented cylindrical battery, a suboptimal grasp pose prevents a successful lift and causes significant displacement, leading the model into unforeseen observational states. (Bottom) The model fails to commit to a specific manipulation strategy during battery adjustment, resulting in a mean action that leads to improper placement in the slot.
Robotic manipulation often requires acting on information that is no longer visible, yet Vision-Language-Action policies are usually evaluated when the current observation largely determines the next action. Existing robotic memory benchmarks expose this gap, but they still rely mainly on final task success and therefore conflate forgetting with manipulation failure. We present MEMOBench, a benchmark for process level memory evaluation in robotic manipulation. MEMOBench includes 30 history dependent tasks, 1{,}500 expert demonstrations, and 4{,}200 executable checkpoint instances from 84 templates. Each checkpoint pairs coarse to fine language with a simulator predicate and labels one memory operation: Storage, Update, or Compression. These annotations define Memory Storage Rate, Memory Update Rate, and Memory Compression Rate, which measure memory fidelity alongside task success. Across standard and memory augmented VLA policies, the strongest memory module baseline reaches only 31.9% average success rate, and high storage often coexists with weak update and compression. Checkpoint language also supervises semantic, contrastive, and framewise memory alignment objectives, yielding modest gains across different memory operations. MEMOBench provides a diagnostic evaluation suite and training supervision for memory grounded robotic policies. The project page is available at https://github.com/Collab-Gen/MEMOBench.
Memory-augmented robotic policies are essential in handling memory-dependent tasks. However, existing approaches typically rely on simply extending the observation window, struggling to simultaneously achieve precise task-state tracking and robust long-horizon retention. To overcome these challenges, inspired by the Atkinson--Shiffrin memory model, we propose MemoAct, a hierarchical memory-augmented policy that leverages distinct memory tiers to tackle specific bottlenecks. Specifically, sensory memory filters immediate perceptual inputs, lossless short-term memory supports precise task-state tracking, and compressed long-term memory facilitates robust long-horizon retention. To enrich the evaluation landscape, we construct MemoryRTBench based on RoboTwin 2.0, comprising 6 manipulation tasks that systematically evaluate policy memory capabilities across three dimensions: sequential, spatial, and episodic memory. Extensive experiments across simulated and real-world scenarios demonstrate that MemoAct achieves superior performance compared to both existing Markovian baselines and history-aware policies. The project page is available at https://tlf-tlf.github.io/MemoActPage/.
Recent advances in robot learning have enabled manipulation policies to perform increasingly diverse tasks and generalize across environments. However, reliable execution often depends on hidden task states that cannot be determined from current observations alone, making interaction history essential. We introduce HIDE, a benchmark for evaluating manipulation memory under partial observability. HIDE comprises 15 tasks covering repetition counting, historical-state recall, and execution-progress tracking, with randomized initial configurations and decision points where similar observations require different actions depending on prior events. We further propose SEEK, a framework combining three complementary memory mechanisms to retain historical evidence and track execution state. Evaluations reveal substantial limitations in existing policies on HIDE, while memory augmentation improves task success in both simulation and real-world experiments. Individual mechanisms benefit some tasks but can degrade others; their combination achieves the highest average success rate on HIDE among the evaluated configurations. These findings highlight the importance of maintaining internal representations of hidden task states and matching memory design to task-specific information requirements.
Yansong Shi, Jiange Yang, Xijie Yang +4
School of Information Science And Technology, University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · State Key Lab of Novel Software Technology, Nanjing University +2