cs.ROMar 1, 2026
SaveRMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design
Organizations: PKU · PsiBot · HKUST (GZ) · THU · SZU · SJTU
Abstract
Robotic manipulation policies have made rapid progress in recent years, yet most existing approaches give limited consideration to memory capabilities. Consequently, they struggle to solve tasks that require reasoning over historical observations and maintaining task-relevant information over time, which are common requirements in real-world manipulation scenarios. Although several memory-aware policies have been proposed, systematic evaluation of memory-dependent manipulation remains underexplored, and the relationship between architectural design choices and memory performance is still not well understood. To address this gap, we introduce RMBench, a simulation benchmark comprising 9 manipulation tasks that span multiple levels of memory complexity, enabling systematic evaluation of policy memory capabilities. We further propose Mem-0, a modular manipulation policy with explicit memory components designed to support controlled ablation studies. Through extensive simulation and real-world experiments, we identify memory-related limitations in existing policies and provide empirical insights into how architectural design choices influence memory performance. The website is available at https://rmbench.github.io/.
Figures & tables
Figure 1: RMBench Tasks. We illustrate the nine memory-dependent tasks in RMBench along with their key execution steps. Tasks detailed description are shown in Appendix. A .
Figure 2: Mem-0 Pipeline . Mem-0 comprises a Planning Module and an Execution Module linked by a Subtask End Classifier. The Planning Module generates high-level subtasks from task instructions, observations, and key-frame memory, while the Execution Module produces low-level actions using the current observation, the subtask, and fused anchor and sliding memories in a diffusion-based policy. Upon subtask completion, a key frame is stored to enable iterative planning and execution until task completion.
| Tasks | TMC | DP | ACT | Pi0.5 | X-VLA | Mem-0 (ours) |
|---|---|---|---|---|---|---|
| Observe and Pick Up | 1% | 1% | 9% | 9% | 4% | |
| Rearrange Blocks | 0% | 29% | 13% | 13% | 89% | |
| Put Back Block | 0% | 0% | 11% | 18% | 90% | |
| Swap Blocks | 11% | 2% | 24% | 16% | 67% | |
| Swap T | 20% | 2% | 15% | 3% | 14% | |
| Average | 6.4% | 6.8% | 14.4% | 11.8% | 52.8%( +38.4% ) |
Table 1: RMBench benchmark results. RMBench includes nine manipulation tasks across the and levels of Task Memory Complexity. We report success rates for five policies, each trained with 50 synthesized demonstrations and evaluated over 100 rollouts. ( Bold : best; Underlined : second-best; Green : relative improvement over the second-best).
Figure 3: Visualization of Baseline Typical Error. Because the baseline predicts the next action solely from the current observation, it struggles to perform reliably on non-Markovian tasks that require persistent memory over time.
| Tasks | Observe and Pick Up | Rearrange Blocks | Put Back Block | Swap Blocks | Swap T | Average |
|---|---|---|---|---|---|---|
| Vanilla (ours) | 4% | 89% | 90% | 67% | 14% | 52.8% |
| w/o Anchor | 4% | 73% | 35% | 15% | 7% | 26.8% |
| w/o Sliding | 3% | 62% | 78% | 39% | 20% | 40.4% |
Table 2: Ablation Studies. ( Bold : the best results; Underlined : the second-best results).
| Tasks | ACT | Pi0.5 | Mem-0 (ours) |
|---|---|---|---|
| Put Back Block | 0.0% | 10.0% | 17.5% |
| Rearrange Blocks | 0.0% | 7.5% | 37.5% |
| Cover Blocks | 0.0% | 0.0% | 12.5% |
| Average | 0.00% | 5.83% | 22.50% |
Table 3: Real-world Experiment results.
Figure 4: Real-world Experiment Tasks. The real-world experimental setup is illustrated above.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Description |
|---|---|
| Observe and Pick Up | A reference object is placed on a shelf, and multiple objects are placed on the table. The robot first observes the reference object while remaining stationary. After the reference object is hidden, the robot must pick up the matching object from the table. |
| Rearrange Blocks | Two pads and a button are placed on the table. One block is positioned between the two pads, and another block is placed on one of the pads. The robot moves the middle block onto a pad, presses the button, and then moves the other block to the middle position. |
| Put Back Block | Four pads are arranged around a central position, with one block placed on one of the pads. The robot moves the block to the center, presses the button, and then returns the block to its original pad. |
| Swap Block | Three pads and a button are placed on the table, with two blocks placed on different pads. The robot uses the empty pad to swap the positions of the two blocks and then presses the button. |
| Swap T | Two T-shaped blocks with different colors are placed on the table. The robot picks up both blocks and swaps their positions and orientations. |
| Battery Try | Two batteries with random orientations and a dual-slot battery holder are placed on the table. The robot repeatedly attempts different insertion orders, placing both batteries into the holder with the correct orientations until the insertion succeeds. |
Table 4: Task descriptions of RMBench benchmark.
| Configuration | Finetuning Type | LoRA Rank | Batch Size | Learning Rate | Epochs | LR Scheduler | Warmup Ratio | Dtype |
|---|---|---|---|---|---|---|---|---|
| Value | LoRA | 8 | 16 | 25 | Cosine | 0.1 | bf16 |
Table 5: Hyperparameters for fine-tuning the Mem-0 Planning Module.
| Configuration | Value |
|---|---|
| Batch Size | 448 |
| Iterations | 30,000 |
| Max Grad. Norm | 2.5 |
| LR Scheduler | Cosine |
| Warmup Ratio | 0.05 |
| Optimizer | AdamW |
Table 6: Hyperparameters for training the Mem-0 Execution Module.
Figure 7: Failure examples of Rearrange Blocks . Mem-0 redundantly presses the button, resulting in task failure.
Figure 8: Failure examples of Cover Blocks . The Classifier fails to accurately detect the completion of the Uncover xxx subtask, thereby preventing a subtask transition. As the instruction remains unchanged, the model is forced to operate under a wrong task context, leading to unintended and erratic behaviors.
Figure 9: Failure examples of Blocks Ranking Try . Upon pressing the button, the system is expected to transition to the next subtask to execute the swapping of designated blocks. However, the Classifier fails to trigger this transition promptly, causing the task to stall in the Press button state. This leads to a coordination conflict between the dual arms: the right hand attempts to initiate manipulation while the left hand remains tethered to the button-pressing instruction.
Figure 10: Failure examples of Press Button . (Top) Insufficient presses: the Classifier issues a false positive termination signal even when the button-press is unsuccessful. (Bottom) Excessive presses: the Classifier fails to recognize a successful subtask completion, leading to redundant execution of the same subtask.
Figure 11: Failure examples of Battery Try . (Top) For the horizontally oriented cylindrical battery, a suboptimal grasp pose prevents a successful lift and causes significant displacement, leading the model into unforeseen observational states. (Bottom) The model fails to commit to a specific manipulation strategy during battery adjustment, resulting in a mean action that leads to improper placement in the slot.