MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation

Haiyang Sun; Haoxiao Wang; Junming Chen; Weicheng Fang; Zihao Su; Jingkun Yi; Wenyou Yi; Hao Chen

cs.ROSep 7, 2026

MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation

Authors: Haiyang Sun, Haoxiao Wang, Junming Chen, Weicheng Fang, Zihao Su, Jingkun Yi, Wenyou Yi, Hao Chen, +1 more

Organizations: Zhejiang University, Hangzhou, China

Abstract

Robotic manipulation often requires acting on information that is no longer visible, yet Vision-Language-Action policies are usually evaluated when the current observation largely determines the next action. Existing robotic memory benchmarks expose this gap, but they still rely mainly on final task success and therefore conflate forgetting with manipulation failure. We present MEMOBench, a benchmark for process level memory evaluation in robotic manipulation. MEMOBench includes 30 history dependent tasks, 1{,}500 expert demonstrations, and 4{,}200 executable checkpoint instances from 84 templates. Each checkpoint pairs coarse to fine language with a simulator predicate and labels one memory operation: Storage, Update, or Compression. These annotations define Memory Storage Rate, Memory Update Rate, and Memory Compression Rate, which measure memory fidelity alongside task success. Across standard and memory augmented VLA policies, the strongest memory module baseline reaches only 31.9% average success rate, and high storage often coexists with weak update and compression. Checkpoint language also supervises semantic, contrastive, and framewise memory alignment objectives, yielding modest gains across different memory operations. MEMOBench provides a diagnostic evaluation suite and training supervision for memory grounded robotic policies. The project page is available at https://github.com/Collab-Gen/MEMOBench.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of MEMOBench, including history dependent manipulation tasks, process level checkpoint annotations, and radar summaries for model evaluation and memory alignment.

Task Properties

Inputs

Annotations

Scale

Benchmark

Mem. Type

Non- Mkv.

Part. Obs.

Dyn. Scene

Vid.

Lang.

Subgoal

Keyframe

Mem. Op.

#Task

#Demo

Avg. Len.

General purpose manipulation benchmarks

RLBench James et al. (2020)

✗

✓

✗

✓

✗

1,800

137

CALVIN Mees et al. (2022)

✗

✓

✗

1,000

584

LIBERO Liu et al. (2023)

✗

✓

✗

130

6,500

162

RoboCerebra Han et al. (2025)

✗

✓

✗

✓

✗

100

1,000

2,972

Table 1: Comparison with existing manipulation benchmarks. Memory types: T = temporal, S = spatial, O = object, P = procedural, Seq = sequential, Cap = memory capacity. RLBench reports the commonly evaluated 18 task subset. RoboMemArena reports its simulation benchmark only, excluding 5 real robot tasks. The final column reports average trajectory length when available. MEMOBench is the first robotic manipulation memory benchmark with process level memory operation annotations, diagnosing which memory operation fails rather than only whether the task succeeds.

Figure 2: Overview of memory alignment from annotations. MEMOBench checkpoints provide subgoal text and frame level supervision for Semantic, Contrastive, and Framewise Memory Alignment.

		Temporal Mem.				Spatial Mem.				Object Mem.				Procedural Mem.				Average
	Model	SR	MSR	MUR	MCR	SR	MSR	MUR	MCR	SR	MSR	MUR	MCR	SR	MSR	MUR	MCR	Avg. SR	Avg. MSR	Avg. MUR	Avg. MCR
step = 5	$\pi_{0}$ Black et al. (2026)	4.4	94.0	12.4	4.0	0.2	0.4	0.4	0.3	27.6	27.6	–	–	12.4	32.4	12.0	–	11.2	38.6	8.3	2.1
	$\pi_{0.5}$ Intelligence et al. (2025)	2.2	67.6	18.4	2.0	0.0	6.4	0.3	6.4	99.6	99.6	–	–	19.6	46.0	22.0	–	30.4	54.9	13.5	4.2
	OpenVLA-OFT Kim et al. (2025)	0.4	5.6	0.5	1.0	0.0	0.5	1.6	0.3	33.2	33.2	–	–	6.4	9.6	15.3	–	10.0	12.2	5.8	0.6
	MemoryVLA † Shi et al. (2026)	0.2	87.8	38.0	0.0	22.4	23.6	24.1	22.3	2.4	2.4	–	–	43.6	58.4	45.7	–	17.2	43.1	35.9	11.1
	HiF-VLA † Lin et al. (2026)	0.2	1.8	0.0	0.0	0.0	0.3	0.0	0.1	79.6	79.6	–	–	4.0	10.4	14.3	–	21.0	23.0	4.8	0.1

Table 2: Main MEMOBench results under replan steps 5, 20, 35, and 50. Avg. columns macro average over applicable memory types. Bold and underline mark the best and second best values within each replan block. Dashes are excluded from averages, and † marks explicit memory mechanisms.

		Temporal Mem.				Spatial Mem.				Object Mem.				Procedural Mem.				Average
	Method	SR	MSR	MUR	MCR	SR	MSR	MUR	MCR	SR	MSR	MUR	MCR	SR	MSR	MUR	MCR	Avg. SR	Avg. MSR	Avg. MUR	Avg. MCR
step = 5	Contrastive Memory Alignment	14.4	91.8	39.6	14.0	1.4	7.6	1.6	8.3	99.2	99.2	–	–	23.6	40.4	23.0	–	34.7	59.8	21.4	11.1
	Semantic Memory Alignment	12.4	97.8	36.5	12.0	0.2	5.6	0.5	4.3	99.2	99.2	–	–	15.6	40.4	15.0	–	31.9	60.8	17.4	8.1
	Framewise Memory Alignment	10.4	99.4	25.6	10.0	1.4	3.6	1.6	4.3	99.2	99.2	–	–	19.6	34.4	19.0	–	32.7	59.2	15.4	7.1
	Memory module	9.8	92.0	32.2	10.0	2.0	8.4	1.7	5.6	99.6	99.6	–	–	16.0	32.4	15.7	–	31.9	58.1	16.5	7.8
step = 20	Contrastive Memory Alignment	12.4	97.8	25.6	12.0	1.4	7.6	1.6	6.3	99.2	99.2	–	–	5.6	14.4	5.0	–	29.7	54.8	10.7	9.1

Table 3: Memory alignment results for the RoboMME

\pi_{0.5}

memory module Dai et al. (2026) . Alignment variants add auxiliary training losses, while Memory module is the unaligned baseline. Bold and underline mark the best and second best values within each replan block. Dashes are excluded from averages.

Figure 3: Metric correlations from Table 2 . Each point is one model, replan step, and memory type entry. Shading marks temporal cases with high MSR but low SR. Pearson correlations use all applicable entries for each process metric.

Figure 4: Failure taxonomy for a representative temporal task. Bars summarize coarse, intermediate, and fine-grained 50 episode rollout logs.

Memory type	SR	MSR	MUR	MCR
Temporal	2.2/54.9	67.6/68.0	18.4/54.2	2.0/53.0
Spatial	0.0/45.8	6.4/77.9	0.3/84.1	6.4/60.7
Object	99.6/96.8	99.6/96.8	–	–
Procedural	19.6/10.0	46.0/13.2	22.0/20.3	–
Average	30.4/51.9	54.9/64.0	13.5/52.9	4.2/56.8

Table 4: Oracle memory decomposition at replan step 5. Each cell reports standard

\pi_{0.5}

and, after the slash, the same backbone fine tuned and evaluated with ground truth memory text appended to the prompt. Dashes mark memory types without the corresponding operation.

Figure 5: Shuffled memory control on the temporal task from Figure 4 . Lines show prefix checkpoint completion, and final labels show successful rollouts.

Figure 6: Annotation granularity ablation on the temporal task from Figure 4 . Lines show prefix checkpoint completion, and final labels show successful rollouts.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Family	Task Variants	Ops.	#Task	#Ckpt.
Temporal	Drawer checking	Top drawer, middle drawer, all three drawers	S/U/C	3	5
	Appliance tracking	Microwave open/close, microwave open/close plus bowl placement	S/U/C	2	5
	Stove transition	Stove on/off, near shelf variant, two frying pan placement variants	S/U	4	10
	Drawer transfer	Check middle drawer, then place black bowl in top drawer and close it	S/U	1	3
Spatial	Single drawer placement	Artichoke, broccoli, brussels sprouts, charger, or tomato into top drawer	S/U/C	5	15
	Two drawer placement	Five paired object variants for top and middle drawers	S/U/C	5	30

Appendix

Table 5: Task family summary for MEMOBench. Ops. gives the memory operations covered by each family, where S = Storage, U = Update, and C = Compression. #Ckpt. reports executable checkpoint templates before expanding them across demonstrations.

Family	Tasks	Episodes	Steps	Avg.	Median	P95	Max	Storage
Object	5	250	121,276	485.10	492.00	526.00	556	44.53 GiB
Procedural	5	250	250,204	1,000.82	876.00	1,649.50	2,012	91.79 GiB
Spatial	10	500	357,865	715.73	727.00	1,101.10	1,513	131.58 GiB
Temporal	10	500	198,099	396.20	400.50	679.05	779	72.66 GiB
Total	30	1,500	927,444	618.30	489.00	1,110.10	2,012	340.56 GiB

Appendix

Table 6: Family level statistics for the expert demonstrations. Steps count action or control steps, and storage is measured from the HDF5 task files.

Length bucket	Episodes	Share
0–199	50	3.33%
200–399	199	13.27%
400–599	687	45.80%
600–799	108	7.20%
800–999	290	19.33%
1,000–1,199	110	7.33%

Appendix

Table 7: Distribution of expert demonstration lengths in action steps.

Statistic	Value
Fleiss’ $\kappa$ (three raters)	0.799
Pairwise Cohen’s $\kappa$	0.695 / 0.835 / 0.868
Exact three rater agreement	70/84 (83.3%)
Majority vote vs. released labels	71/84 (84.5%)
Pooled vs. released labels	203/252 (80.6%)
Per rater vs. released labels	77.4% / 79.8% / 84.5%

Appendix

Table 8: Agreement statistics for the blind relabeling study over the 84 checkpoint templates. Pooled agreement counts each of the three annotator decisions per template, for 252 decisions in total.

Released label	Ann. S	Ann. U	Ann. C
S (35 templates)	100	5	0
U (32 templates)	2	90	4
C (17 templates)	0	38	13

Appendix

Table 9: Pooled confusion matrix between released operation labels and annotator labels in the blind relabeling study. Each row pools the three annotator decisions per template, so rows sum to three times the template count.

Figure 7: Radar summaries for Table 2 (left) and Table 3 (right). Each axis takes the best value over replan settings for each method and metric, then normalizes by the axis maximum.

Memory type	SR	MSR	MUR	MCR
Temporal	62.4	63.8	61.7	63.0
Spatial	50.4	84.3	52.8	77.6
Object	93.2	93.2	–	–
Procedural	19.6	20.4	21.0	–
Average	56.4	65.4	45.2	70.3

Appendix

Table 10: MemER results on MEMOBench. MemER receives checkpoint based keyframe and subtask supervision, so we report it as a privileged supervision reference row rather than a matched baseline for Table 2 . Dashes mark memory types without the corresponding operation.

Item	Setting
Train data	1,500 expert demonstrations, 50 per task
Evaluation	1,500 held out simulator rollouts, 50 per task
Evaluation seeds	base seed 7, episode seed $7+i$
Rollout horizon	10 stabilization steps and 1,200 policy steps
Image size	$256\times 256$ render, $224\times 224$ policy input
Action execution	50 action predictions, replan steps $k\in\{5,20,35,50\}$

Appendix

Table 11: Reproducibility settings for MEMOBench evaluation and the controlled

\pi_{0.5}

memory alignment runs.

Explore similar work

RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design
Mar 1, 2026Tianxing Chen, Yuran Wang, Mingleyang Li +16Robotic Manipulation Robot Policy Learning
PKU · PsiBot · HKUST (GZ) +3
RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies
Mar 4, 2026Yinpei Dai, Hongze Fu, Jayjun Lee +6Memory-Augmented VLMs Robot Manipulation Benchmarks
University of Michigan · Stanford University · Figure AI
MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
Sep 30, 2026Egor Cherepanov, Nikita Kachaev, Aleksandr I. Panov +1Memory-Augmented VLMs Long-Horizon Robotic Manipulation
AXXX, Moscow, Russia · MIRIAI, Moscow, Russia · HSE University, Moscow, Russia

Mar 1, 2026J/K move · Enter open · S save

RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design

Tianxing Chen, Yuran Wang, Mingleyang Li, Yan Qin+15

PKU · PsiBot · HKUST (GZ)+3

Robotic Manipulation Robot Policy Learning Robot Manipulation Benchmarks Robotic Policy Evaluation

Open paper AskPDF arXiv

Robotic manipulation policies have made rapid progress in recent years, yet most existing approaches give limited consideration to memory capabilities. Consequently, they struggle to solve tasks that require reasoning over historical observations and maintaining task-relevant information over time, which are common requirements in real-world manipulation scenarios. Although several memory-aware policies have been proposed, systematic evaluation of memory-dependent manipulation remains underexplored, and the relationship between architectural design choices and memory performance is still not well understood. To address this gap, we introduce RMBench, a simulation benchmark comprising 9 manipulation tasks that span multiple levels of memory complexity, enabling systematic evaluation of policy memory capabilities. We further propose Mem-0, a modular manipulation policy with explicit memory components designed to support controlled ablation studies. Through extensive simulation and real-world experiments, we identify memory-related limitations in existing policies and provide empirical insights into how architectural design choices influence memory performance. The website is available at https://rmbench.github.io/.

Abstract

Explore similar work

RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design

RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies

MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation