Memory is critical for long-horizon and history-dependent robotic manipulation. Such tasks often involve counting repeated actions or manipulating objects that become temporarily occluded. Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms; however, their evaluations remain confined to narrow, non-standardized settings. This limits systematic understanding, comparison, and progress measurement. To address these challenges, we introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models in long-horizon, history-dependent scenarios. Our benchmark comprises 16 manipulation tasks constructed under a carefully designed taxonomy that evaluates temporal, spatial, object, and procedural memory. We further develop a suite of 14 memory-augmented VLA variants built on the π0.5 backbone to systematically explore different memory representations across multiple integration strategies. Experimental results show that the effectiveness of memory representations is highly task-dependent, with each design offering distinct advantages and limitations across different tasks. Videos and code can be found at our website https://robomme.github.io.
Figures & tables
Task Name
Memory Type
Avg. #Steps
Task Challenge
Brief Description
Task Suite: Counting
PickXTimes
T
538
count
Pick and place a cube of a given color for a specified number of repetitions.
BinFill
T
604
count
Place a specified number of cubes of a given color into the bin as the cubes appear over time.
SwingXTimes
T
435
count, swing-motion
Swing a cube back and forth between two targets for a specified number of cycles.
StopCube
T
317
count, time-critical
Press a button exactly when a moving cube reaches the target at a specified occurrence.
Table 1: Task Summary . Overview of all tasks, their corresponding memory types, average total timesteps, and key challenges.
Dataset
Environment Complexity
Condition
Annotation
Memory Type
Total #Tasks
Total #Demos
Avg. #Steps
Non-Markov
Partial Obs.
Dynamic
Vid.
Lang.
Subgoal
Keyframe
RLBench18 [ 21 , 39 ]
✗
✗
✗
✗
✓
✗
✓
T
18
1,800
137
CALVIN [ 34 ]
✗
✗
✗
✗
✓
✓
✓
T
34
1,000
584
LIBERO [ 31 ]
✗
✗
✗
✗
✓
✗
✗
T
130
6,500
162
VLABench [ 50 ]
✗
✗
✗
✗
✓
✗
✗
T
100
1,600
115
RoboCerebra [ 19 ]
✗
✗
✓
✗
✓
✓
✗
T
100
1,000
2,972
Table 2: Benchmark Comparison. The upper section lists general-purpose benchmarks and the lower section lists memory-based benchmarks. Memory types include temporal (T), spatial (S), object (O), and procedural (P). Environment complexity captures non-Markovian structure, partial observability, and dynamic scene change. Condition denotes initial inputs (Vid. for video observations; Lang. for language instructions). Annotations indicate the availability of language-based subgoals or keyframe timesteps for auxiliary supervision.
Figure 2: Framework of MME-VLA Suite. The top part illustrates three memory representations, each with two instantiations: (1) Symbolic Memory summarizes past interactions as high-level abstractions via language-based subgoals, optionally grounded to image pixels; (2) Perceptual Memory encodes history as raw visual tokens, using either token dropping to remove redundancy or uniform frame sampling to preserve essential context; (3) Recurrent Memory compresses history into fixed-size latent states through test-time training or recurrent memory transformers. The bottom part shows three integration strategies consistent with π0.5 : (1) Memory-as-Context concatenates memory tokens with observation tokens (e.g., images, task instructions, and proprioception); (2) Memory-as-Modulator applies adaptive LayerNorm to modulate the action expert, where action features cross-attend to memory tokens via multi-head attention; (3) Memory-as-Expert adds a separate memory expert that interacts with other experts through block-wise causal attention.
Method
Counting
Permanence
Reference
Imitation
AVG
Bin Fill
Pick Xtimes
Swing Xtimes
Stop Cube
Video Umsk
Button Umsk
Video UmskS
Button UmskS
Pick HighL
Video Repick
Video PlcBtn
Video PlcOrd
Move Cube
Insert Peg
Pattern Lock
Route Stick
Human Performance
96.00
100.0
80.00
78.00
90.00
92.00
92.00
90.00
92.00
92.00
98.00
90.00
90.00
98.00
84.00
86.00
90.50
MME-VLA w/ Symbolic Memory
SimpleSG
Oracle
85.78
99.78
100.0
44.67
33.11
22.00
15.56
15.56
44.00
27.78
31.33
26.00
87.33
10.00
95.33
55.11
49.58
GroundSG
Oracle
85.78
100.0
100.0
49.67
98.78
95.00
99.22
80.22
83.33
97.33
100.0
100.0
87.78
15.56
97.00
55.56
84.08
Gemini
46.00
63.00
45.00
2.00
29.00
9.00
14.00
2.00
20.00
15.00
26.00
29.00
61.00
4.00
7.00
0.00
23.25
Table 3: Main Results. Our MME-VLA suite integrates π0.5 with three memory representations: (1) symbolic memory, implemented as simple or grounded subgoals ( SimpleSG , GroundSG ); (2) perceptual memory, using token dropping ( TokenDrop ) or frame sampling ( FrameSamp ); and (3) recurrent memory, using test-time training ( TTT ) or a recurrent memory transformer ( RMT ). Symbolic subgoals can be generated by Gemini-2.5-Pro ( Gemini ), a fine-tuned Qwen3-VL-4B ( QwenVL ), or simulator ground truth ( Oracle ). Perceptual and recurrent memory are integrated end-to-end via three mechanisms: memory-as-context ( Context ), memory-as-modulator ( Modul ), and memory-as-expert ( Expert ). Red marks the best per section; marks the overall best for non-oracle models.
Figure 3: Performance comparison across task characteristics.
Figure 4: Efficiency-performance comparison among models.
Method
Put Fruits
Track Cube
Repick Block
Draw Pattern
Total
π0.5
2/10
1/10
1/10
0/10
4/40
GroundSG + QwenVL
9/10
3/10
5/10
2/10
19/40
FrameSamp + Modul
6/10
5/10
6/10
8/10
25/40
Table 4: Real-world experiment results.
Appendix figures & tables51 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Batch Size
48
Warmup Ratio
0.05
LR
1×10−4
Train Epochs
2
LoRA
rank 16, alpha 32
DeepSpeed
ZeRO-2
Appendix
Table 5: Training configuration for VLM subgoal predictor.
Figure 5: Example Prompt for VLM Simple Subgoal Prediction. This is used to generate simple subgoals for the SimpleSG VLA policy.
Figure 6: Example Prompt for VLM Grounded Subgoal Prediction. This is used to generate grounded subgoals for the GroundSG VLA policy.
Figure 7: Example Prompt for MemER Baseline. We use grounded subgoals for MemER because we found that grounded subgoals perform better.
Hyperparameter
Value
Base model
π0.5 base
Batch size
64 for symbolic/perceptual, 16 for recurrent
Action horizon
20
Action space
Joint-space
Proprioception
False
Total steps
80,000
Appendix
Table 6: Training hyperparameters for MME-VLA models.
Hyperparameters
SAM2Act Training
SAM2Act+ Training
Batch size
48
80
LR
6.0×10−4
1.0×10−4
Optimizer
LAMB
LAMB
LR schedule
warmup + cosine
warmup + cosine
Weight decay
1×10−4
1×10−4
Warmup steps
2000
2000
Appendix
Table 7: Training hyperparameters for SAM2Act and SAM2Act+.
Hyperparameter
Value
Base model
OpenVLA-7B (LoRA, rank 32 )
Batch size
32 (symbolic), 24 (perceptual)
Action head
L1 regression, parallel decoding
Action horizon
20
Action space
Joint-space
Proprioception
True
Appendix
Table 8: Training hyperparameters for OpenVLA-OFT.
2 cameras + 8 -D state + CLIP ViT-B/32 text (frozen)
Diffusion
DDPM, 100 steps, cosine, ϵ -pred, clip sample
Appendix
Table 10: Training hyperparameters for Diffusion Policy on RoboMME.
Figure 8: Human Study GUI.
Method
Counting
Permanence
Reference
Imitation
AVG
Bin Fill
Pick Xtimes
Swing Xtimes
Stop Cube
Video Umsk
Button Umsk
Video UmskS
Button UmskS
Pick HighL
Video Repick
Video PlcBtn
Video PlcOrd
Move Cube
Insert Peg
Draw Pattern
Route Stick
Human
96.00
100.0
80.00
78.00
90.00
92.00
92.00
90.00
92.00
92.00
98.00
90.00
90.00
98.00
84.00
86.00
90.50
Gemini-2.5-Pro
78.00
80.00
58.00
32.00
86.00
82.00
18.00
12.00
78.00
28.00
50.00
60.00
66.00
32.00
4.00
2.00
47.88
GPT4o
50.00
58.00
32.00
18.00
32.00
60.00
28.00
34.00
30.00
14.00
18.00
20.00
70.00
18.00
6.00
2.00
30.63
Qwen3-VL-32B
54.00
60.00
54.00
32.00
26.00
28.00
22.00
14.00
14.00
32.00
26.00
14.00
66.00
2.00
2.00
0.00
27.88
Appendix
Table 11: Performance using the oracle planner for both foundation models and human participants.
Representation
Policy
Integration/VLM
Counting
Permanence
Reference
Imitation
AVG
No Memory ( π0.5 )
–
–
28.78
17.00
17.17
8.78
17.93 ± 0.61
Symbolic
SimpleSG
Oracle
82.56
21.56
32.28
61.94
49.58 ± 1.33
GroundSG
Oracle
83.86
93.31
95.17
63.98
84.08 ± 1.86
Gemini
39.00
13.50
22.50
18.00
23.25 ± 0.35
SimpleSG
QwenVL
44.61
19.61
25.22
26.56
29.00 ± 0.39
Gemini
12.75
15.75
11.25
6.5
11.56 ± 0.97
Appendix
Table 12: Comparison over MME-VLA suite and prior methods per task suite. Red marks the best per section; marks the overall best for non-oracle models.
Precise temporal coordination; success depends critically on acting at the correct moment
StopCube
Short-Horizon Video Reasoning
Grounding objects from short video history in mostly static or mildly dynamic scenes
VideoUnmask , VideoRepick , VideoUnmaskSwap
Long-Horizon Video Reasoning
Integrating information across extended temporal sequences (often >1000 steps) before acting
VideoPlaceButton , VideoPlaceOrder
Dynamic Online Scene-Change
Continual belief updates and online adaptation under concurrent environmental changes during execution
ButtonUnmask , ButtonUnmaskSwap , PickHighlight
Event-Salient
progress is marked by visually salient, discrete events that clearly signal subtask completion or manipulation intent
PickXtimes , BinFill , MoveCube
Appendix
Table 13: Task categorization based on functional characteristics.
Representation
Policy
Integration/ VLM Type
Motion- Centric
Time- Sensitive
Short- Horizon
Long- Horizon
Scene- Change
Event- Salient
No Memory ( π0 )
11.17
6.67
16.39
28.45
13.41
32.96
Symbolic
SimpleSG
Gemini
14.00
2.00
19.33
27.50
10.33
56.67
QwenVL
7.34
0.44
24.96
29.22
15.33
84.96
GroundSG
Gemini
3.25
3.00
22.00
9.50
7.67
20.33
QwenVL
5.83
0.00
48.22
42.89
17.70
72.08
Perceptual
TokenDrop
Context
32.84
3.11
25.93
28.22
22.74
71.70
Appendix
Table 14: Performance grouped by task functional requirements. Results are averaged within each category.
Representation
Policy
Integration/ VLM Type
Counting
Permanence
BinFill
PickXTimes
SwingXTimes
StopCube
VideoUnmask
ButtonUnmask
VideoUnmaskSwap
ButtonUnmaskSwap
Symbolic
SimpleSG
Oracle
85.78 ± 2.91
99.78 ± 0.67
100.0 ± 0.00
44.67 ± 22.25
33.11 ± 2.67
22.00 ± 3.06
15.56 ± 2.60
15.56 ± 3.84
GroundSG
Oracle
85.78 ± 1.67
100.00 ± 0.00
100.0 ± 0.00
49.67 ± 20.02
98.78 ± 1.04
95.00 ± 2.83
99.22 ± 1.04
80.22 ± 15.69
Symbolic
SimpleSG
Gemini
46.00 ± 7.14
63.00 ± 7.07
45.00 ± 4.24
2.00 ± 0.00
29.00 ± 1.41
9.00 ± 7.07
14.00 ± 2.83
2.00 ± 2.83
SimpleSG
QwenVL
77.56 ± 2.19
95.33 ± 2.00
5.11 ± 1.45
0.44 ± 0.88
34.22 ± 3.53
19.33 ± 8.31
15.33 ± 2.83
9.56 ± 4.56
GroundSG
Gemini
26.00 ± 8.49
18.00 ± 0.00
4.00 ± 0.00
3.00 ± 1.41
36.00 ± 0.00
14.00 ± 2.83
13.00 ± 1.41
0.00 ± 0.00
Appendix
Table 15: Full results for different models. Red marks the best per section; marks the overall best for non-oracle models.
Memory Tokens
FrameSamp + Modul
TokenDrop + Modul
64
30.42
18.11
128
36.54
32.76
256
42.90
36.36
512
44.51
38.02
1024
45.87
40.17
Appendix
Table 16: Effect of memory token budget on perceptual memory methods. Increasing the memory budget generally improves performance.
Method
Memory
Observation
Language
Success
FrameSamp + Context
1.5 / 8.9
60.2 / 84.9
38.3 / 63.9
30.68
TokenDrop + Context
4.8 / 17.0
45.4 / 72.9
49.2 / 94.8
34.50
TokenDrop + Expert
0.7 / 12.7
47.5 / 71.9
51.8 / 97.8
34.86
FrameSamp + Expert
1.3 / 21.8
46.0 / 68.2
52.7 / 96.4
36.25
Appendix
Table 17: Attention allocation among memory, observation, and language tokens. We report mean / max attention probability mass (%) aggregated across layers.
Method
Scale
Bias
Success
TokenDrop + Modul
5.99±38.67
0.024±14.19
38.04
FrameSamp + Modul
3.74±22.55
0.012±9.45
44.51
Appendix
Table 18: Statistics of memory-conditioned modulation. We report mean ± standard deviation of the predicted scale and bias values.
Figure 9: We design four history-dependent real-world tasks for policy evaluation. The upper dashed box indicates selected frames from previous video history. The bottom large box indicates current observation for execution.
Figure 10: Real Robot Experiment Platform Setup.
Figure 11: Overview of the main assets used in RoboMME.
Difficulty
#Colors
#Total Cubes
#Goal Cubes
Easy
1
4–6
1
Medium
2
8–10
1–2
Hard
3
10–12
2–3
Appendix
Table 19: BinFill Task Configuration.
Figure 12: BinFill Task Example. In this instance, the goal is to place one blue cube into the bin and then press the button to stop.
Difficulty
#Total Cubes
#Repetitions
Easy
1
1–3
Medium
3 (2 distractors)
1–3
Hard
3 (2 distractors)
4–5
Appendix
Table 20: PickXtimes Task Configuration.
Figure 13: PickXtimes Task Example. In this instance, the goal is to pick up the green cube and place it on the target twice, then press the button to stop.
Difficulty
#Total Cubes
#Repetitions
Easy
1
1–3
Medium
3 (2 distractors)
1–2
Hard
3 (2 distractors)
3
Appendix
Table 21: SwingXtimes Task Configuration.
Figure 14: SwingXtimes Task Example. In this instance, the goal is pick up the green cube, first move it to the right-side target, then put down the cube on the left-side target (i.e., swing between targets one time), finally press the button to stop. Spatial directions (e.g., left/right) follow the robot base coordinate frame rather than the front camera frame.
Difficulty
#Passes
Move Speed
Easy
2–3
slow/medium/fast
Medium
3–4
slow/medium/fast
Hard
4–5
slow/medium/fast
Appendix
Table 22: StopCube Task Configuration.
Figure 15: StopCube Task Example. In this instance, the goal is press the button to stop the cube exactly at the target on its second visit.
Difficulty
#Total Containers
#Goal Containers
#Total Cubes
Easy
3
1
3 (red, green, blue)
Medium
5
1
3 (red, green, blue)
Hard
10
2
3 (red, green, blue)
Appendix
Table 23: VideoUnmask Task Configuration.
Figure 16: VideoUnmask Task Example. In this instance, after watching a video in which all cubes are masked, the robot must pick up the container hiding the green cube. Red-bordered frames denote the video-based observation prior to execution.
Difficulty
#Total Containers
#Goal Containers
#Total Cubes
Easy
3
1
3 (red, green, blue)
Medium
5
1
3 (red, green, blue)
Hard
15
2
3 (red, green, blue)
Appendix
Table 24: ButtonUnmask Task Configuration.
Figure 17: ButtonUnmask Task Example. In this instance, the robot first presses the button on the table and then selects the container hiding the green cube. During the button press, all cubes are masked.
Difficulty
#Total Containers
#Swap Times
#Goal Containers
#Total Cubes
Easy
3
1–2
1–2
3 (red, green, blue)
Medium
4
1–2
1
3 (red, green, blue)
Hard
4
2–3
2
3 (red, green, blue)
Appendix
Table 25: VideoUnmaskSwap Task Configuration.
Figure 18: VideoUnmaskSwap Task Example. In this instance, the robot first watches the video, then picks up the container hiding the blue cube, followed by the one hiding the green cube. The video shows that all cubes are being masked by containers, with some containers swapped between locations. Red-bordered frames denote the video-based initial observations provided before execution.
Difficulty
#Total Containers
#Goal Containers
#Swap Times
#Total Cubes
Easy
3
1–2
1–2
3 (red, green, blue)
Medium
4
1
1–2
3 (red, green, blue)
Hard
4
2
2–3
3 (red, green, blue)
Appendix
Table 26: ButtonUnmaskSwap Task Configuration.
Figure 19: ButtonUnmaskSwap Task Example. In this instance, the robot first presses both buttons on the table, then picks up the container hiding the green cube, followed by the one hiding the blue cube. During the button presses, all cubes are masked and their locations are swapped.
Difficulty
#Total Cubes
#Goal Cubes
Easy
3
1
Medium
4
2
Hard
6 (cluster)
3
Appendix
Table 27: PickHighlight Task Configuration.
Figure 20: PickHighlight Task Example. In this instance, the goal is to first press the button, then pick up all cubes highlighted by a white area , finally press the button again to stop. During the button pressing, the green cube was highlighted with a white area on the table, the robot needs to capture this information, and pick up the correct cubes.
Difficulty
#Cubes
#Swap Times
#Repetition
Easy
3 (same color)
1–2
1–3
Medium
3 (same color)
2–3
1–3
Hard
15 (different colors, clutter scene)
0
1–3
Appendix
Table 28: VideoRepick Task Configuration.
Figure 21: VideoRepick Task Example. In this instance, the robot first watches a video, then picks up the same previously picked cube twice, and finally presses the button to stop. Red-bordered frames denote the video-based observations shown prior to execution.
Difficulty
#Cubes
#Targets
Target Swapping
Easy
1
3
No
Medium
3
4
No
Hard
3
4
Yes
Appendix
Table 29: VideoPlaceButton Task Configuration.
Figure 22: VideoPlaceButton Task Example. In this example, the robot first observes a video depicting an interleaved sequence of cube placements and button presses, and then places the cube onto the correct target corresponding to its location prior to the button press. Red-bordered frames indicate the video-based observations provided before execution.
Difficulty
#Cubes
#Targets
Target Swapping
Easy
1
4
No
Medium
3
4
No
Hard
3
4
Yes
Appendix
Table 30: VideoPlaceOrder Task Configuration.
Figure 23: VideoPlaceOrder Task Example. In this instance, the goal is to place the red cube on the third target where it was previously placed in the given video. Red-bordered frames denote the video-based observation prior to execution.
Figure 24: MoveCube Task Example. In this instance, the robot must watch the video carefully, then move the cube to the target in the same manner as demonstrated. Red-bordered frames denote the video-based observation prior to execution.
Figure 25: InsertPeg Task Example. In this instance, the robot must watch the video carefully, then grasp the same peg at the same end and insert it into the same side of the box as demonstrated. Red-bordered frames denote the video-based observation prior to execution.
Difficulty
Grid Size
Pattern Length
Easy
3 × 3
2–4
Medium
4 × 4
3–5
Hard
5 × 5
4–8
Appendix
Table 31: PatternLock Task Configuration
Figure 26: PatternLock Task Example. In this instance, the goal is to watch the video carefully and then use the stick attached to the robot to retrace the same pattern shown in the video. Red-bordered frames denote the video-based observations prior to execution.
Difficulty
Path Length
Backtracking
Easy
2–3
No
Medium
4–5
No
Hard
4–7
Yes
Appendix
Table 32: RouteStick Task Configuration
Figure 27: RouteStick Task Example. In this instance, the robot watches the video and uses the stick attached to the end-effector to navigate around obstacles on the table, reproducing the demonstrated path.
Robotic manipulation often requires acting on information that is no longer visible, yet Vision-Language-Action policies are usually evaluated when the current observation largely determines the next action. Existing robotic memory benchmarks expose this gap, but they still rely mainly on final task success and therefore conflate forgetting with manipulation failure. We present MEMOBench, a benchmark for process level memory evaluation in robotic manipulation. MEMOBench includes 30 history dependent tasks, 1{,}500 expert demonstrations, and 4{,}200 executable checkpoint instances from 84 templates. Each checkpoint pairs coarse to fine language with a simulator predicate and labels one memory operation: Storage, Update, or Compression. These annotations define Memory Storage Rate, Memory Update Rate, and Memory Compression Rate, which measure memory fidelity alongside task success. Across standard and memory augmented VLA policies, the strongest memory module baseline reaches only 31.9% average success rate, and high storage often coexists with weak update and compression. Checkpoint language also supervises semantic, contrastive, and framewise memory alignment objectives, yielding modest gains across different memory operations. MEMOBench provides a diagnostic evaluation suite and training supervision for memory grounded robotic policies. The project page is available at https://github.com/Collab-Gen/MEMOBench.
Vision-language-action policies often see only one or a few recent frames, which makes it difficult to evaluate how they use information that disappears during a task. We introduce MIKASA-Robo-VLA, a benchmark of 90 language-conditioned manipulation tasks. All but 10 hide the cue an action depends on. Those 10 are reactive controls. MIKASA-Robo, the suite it rebuilds, has 32 tasks and uses language only in a representative VLA subset. Here every task provides an instruction, while memory-dependent tasks hide a task-relevant cue and reactive controls keep it available. For 70 tasks, environment phase timings specify an information gap, and for 28 of them the gap exceeds the 16-frame window of the widest fixed-context VLA we survey. The gap counts only the interval the cue is provably absent, not the full duration a policy must retain it, so every memory-dependent task still requires memory by construction, including the ones whose measured gap is short. We release 22,500 oracle trajectories across 10 memory types in RLDS and LeRobotDataset v3. A reference π0.5 baseline with current images and proprioception, but no observation history or explicit memory module, is fine-tuned on 14 tasks and achieves 0.211 ± 0.044 mean task success. Its lower success on the evaluated Long-split tasks is confounded by open-loop chunking and the memory types represented in that subset. Project page: https://mikasarobo.github.io/
Egor Cherepanov, Nikita Kachaev, Aleksandr I. Panov +1
AXXX, Moscow, Russia · MIRIAI, Moscow, Russia · HSE University, Moscow, Russia
Memory is essential for long-horizon, partially observed robotic manipulation: a robot must remember which object was placed in a drawer, whose cup it moved, or how many action cycles have elapsed. Recent vision-language-action (VLA) models embed memory directly inside the policy, but benchmarks show no single in-policy mechanism covers all spatio-temporal dimensions, trailing oracle methods by a wide margin. We argue that memory type dictates where memory should reside: short-term perceptual memory (repetition, timing, retracing) belongs inside the policy, while long-term object memory (persistent spatial state, containment, event history) belongs outside as an explicit, readable record. We present Ledger, a harness that realizes this split over a single fine-tuned π0.5 policy by pairing an in-policy frame-sampling memory with an external spatio-temporal object memory, the ledger, built from a SAM3 tracker and a VLM captioner of the demonstration and read by an LLM planner that decides at step boundaries. On RoboMME, Ledger reaches the highest four-suite average among the evaluated methods, 64.3% (vs. 45.9% for the strongest prior method under identical evaluation), leading object reference (60.7% vs. 40.3%) and object permanence (86.7% vs. 56.2%) using a single set of weights. Choosing the memory source at runtime, from the instruction and the record, removes the need for a task-level router.