cs.ROOct 12, 2026
SaveCan a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation?
Organizations: Harvard University
Abstract
We use a frozen frontier vision-language model, GPT-6-Astra, directly as the robot policy on RoboMME, a benchmark of 16 memory-dependent manipulation tasks spanning temporal, spatial, object, and procedural memory. The model keeps the whole episode in context and outputs every end-effector waypoint and gripper command, with no RoboMME-specific training. On the 800 test episodes it succeeds in 92.1%, above the previously reported human score of 90.5%, which was obtained with an oracle skill interface, and far ahead of the best leaderboard entry (60.8%) and of a hierarchical system that pairs the same model with a RoboMME-trained policy (79.1%). It matches or exceeds the human score on 13 of the 16 tasks. Its 63 failures concentrate in two tasks and are mostly premature completions and mistimed or unintended button presses. These results blur the usual division between long-horizon planning, memory and end-effector control, and raise the question of how much of this stack a single general-purpose model can take over in embodied systems. Project website: https://embodied-astra.github.io. Code, prompts and all episode logs: https://github.com/jackbjed/embodied-astra-robomme.
Figures & tables
Figure 1: Success on the 800 RoboMME test episodes, as reported by each source. GPT-6-Astra keeps the whole episode in context and commands every end-effector waypoint, with no RoboMME-specific training (single run). Light grey bars use RoboMME’s oracle skill interface, in which a privileged planner executes chosen skills. Details in Tables 1 and 3 .
| System | Decision model | Action interface | RoboMME-trained parameters | Status | Runs | Success |
|---|---|---|---|---|---|---|
| Human players [ 9 ] | 18 participants | skill selection, oracle planner | – | reference | 1 | 90.5 |
| PonderPounce-9B [ 8 ] | Qwen3.5-9B context engine + -initialized controller | joint angles | yes (1,587 demos) | leaderboard | 3 | 60.8 |
| 9 data | same | joint angles | yes (14,400 episodes) | paper only | 3 | 75.5 |
| Three-tier [ 5 ] | GPT-6-Astra planner + RoboMME VLA + learned monitor | joint angles (VLA) | yes (VLA, monitor) | project report | 1 | 79.1 |
| Ours | GPT-6-Astra, full episode context | Cartesian waypoints, RoboMME motion planner | none | this paper (not yet validated) | 1 | 92.1 |
Table 1: Results on the 800 RoboMME test episodes (%), as reported by each source; leaderboard snapshot of 2026-09-29, accessed 2026-10-10 [ 19 ] . Only the human reference uses the oracle skill interface; all other rows issue low-level or waypoint actions. Our result has not been validated by the benchmark maintainers. RoboMME’s leaderboard also lists a baseline that executes ground-truth subgoals (84.1%).
Figure 2: The model receives the goal, the task video and subsampled frames of everything that happens, and keeps its own previous answers in context. In the waypoint setting (leaderboard) it commands every end-effector pose; in RoboMME’s oracle-planner setting (diagnostic only) it selects a skill that an oracle executes.
| Suite | Task | Human † | PP-9B ‡ | GPT-6-Astra |
| Counting | BinFill | 96 | 50.7 | 98 |
| PickXtimes | 100 | 93.3 | 100 | |
| SwingXtimes | 80 | 72.0 | 98 | |
| StopCube | 78 | 82.7 | 54 | |
| Permanence | VideoUnmask | 90 | 94.7 | 94 |
| VideoUnmaskSwap | 92 | 25.3 | 96 |
Table 2: Success rate (%) on the 50 test episodes of each task. † Previously reported human-reference score [ 9 ] , obtained with the oracle skill interface rather than waypoints. ‡ PonderPounce-9B leaderboard entry, joint-angle control, mean of 3 runs [ 8 ] . GPT-6-Astra is a single run. Bold marks scores at or above the human reference.
Figure 3: Success rate (%) on the RoboMME test split by suite, with the memory type each suite targets (left; mean of its four tasks), and by task (right). GPT-6-Astra is a single run; the human score was obtained with the oracle skill interface; PonderPounce-9B is the best leaderboard entry (joint-angle control, mean of 3 runs). Values as in Table 2 .
Figure 4: All 63 failures of the run (737 of 800 episodes succeeded), by observed class and by the task and behaviour in which they occur. Widths are proportional to episode counts. Classes are assigned from episode records (Appendix A ); they describe behaviour, not internal causes.
Figure 5: Recorded front-camera frames of failed rollouts from the three most frequent groups; is simulated time and quotes are the model’s own descriptions. Frames with a plain border were sent to the model as observations; frames with a dashed red border are execution or terminal frames from the episode recording that the model never received, because the episode ended before its next request. Top: MoveCube (stick-push variant), premature completion while still holding the stick. Middle: StopCube, unintended button contact while positioning. Bottom: VideoUnmask, a set-aside waypoint beyond the arm’s reach that the executor runs until the step limit.
| Setting | GPT-6-Astra | GPT-4o | Gemini-2.5-Pro | Qwen3-VL-32B | Human |
|---|---|---|---|---|---|
| RoboMME’s evaluation [ 9 ] | – | 30.6 | 47.9 | 27.9 | 90.5 |
| Our harness | 77.8 | 30.8 | – | – | – |
| + D, G | 92.3 | 31.9 | – | – | – |
| + D, G, F+T (diagnostic) | 94.6 | 32.4 | – | – | – |
Table 3: Oracle-planner setting: average success rate (%) over the 800 test episodes. Corrections apply to the tasks of Table 4 only; all other tasks are unchanged.
| GPT-6-Astra | GPT-4o | ||||||
| Variant | Tasks | base | variant | gained / lost | base | variant | gained / lost |
| D | 5 video-memory tasks | 159 | 239 | 83 / 3 | 32 | 37 | 19 / 14 |
| G | PickHighlight | 14 | 50 | 36 / 0 | 20 | 24 | 7 / 3 |
| D and G (300 episodes) | 173 | 289 | 119 / 3 | 52 | 61 | 26 / 17 | |
| McNemar exact | |||||||
| F+T | StopCube (with D, diagnostic) | 24 | 43 | 23 / 4 | 21 | 25 | 12 / 8 |
Table 4: Paired control in the oracle-planner setting. Successful episodes (of 50 per task) without and with each correction, on the same test episodes. “Gained / lost” counts episodes that switch outcome.
Figure 6: Paired outcomes of the 300 test episodes affected by D and G, for GPT-6-Astra and GPT-4o, drawn to the same scale. Unlike two aggregate bars, the flows show both recovered failures (green) and new failures (red). F+T is not included.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Suite | Task | Human | Gemini † | GPT-4o | GPT-6-Astra | +corr. | corr. |
| Counting | BinFill | 96 | 78 | 58 | 98 | 98 | – |
| PickXtimes | 100 | 80 | 56 | 96 | 96 | – | |
| SwingXtimes | 80 | 58 | 18 | 100 | 100 | – | |
| StopCube | 78 | 32 | 42 | 48 | 86 | F+T ∗ | |
| Permanence | VideoUnmask | 90 | 86 | 34 | 100 | 100 | – |
| VideoUnmaskSwap | 92 | 18 | 20 | 78 | 100 | D |
Table 5: Oracle-planner success rate (%) on the 50 test episodes of each task. Human and Gemini-2.5-Pro ( † ) are as reported [ 9 ] ; GPT-4o is our run in the same harness (RoboMME reports 30.6%). “+corr.” applies the correction in the last column to that task only. ∗ F+T is an oracle-interface diagnostic; without it the average is 92.3%. Bold marks GPT-6-Astra results at or above the human score.