Robotic manipulation often requires remembering information that is no longer visible, such as an object's location before occlusion or the order of a demonstrated action sequence. A large vision-language planner can read this history, but calling it before every action chunk is slow and costly. We present a three-tier system for the RoboMME benchmark. GPT-6 Astra (System 2) selects a grounded subtask from stored visual history, a π0.5 vision-language-action policy (System 1) executes it, and a fine-tuned Qwen3-VL-4B monitor (System 1.5) checks after each action chunk whether the subtask is finished. The instruction is kept until the monitor reports completion or a task-specific event occurs, while the policy keeps acting on new observations. On RoboMME, the system succeeds in 79.13% of episodes with an average of 3.63 Astra calls and 80.84 seconds per episode. This is 34.62 points above the best published non-oracle baseline and 4.95 points below GroundSG+Oracle, which gives the low-level policy ground-truth subgoals.
Figures & tables
Figure 1: Planning, action, and monitoring at different frequencies. (a) Astra uses task history to choose a subtask instruction lk . The VLA executes actions at in 16-step chunks; the environment returns observations ot , and a local monitor checks progress after each chunk. The instruction is kept until an event requires a new high-level decision (Section 2.4 ). (b) Overall success on RoboMME compared with two published baselines from Table 1 .
Figure 2: Instruction reuse between planning events. Three frames from a successful recorded episode. A negative check at step 64 keeps lk . Completion at step 128 triggers Astra’s second call and the next instruction lk+1 ; the positive check shown refers to lk . At step 160, a negative check continues lk+1 . Details are in Appendix A .
Method
Counting
Permanence
Reference
Imitation
Overall
π0.5 [ 1 ]
28.78
17.00
17.16
8.78
17.93
FrameSamp+Modul [ 3 ]
65.22
25.11
36.33
51.39
44.51
SAM2Act+ [ 4 ]
35.33
26.00
16.83
7.33
21.37
MemER [ 8 ]
48.83
53.17
38.00
29.50
42.38
GroundSG+QwenVL [ 3 ]
38.00
39.34
31.56
21.89
32.70
GroundSG+Gemini [ 3 ]
12.75
15.75
11.25
6.50
11.56
Table 1: Success rates (%) on RoboMME. Baselines are published RoboMME results [ 3 ] . GroundSG+Oracle uses ground-truth subgoals. Bold marks the best non-oracle result.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Planning, action, and monitoring during execution. BinFill, hard test episode 023. The first pickup instruction is kept at steps 0 and 64. Completion at step 128 triggers the placement instruction. A negative check at step 160 continues that instruction; completion at step 192 triggers the next pickup. Green badges mark completion events, red badges mark continued execution, and purple marks initialization. At a completion event, the displayed instruction is the next instruction issued by Astra; the monitor decision refers to the preceding subtask.
Figure 4: From demonstration history to grounded action. VideoUnmaskSwap, hard test episode 035. The upper row shows selected demonstration frames: visible objects, covering, and the later scene after container motion. Astra uses the demonstration history to select the container in the current execution scene. The lower row shows the grounded pickup instruction, continued execution under a negative monitor decision, and completion at step 112 followed by the next instruction. Demonstration labels are video frame indices; execution labels are environment steps.
Suite
Task
Success
Fail
Timeout
Rate (%)
Counting
BinFill
41
6
3
82.00
StopCube
9
41
0
18.00
PickXtimes
48
1
1
96.00
SwingXtimes
41
9
0
82.00
Permanence
ButtonUnmask
50
0
0
100.00
VideoUnmask
46
1
3
92.00
Appendix
Table 2: Per-task outcomes on the RoboMME test set (50 episodes per task).
We use a frozen frontier vision-language model, GPT-6-Astra, directly as the robot policy on RoboMME, a benchmark of 16 memory-dependent manipulation tasks spanning temporal, spatial, object, and procedural memory. The model keeps the whole episode in context and outputs every end-effector waypoint and gripper command, with no RoboMME-specific training. On the 800 test episodes it succeeds in 92.1%, above the previously reported human score of 90.5%, which was obtained with an oracle skill interface, and far ahead of the best leaderboard entry (60.8%) and of a hierarchical system that pairs the same model with a RoboMME-trained policy (79.1%). It matches or exceeds the human score on 13 of the 16 tasks. Its 63 failures concentrate in two tasks and are mostly premature completions and mistimed or unintended button presses. These results blur the usual division between long-horizon planning, memory and end-effector control, and raise the question of how much of this stack a single general-purpose model can take over in embodied systems. Project website: https://embodied-astra.github.io. Code, prompts and all episode logs: https://github.com/jackbjed/embodied-astra-robomme.
Embodied AI systems are often organized into System 1 and System 2. System 1 is typically a pretrained policy that generates actions at high frequency, whereas System 2 is often instantiated as a vision-enabled language model for high-level planning. We ask whether a large language model (LLM) can act as the policy for robot manipulation without task-specific finetuning. We call this setting LLM as policy. We evaluate three LLMs on all 42 RoboDojo tasks and compare their scores with 40 public policies. Astra and GPT-5.5 use the official 50-episode-per-task protocol; DeepSeek-Flash uses 10 episodes per task. GPT-6 Astra achieves 22.48% average success rate and 28.97 Score over 2,100 trials, ranking above every public entry. Yet GPT-5.5 and DeepSeek-Flash reach only 0.88% and 1.92% average success rate with the same post-processing. We find that Astra exhibits a sharply polarized capability profile. It generalizes well to tasks that require semantic understanding but not high-precision control. In contrast, it performs poorly on tasks that require precision, dynamic control, or complex bimanual coordination. In-context experiments show no aggregate benefit from one-shot demonstrations, while selected interaction traces show within-episode corrections under perturbations. Overall, the evaluated LLMs vary substantially in manipulation performance. Astra stands out and provides initial evidence for the potential of a general-purpose manipulation model, although reliable precision and dynamic control remain limitations in the evaluated setting.
Wenbo Zhang, Kaixuan Wang, Yutao Ouyang +9
RoboProbe · RoboDojo · The University of Hong Kong +5
Vision-language model (VLM) agents can control robots through visual feedback and action primitives, but repeated LLM calls and redundant observations incur substantial token overhead. We introduce PyRUA-Lean, an interactive code-execution framework in which the agent composes robot primitives, including vision-language-action (VLA) policies, into Python cells that check and retry locally and return only the images and state they request. Across 700 simulated task instances from LIBERO-PRO, RoboTwin 2.0, and RoboCasa365, it raises success from 63.1% to 71.7% over tool calling with the same GPT-6 Astra planner and primitives, using 49% fewer LLM calls and 65% fewer input tokens on jointly solved instances. On RoboDojo, without any VLA policy or privileged information, it reaches 51.6% average success rate (Score 58.2), 19.4 points above the top entry of the public leaderboard. Based on the PyRUA-Lean framework, we train PyRUA-Lean Coder v0.1-4B, an open 4B model, with GRPO on 45 LIBERO-PRO instances; it improves from 18.8% to 44.1% on unseen task instances, surpassing the Qwen3.5-27B base model.
Ruiyang Si, Jianxin Bi, Shunyu Yang +12
Peking University · National University of Singapore · NVIDIA +1