Frontier multimodal large language models (MLLMs) can control robots without robot-specific training, but how their capabilities compare and interact with post-trained vision-language-action (VLA) and world action model (WAM) policies across manipulation demands remains to be thoroughly evaluated. In this report, we evaluate GPT-6-Astra on EBench as the core of an agent that controls a robot through an agent harness under the same visual and proprioceptive observation protocol as post-trained policies. We study two agent configurations: direct control, in which the agent generates waypoint commands directly, and harness with policy, in which it can also invoke a post-trained VLA or WAM policy for action execution. Under direct control, the agent achieves competitive overall performance relative to post-trained policies. It leads on short-horizon tasks and exhibits the least performance variation across generalization dimensions, whereas it underperforms post-trained policies on long-horizon tasks, tabletop manipulation, and high-precision contact-rich tasks. In the harness with policy, invoking a post-trained VLA or WAM combines the complementary strengths of both components, raising overall task success to the level of the strongest compared post-trained policies while substantially reducing token cost. However, performance remains below the per-task upper bound of the standalone components due to suboptimal policy invocation, fixed-instruction execution without subgoals, and grasp disruption when the agent resumes control. Overall, our study characterizes the capability boundary of a frontier MLLM agent and suggests directions for improving future agent harness design. Project page: https://internrobotics.github.io/EBench-Astra-Tech-Report/
Figures & tables
Figure 1: Overview of the GPT-6-Astra control loop on EBench and executor timelines on a Microwave test scene. The input prompt combines the task instruction with a detailed task description annotated from a single training episode. The timeline bars compare active executors across an episode under direct control and under the harness with π0.5 , with blue segments executed by the agent and purple segments by π0.5 , and with keyframes showing right-wrist camera views from that episode at the marked steps.
Overall
Mobile
Tabletop
Policy
Type
SR (%)
Score
SR (%)
Score
SR (%)
Score
Post-trained policies
OpenWAM- α
WAM
55.3
0.701
60.2
0.740
42.1
0.594
InternW0- Δ
WAM
49.2
0.660
55.0
0.700
33.6
0.552
Qwen-RobotManip
VLA
45.6
0.608
43.9
0.576
50.0
0.696
π0.5
VLA
41.4
0.544
45.9
0.578
29.3
0.452
Table 1: SR and score on the EBench test set, overall and by mobility group, of the eight post-trained policies with per-task results on the EBench leaderboard and of the GPT-6-Astra-based agent under direct control and in the harness with each policy. Best and second-best values in each column are in bold and underlined. MLLM denotes an agent whose motion commands all come from its MLLM core, and Hybrid an agent that also invokes a post-trained policy.
Figure 2: SR of the direct-control agent and of OpenWAM- α , InternW0- Δ , Qwen-RobotManip, and π0.5 (a) across mobility, horizon, and precision task groups and (b) across the four generalization dimensions of the EBench test set. The radial axis of each panel ends at its largest SR, rounded up to a multiple of 10%.
Figure 3: The direct-control agent across the 26 EBench test tasks by task group. (a) SR of the direct-control agent, range across the eight post-trained policies, and best post-trained policy, with its rank among the agent and the eight post-trained policies in the column on the right of the panel. (b) Shares of its episodes ending in complete success, partial credit, or a score of zero, averaged equally across tasks within each group and rounded to whole percent so that each bar sums to 100.
Figure 4: Adaptation in two episodes of the direct-control agent, with the agent’s stated reasons below the frames. Orange boxes show enlarged views of the contact and manipulation details. (a) Apple to bowl: after a slip, the agent takes a wider view and carries the apple with the arm alone, reaching a 1.00 score. (b) Coffee beans: after a sweep leaves the beans in place, the agent lowers the edge and tilts the fingertips, reaching a 0.50 score.
Figure 5: Rollouts of the direct-control agent evaluated zero-shot, π0.5 , and OpenWAM- α on a composite instruction outside the standard EBench tasks: placing a bookmark and then a pen onto the same open book. Keyframes are shown in chronological order with video time.
Tasks
Direct control
Policy alone
Harness with
π0.5
InternW0- Δ
π0.5
InternW0- Δ
All tasks (26)
SR (%)
46.7
41.4
49.2
53.0
56.0
Score
0.654
0.544
0.660
0.692
0.744
Precision
Low (14)
60.6
49.6
58.7
68.9
68.2
Table 2: Comparison of the GPT-6-Astra-based agent under direct control, each post-trained policy evaluated alone, and the harness with π0.5 or InternW0- Δ across all 510 test episodes of EBench. Best and second-best values in each row are shown in bold and underlined.
Figure 6: SR of the GPT-6-Astra-based agent under direct control, of each policy alone, and of the harness with that policy, for π0.5 in (a, b) and InternW0- Δ in (c, d), across mobility, horizon, and precision task groups in (a, c) and across the four generalization dimensions of the EBench test set in (b, d). Panels in the same column share a radial axis that ends at the largest SR of the column, rounded up to a multiple of 10%.
Figure 7: Per-task SR in percent across the 26 test tasks for the GPT-6-Astra-based agent under direct control, for each post-trained policy alone, and for the harness with π0.5 and with InternW0- Δ , together with the share in percent of physics steps that each post-trained policy executes in its harness configuration. Tasks are grouped by precision and ordered within each group by the mean SR of the two post-trained policies alone. One blue-to-red scale shades SR and step share alike on an absolute 0–100% scale, values are rounded to whole percent, and bold marks the highest nonzero SR of each task. Table C.1 lists the full task names.
Direct control
Harness with
π0.5
InternW0- Δ
Outcome of the 510 episodes
Success
237
269
284
Partial credit
188
162
168
Score of zero
85
79
58
Ending of the unsuccessful episodes
Table 3: Episode outcomes across the 510 test episodes under direct control and in the harness with π0.5 or InternW0- Δ , termination modes of unsuccessful episodes, and policy invocation statistics. A reported failure denotes an early termination in which the agent explicitly reports the task as incomplete. Recovery requests denote policy invocations whose stated reason is coded lexically as asking the policy to recover from a slip, missed grasp, or displaced object, as detailed in Appendix C .
Figure 8: Policy invocation in the two harness configurations. (a) Share of the physics steps executed by the policy in each tenth of an episode, averaged over the successful and over the unsuccessful episodes. (b) Per task, the change in SR from direct control to the harness with policy against the SR difference between the policy alone and direct control, with the identity line. (c) Per task, the share of physics steps executed by the policy against the SR of the policy alone. Each marker in (b) and (c) is one of the 26 tasks, and r is the Pearson correlation over tasks.
Figure 9: Token use of the GPT-6-Astra-based agent under direct control and in the harness with each policy on the 510 EBench test episodes. (a) Mean tokens per episode in successful and unsuccessful episodes. (b) Tokens per success on short-horizon and long-horizon tasks, dividing all tokens of each group, including those of its unsuccessful episodes, by its number of successful episodes.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Tool
Arguments
Available
Description as shown to the model
execute_eef
waypoints : 1–6 of { left , right (position, quaternion, and two finger targets, or null), base pose or null, steps 1–64, and optional motion_steps }
all
“Execute 1..6 native GenManip absolute EEF waypoints in order. Each arm target has xyz(m), unit quaternion(wxyz) in that arm’s planner-base frame, and two finger joint targets(m): [0.044,0.044] open, [0,0] close. […] The simulator performs IK; targets may remain unreachable, so inspect returned measured poses. No automatic grasp approval or task-stage rules are imposed. Returns final robot state and selected RGB cameras; sequences have no intermediate model observation. […]”
set_gripper
side , opening in [0,1], steps 1–64
all
“Send a normalized gripper opening command: opening=0 fully closed, opening=1 fully open. […] Use this for closing at rest; measured opening may remain nonzero due to contact. Returns selected RGB and measured robot state.”
observe
none
all
“Return selected RGB views and measured robot state from the latest acknowledged simulator step. Does not advance physics. […]”
execute_pi05
steps 1–300, instruction (string or null)
VLA
“Delegate a bounded manipulation segment to the trained EBench pi0.5 policy. It uses the CURRENT RGB images and robot state, predicts absolute joint/gripper targets, and replans every 30 steps. Choose this when learned dexterity may help with grasping, alignment, insertion or other fine manipulation. […] After the requested steps, control automatically returns to you with fresh RGB and state; […] pass instruction=null. […]”
execute_wam
steps 1–300, instruction (string or null)
WAM
“Delegate a bounded manipulation segment to the trained EBench InternW0-delta world-action-model policy. It uses the CURRENT RGB images and robot state together with its own frames from earlier in this episode, predicts absolute joint/gripper targets, and replans every 10 steps. […] After the requested steps, control automatically returns to you with fresh RGB and state; […] pass instruction=null. […]”
query_icl
view (text, frames, or all), frame_indices (keyframe indices)
demo
“The saved ICL demonstration remains available through query_icl throughout this episode. […] Querying does not advance simulation. Ground actions in the current RGB/state.”
Appendix
Table A.1: Agent tools, with descriptions reproduced from the tool declarations (omissions marked […]). All tools except query_icl also take cameras (one to three of overlook , left_wrist , and right_wrist ), and execute_eef , execute_pi05 , and execute_wam a free-text reason . Availability: all denotes every configuration, VLA the harness with π0.5 , WAM the harness with InternW0- Δ , and demo both harness configurations, in which the demonstration of the first turn also remains available on request.
Part
Content as shown to the model
Header
“HISTORICAL DEMONSTRATION – reference from a separate recorded episode, not the current observation or the current episode’s action history.”
Source and instruction
“Historical demonstration of bowl_to_plate, source episode 600. Reference instruction: Take out one of the three bowls and stack on the plate.”
Arm roles
“Right arm; base shifts from the sink toward the plate.”
Observed sequence
“Reach into the sink and grip the rim of the brown bowl. Lift it clear of the other bowls and sink rim, reposition the base toward the plate on the counter, place the bowl over the plate, lower it onto the plate, and open the fingers.”
Observed outcome
“The brown bowl is visibly supported on the white plate after the right fingers open; two patterned bowls remain in the sink.”
Evidence limits
“Final contact is partly occluded by the gripper. No official success label or force measurement is supplied.”
Appendix
Table A.2: Parts of the annotated bowl-to-plate demonstration in the first turn, with text reproduced from the prompt and omissions marked […]. The numeric record of one keyframe is shown with values rounded to three decimals.
Figure A.1: The ten keyframes of the annotated bowl-to-plate demonstration (training episode 600) with their captions, as they enter the first turn. Each frame shows the video time of the source recording and the camera when it is not the overview camera. The agent does not observe the top camera during execution.
Field
Meaning
executed_waypoints (EEF)
One record per executed waypoint: its running index within the episode, the physics steps executed, and the measured state after it.
executed_waypoints ( π0.5 )
The controller, the requested and executed physics steps, the number of policy inferences (one per 30 steps), and whether control returned to the agent or the episode ended.
executed_waypoints (InternW0- Δ )
As for π0.5 , with one inference per 10 steps, together with the fixed policy prompt and its source, scope, and SHA-256 hash, and the number of inferences in the call whose frame from 32 steps earlier had not been observed and was padded.
state.gripper , state.base
The four finger-joint positions in meters, left arm first (0.044 per finger is fully open), and the planar base pose ( x , y in meters, yaw in radians).
state.ee_pose , timestep
For each arm, the EEF position in meters and its wxyz quaternion in that arm’s planner-base frame, together with the physics steps executed since the start of the episode.
remaining_tool_calls , remaining_physics_steps
Always null, since no local limit applies and the benchmark’s step limit is not disclosed.
Appendix
Table A.3: Fields of a tool response.
Figure B.1: Initial overview frame of one direct-control test episode of each EBench task, by mobility and horizon, with the precision label of each task. Task names as in Table B.2 . The tabletop frames show the upper central 80% of the view.
Overall
Mobile
Tabletop
Policy
Type
SR (%)
Score
SR (%)
Score
SR (%)
Score
Post-trained policies
PICA-WAM ( Zou et al., 2026 )
WAM
61.4
0.749
67.5
0.783
45.0
0.663
OpenWAM- α ( Wang et al., 2026b )
WAM
55.3
0.701
60.2
0.740
42.1
0.594
Vela ( Li et al., 2026b )
VLA
49.7
0.655
51.7
0.661
44.3
0.641
InternW0- Δ ( Miao et al., 2026 )
WAM
49.2
0.660
55.0
0.700
33.6
0.552
Appendix
Table B.1: SR and score of all entries of the EBench leaderboard, overall and by mobility (19 mobile and 7 tabletop tasks, each averaged equally over tasks), with the best and second-best values in each column shown in bold and underlined. The results of the post-trained policies are taken from the leaderboard.
Task
Prec.
Mob.
Hor.
OWAM
IW0- Δ
Astra
Qwen
π0.5
IVLA
π0
Giga
FWAM
Apple from shelf
Low
Mob.
Short
70.0
15.0
70.0
65.0
75.0
35.0
40.0
55.0
40.0
Apple to fruit bowl
Low
Mob.
Short
95.0
95.0
90.0
50.0
25.0
20.0
10.0
20.0
70.0
Bookmark on book
Low
Mob.
Short
35.0
35.0
90.0
40.0
55.0
45.0
25.0
30.0
30.0
Bowl to plate
Low
Mob.
Short
85.0
80.0
75.0
75.0
85.0
60.0
60.0
65.0
75.0
Gear installation
High
Tab.
Short
40.0
50.0
15.0
50.0
10.0
15.0
30.0
20.0
15.0
Perfume to cosmetics rack
Low
Mob.
Short
75.0
30.0
75.0
55.0
60.0
45.0
25.0
25.0
50.0
Appendix
Table B.2: SR (%) per task on the EBench test set, with the best and second-best value of each task in bold and underlined. OWAM: OpenWAM- α , IW0- Δ : InternW0- Δ , IVLA: InternVLA-A1.5, Giga: GigaBrain-0.7, FWAM: Fast-WAM. The direct-control run evaluated Bottle and Shop with a step limit of 3,000 steps instead of 5,000.
Task
Prec.
Mob.
Hor.
OWAM
IW0- Δ
Astra
Qwen
π0.5
IVLA
π0
Giga
FWAM
Apple from shelf
Low
Mob.
Short
0.70
0.15
0.70
0.65
0.75
0.35
0.40
0.55
0.40
Apple to fruit bowl
Low
Mob.
Short
0.95
0.95
0.90
0.50
0.25
0.20
0.10
0.20
0.70
Bookmark on book
Low
Mob.
Short
0.35
0.35
0.90
0.40
0.55
0.45
0.25
0.30
0.30
Bowl to plate
Low
Mob.
Short
0.85
0.80
0.75
0.75
0.85
0.60
0.60
0.65
0.75
Gear installation
High
Tab.
Short
0.50
0.60
0.20
0.58
0.13
0.20
0.38
0.28
0.15
Perfume to cosmetics rack
Low
Mob.
Short
0.78
0.30
0.88
0.60
0.68
0.60
0.30
0.33
0.50
Appendix
Table B.3: Score per task on the EBench test set, with the best and second-best value of each task in bold and underlined. Abbreviations and step limits as in Table B.2 .
Object
Background
Instruction
Mix
Policy
SR (%)
Score
SR (%)
Score
SR (%)
Score
SR (%)
Score
OpenWAM- α
52.5
0.657
61.5
0.767
60.0
0.751
44.6
0.604
InternW0- Δ
49.2
0.645
51.5
0.672
52.3
0.694
44.6
0.619
GPT-6-Astra (direct control)
44.2
0.648
50.8
0.692
44.6
0.620
46.2
0.643
Qwen-RobotManip
40.8
0.565
55.4
0.722
51.5
0.655
33.8
0.485
π0.5
41.7
0.534
42.3
0.575
46.2
0.597
34.6
0.457
Appendix
Table B.4: SR (%) and score of the GPT-6-Astra-based agent under direct control and the eight post-trained policies in the Object, Background, Instruction, and Mix generalization dimensions of the EBench test set, with the best and second-best value of each column in bold and underlined. Rows below the midrule are excluded from this ranking and report the agent in the harness with each policy, as analyzed in Section 6.3 .
Figure B.2: SR of OpenWAM- α , InternW0- Δ , and the GPT-6-Astra-based agent (direct control), the three highest-ranked systems in the per-task comparison, by precision, mobility, horizon, and generalization dimension on the 26 EBench test tasks.
Direct control
π0.5 in the harness
Chunk transitions measured
109,079
88,573
Median wrist rotation ( ∘ per step)
0.00
0.08
99th percentile ( ∘ per step)
2.81
2.58
99.9th percentile ( ∘ per step)
4.06
3.81
Transitions above 3 ∘ per step (episodes)
643 (147)
196 (94)
Commands turning a wrist by 90∘ within a window of 24 steps (episodes)
38 (31)
0 (0)
Appendix
Table B.5: Wrist and joint speed between consecutive chunks of one command under direct control and in the segments of π0.5 in the harness with π0.5 , over all 510 test episodes of each configuration. Parentheses give the number of episodes. A window spans three consecutive chunk transitions of one command.
Task
Direct control
Policy alone
Harness with
Policy step share
π0.5
InternW0- Δ
π0.5
InternW0- Δ
π0.5
InternW0- Δ
Teacup, saucer & teapot
85.0
30.0
45.0
30.0
55.0
51.9
48.9
Peg in hole
20.0
10.0
10.0
5.0
0.0
61.1
63.4
Microwave
73.3
40.0
6.7
60.0
60.0
75.0
71.2
Fruit
40.0
10.0
60.0
30.0
45.0
57.3
64.0
Pen
15.0
0.0
15.0
5.0
0.0
53.5
53.4
Appendix
Table C.1: Per-task SR (%) of GPT-6-Astra under direct control and in the harness with each policy, of each policy alone, and the share (%) of physics steps executed by the policy in each harness configuration, with the best and second-best SR of each task in bold and underlined. The direct-control run evaluated Bottle and Shop with a step limit of 3,000 steps instead of 5,000.
Figure C.1: SR by atomic skill of the GPT-6-Astra-based agent under direct control, in the harness with π0.5 and InternW0- Δ , and of each policy standalone. Each skill averages over the tasks that require it: placing (23 tasks), moving (17), handover (5), sweeping (4), inserting (3), pressing (3), flipping (2), pouring (2), pulling (2), and pushing (1). Grasping, which is required by all 26 tasks, is omitted. The radial axis ends at the largest SR, rounded up to a multiple of 10%.
Harness with π0.5
Harness with InternW0- Δ
Direct control
n
Succ.
Limit
Claim
Fail
Succ.
Limit
Claim
Fail
Success
237
178
52
3
4
188
33
2
13
Step limit
229
72
140
6
10
76
135
8
9
Ended, claims completion
21
10
3
6
2
11
5
3
2
Ended, reports failure
22
8
6
0
8
9
4
0
9
Other
1
1
0
0
0
0
1
0
0
Appendix
Table C.2: How the 510 episodes under direct control end and how the same scenes end in each harness configuration, by success, step limit, a completion claim, or a failure report of the agent, with the few episodes of other kinds omitted from the columns of the harness configurations.
Decisive incident
With π0.5
With InternW0- Δ
Object lost or misplaced during a policy segment or when the agent resumes control
16
30
Policy without the object, on the wrong object or subgoal, or undoing a stage
5
1
Agent’s own grasp slips
30
12
Completion claimed while a stage remains unmet
3
2
Other
5
4
Total
59
49
Appendix
Table C.3: Decisive incident in the episodes that the agent solved under direct control and each harness configuration lost.
Decisive incident
π0.5
InternW0- Δ
Same in both
Main tasks
Agent’s own grasp slips (lift test or transport)
58
42
19
Utensils to holder, Peg in hole, Pen
Object lost or misplaced in a policy segment
37
53
10
Peg in hole, Pen, Shop
Contact stage never completed
19
24
12
Tighten nut, Gear installation, Peg in hole
Policy segment empty-handed or on the wrong object
16
15
4
Shop, Bottle
Step limit without a decisive loss
16
11
4
Bottle
Slip after resuming control without a maintained grasp
14
10
7
Collect coffee beans
Appendix
Table C.4: Decisive incidents of the 164 test episodes that fail with both harness configurations, read call by call, per configuration, with the number of episodes whose two runs show the same incident and the tasks that contribute most.
Harness with π0.5
Harness with InternW0- Δ
Successful
Unsuccessful
Successful
Unsuccessful
Episodes
269
241
284
226
Motion commands per episode
14.7
32.0
15.2
30.4
Transitions per episode
4.2
7.2
3.8
6.8
Transitions per motion command
0.33
0.26
0.28
0.24
Episodes ending with an agent command (%)
42.8
68.0
47.2
71.7
Appendix
Table C.5: Policy invocation in the successful and unsuccessful episodes of the two harness configurations. Transitions are changes of executor between consecutive motion commands, and disruption reports are motion commands whose stated reasons report a slip, a drop, or a displaced object.
Configuration
Tokens (M)
Cached (%)
Uncached (k)
Writes (k)
Output (k)
USD/episode
USD/success
Direct control
3.77
91.5
75.6
244.8
11.2
7.82
16.82
Harness with π0.5
1.83
96.5
64.2
0
4.6
2.63
4.99
Harness with InternW0- Δ
2.22
97.0
65.7
0
4.9
3.06
5.49
Appendix
Table C.6: Mean token composition and API-equivalent cost per episode of GPT-6-Astra on the 510 EBench test episodes, with input split into cached input, cache writes, and other uncached input, and the cost per success dividing the total cost by the number of successes.
Figure D.1: Two successful direct-control episodes of the GPT-6-Astra-based agent: Bookmark on book and Remote to holder. Overview frames with video time, with insets enlarging the outlined grasp regions.
Figure D.2: Two unsuccessful direct-control episodes of the GPT-6-Astra-based agent, each ending with score 0.50: Peg in hole (right-wrist view) and Bottle (overview). Video time is shown on each frame.
Figure D.3: Three unsuccessful direct-control episodes of the GPT-6-Astra-based agent, with the agent’s stated reasons below the frames. Orange boxes are enlarged in the insets. (a) Peg in hole: the peg tip lands beside the hole, and the peg ends lying across the receiver at the step limit (score 0.50). (b) Tighten nut: the nut falls beside the bolt, is re-acquired, and rests unthreaded on the bolt tip at the step limit (score 0.50). (c) Flip cup & collect cookies: during the second handover the supporting hand withdraws, and the cup falls off the table (score 0.10).
Figure D.4: Three direct-control episodes of the GPT-6-Astra-based agent with safety-relevant behavior, with the agent’s stated reasons below the frames. Orange boxes are enlarged in the insets. (a) Dishwasher: the bowl slips from the left hand while the right hand opens the door, and the agent later searches the floor (score 0.67). (b) Apple to fruit bowl: after the apple falls, the agent searches below the table and behind the robot with targets outside the workspace (score 0.00). (c) Collect coffee beans: the held jar and later the spoon contact the table edge (score 0.07).
Figure D.5: Two unsuccessful direct-control episodes that the GPT-6-Astra-based agent ended itself, with the video time and camera of each frame. (a) Dishwasher: the final message reports both bowls in the rack while the door is open. (b) Collect coffee beans: the held jar is lost at the table edge.
Figure D.6: The GPT-6-Astra-based agent (direct control), π0.5 , and OpenWAM- α on two tasks: (a) Teacup, saucer & teapot and (b) Glass into glassbox. Keyframes are in chronological order with video time. The outcome and score of each rollout are given at the right, and insets show enlarged views of the outlined manipulation regions. The rollouts differ in object variants and viewpoints.
Figure D.7: The GPT-6-Astra-based agent (direct control) without (zero-shot) and with the demonstration on the same test episodes: (a) Frame placement and (b) Gear installation. Keyframes are in chronological order with video time, and columns do not correspond to matched elapsed time.
Figure E.1: Three unsuccessful episodes in the harness with policy on scenes that the agent solved under direct control: (a) Gear installation with π0.5 , (b) Detergent with InternW0- Δ , and (c) Teacup, saucer & teapot with π0.5 . Each frame shows the video time, the camera when it is a wrist camera, and the executor, with EEF marking the agent’s own commands.
Figure E.2: Keyframes of the GPT-6-Astra-based agent under direct control and in the harness with π0.5 as a tool, with the video time, the executor of each frame, and the outcome of each episode. (a, b) Glass into glassbox on one test scene, under direct control and with π0.5 . (c, d) Teacup, saucer & teapot on one test scene, under direct control and with π0.5 .
Figure E.3: A Shop episode in the harness with InternW0- Δ , with the video time, the executor of each frame, and the agent’s stated reasons in quotation marks. Orange boxes are enlarged in the insets.
We use a frozen frontier vision-language model, GPT-6-Astra, directly as the robot policy on RoboMME, a benchmark of 16 memory-dependent manipulation tasks spanning temporal, spatial, object, and procedural memory. The model keeps the whole episode in context and outputs every end-effector waypoint and gripper command, with no RoboMME-specific training. On the 800 test episodes it succeeds in 92.1%, above the previously reported human score of 90.5%, which was obtained with an oracle skill interface, and far ahead of the best leaderboard entry (60.8%) and of a hierarchical system that pairs the same model with a RoboMME-trained policy (79.1%). It matches or exceeds the human score on 13 of the 16 tasks. Its 63 failures concentrate in two tasks and are mostly premature completions and mistimed or unintended button presses. These results blur the usual division between long-horizon planning, memory and end-effector control, and raise the question of how much of this stack a single general-purpose model can take over in embodied systems. Project website: https://embodied-astra.github.io. Code, prompts and all episode logs: https://github.com/jackbjed/embodied-astra-robomme.
Robotic manipulation has increasingly adopted vision-language-action (VLA) models, which achieve strong performance but typically require task-specific demonstrations and fine-tuning, and often generalize poorly under domain shift. We investigate whether general-purpose large language model (LLM) agent frameworks, originally developed for software engineering, can serve as an alternative control paradigm for embodied manipulation. We introduce FAEA (Frontier Agent as Embodied Agent), which applies an LLM agent framework directly to embodied manipulation without modification. Using the same iterative reasoning that enables software agents to debug code, FAEA enables embodied agents to reason through manipulation strategies. We evaluate an unmodified frontier agent, Claude Agent SDK, across the LIBERO, ManiSkill3, and MetaWorld benchmarks. With privileged environment state access, FAEA achieves success rates of 84.9%, 85.7%, and 96%, respectively. This level of task success approaches that of VLA models trained with less than 100 demonstrations per task, without requiring demonstrations or fine-tuning. With one round of human feedback as an optional optimization, performance increases to 88.2% on LIBERO. This demonstration-free capability has immediate practical value: FAEA can autonomously explore novel scenarios in simulation and generate successful trajectories for training data augmentation in embodied learning. Our results indicate that general-purpose agents are sufficient for a class of manipulation tasks dominated by deliberative, task-level planning. This opens a path for robotics systems to leverage actively maintained agent infrastructure and benefit directly from ongoing advances in frontier models. Code is available at https://github.com/robiemusketeer/faea-sim
Embodied AI systems are often organized into System 1 and System 2. System 1 is typically a pretrained policy that generates actions at high frequency, whereas System 2 is often instantiated as a vision-enabled language model for high-level planning. We ask whether a large language model (LLM) can act as the policy for robot manipulation without task-specific finetuning. We call this setting LLM as policy. We evaluate three LLMs on all 42 RoboDojo tasks and compare their scores with 40 public policies. Astra and GPT-5.5 use the official 50-episode-per-task protocol; DeepSeek-Flash uses 10 episodes per task. GPT-6 Astra achieves 22.48% average success rate and 28.97 Score over 2,100 trials, ranking above every public entry. Yet GPT-5.5 and DeepSeek-Flash reach only 0.88% and 1.92% average success rate with the same post-processing. We find that Astra exhibits a sharply polarized capability profile. It generalizes well to tasks that require semantic understanding but not high-precision control. In contrast, it performs poorly on tasks that require precision, dynamic control, or complex bimanual coordination. In-context experiments show no aggregate benefit from one-shot demonstrations, while selected interaction traces show within-episode corrections under perturbations. Overall, the evaluated LLMs vary substantially in manipulation performance. Astra stands out and provides initial evidence for the potential of a general-purpose manipulation model, although reliable precision and dynamic control remain limitations in the evaluated setting.
Wenbo Zhang, Kaixuan Wang, Yutao Ouyang +9
RoboProbe · RoboDojo · The University of Hong Kong +5