Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy

Yifei Yao; Yude Zou; Guanbo Jiao; Xing Gao; Ning Gao; Yukai Wang; Jiantong Chen; Zihou Zhu

cs.ROOct 26, 2026

Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy

Authors: Yifei Yao, Yude Zou, Guanbo Jiao, Xing Gao, Ning Gao, Yukai Wang, Jiantong Chen, Zihou Zhu,

Organizations: Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Fudan University · Zhejiang University

Abstract

Frontier multimodal large language models (MLLMs) can control robots without robot-specific training, but how their capabilities compare and interact with post-trained vision-language-action (VLA) and world action model (WAM) policies across manipulation demands remains to be thoroughly evaluated. In this report, we evaluate GPT-6-Astra on EBench as the core of an agent that controls a robot through an agent harness under the same visual and proprioceptive observation protocol as post-trained policies. We study two agent configurations: direct control, in which the agent generates waypoint commands directly, and harness with policy, in which it can also invoke a post-trained VLA or WAM policy for action execution. Under direct control, the agent achieves competitive overall performance relative to post-trained policies. It leads on short-horizon tasks and exhibits the least performance variation across generalization dimensions, whereas it underperforms post-trained policies on long-horizon tasks, tabletop manipulation, and high-precision contact-rich tasks. In the harness with policy, invoking a post-trained VLA or WAM combines the complementary strengths of both components, raising overall task success to the level of the strongest compared post-trained policies while substantially reducing token cost. However, performance remains below the per-task upper bound of the standalone components due to suboptimal policy invocation, fixed-instruction execution without subgoals, and grasp disruption when the agent resumes control. Overall, our study characterizes the capability boundary of a frontier MLLM agent and suggests directions for improving future agent harness design. Project page: https://internrobotics.github.io/EBench-Astra-Tech-Report/

Figures & tables

Figure 1: Overview of the GPT-6-Astra control loop on EBench and executor timelines on a Microwave test scene. The input prompt combines the task instruction with a detailed task description annotated from a single training episode. The timeline bars compare active executors across an episode under direct control and under the harness with $\pi_{0.5}$ , with blue segments executed by the agent and purple segments by $\pi_{0.5}$ , and with keyframes showing right-wrist camera views from that episode at the marked steps.

		Overall		Mobile		Tabletop
Policy	Type	SR (%)	Score	SR (%)	Score	SR (%)	Score
Post-trained policies
OpenWAM- $\alpha$	WAM	55.3	0.701	60.2	0.740	42.1	0.594
InternW0- $\Delta$	WAM	49.2	0.660	55.0	0.700	33.6	0.552
Qwen-RobotManip	VLA	45.6	0.608	43.9	0.576	50.0	0.696
$\pi_{0.5}$	VLA	41.4	0.544	45.9	0.578	29.3	0.452

Table 1: SR and score on the EBench test set, overall and by mobility group, of the eight post-trained policies with per-task results on the EBench leaderboard and of the GPT-6-Astra-based agent under direct control and in the harness with each policy. Best and second-best values in each column are in bold and underlined. MLLM denotes an agent whose motion commands all come from its MLLM core, and Hybrid an agent that also invokes a post-trained policy.

Figure 2: SR of the direct-control agent and of OpenWAM-

\alpha

, InternW0-

\Delta

, Qwen-RobotManip, and

\pi_{0.5}

(a) across mobility, horizon, and precision task groups and (b) across the four generalization dimensions of the EBench test set. The radial axis of each panel ends at its largest SR, rounded up to a multiple of 10%.

Figure 3: The direct-control agent across the 26 EBench test tasks by task group. (a) SR of the direct-control agent, range across the eight post-trained policies, and best post-trained policy, with its rank among the agent and the eight post-trained policies in the column on the right of the panel. (b) Shares of its episodes ending in complete success, partial credit, or a score of zero, averaged equally across tasks within each group and rounded to whole percent so that each bar sums to 100.

Figure 4: Adaptation in two episodes of the direct-control agent, with the agent’s stated reasons below the frames. Orange boxes show enlarged views of the contact and manipulation details. (a) Apple to bowl: after a slip, the agent takes a wider view and carries the apple with the arm alone, reaching a 1.00 score. (b) Coffee beans: after a sweep leaves the beans in place, the agent lowers the edge and tilts the fingertips, reaching a 0.50 score.

Figure 5: Rollouts of the direct-control agent evaluated zero-shot, $\pi_{0.5}$ , and OpenWAM- $\alpha$ on a composite instruction outside the standard EBench tasks: placing a bookmark and then a pen onto the same open book. Keyframes are shown in chronological order with video time.

All tasks (26)
Tasks	Direct control	Policy alone		Harness with
Tasks		$\pi_{0.5}$	InternW0- $\Delta$	$\pi_{0.5}$	InternW0- $\Delta$
SR (%)	46.7	41.4	49.2	53.0	56.0
Score	0.654	0.544	0.660	0.692	0.744
Precision
Low (14)	60.6	49.6	58.7	68.9	68.2

Table 2: Comparison of the GPT-6-Astra-based agent under direct control, each post-trained policy evaluated alone, and the harness with

\pi_{0.5}

or InternW0-

\Delta

across all 510 test episodes of EBench. Best and second-best values in each row are shown in bold and underlined.

Figure 6: SR of the GPT-6-Astra-based agent under direct control, of each policy alone, and of the harness with that policy, for

\pi_{0.5}

in (a, b) and InternW0-

\Delta

in (c, d), across mobility, horizon, and precision task groups in (a, c) and across the four generalization dimensions of the EBench test set in (b, d). Panels in the same column share a radial axis that ends at the largest SR of the column, rounded up to a multiple of 10%.

Figure 7: Per-task SR in percent across the 26 test tasks for the GPT-6-Astra-based agent under direct control, for each post-trained policy alone, and for the harness with

\pi_{0.5}

and with InternW0-

\Delta

, together with the share in percent of physics steps that each post-trained policy executes in its harness configuration. Tasks are grouped by precision and ordered within each group by the mean SR of the two post-trained policies alone. One blue-to-red scale shades SR and step share alike on an absolute 0–100% scale, values are rounded to whole percent, and bold marks the highest nonzero SR of each task. Table C.1 lists the full task names.

Outcome of the 510 episodes
	Direct control	Harness with
		$\pi_{0.5}$	InternW0- $\Delta$
Success	237	269	284
Partial credit	188	162	168
Score of zero	85	79	58
Ending of the unsuccessful episodes

Table 3: Episode outcomes across the 510 test episodes under direct control and in the harness with

\pi_{0.5}

or InternW0-

\Delta

, termination modes of unsuccessful episodes, and policy invocation statistics. A reported failure denotes an early termination in which the agent explicitly reports the task as incomplete. Recovery requests denote policy invocations whose stated reason is coded lexically as asking the policy to recover from a slip, missed grasp, or displaced object, as detailed in Appendix C .

Figure 8: Policy invocation in the two harness configurations. (a) Share of the physics steps executed by the policy in each tenth of an episode, averaged over the successful and over the unsuccessful episodes. (b) Per task, the change in SR from direct control to the harness with policy against the SR difference between the policy alone and direct control, with the identity line. (c) Per task, the share of physics steps executed by the policy against the SR of the policy alone. Each marker in (b) and (c) is one of the 26 tasks, and

r

is the Pearson correlation over tasks.

Figure 9: Token use of the GPT-6-Astra-based agent under direct control and in the harness with each policy on the 510 EBench test episodes. (a) Mean tokens per episode in successful and unsuccessful episodes. (b) Tokens per success on short-horizon and long-horizon tasks, dividing all tokens of each group, including those of its unsuccessful episodes, by its number of successful episodes.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Arguments	Available	Description as shown to the model
execute_eef	waypoints : 1–6 of { left , right (position, quaternion, and two finger targets, or null), base pose or null, steps 1–64, and optional motion_steps }	all	“Execute 1..6 native GenManip absolute EEF waypoints in order. Each arm target has xyz(m), unit quaternion(wxyz) in that arm’s planner-base frame, and two finger joint targets(m): [0.044,0.044] open, [0,0] close. […] The simulator performs IK; targets may remain unreachable, so inspect returned measured poses. No automatic grasp approval or task-stage rules are imposed. Returns final robot state and selected RGB cameras; sequences have no intermediate model observation. […]”
set_gripper	side , opening in [0,1], steps 1–64	all	“Send a normalized gripper opening command: opening=0 fully closed, opening=1 fully open. […] Use this for closing at rest; measured opening may remain nonzero due to contact. Returns selected RGB and measured robot state.”
observe	none	all	“Return selected RGB views and measured robot state from the latest acknowledged simulator step. Does not advance physics. […]”
execute_pi05	steps 1–300, instruction (string or null)	VLA	“Delegate a bounded manipulation segment to the trained EBench pi0.5 policy. It uses the CURRENT RGB images and robot state, predicts absolute joint/gripper targets, and replans every 30 steps. Choose this when learned dexterity may help with grasping, alignment, insertion or other fine manipulation. […] After the requested steps, control automatically returns to you with fresh RGB and state; […] pass instruction=null. […]”
execute_wam	steps 1–300, instruction (string or null)	WAM	“Delegate a bounded manipulation segment to the trained EBench InternW0-delta world-action-model policy. It uses the CURRENT RGB images and robot state together with its own frames from earlier in this episode, predicts absolute joint/gripper targets, and replans every 10 steps. […] After the requested steps, control automatically returns to you with fresh RGB and state; […] pass instruction=null. […]”
query_icl	view (text, frames, or all), frame_indices (keyframe indices)	demo	“The saved ICL demonstration remains available through query_icl throughout this episode. […] Querying does not advance simulation. Ground actions in the current RGB/state.”

Appendix

Table A.1: Agent tools, with descriptions reproduced from the tool declarations (omissions marked […]). All tools except query_icl also take cameras (one to three of overlook , left_wrist , and right_wrist ), and execute_eef , execute_pi05 , and execute_wam a free-text reason . Availability: all denotes every configuration, VLA the harness with

\pi_{0.5}

, WAM the harness with InternW0-

\Delta

, and demo both harness configurations, in which the demonstration of the first turn also remains available on request.

Part	Content as shown to the model
Header	“HISTORICAL DEMONSTRATION – reference from a separate recorded episode, not the current observation or the current episode’s action history.”
Source and instruction	“Historical demonstration of bowl_to_plate, source episode 600. Reference instruction: Take out one of the three bowls and stack on the plate.”
Arm roles	“Right arm; base shifts from the sink toward the plate.”
Observed sequence	“Reach into the sink and grip the rim of the brown bowl. Lift it clear of the other bowls and sink rim, reposition the base toward the plate on the counter, place the bowl over the plate, lower it onto the plate, and open the fingers.”
Observed outcome	“The brown bowl is visibly supported on the white plate after the right fingers open; two patterned bowls remain in the sink.”
Evidence limits	“Final contact is partly occluded by the gripper. No official success label or force measurement is supplied.”

Appendix

Table A.2: Parts of the annotated bowl-to-plate demonstration in the first turn, with text reproduced from the prompt and omissions marked […]. The numeric record of one keyframe is shown with values rounded to three decimals.

Figure A.1: The ten keyframes of the annotated bowl-to-plate demonstration (training episode 600) with their captions, as they enter the first turn. Each frame shows the video time of the source recording and the camera when it is not the overview camera. The agent does not observe the top camera during execution.

Field	Meaning
executed_waypoints (EEF)	One record per executed waypoint: its running index within the episode, the physics steps executed, and the measured state after it.
executed_waypoints ( $\pi_{0.5}$ )	The controller, the requested and executed physics steps, the number of policy inferences (one per 30 steps), and whether control returned to the agent or the episode ended.
executed_waypoints (InternW0- $\Delta$ )	As for $\pi_{0.5}$ , with one inference per 10 steps, together with the fixed policy prompt and its source, scope, and SHA-256 hash, and the number of inferences in the call whose frame from 32 steps earlier had not been observed and was padded.
state.gripper , state.base	The four finger-joint positions in meters, left arm first (0.044 per finger is fully open), and the planar base pose ( $x$ , $y$ in meters, yaw in radians).
state.ee_pose , timestep	For each arm, the EEF position in meters and its wxyz quaternion in that arm’s planner-base frame, together with the physics steps executed since the start of the episode.
remaining_tool_calls , remaining_physics_steps	Always null, since no local limit applies and the benchmark’s step limit is not disclosed.

Appendix

Table A.3: Fields of a tool response.

Figure B.1: Initial overview frame of one direct-control test episode of each EBench task, by mobility and horizon, with the precision label of each task. Task names as in Table B.2 . The tabletop frames show the upper central 80% of the view.

		Overall		Mobile		Tabletop
Policy	Type	SR (%)	Score	SR (%)	Score	SR (%)	Score
Post-trained policies
PICA-WAM ( Zou et al., 2026 )	WAM	61.4	0.749	67.5	0.783	45.0	0.663
OpenWAM- $\alpha$ ( Wang et al., 2026b )	WAM	55.3	0.701	60.2	0.740	42.1	0.594
Vela ( Li et al., 2026b )	VLA	49.7	0.655	51.7	0.661	44.3	0.641
InternW0- $\Delta$ ( Miao et al., 2026 )	WAM	49.2	0.660	55.0	0.700	33.6	0.552

Appendix

Table B.1: SR and score of all entries of the EBench leaderboard, overall and by mobility (19 mobile and 7 tabletop tasks, each averaged equally over tasks), with the best and second-best values in each column shown in bold and underlined. The results of the post-trained policies are taken from the leaderboard.

Task	Prec.	Mob.	Hor.	OWAM	IW0- $\Delta$	Astra	Qwen	$\pi_{0.5}$	IVLA	$\pi_{0}$	Giga	FWAM
Apple from shelf	Low	Mob.	Short	70.0	15.0	70.0	65.0	75.0	35.0	40.0	55.0	40.0
Apple to fruit bowl	Low	Mob.	Short	95.0	95.0	90.0	50.0	25.0	20.0	10.0	20.0	70.0
Bookmark on book	Low	Mob.	Short	35.0	35.0	90.0	40.0	55.0	45.0	25.0	30.0	30.0
Bowl to plate	Low	Mob.	Short	85.0	80.0	75.0	75.0	85.0	60.0	60.0	65.0	75.0
Gear installation	High	Tab.	Short	40.0	50.0	15.0	50.0	10.0	15.0	30.0	20.0	15.0
Perfume to cosmetics rack	Low	Mob.	Short	75.0	30.0	75.0	55.0	60.0	45.0	25.0	25.0	50.0

Appendix

Table B.2: SR (%) per task on the EBench test set, with the best and second-best value of each task in bold and underlined. OWAM: OpenWAM-

\alpha

, IW0-

\Delta

: InternW0-

\Delta

, IVLA: InternVLA-A1.5, Giga: GigaBrain-0.7, FWAM: Fast-WAM. The direct-control run evaluated Bottle and Shop with a step limit of 3,000 steps instead of 5,000.

Task	Prec.	Mob.	Hor.	OWAM	IW0- $\Delta$	Astra	Qwen	$\pi_{0.5}$	IVLA	$\pi_{0}$	Giga	FWAM
Apple from shelf	Low	Mob.	Short	0.70	0.15	0.70	0.65	0.75	0.35	0.40	0.55	0.40
Apple to fruit bowl	Low	Mob.	Short	0.95	0.95	0.90	0.50	0.25	0.20	0.10	0.20	0.70
Bookmark on book	Low	Mob.	Short	0.35	0.35	0.90	0.40	0.55	0.45	0.25	0.30	0.30
Bowl to plate	Low	Mob.	Short	0.85	0.80	0.75	0.75	0.85	0.60	0.60	0.65	0.75
Gear installation	High	Tab.	Short	0.50	0.60	0.20	0.58	0.13	0.20	0.38	0.28	0.15
Perfume to cosmetics rack	Low	Mob.	Short	0.78	0.30	0.88	0.60	0.68	0.60	0.30	0.33	0.50

Appendix

Table B.3: Score per task on the EBench test set, with the best and second-best value of each task in bold and underlined. Abbreviations and step limits as in Table B.2 .

	Object		Background		Instruction		Mix
Policy	SR (%)	Score	SR (%)	Score	SR (%)	Score	SR (%)	Score
OpenWAM- $\alpha$	52.5	0.657	61.5	0.767	60.0	0.751	44.6	0.604
InternW0- $\Delta$	49.2	0.645	51.5	0.672	52.3	0.694	44.6	0.619
GPT-6-Astra (direct control)	44.2	0.648	50.8	0.692	44.6	0.620	46.2	0.643
Qwen-RobotManip	40.8	0.565	55.4	0.722	51.5	0.655	33.8	0.485
$\pi_{0.5}$	41.7	0.534	42.3	0.575	46.2	0.597	34.6	0.457

Appendix

Table B.4: SR (%) and score of the GPT-6-Astra-based agent under direct control and the eight post-trained policies in the Object, Background, Instruction, and Mix generalization dimensions of the EBench test set, with the best and second-best value of each column in bold and underlined. Rows below the midrule are excluded from this ranking and report the agent in the harness with each policy, as analyzed in Section 6.3 .

Figure B.2: SR of OpenWAM-

\alpha

, InternW0-

\Delta

, and the GPT-6-Astra-based agent (direct control), the three highest-ranked systems in the per-task comparison, by precision, mobility, horizon, and generalization dimension on the 26 EBench test tasks.

	Direct control	$\pi_{0.5}$ in the harness
Chunk transitions measured	109,079	88,573
Median wrist rotation ( ∘ per step)	0.00	0.08
99th percentile ( ∘ per step)	2.81	2.58
99.9th percentile ( ∘ per step)	4.06	3.81
Transitions above 3 ∘ per step (episodes)	643 (147)	196 (94)
Commands turning a wrist by $90^{\circ}$ within a window of 24 steps (episodes)	38 (31)	0 (0)

Appendix

Table B.5: Wrist and joint speed between consecutive chunks of one command under direct control and in the segments of

\pi_{0.5}

in the harness with

\pi_{0.5}

, over all 510 test episodes of each configuration. Parentheses give the number of episodes. A window spans three consecutive chunk transitions of one command.

Task	Direct control	Policy alone		Harness with		Policy step share
Task		$\pi_{0.5}$	InternW0- $\Delta$	$\pi_{0.5}$	InternW0- $\Delta$	$\pi_{0.5}$	InternW0- $\Delta$
Teacup, saucer & teapot	85.0	30.0	45.0	30.0	55.0	51.9	48.9
Peg in hole	20.0	10.0	10.0	5.0	0.0	61.1	63.4
Microwave	73.3	40.0	6.7	60.0	60.0	75.0	71.2
Fruit	40.0	10.0	60.0	30.0	45.0	57.3	64.0
Pen	15.0	0.0	15.0	5.0	0.0	53.5	53.4

Appendix

Table C.1: Per-task SR (%) of GPT-6-Astra under direct control and in the harness with each policy, of each policy alone, and the share (%) of physics steps executed by the policy in each harness configuration, with the best and second-best SR of each task in bold and underlined. The direct-control run evaluated Bottle and Shop with a step limit of 3,000 steps instead of 5,000.

Figure C.1: SR by atomic skill of the GPT-6-Astra-based agent under direct control, in the harness with

\pi_{0.5}

and InternW0-

\Delta

, and of each policy standalone. Each skill averages over the tasks that require it: placing (23 tasks), moving (17), handover (5), sweeping (4), inserting (3), pressing (3), flipping (2), pouring (2), pulling (2), and pushing (1). Grasping, which is required by all 26 tasks, is omitted. The radial axis ends at the largest SR, rounded up to a multiple of 10%.

		Harness with $\pi_{0.5}$				Harness with InternW0- $\Delta$
Direct control	$n$	Succ.	Limit	Claim	Fail	Succ.	Limit	Claim	Fail
Success	237	178	52	3	4	188	33	2	13
Step limit	229	72	140	6	10	76	135	8	9
Ended, claims completion	21	10	3	6	2	11	5	3	2
Ended, reports failure	22	8	6	0	8	9	4	0	9
Other	1	1	0	0	0	0	1	0	0

Appendix

Table C.2: How the 510 episodes under direct control end and how the same scenes end in each harness configuration, by success, step limit, a completion claim, or a failure report of the agent, with the few episodes of other kinds omitted from the columns of the harness configurations.

Decisive incident	With $\pi_{0.5}$	With InternW0- $\Delta$
Object lost or misplaced during a policy segment or when the agent resumes control	16	30
Policy without the object, on the wrong object or subgoal, or undoing a stage	5	1
Agent’s own grasp slips	30	12
Completion claimed while a stage remains unmet	3	2
Other	5	4
Total	59	49

Appendix

Table C.3: Decisive incident in the episodes that the agent solved under direct control and each harness configuration lost.

Decisive incident	$\pi_{0.5}$	InternW0- $\Delta$	Same in both	Main tasks
Agent’s own grasp slips (lift test or transport)	58	42	19	Utensils to holder, Peg in hole, Pen
Object lost or misplaced in a policy segment	37	53	10	Peg in hole, Pen, Shop
Contact stage never completed	19	24	12	Tighten nut, Gear installation, Peg in hole
Policy segment empty-handed or on the wrong object	16	15	4	Shop, Bottle
Step limit without a decisive loss	16	11	4	Bottle
Slip after resuming control without a maintained grasp	14	10	7	Collect coffee beans

Appendix

Table C.4: Decisive incidents of the 164 test episodes that fail with both harness configurations, read call by call, per configuration, with the number of episodes whose two runs show the same incident and the tasks that contribute most.

	Harness with $\pi_{0.5}$		Harness with InternW0- $\Delta$
	Successful	Unsuccessful	Successful	Unsuccessful
Episodes	269	241	284	226
Motion commands per episode	14.7	32.0	15.2	30.4
Transitions per episode	4.2	7.2	3.8	6.8
Transitions per motion command	0.33	0.26	0.28	0.24
Episodes ending with an agent command (%)	42.8	68.0	47.2	71.7

Appendix

Table C.5: Policy invocation in the successful and unsuccessful episodes of the two harness configurations. Transitions are changes of executor between consecutive motion commands, and disruption reports are motion commands whose stated reasons report a slip, a drop, or a displaced object.

Configuration	Tokens (M)	Cached (%)	Uncached (k)	Writes (k)	Output (k)	USD/episode	USD/success
Direct control	3.77	91.5	75.6	244.8	11.2	7.82	16.82
Harness with $\pi_{0.5}$	1.83	96.5	64.2	0	4.6	2.63	4.99
Harness with InternW0- $\Delta$	2.22	97.0	65.7	0	4.9	3.06	5.49

Appendix

Table C.6: Mean token composition and API-equivalent cost per episode of GPT-6-Astra on the 510 EBench test episodes, with input split into cached input, cache writes, and other uncached input, and the cost per success dividing the total cost by the number of successes.

Figure D.1: Two successful direct-control episodes of the GPT-6-Astra-based agent: Bookmark on book and Remote to holder. Overview frames with video time, with insets enlarging the outlined grasp regions.

Figure D.2: Two unsuccessful direct-control episodes of the GPT-6-Astra-based agent, each ending with score 0.50: Peg in hole (right-wrist view) and Bottle (overview). Video time is shown on each frame.

Figure D.3: Three unsuccessful direct-control episodes of the GPT-6-Astra-based agent, with the agent’s stated reasons below the frames. Orange boxes are enlarged in the insets. (a) Peg in hole: the peg tip lands beside the hole, and the peg ends lying across the receiver at the step limit (score 0.50). (b) Tighten nut: the nut falls beside the bolt, is re-acquired, and rests unthreaded on the bolt tip at the step limit (score 0.50). (c) Flip cup & collect cookies: during the second handover the supporting hand withdraws, and the cup falls off the table (score 0.10).

Figure D.4: Three direct-control episodes of the GPT-6-Astra-based agent with safety-relevant behavior, with the agent’s stated reasons below the frames. Orange boxes are enlarged in the insets. (a) Dishwasher: the bowl slips from the left hand while the right hand opens the door, and the agent later searches the floor (score 0.67). (b) Apple to fruit bowl: after the apple falls, the agent searches below the table and behind the robot with targets outside the workspace (score 0.00). (c) Collect coffee beans: the held jar and later the spoon contact the table edge (score 0.07).

Figure D.5: Two unsuccessful direct-control episodes that the GPT-6-Astra-based agent ended itself, with the video time and camera of each frame. (a) Dishwasher: the final message reports both bowls in the rack while the door is open. (b) Collect coffee beans: the held jar is lost at the table edge.

Figure D.6: The GPT-6-Astra-based agent (direct control), $\pi_{0.5}$ , and OpenWAM- $\alpha$ on two tasks: (a) Teacup, saucer & teapot and (b) Glass into glassbox. Keyframes are in chronological order with video time. The outcome and score of each rollout are given at the right, and insets show enlarged views of the outlined manipulation regions. The rollouts differ in object variants and viewpoints.

Figure D.7: The GPT-6-Astra-based agent (direct control) without (zero-shot) and with the demonstration on the same test episodes: (a) Frame placement and (b) Gear installation. Keyframes are in chronological order with video time, and columns do not correspond to matched elapsed time.

Figure E.1: Three unsuccessful episodes in the harness with policy on scenes that the agent solved under direct control: (a) Gear installation with $\pi_{0.5}$ , (b) Detergent with InternW0- $\Delta$ , and (c) Teacup, saucer & teapot with $\pi_{0.5}$ . Each frame shows the video time, the camera when it is a wrist camera, and the executor, with EEF marking the agent’s own commands.

Figure E.2: Keyframes of the GPT-6-Astra-based agent under direct control and in the harness with $\pi_{0.5}$ as a tool, with the video time, the executor of each frame, and the outcome of each episode. (a, b) Glass into glassbox on one test scene, under direct control and with $\pi_{0.5}$ . (c, d) Teacup, saucer & teapot on one test scene, under direct control and with $\pi_{0.5}$ .

Figure E.3: A Shop episode in the harness with InternW0- $\Delta$ , with the video time, the executor of each frame, and the agent’s stated reasons in quotation marks. Orange boxes are enlarged in the insets.

Language Model-Based Control

Language model-based control uses language models to interpret goals, choose actions, and steer AI systems or physical processes.

See all 180 papers

Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation?
Oct 12, 2026Jack B. Jedlicki, Amish Sethi, Heng YangRobotic Manipulation Long-Horizon Robotic Manipulation
Harvard University
Demonstration-Free Robotic Control via LLM Agents
Jan 28, 2026Brian Y. Tsui, Alan Y. Fang, Tiffany J. HwuRobotic Control Language Model-Based Control
All authors are independent researchers.
An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond
Sep 21, 2026Wenbo Zhang, Kaixuan Wang, Yutao Ouyang +9Language-Conditioned Robot Manipulation Generalization in Robotic Manipulation
RoboProbe · RoboDojo · The University of Hong Kong +5

Oct 12, 2026J/K move · Enter open · S save

Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation?

Jack B. Jedlicki, Amish Sethi, Heng Yang

Harvard University

Robotic Manipulation Long-Horizon Robotic Manipulation VLMs for Robotics ICL for Robotics

Open paper AskPDF arXiv

We use a frozen frontier vision-language model, GPT-6-Astra, directly as the robot policy on RoboMME, a benchmark of 16 memory-dependent manipulation tasks spanning temporal, spatial, object, and procedural memory. The model keeps the whole episode in context and outputs every end-effector waypoint and gripper command, with no RoboMME-specific training. On the 800 test episodes it succeeds in 92.1%, above the previously reported human score of 90.5%, which was obtained with an oracle skill interface, and far ahead of the best leaderboard entry (60.8%) and of a hierarchical system that pairs the same model with a RoboMME-trained policy (79.1%). It matches or exceeds the human score on 13 of the 16 tasks. Its 63 failures concentrate in two tasks and are mostly premature completions and mistimed or unintended button presses. These results blur the usual division between long-horizon planning, memory and end-effector control, and raise the question of how much of this stack a single general-purpose model can take over in embodied systems. Project website: https://embodied-astra.github.io. Code, prompts and all episode logs: https://github.com/jackbjed/embodied-astra-robomme.

Abstract

Language Model-Based Control

Explore similar work

Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation?

Demonstration-Free Robotic Control via LLM Agents

An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond