General-purpose multimodal agents can write robot-control programs, but repeated exploration and model-mediated action selection can make execution slow. We study how external body knowledge, successful experience, and executable skills improve an XLeRobot controlled by GPT-6-Astra in a simulated and a physical elevator-button task. In 30 fixed-start simulation trials, complete robot geometry and camera information reduce mean completion time by 57.4% relative to a baseline with only the common control interface and no prior experience; images with synchronized action and state records reduce it by 68.6% without additional body assets. In nine paired comparisons (18 trials) at starts displaced by 10-100 cm, experience recorded at the original start reduces mean time by 58-63% relative to no experience, demonstrating generalization to the tested new starting positions. During experience experiments, GPT-6-Astra spontaneously generates a short visual-feedback program. Researcher-refactored versions reduce mean local-task time by 29-31% in 27 simulation trials. Finally, 12 real-robot trials using operator-confirmed button contact demonstrate sim2real reuse: at a shared nominal start, simulation XML assets and simulation experience reduce mean time by 53.0% and 49.9%, respectively; real experience also transfers to two new starts. These results suggest a practical way to build general-purpose manipulation experiments around GPT-6-Astra: supply machine-readable body descriptions and synchronized demonstrations, and turn useful agent-generated feedback routines into reusable skills, while the agent adapts actions from current images. We release all task prompts, trial-level experimental data, and acquired skill implementations at https://github.com/hesd10/astra-robot-sim2real.
Figures & tables
Figure 1: Faster elevator-button manipulation with GPT-6-Astra in simulation and on a real robot. Robot geometry and camera information (blue), or synchronized images, actions, and robot states from a successful simulated attempt (orange), reduce task time relative to the same agent given neither resource. The sim2real arrow denotes reuse of simulation knowledge and experience on a calibrated physical robot. Bottom: researchers derived reusable skills from an agent-generated routine to accelerate the final approach in simulation. STEP performs one short movement and returns an image to the agent; LOOP repeats movements and image checks internally until the button’s visual activation signal is detected or an execution limit is reached. Their baseline has no supplied skill. Bars show mean task time, including observation, model interaction, and execution (three trials per condition above; nine below). Percentages denote reductions relative to each baseline; bar pairs start at zero and are scaled independently. Images illustrate the task.
Figure 2: Three external resources support execution at different levels: body knowledge informs geometric calculations, recorded experience provides action references, and reusable skills execute local motion and feedback. GPT-6-Astra interprets current observations and selects actions.
Figure 3: The MuJoCo elevator lobby used in the simulation experiments. (a) The cutaway view shows the robot at the original starting state, facing three elevators. (b) The enlarged panel shows the upward call button beside the middle elevator. The agent observes the scene through its head and wrist cameras.
Body information
I0
Common control API, actuator names, units, usable limits, and current feedback.
I1
I0 plus a qualitative description of the robot’s structure.
I2
I1 plus mechanical XML, meshes, and joint mapping.
I3
I2 plus camera geometry and calibration.
Historical experience
E0
No historical experience.
Table 1: Condition definitions and evaluation design. Simulation labels combine a body-information level and an experience level, such as I0E3. The real-robot labels distinguish no prior material, simulation assets, simulation experience, and real experience.
Figure 4: Fixed-start simulation results: (a) body-information levels without experience; (b) experience representations at the minimum body-information level. Bars show means; dots show all three trials per condition. Geometry and synchronized action experience have the clearest gains. The full table also includes I3 experience conditions.
Figure 5: Reusing a working right-arm pose in simulation. (a) Initial configuration. (b) The selected experience source at 481 s, after it has found the arm pose used for pressing. (c) A subsequent I0E3 trial at 141 s, after adopting the same key joint targets. The shoulder-pitch, elbow, and wrist-pitch targets are (q2,q3,q4)=(1.0,1.2,−0.84) rad in (b,c). Views are rendered from recorded simulation configurations using the same camera angle relative to the robot. The agent continues to adjust the base from current observations.
Figure 6: A complete simulated approach using recorded experience, completed in 310 s. (a) Initial position. (b) Advancing with an intermediate arm configuration. (c) Extending into the working pose. (d) Approaching the middle elevator’s panel. (e) Final alignment before activation. (f) The gripper activates the upward call button; its indicator turns red. Timestamps use the task clock. Frames are rendered from recorded states with a fixed external camera focused on the robot, middle elevator, and panel; insets in (e,f) enlarge the button interaction.
Figure 7: Experience reuse from new starting positions at (a) 10 cm, (b) 50 cm, and (c) 100 cm offset radii. Each line compares a trial without experience (E0) with a trial using experience recorded at the original start (E3), from the same displaced position. Colors identify points within each offset radius. Experience reduces task time at every tested point.
Figure 8: Near-button task times with no supplied routine (FREE), a single-step skill (STEP), or an internally repeating skill (LOOP). S1–S3 progressively increase the initial distance to the button; each state–condition combination has three trials. STEP and LOOP reduce the overall mean, with most of the gain already achieved by STEP.
Condition
Material
Trial 1
Trial 2
Trial 3
Mean
A
None
808.9
431.5
980.1
740.1
B
Simulation XML
284.3
408.5
351.2
348.0
C
Simulation E3
366.5
381.3
364.3
370.7
D
Real E3
250.3
269.7
360.0
293.4a
Table 2: Real-task times in seconds, using operator-confirmed gripper-tip contact as success. A/B/C each have three repeats at a shared nominal start.
Figure 9: Real-robot results. (a) Baseline, simulation assets, and simulation experience at a shared nominal start, with three trials per condition. (b) Real experience reused once at the source start and once at each of two new starts. D2/D3 test transfer across placements.
Figure 10: External views of real-experience transfer in D3: (a) initial view, (b) approach, and (c) near-button interaction. Labels indicate time within the demonstration video.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Condition
n
Mean (s)
SD (s)
Motion requests
I0E0
3
1024.9
118.7
66.7
I1E0
3
1160.4
136.3
67.3
I2E0
3
579.8
92.5
48.0
I3E0
3
437.1
47.5
33.3
I0E1
3
878.1
105.2
55.0
I0E2
3
850.7
220.7
57.7
Appendix
Table 3: Complete fixed-start results. SD is sample standard deviation. Every condition succeeds 3/3; time uses the simulation service task clock.
State
FREE
STEP
LOOP
S1
126.9±21.6
93.8±22.4
84.8±4.3
S2
209.2±26.2
136.1±5.8
116.9±21.9
S3
171.9±3.4
130.8±12.0
151.1±56.5
Appendix
Table 4: Local-task mean and sample SD in seconds, three trials per cell.
S3 repetition
STEP (s)
LOOP (s)
LOOP calls
Motion pulses
1
117.1
103.1
3
5
2
139.1
213.3
7
9
3
136.4
137.0
5
13
Appendix
Table 5: S3 diagnostic breakdown. Calls count separate invocations by Astra; pulses count movements inside those invocations. The second repetition drives most of LOOP’s larger S3 mean.
Operation
Arguments
Returned information or effect
schema
None
Camera roles, joint bounds, and motion limits.
state
None
Timestamped joint positions, targets, and motion status.
observe
None
Fresh head, left-wrist, and right-wrist images with capture timestamps.
move
Joint targets; duration
Coordinated bounded joint motion; returns before execution finishes.
base
vx,vy,ωz ; duration
Timed base velocity; actual motion is checked through subsequent observations.
stop
None
Stop base motion and cancel the joint segment, retaining applied joint targets.
Appendix
Table 6: Shared low-level operations. Joint angles use radians, base translation speed m/s, yaw speed rad/s, and durations seconds.
GPT-6-Astra demonstrates a level of competence in robotic manipulation and navigation that challenges the conventional paradigm of building specialized models for individual tasks and embodiments. In this report, we ask not only how capable GPT-6-Astra is, but what its capabilities imply for the field. First, we probe the limits of standalone GPT-6-Astra. We conduct a large-scale evaluation across 12 representative benchmarks spanning tabletop manipulation, humanoids, mobile platforms, and dexterous hands. Under a unified protocol that allows in-context learning (ICL) from training-set demonstrations and substantially increases the number of execution steps, GPT-6-Astra achieves unprecedented performance (Fig. 1). Future research may focus on improving 1) the generality of ICL 2) cost and execution time, 3) physical-state verification, 4) recovery from irreversible failures, and 5) long-horizon preservation of progress. Second, we study the agentic harness for embodied control. We identify that close-loop coordination between a high-level "System 2" reasoner (i.e., GPT-6-Astra) and a low-level "System 1" controller (e.g., π0.5) is costly but effective. Third, we examine GPT-6-Astra on real-world deployment. Our experiments on 8 robot configurations reveal substantial gaps: 1) robot demonstrations required for ICL are expensive to collect, and 2) it is implausible to trade longer execution for higher success rates with physical constraints. Fourth, we envision the next generation of embodied benchmarks. We develop a prototype that 1) derives tasks from real-world needs, 2) provides no in-domain demonstrations, and 3) judges agents on a world clock that keeps running while they deliberate. Taken together, our results suggest that GPT-6-Astra has largely crossed the threshold of embodied competence, but not yet that of practical embodied intelligence.
GPT-6 Astra exhibits a remarkable ability to generate numerical robot actions, extending its role beyond high-level planning. To assess Astra's capabilities as general-purpose embodied policies, we conduct comprehensive evaluations across six domains, examining direct control, cooperation with learned policies, and feedback-driven adaptation. In gripper manipulation, Astra can correct task targets and prepare contact conditions for subsequent policy execution; hybrid control with π0.5 achieves 48% success on the evaluated RoboDojo subset. In dexterous manipulation, hybrid control achieves 50% success in ten experience-guided DexJoCo trials, while direct in-hand control struggles to coordinate finger contacts. In mobile manipulation, hybrid control reaches 38.7% success on the evaluated RoboCasa365. In navigation, Astra leads our local comparisons, reaching 92% success on RxR instruction following and 82% on HM3D object search, although search incurs substantial detours. In locomotion, dense motion-reference generation remains unreliable: none of five sequential attempts on a single obstacle course reaches the goal, despite improvements in stability and forward progress. In humanoid loco-manipulation, Astra exceeds baseline methods on 13 of 30 HumanoidBench tasks with pretrained whole-body controllers. These findings reveal a gap between useful task decisions and reliable physical control. Inference latency further constrains practical control: across 50 RoboDojo instances per condition, policy-assisted and direct control consume 624.8 million and 1.132 billion tokens. A 30-second locomotion run requires 250 model calls averaging 39.86 seconds each, with physics paused during inference.
Transferring robotic skills from simulation to reality requires task knowledge that remains usable across differences in perception, dynamics, and embodiment. We introduce Skill2Real, an agentic policy framework that learns executable skills through a shared application programming interface (API). A Proposer-Verifier-Governor (PVG) loop uses privileged simulation evidence to diagnose outcomes and validate updates, while keeping learned skills grounded in public observations and API semantics. The Cerebellum first acquires local manipulation skills; the Brain then learns task-level composition with the Cerebellum frozen. Both memories transfer to the real robot without task-policy fine-tuning or skill-memory updates. As GPT-5.6 Sol learns skills on LIBERO-90, evaluating each frozen checkpoint with GPT-6 Astra raises LIBERO-Pro Long success from 2.0% to 56.3%, without training on Pro Long. Independent Robosuite training reaches 85.1% and 89.4% mean success with Sol and Opus 5 across seven tasks, respectively. Frozen Sol-trained LIBERO-90 skills achieve 78.75% mean completion across four real-world manipulation tasks with Astra. Removing the Verifier or Governor during LIBERO-90 training lowers final Pro Long success by 17.3 and 13.3 percentage points, respectively. These results support learning and transferring a hierarchy of executable skills through a common robot interface.
Xincheng He, Siyu Ma, Chang Yu +5
University of California, Los Angeles · Shanghai Jiao Tong University · University of Utah +1