Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning
Authors: Youwei Zhou, Bohan Hou, Yankai Fu, Yaoxu Lyu, Yifan Ye, Shilin Shan, Chen Guo, Jiaming Liu, Ran He, Caifeng Shan, Shanghang Zhang, Jianfei Yang, Zhihe Lu
Organizations: Nanjing University · Nanyang Technological University · Peking University · Independent Researcher · Institute of Automation of the Chinese Academy of Sciences
Recent advances in visuospatial understanding and agentic reasoning enable direct robotic manipulation with general-purpose multimodal models, but their embodied capabilities remain poorly understood. This paper systematically evaluate GPT-6 Astra for closed-loop robotic manipulation using a lightweight agent execution framework. We first probe three fundamental embodied capabilities-depth perception, embodied in-context learning, and tactile understanding-and demonstrate that the model can exploit depth information, acquire simple manipulation strategies from contextual demonstrations, and use tactile feedback to correct its actions. We then evaluate GPT-6 Astra across five established manipulation benchmarks along six capability dimensions. Despite competitive performance on several benchmarks-26.92% success on RoboDojo and 76.0% on RoboTwin 2.0-our analysis reveals a pronounced gap between semantic reasoning and physical execution, particularly in consequence-aware action planning, precision, and complex motion coordination. To further examine these limitations, we introduce LIBERO-GeoMotion, a new benchmark of geometry-constrained tasks that require precise pose alignment, coordinated translation and rotation, and anticipation of physical action consequences. GPT-6 Astra achieves only 11.1% zero-shot success, exposing the difficulty of translating spatial understanding into physically feasible motion. Real-world experiments corroborate these findings: zero-shot execution is feasible for simple tasks, yet persistent deficits remain in motion feasibility, precise control, and physical safety, while substantial inference latency further hinders practical deployment. Overall, our results characterize the emerging embodied capabilities of general-purpose multimodal models and delineate the key challenges toward safe, efficient, and reliable robotic manipulation.
Figures & tables
Figure 1: Overview of the systematic evaluation of GPT-6 Astra for robotic manipulation. (a) illustrates the closed-loop control framework. (b) represents the analysis of depth perception, embodied in-context learning, and tactile understanding. (c) summarizes benchmark performance, embodied capability profile and real-world validation. (d) highlights limitations in rotation orientation control through benchmark LIBERO-GeoMotion.
Figure 2: GPT-GEAR: a tool-mediated framework for robotic manipulation driven by GPT-6 Astra. The model generates commands from multimodal inputs and execution feedback. Observe retrieves current or historical observations, Reference supplies optional demonstration context, and Act executes model-generated commands. Status verifies task completion, while Finish terminates the interaction.
Observation
Success Rate (%)
Action Rounds
Total Tokens
w/ RGB
47.5
34.61
1,046,755
w/ RGB + Camera parameters
60.0
26.32
986,066
w/ RGB + Camera parameters + Explicit depth estimation
62.5
28.85
1,119,366
w/ RGB + Depth map
67.5
24.23
1,101,271
w/ RGB + Back-projection tool
60.0
27.00
1,377,672
Table 1: Depth perception ablation with representative tasks from benchmark LIBERO-Plus. The Action Rounds and Total Tokens are averaged over all evaluated episodes.
Figure 3: The Visualization of depth perception ablation. The selected task is open the middle drawer of the cabinet from benchmark LIBERO-Plus. Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 4: Depth estimates compared with ground-truth depth. Predictions and ground truth are shown across observation steps for three task examples. MAE denotes mean absolute error, and N is the number of observations shown.
Context Formulation
Success Rate (%)
Action Rounds
Total Tokens
w/o context
64.0
23.00
1,187,814
w/ First and last frames
46.0
21.18
1,448,025
w/ Video sequence frames
64.0
20.72
1,442,084
w/ Textual step description
78.0
21.30
1,052,921
w/ Action trajectory + first and last frames
66.0
20.64
1,218,883
w/ Action trajectory + video sequence frames
82.0
19.92
1,222,021
Table 2: Embodied in-context-learning ablation with representative tasks from dataset LIBERO-Plus. The Action Rounds and Total Tokens are averaged over all evaluated episodes. All the contexts are from expert trajectories for the same task under different scene perturbations.
Figure 5: The Visualization of in-context learning ablation. The shown tasks are pick the akita black bowl in the top layer of the wooden cabinet and place it on the plate and put the yellow and white mug in the microwave and close it from dataset LIBERO-Plus. Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Observation
Success Rate (%)
Action Rounds
Total Tokens
w/ RGB
14.0
10.57
423.90k
w/ Tactile (Tactile-only)
16.0
12.38
569.02k
w/ RGB + Tactile (image)
30.0
10.53
646.06k
w/ RGB + Tactile (text)
50.0
9.44
451.19k
w/ RGB + Tactile (image + text)
48.0
9.75
553.36k
Table 3: Tactile-understanding ablation with representative tasks from UniVTAC benchmark. The Action Rounds and Total Tokens are averaged over all evaluated episodes.
Figure 6: The visualization of tactile understanding ablation. The shown task is put bottle in shelf from dataset UniVTAC. Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 7: The CoT chain of tactile understanding under the RGB + tactile (image + text) condition. ΣF denotes the summed force in (a), while FL and FR denote the left and right pad vertical forces.
Figure 8: The comprehensive comparison of GPT-6 Astra with existing solutions [ 30 , 4 , 51 , 47 , 49 , 64 , 65 , 19 , 8 , 60 , 54 , 59 , 44 , 42 , 28 , 29 , 53 , 27 , 15 , 17 ] . Bars indicate success rate (SR), and diamonds additionally report the process-oriented score on RoboDojo. Higher values indicate better performance. * represents the reevaluated methods with our protocol. GPT-6 Astra is highlighted with red.
Dimension
Operational requirement
Number
Representative tasks
Generalization
Adapt to changes in objects, scene layouts, or observation conditions.
120
LIBERO-Plus: Soup to Basket; Milk to Basket, etc. RoboTwin 2.0: Pick Diverse Bottles; Pick Dual Bottles, etc. RoboDojo: Stack Blocks; Fold Clothes, etc. RoboCasa GR1: Novel Object: Plate to Box; Tray to Pot, etc.
Memory
Retain object identities, initial states, sequences or counts needed by later actions.
6
RoboDojo: Match and Pick from Conveyor; Cover Blocks, etc.
Task Understanding
Ground linguistic references, spatial relations, symbolic goals, and task rules.
57
LIBERO-Plus: Bowl beside Ramekin to Plate, etc. RoboTwin 2.0: Blocks Ranking RGB; Place A Left of B, etc. RoboDojo: General Pickup; Solve Equation, etc. UniVTAC: Grasp Classify.
Long-horizon
Complete a task through an extended sequence of interdependent subgoals.
60
LIBERO-Plus: Bowl to Bottom Drawer and Close, etc. RoboTwin 2.0: Handover Block; Stack Three Bowls, etc. RoboDojo: Organize Table; Fill Egg Holder, etc. RoboCasa GR1: Potato to Microwave and Close, etc.
High-order operation
Coordinate orientation changes, rotation, pouring, or motion around articulated joints.
31
RoboTwin 2.0: Rotate QR Code; Hanging Mug, etc. RoboDojo: Fasten Screws; Insert Key and Turn, etc. UniVTAC: Insert Hole (oblique hole), etc. RoboCasa GR1: Bottle to Cabinet and Close, etc.
Precision
Meet tight alignment, contact, insertion, support, or release tolerances.
74
LIBERO-Plus: Wine Bottle to Rack; Book to Caddy, etc. RoboTwin 2.0: Click Bell; Place Phone on Stand, etc. RoboDojo: Insert Tubes; Plug in Charger, etc. UniVTAC: Insert Tube; Insert HDMI, etc.
Table 4: The statistics of six capability dimensions and assigned tasks across the five benchmarks. Long task names are abbreviated. There are a total of 164 tasks and every task can count toward multiple dimensions. Counts include all tasks assigned to each dimension through a primary or secondary requirement, beyond the displayed examples.
Dataset
Generalization
Memory
Task Understanding
Long-horizon
High-order operation
Precision
LIBERO-Plus
83.50% (334/400)
–
87.43% (306/350)
67.00% (67/100)
60.00% (12/20)
58.33% (35/60)
RoboTwin 2.0
76.00% (380/500)
–
91.67% (55/60)
62.86% (88/140)
58.89% (53/90)
77.50% (124/160)
RoboDojo
19.17% (23/120)
56.67% (34/60)
30.67% (46/150)
17.33% (52/300)
9.17% (11/120)
15.71% (33/210)
RoboCasa GR1
22.22% (80/360)
–
–
9.17% (11/120)
7.50% (6/80)
18.96% (91/480)
UniVTAC
–
–
95.00% (19/20)
–
66.25% (53/80)
69.29% (97/140)
Overall
59.20% (817/1380)
56.67% (34/60)
73.45% (426/580)
33.03% (218/660)
34.62% (135/390)
36.19% (380/1050)
Table 5: Success rates across all 164 tasks by capability dimension, using primary and secondary assignments. Cells show success rate and successful/evaluated episodes. ”–” indicates that the benchmark has no task assigned to that dimension.
Figure 9: The visualization of representative task executions demonstrating generalization . Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 10: The visualization of representative task executions demonstrating task understanding . Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 11: The visualization of representative task executions demonstrating memory . Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 12: The visualization of representative task executions illustrating limitations in long horizon . Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 13: The visualization of representative task executions illustrating limitations in precision . Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 14: The visualization of representative task executions illustrating limitations in high-order operation for gripper-equipped robots and robots equipped with dexterous hands. Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 15: The average execution latency for comparison. Lower is better. Red bars denote GPT-6 Astra, and red annotations report its execution-time ratio relative to the other evaluated solutions.
Figure 16: Overview of the nine task families in LIBERO-GeoMotion. Each panel illustrates a geometry-constrained manipulation task, with green dashed outlines indicating target poses or configurations and arrows highlighting spatial, orientation, or interaction requirements.
Task
(a) Passage
(b) Book slot
(c) Bottle
(d) Plate
(e) Book drawer
(f) Mug
(g) Mango
(h) Package
(i) Two mugs
w/o ICL
0.0
0.0
50.0
0.0
0.0
0.0
50.0
0.0
0.0
w/ ICL
25.0
50.0
75.0
25.0
25.0
75.0
75.0
0.0
100.0
Δ (pp)
+25.0
+50.0
+25.0
+25.0
+25.0
+75.0
+25.0
0.0
+100.0
Table 6: Success rates (%) on LIBERO-GeoMotion. Columns correspond to the nine task families illustrated in Figure 16 . Each family contains four scene variants, and Δ denotes the improvement in percentage points.
Figure 17: Representative failure trajectories in LIBERO-GeoMotion. (a) Mango transport: unintended contact knocks over a nearby vase. (b) Narrow-passage transport: unsuccessful coordination of translation and rotation prevents the elongated bar from passing through the opening. (c) Upright book placement: the book tips over during manipulation, and subsequent recovery attempts fail. (d) Package storage: the package remains outside the drawer, leaving placement and closure incomplete. Red boxes highlight representative failure-related states.
Figure 24
Task
Zero-shot
One-example ICL
Pick-and-place
100% (5/5)
100% (5/5)
Mango grasping
0% (0/5)
60% (3/5)
Block-in-cup insertion
40% (2/5)
80% (4/5)
Pouring into a drawer
40% (2/5)
100% (5/5)
Shelf rearrangement
20% (1/5)
60% (3/5)
Overall
40% (10/25)
80% (20/25)
Table 7: Real-world success rates under zero-shot and one-example ICL conditions. Each task is evaluated over five trials per condition. Counts indicate successful trials divided by total trials.
Figure 18: Representative real-world execution trajectories of GPT-6 Astra. Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
GPT-6-Astra demonstrates a level of competence in robotic manipulation and navigation that challenges the conventional paradigm of building specialized models for individual tasks and embodiments. In this report, we ask not only how capable GPT-6-Astra is, but what its capabilities imply for the field. First, we probe the limits of standalone GPT-6-Astra. We conduct a large-scale evaluation across 12 representative benchmarks spanning tabletop manipulation, humanoids, mobile platforms, and dexterous hands. Under a unified protocol that allows in-context learning (ICL) from training-set demonstrations and substantially increases the number of execution steps, GPT-6-Astra achieves unprecedented performance (Fig. 1). Future research may focus on improving 1) the generality of ICL 2) cost and execution time, 3) physical-state verification, 4) recovery from irreversible failures, and 5) long-horizon preservation of progress. Second, we study the agentic harness for embodied control. We identify that close-loop coordination between a high-level "System 2" reasoner (i.e., GPT-6-Astra) and a low-level "System 1" controller (e.g., π0.5) is costly but effective. Third, we examine GPT-6-Astra on real-world deployment. Our experiments on 8 robot configurations reveal substantial gaps: 1) robot demonstrations required for ICL are expensive to collect, and 2) it is implausible to trade longer execution for higher success rates with physical constraints. Fourth, we envision the next generation of embodied benchmarks. We develop a prototype that 1) derives tasks from real-world needs, 2) provides no in-domain demonstrations, and 3) judges agents on a world clock that keeps running while they deliberate. Taken together, our results suggest that GPT-6-Astra has largely crossed the threshold of embodied competence, but not yet that of practical embodied intelligence.
Recent advances in multimodal models have enabled GPT-6 Astra to perform robotic manipulation through in-context learning. Yet contact-rich control and tactile perception are often regarded as strengths of specialist policies, leaving open whether Astra can directly understand and use tactile observations. We provide, to our knowledge, the first demonstration that Astra directly interprets vision-based tactile images and uses their contact cues for manipulation. Tactile RGB images enter through its existing visual interface, enabling few-shot closed-loop control on nineteen simulated UniVTAC and OpenViTac tasks with 210 sampled scene seeds per setting. With visuo-tactile input, Astra achieves mean success rates of 93.8% and 81.8%, exceeding the strongest reported specialist baselines in our comparison by 22.9 and 13.1 percentage points, respectively. Vision-only performance also leads these baselines, reaching 76.3% and 80.6%. Trajectory analysis reveals active probing, cautious grasping, contact-guided alignment, and recovery, alongside difficulties with tangential loading and corrective execution. Our findings challenge the exclusivity of specialist policies in contact-rich manipulation and show that fine-grained vision can partially compensate for missing touch. They also identify vision-based tactile sensing as an interface for scaling contact understanding with foundation models and reveal signs of benchmark saturation. These results motivate joint visual-tactile pretraining, harder contact-rich evaluations, and physical-robot validation.
General-purpose multimodal models have shown promise as direct robot action policies, yet whether they grasp the physics underlying real-world interactions and can apply that understanding to guide action remains underexplored. We investigate this question using contact-rich manipulation as a representative testbed, examining tactile understanding as a concrete instance of this broader capability. Contact-rich manipulation requires interpreting tactile feedback and adjusting actions after obstruction. Existing learned tactile policies acquire these capabilities through specialized pretraining and per-task optimization, with substantial data and training costs. We present GPT-as-Tactile-Policy, which summarizes expert demonstrations into task guidance and tactile references for contact assessment and corrective action. GPT-6-Astra combines this context with multimodal observations and interaction history to generate control commands and adjust actions through contact feedback. Without additional robotics-specific pretraining or per-task policy fine-tuning, our method achieves 93.8% mean success across five UniVTAC tasks. Overall performance is comparable to state-of-the-art policies, with higher success on selected tasks. On Insert Hole, tactile feedback improves success from 31% to 99%. Execution analysis further demonstrates contact-guided probing and correction, supporting manipulation progress through explicit expert experience and online feedback.
Ziyong Wang, Yupei Lin, Runxuan Huang +3
Sun Yat-sen University · X-Era AI Lab · The Hong Kong Polytechnic University