cs.ROOct 20, 2026
SaveLarge Multimodal Models for Direct Robot Control: GPT-6 as a Beginning
Organizations: Nanjing University · Nanyang Technological University · Peking University · Independent Researcher ·
Abstract
Recent advances in visuospatial understanding and agentic reasoning enable direct robotic manipulation with general-purpose multimodal models, but their embodied capabilities remain poorly understood. This paper systematically evaluate GPT-6 Astra for closed-loop robotic manipulation using a lightweight agent execution framework. We first probe three fundamental embodied capabilities-depth perception, embodied in-context learning, and tactile understanding-and demonstrate that the model can exploit depth information, acquire simple manipulation strategies from contextual demonstrations, and use tactile feedback to correct its actions. We then evaluate GPT-6 Astra across five established manipulation benchmarks along six capability dimensions. Despite competitive performance on several benchmarks-26.92% success on RoboDojo and 76.0% on RoboTwin 2.0-our analysis reveals a pronounced gap between semantic reasoning and physical execution, particularly in consequence-aware action planning, precision, and complex motion coordination. To further examine these limitations, we introduce LIBERO-GeoMotion, a new benchmark of geometry-constrained tasks that require precise pose alignment, coordinated translation and rotation, and anticipation of physical action consequences. GPT-6 Astra achieves only 11.1% zero-shot success, exposing the difficulty of translating spatial understanding into physically feasible motion. Real-world experiments corroborate these findings: zero-shot execution is feasible for simple tasks, yet persistent deficits remain in motion feasibility, precise control, and physical safety, while substantial inference latency further hinders practical deployment. Overall, our results characterize the emerging embodied capabilities of general-purpose multimodal models and delineate the key challenges toward safe, efficient, and reliable robotic manipulation.
Figures & tables
Figure 1: Overview of the systematic evaluation of GPT-6 Astra for robotic manipulation. (a) illustrates the closed-loop control framework. (b) represents the analysis of depth perception, embodied in-context learning, and tactile understanding. (c) summarizes benchmark performance, embodied capability profile and real-world validation. (d) highlights limitations in rotation orientation control through benchmark LIBERO-GeoMotion.
Figure 2: GPT-GEAR: a tool-mediated framework for robotic manipulation driven by GPT-6 Astra. The model generates commands from multimodal inputs and execution feedback. Observe retrieves current or historical observations, Reference supplies optional demonstration context, and Act executes model-generated commands. Status verifies task completion, while Finish terminates the interaction.
| Observation | Success Rate (%) | Action Rounds | Total Tokens |
| w/ RGB | 47.5 | 34.61 | 1,046,755 |
| w/ RGB + Camera parameters | 60.0 | 26.32 | 986,066 |
| w/ RGB + Camera parameters + Explicit depth estimation | 62.5 | 28.85 | 1,119,366 |
| w/ RGB + Depth map | 67.5 | 24.23 | 1,101,271 |
| w/ RGB + Back-projection tool | 60.0 | 27.00 | 1,377,672 |
Table 1: Depth perception ablation with representative tasks from benchmark LIBERO-Plus. The Action Rounds and Total Tokens are averaged over all evaluated episodes.
Figure 3: The Visualization of depth perception ablation. The selected task is open the middle drawer of the cabinet from benchmark LIBERO-Plus. Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 4: Depth estimates compared with ground-truth depth. Predictions and ground truth are shown across observation steps for three task examples. MAE denotes mean absolute error, and is the number of observations shown.
| Context Formulation | Success Rate (%) | Action Rounds | Total Tokens |
| w/o context | 64.0 | 23.00 | 1,187,814 |
| w/ First and last frames | 46.0 | 21.18 | 1,448,025 |
| w/ Video sequence frames | 64.0 | 20.72 | 1,442,084 |
| w/ Textual step description | 78.0 | 21.30 | 1,052,921 |
| w/ Action trajectory + first and last frames | 66.0 | 20.64 | 1,218,883 |
| w/ Action trajectory + video sequence frames | 82.0 | 19.92 | 1,222,021 |
Table 2: Embodied in-context-learning ablation with representative tasks from dataset LIBERO-Plus. The Action Rounds and Total Tokens are averaged over all evaluated episodes. All the contexts are from expert trajectories for the same task under different scene perturbations.
Figure 5: The Visualization of in-context learning ablation. The shown tasks are pick the akita black bowl in the top layer of the wooden cabinet and place it on the plate and put the yellow and white mug in the microwave and close it from dataset LIBERO-Plus. Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
| Observation | Success Rate (%) | Action Rounds | Total Tokens |
| w/ RGB | 14.0 | 10.57 | 423.90k |
| w/ Tactile (Tactile-only) | 16.0 | 12.38 | 569.02k |
| w/ RGB + Tactile (image) | 30.0 | 10.53 | 646.06k |
| w/ RGB + Tactile (text) | 50.0 | 9.44 | 451.19k |
| w/ RGB + Tactile (image + text) | 48.0 | 9.75 | 553.36k |
Table 3: Tactile-understanding ablation with representative tasks from UniVTAC benchmark. The Action Rounds and Total Tokens are averaged over all evaluated episodes.
Figure 6: The visualization of tactile understanding ablation. The shown task is put bottle in shelf from dataset UniVTAC. Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 7: The CoT chain of tactile understanding under the RGB + tactile (image + text) condition. denotes the summed force in (a), while and denote the left and right pad vertical forces.
Figure 8: The comprehensive comparison of GPT-6 Astra with existing solutions [ 30 , 4 , 51 , 47 , 49 , 64 , 65 , 19 , 8 , 60 , 54 , 59 , 44 , 42 , 28 , 29 , 53 , 27 , 15 , 17 ] . Bars indicate success rate (SR), and diamonds additionally report the process-oriented score on RoboDojo. Higher values indicate better performance. * represents the reevaluated methods with our protocol. GPT-6 Astra is highlighted with red.
| Dimension | Operational requirement | Number | Representative tasks |
| Generalization | Adapt to changes in objects, scene layouts, or observation conditions. | 120 | LIBERO-Plus: Soup to Basket; Milk to Basket, etc. RoboTwin 2.0: Pick Diverse Bottles; Pick Dual Bottles, etc. RoboDojo: Stack Blocks; Fold Clothes, etc. RoboCasa GR1: Novel Object: Plate to Box; Tray to Pot, etc. |
| Memory | Retain object identities, initial states, sequences or counts needed by later actions. | 6 | RoboDojo: Match and Pick from Conveyor; Cover Blocks, etc. |
| Task Understanding | Ground linguistic references, spatial relations, symbolic goals, and task rules. | 57 | LIBERO-Plus: Bowl beside Ramekin to Plate, etc. RoboTwin 2.0: Blocks Ranking RGB; Place A Left of B, etc. RoboDojo: General Pickup; Solve Equation, etc. UniVTAC: Grasp Classify. |
| Long-horizon | Complete a task through an extended sequence of interdependent subgoals. | 60 | LIBERO-Plus: Bowl to Bottom Drawer and Close, etc. RoboTwin 2.0: Handover Block; Stack Three Bowls, etc. RoboDojo: Organize Table; Fill Egg Holder, etc. RoboCasa GR1: Potato to Microwave and Close, etc. |
| High-order operation | Coordinate orientation changes, rotation, pouring, or motion around articulated joints. | 31 | RoboTwin 2.0: Rotate QR Code; Hanging Mug, etc. RoboDojo: Fasten Screws; Insert Key and Turn, etc. UniVTAC: Insert Hole (oblique hole), etc. RoboCasa GR1: Bottle to Cabinet and Close, etc. |
| Precision | Meet tight alignment, contact, insertion, support, or release tolerances. | 74 | LIBERO-Plus: Wine Bottle to Rack; Book to Caddy, etc. RoboTwin 2.0: Click Bell; Place Phone on Stand, etc. RoboDojo: Insert Tubes; Plug in Charger, etc. UniVTAC: Insert Tube; Insert HDMI, etc. |
Table 4: The statistics of six capability dimensions and assigned tasks across the five benchmarks. Long task names are abbreviated. There are a total of 164 tasks and every task can count toward multiple dimensions. Counts include all tasks assigned to each dimension through a primary or secondary requirement, beyond the displayed examples.
| Dataset | Generalization | Memory | Task Understanding | Long-horizon | High-order operation | Precision |
| LIBERO-Plus | 83.50% (334/400) | – | 87.43% (306/350) | 67.00% (67/100) | 60.00% (12/20) | 58.33% (35/60) |
| RoboTwin 2.0 | 76.00% (380/500) | – | 91.67% (55/60) | 62.86% (88/140) | 58.89% (53/90) | 77.50% (124/160) |
| RoboDojo | 19.17% (23/120) | 56.67% (34/60) | 30.67% (46/150) | 17.33% (52/300) | 9.17% (11/120) | 15.71% (33/210) |
| RoboCasa GR1 | 22.22% (80/360) | – | – | 9.17% (11/120) | 7.50% (6/80) | 18.96% (91/480) |
| UniVTAC | – | – | 95.00% (19/20) | – | 66.25% (53/80) | 69.29% (97/140) |
| Overall | 59.20% (817/1380) | 56.67% (34/60) | 73.45% (426/580) | 33.03% (218/660) | 34.62% (135/390) | 36.19% (380/1050) |
Table 5: Success rates across all 164 tasks by capability dimension, using primary and secondary assignments. Cells show success rate and successful/evaluated episodes. ”–” indicates that the benchmark has no task assigned to that dimension.
Figure 9: The visualization of representative task executions demonstrating generalization . Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 10: The visualization of representative task executions demonstrating task understanding . Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 11: The visualization of representative task executions demonstrating memory . Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 12: The visualization of representative task executions illustrating limitations in long horizon . Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 13: The visualization of representative task executions illustrating limitations in precision . Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 14: The visualization of representative task executions illustrating limitations in high-order operation for gripper-equipped robots and robots equipped with dexterous hands. Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
Figure 15: The average execution latency for comparison. Lower is better. Red bars denote GPT-6 Astra, and red annotations report its execution-time ratio relative to the other evaluated solutions.
Figure 16: Overview of the nine task families in LIBERO-GeoMotion. Each panel illustrates a geometry-constrained manipulation task, with green dashed outlines indicating target poses or configurations and arrows highlighting spatial, orientation, or interaction requirements.
| Task | (a) Passage | (b) Book slot | (c) Bottle | (d) Plate | (e) Book drawer | (f) Mug | (g) Mango | (h) Package | (i) Two mugs |
| w/o ICL | 0.0 | 0.0 | 50.0 | 0.0 | 0.0 | 0.0 | 50.0 | 0.0 | 0.0 |
| w/ ICL | 25.0 | 50.0 | 75.0 | 25.0 | 25.0 | 75.0 | 75.0 | 0.0 | 100.0 |
| (pp) | +25.0 | +50.0 | +25.0 | +25.0 | +25.0 | +75.0 | +25.0 | 0.0 | +100.0 |
Table 6: Success rates (%) on LIBERO-GeoMotion. Columns correspond to the nine task families illustrated in Figure 16 . Each family contains four scene variants, and denotes the improvement in percentage points.
Figure 17: Representative failure trajectories in LIBERO-GeoMotion. (a) Mango transport: unintended contact knocks over a nearby vase. (b) Narrow-passage transport: unsuccessful coordination of translation and rotation prevents the elongated bar from passing through the opening. (c) Upright book placement: the book tips over during manipulation, and subsequent recovery attempts fail. (d) Package storage: the package remains outside the drawer, leaving placement and closure incomplete. Red boxes highlight representative failure-related states.
| Task | Zero-shot | One-example ICL |
| Pick-and-place | 100% (5/5) | 100% (5/5) |
| Mango grasping | 0% (0/5) | 60% (3/5) |
| Block-in-cup insertion | 40% (2/5) | 80% (4/5) |
| Pouring into a drawer | 40% (2/5) | 100% (5/5) |
| Shelf rearrangement | 20% (1/5) | 60% (3/5) |
| Overall | 40% (10/25) | 80% (20/25) |
Table 7: Real-world success rates under zero-shot and one-example ICL conditions. Each task is evaluated over five trials per condition. Counts indicate successful trials divided by total trials.
Figure 18: Representative real-world execution trajectories of GPT-6 Astra. Green grounding boxes highlight successful actions, while red grounding boxes highlight failure-inducing actions.
More in this topic
Robotic Manipulation
Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.
See all 1,671 papers