Frontier LLMs Enter the World of Robotics: A Cross-Model Evaluation on RoboColiseum

Da Huang; Jichao Wang; Junhui Wu; Tong Ran; Linting Guo; Siying Zhang; Chuang Men; Sukai Wang

cs.ROOct 20, 2026

Frontier LLMs Enter the World of Robotics: A Cross-Model Evaluation on RoboColiseum

Authors: Da Huang, Jichao Wang, Junhui Wu, Tong Ran, Linting Guo, Siying Zhang, Chuang Men, Sukai Wang,

Abstract

Recent studies have explored the potential of multimodal large language models for robot control, yet the models' embodied capabilities remain insufficiently characterized across models, tasks, and execution settings. We conduct a multidimensional comparative evaluation on RoboColiseum across four tracks: Instruction Following, Spatial Reasoning, General Manipulation, and Robustness. We implement Agent2Act, a control harness that integrates GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5, and Gemini 4 Argon into the evaluation environment through shared observation, action execution, and feedback interfaces. Under common task settings and scoring criteria, all four models exhibit a consistent capability profile: high instruction-following scores, with three approaching perfect performance; strong spatial reasoning; and general manipulation scores below those of the other three tracks. Within the evaluated configurations, Astra leads instruction following and robustness, while Opus leads spatial reasoning, general manipulation, and overall performance. These results reveal complementary strengths across models and a persistent gap between task understanding and reliable physical execution. Component comparisons on selected tasks further show that harness design affects task success, planning cost, and execution time. Additionally, case studies with Astra examine whole-body coordination, manipulation across robot embodiments, precision manipulation, dexterous-hand manipulation, and real-robot tasks, providing further insight into the model's capabilities and limitations across embodied tasks.

Figures & tables

Model	overall	instr	spatial	manip	robust
Claude Opus 5.5	76.28	96.60	82.50	46.70	79.30
GPT-6 Astra	75.99	99.35	78.33	45.86	80.42
GPT-6.1 Sol	70.91	95.50	77.71	40.01	70.40
Gemini 4 Argon	66.86	85.89	69.38	38.79	73.36

TABLE I: Four-model comparison: Overall and four track scores (%), ordered by Overall.

Fig. 1: Agent2Act’s LLM-driven control loop. The multimodal LLM uses task context and observations, queries read-only sensor evidence, and proposes actions. Agent2Act validates and converts proposals into robot commands; subsequent observations, robot responses, and task progress inform the next decision. RoboColiseum determines task outcomes through native evaluation. Camera panels show recorded observations.

Module	Information actually supplied to the model	Use and scope
Original task instruction	The benchmark’s original natural-language request, including the requested object, arm, relation, or operation.	Provided as the task instruction; kept separate from authored guidance.
General robot instructions	Action schema, coordinate and pose conventions, robot/gripper geometry, available controls, camera calibration, and execution constraints.	Shared control interface; No geometry and No projection remove the specified information or requirement.
Authored operation guidance	Optional task-specific requirements and general manipulation suggestions, such as contact or operation strategies.	Configurable input, separate from the original instruction; not supplied in the benchmark evaluation on any of the four tracks.
Visual and measured state	Current RGB observations, measured robot poses/joints and gripper state, plus depth observations with simulated noise through the configured input/query interface.	Supplied observations; No depth removes depth inputs and queries.
Execution feedback	Requested-versus-observed motion summaries and action-validation/rejection information.	Supports correction; No feedback removes deviation summaries while retaining current observations.
Recent history	Previous decisions, queries, execution records, and configured observation history.	Supplied in the full setting; removed in the No history comparison.

TABLE II: Model input modules and their use in the reported setting.

Rank	Model	Overall	Instruction Following	Spatial Reasoning	General Manipulation	Robustness
1	Paimon	0.7754	0.8961	0.6791	0.6613	0.8653
2	HiDream-O1-Embodied	0.7692	0.8913	0.7437	0.6442	0.7978
3	Claude Opus 5.5	0.7628	0.9660	0.8250	0.4670	0.7930
4	GPT-6 Astra	0.7599	0.9935	0.7833	0.4586	0.8042
5	GPT-6.1 Sol	0.7091	0.9550	0.7771	0.4001	0.7040
6	DM0.5	0.7076	0.8444	0.6146	0.6370	0.7344

TABLE III: Model leaderboard: Overall and four track scores on a 0–1 scale.

Task	GPT-6 Astra	GPT-6.1 Sol	Claude Opus 5.5	Gemini 4 Argon
Billiards color	1.00	0.98	0.99	0.87
Block color	1.00	0.99	1.00	0.90
Block number	1.00	1.00	1.00	0.94
Block shape	0.99	0.91	0.91	0.85
Block size	0.98	0.95	0.95	0.90
Common sense	0.99	0.96	1.00	0.89

TABLE IV: Instruction Following: reported score for each task on a 0–1 scale.

Task	GPT-6 Astra	GPT-6.1 Sol	Claude Opus 5.5	Gemini 4 Argon
Absolute pickup	0.92	0.83	1.00	0.88
Relative pickup	0.93	0.97	1.00	0.80
Beverage placement	0.93	0.88	0.98	0.85
Relative placement	0.85	0.93	0.92	0.82
Size ordering	0.90	1.00	0.93	0.77
Number ordering	0.87	0.60	0.90	0.77

TABLE V: Spatial Reasoning: reported score for each task on a 0–1 scale.

Task	GPT-6 Astra	GPT-6.1 Sol	Claude Opus 5.5	Gemini 4 Argon
Clean desktop	0.07	0.05	0.20	0.18
Pot holding	0.41	0.23	0.60	0.00
Open door	0.86	0.81	0.68	0.81
Block-in-box	0.83	0.80	0.81	0.75
Pour workpiece	0.11	0.10	0.24	0.16
Scoop popcorn	0.43	0.36	0.29	0.20

TABLE VI: General Manipulation: reported score for each task on a 0–1 scale.

	GPT-6 Astra		GPT-6.1 Sol		Claude Opus 5.5		Gemini 4 Argon
Setting	Score	$\Delta$	Score	$\Delta$	Score	$\Delta$	Score	$\Delta$
IF baseline	0.99	—	0.96	—	0.97	—	0.86	—
Background and lighting	0.84	0.16	0.80	0.16	0.81	0.16	0.78	0.08
Camera position	0.85	0.14	0.86	0.10	0.77	0.20	0.72	0.14
Image quality	0.68	0.32	0.18	0.77	0.76	0.20	0.60	0.26
Instruction phrasing	0.86	0.13	0.85	0.11	0.79	0.18	0.77	0.09

TABLE VII: Robustness by perturbation type, compared with each model’s Instruction Following (IF) baseline. Scores and

\Delta

are on a 0–1 scale, rounded to two decimal places;

\Delta

is the IF baseline minus the perturbation score.

Component	Set	Intervention and purpose
Execution feedback	$\mathcal{I}_{10}$	Remove requested-versus-observed motion summaries; retain current images, measured state, depth, and history. Examine explicit feedback for correction and task progress.
Depth queries	$\mathcal{I}_{10}$	Remove simulated noisy depth inputs and query tools; retain RGB, camera calibration, robot geometry/state, feedback, and history. Examine completion and planning cost without depth.
IK checking	$\mathcal{I}_{10}$	Hard IK rejects targets that fail strict IK validation and requests replanning. Soft IK allows approximate execution when strict pose acceptance is not met. Examine task completion and subsequent motion correction.
Camera projection	$\mathcal{I}_{10}$	Remove the requirement to compute and report the first motion waypoint’s image projection for self-checking. Retain RGB, calibration, depth, geometry, and local IK. Examine explicit geometric checking.
Gripper geometry	$\mathcal{I}_{10}$	Remove gripper dimensions, tool axes, asymmetric-finger descriptions, and contact guidance; retain vision, depth, calibration, and the fixed wrist-to-TCP transform. Examine approach orientation, finger-pad adjustment, and grasping.
Action representation	$\mathcal{I}_{10}$	Replace EEF targets and their IK conversion with joint increments planned from measured joints and forward-kinematics poses. Enable both arms and grippers, with waist, head, and base control disabled.

TABLE VIII: The seven configuration changes and their purposes.

Fig. 2: Configuration comparison of success rate, mean time, planning decisions, and issued steps. Solid bars denote the full system; hatched bars denote variant configurations.

Fig. 3: Whole-body control during trash disposal, combining base movement, torso adjustment, and arm manipulation.

Fig. 4: Bowl-stacking task where hand and wrist adjustments do not establish a stable grasp.

Fig. 5: Representative precision-manipulation sequences for (a) block insertion into a matching hole, (b) phone charging, and (c) Ethernet cable insertion. External-view frames progress left to right, showing approach, grasping, and positioning during the recorded attempts.

Fig. 6: Chess-piece manipulation on ARX, AgileX, G1, and G2 through the common end-effector interface.

Panel of Figure 15 — Fig. 7: Spatial exploration and mobile interaction in simulation.

Fig. 8: Effect of instruction detail on pot holding. (a, left) Poorly aligned grasp with the original task instruction. (b, right) Better handle alignment and more stable handling with detailed grasping and interaction guidance.

Fig. 9: Final recorded states of ARX Lift2 tasks in third-person views: (a) cup pickup, (b) color-based block pickup, (c) size-based block pickup, (d) block stacking, (e) towel folding, and (f) garment folding in half.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 18 — Fig. 10: Rendered scene examples from the four RoboColiseum tracks: (a) Instruction Following, (b) Spatial Reasoning, (c) General Manipulation, and (d) Robustness. The Robustness examples show the same object-selection task under five perturbation types. These native camera previews illustrate the task settings and do not depict model execution outcomes.

Robotic Control

Robotic control studies how machines use sensing and computation to produce reliable movement and interaction in changing environments.

See all 581 papers

Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning
Oct 20, 2026Youwei Zhou, Bohan Hou, Yankai Fu +10Robotic Manipulation Zero-Shot Robotic Manipulation
Nanjing University · Nanyang Technological University · Peking University +2
Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
Oct 20, 2025Yulin Luo, Chun-Kai Fan, Menghang Dong +19Long-Horizon Robotic Manipulation Multimodal Large Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer2026 Science, Peking University · Institute for Brain and Intelligence, Fudan University · University of Science andJul Technology Beijing +2
The Bitter Lesson Comes to Robotics
Oct 18, 2026Christopher Leet, Achyutha Menon, Sravanthi Machcha +3Language Model-Based Control Robotic Manipulation
Robocurve · University of Southern California · University of California, Irvine

Oct 20, 2026J/K move · Enter open · S save

Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning

Youwei Zhou, Bohan Hou, Yankai Fu, Yaoxu Lyu+9

Nanjing University · Nanyang Technological University · Peking University+2

Robotic Manipulation Zero-Shot Robotic Manipulation Robot Manipulation Benchmarks Safe Robot Manipulation

Open paper AskPDF arXiv

Recent advances in visuospatial understanding and agentic reasoning enable direct robotic manipulation with general-purpose multimodal models, but their embodied capabilities remain poorly understood. This paper systematically evaluate GPT-6 Astra for closed-loop robotic manipulation using a lightweight agent execution framework. We first probe three fundamental embodied capabilities-depth perception, embodied in-context learning, and tactile understanding-and demonstrate that the model can exploit depth information, acquire simple manipulation strategies from contextual demonstrations, and use tactile feedback to correct its actions. We then evaluate GPT-6 Astra across five established manipulation benchmarks along six capability dimensions. Despite competitive performance on several benchmarks-26.92% success on RoboDojo and 76.0% on RoboTwin 2.0-our analysis reveals a pronounced gap between semantic reasoning and physical execution, particularly in consequence-aware action planning, precision, and complex motion coordination. To further examine these limitations, we introduce LIBERO-GeoMotion, a new benchmark of geometry-constrained tasks that require precise pose alignment, coordinated translation and rotation, and anticipation of physical action consequences. GPT-6 Astra achieves only 11.1% zero-shot success, exposing the difficulty of translating spatial understanding into physically feasible motion. Real-world experiments corroborate these findings: zero-shot execution is feasible for simple tasks, yet persistent deficits remain in motion feasibility, precise control, and physical safety, while substantial inference latency further hinders practical deployment. Overall, our results characterize the emerging embodied capabilities of general-purpose multimodal models and delineate the key challenges toward safe, efficient, and reliable robotic manipulation.

Abstract

Robotic Control

Explore similar work

Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

The Bitter Lesson Comes to Robotics