Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.
Figures & tables
Table 1: Comparison with existing benchmarks. Among the compared benchmarks, only Mind2Act World assesses the same task demands both in isolation and in composition, and attributes each failure to reasoning, execution or their composition, with per-model summaries in profile cards. Scale : # Tasks in each benchmark’s own unit; Avg. demo , mean length of a training demonstration. Task design : Workflow , tasks derived from real-world requirements. Capability : Composition , decisions conditioned on realized outcomes are tested; Iso.+Comp. , the same task semantics are assessed in isolation and in composition. Systems : VLA includes other end-to-end policies. Diagnosis : Stage loc. , failures are localized to the stage at which the episode broke down; Attribution , failures are attributed to reasoning, execution or their composition; Harness , a dedicated tool analyses episode logs beyond success and score; Profile card , a model receives a card summarizing where it fails and its behavioral tendencies. stated focus, or evaluated in the original paper or on the official leaderboard; × not; “-” not reported or not applicable.
Figure 1: All 30 tasks, grouped by view and capability. Mind tasks emphasize decisions based on historical observation, Act tasks precise and timely physical execution, and Mind2Act tasks the closed-loop composition of the two, in which action outcomes guide subsequent decisions. Each view has ten tasks, and together they cover twelve capability categories and five application domains. The central wheel groups the capability categories by view; the surrounding tiles show each task’s ID and scene, framed by view. Each task has three difficulty levels and two scene settings ( Clean and Random ), giving 180 configurations.
Application domain
#
Real-world need
Tasks
Home & Food Service
8
Kitchen prep, meal service, tidying
P1 P2 S1 PR4 CP01 CP02 CP03 CP04
Manufacturing & Logistics
5
Part checks, traceability, sorting, interlocks, transport
E1 M1 DR1 DR3 DR4
Instruments & Tool Use
9
Lab instruments, control panels, HMIs, office documents
E2 F1 DR2 PR1 PR2 PR3 PR5 CP06 CP09
Assembly & Physical Reasoning
3
Building from drawings, limited stock, chain mechanisms
PR6 CP07 CP08
Companion & Edutainment
5
Cognitive, music and sports training; puzzle play
F2 M2 S2 CP05 CP10
Table 2: Application domains of Mind2Act World. Each task is built from a need in one of five domains. Chips give task IDs colored by view: Mind Act Mind2Act .
Figure 2: Difficult steps inside eight Hard-level tasks. Each panel follows one Hard-level demonstration through four intermediate steps and shows what makes each step hard: the bold label under a frame names the operation, and the italic line the demand that makes it difficult. A failure at any of these steps blocks the stages after it. The color of the task ID gives its view, and the tag beside the task name the Hard-level setting; Fig. 3 compares the levels themselves.
Figure 3: Easy, Medium and Hard levels of twelve tasks. Four tasks per view at Easy , Medium and Hard . A harder level keeps the goal and raises one or more task demands: Mind tasks add more to remember or infer, Act tasks require more spatial accuracy and physical stability, and Mind2Act tasks lengthen the workflow or shorten the response window. The label under each frame gives the parameter that changes, and the icon beside each task name gives the kind of demand (key below the panels; its counts cover all 30 tasks).
Figure 4: Examples of the Clean and Random settings. Every task is evaluated in both settings; shown are the first head-camera frames of three example tasks. Clean scenes use default materials and lighting without clutter, and each Random slot adds clutter and varies surfaces and lighting (dots: table color, light tint) while keeping the task’s objects, targets and events.
Figure 5: Training, evaluation and diagnosis pipeline. Training : end-to-end policies are fine-tuned on 4,500 demonstrations generated by scripted experts. Evaluation : the policies predict action chunks and the agents call motion tools, while the evaluator decides success from simulator state hidden from both. Diagnosis : episode logs feed a model profile card, which records where each failed episode first breaks down and whether the failure lies in Mind , Act or their composition (Section 6.1 ). Dashed elements are the next stage of the loop. The failure codes on a card become concrete changes (Table 7 ), and the improved model is evaluated again (Section 7 ). The evaluation and diagnosis panels trace one recorded GPT-6 Astra episode on S1 Medium, and the card panel is illustrative.
Figure 6: Task setups grouped by application domain. Nineteen of the thirty tasks are shown with the objects and equipment each one uses. All thirty tasks are performed by the same two-arm ARX X5 robot.
Allocation
Agent
End-to-end policy
Tasks / difficulty levels / scene settings
30 / 3 / 2
30 / 3 / 2
Configurations
180
180
Episodes per configuration
5
50
Clean + Random episodes per task and difficulty
10
100
Episodes per task, across three difficulties
30
300
Episodes per view, ten tasks
300
3,000
Table 3: Evaluation episodes per system. Every system runs the same 180 configurations: thirty tasks at three difficulty levels, each in a Clean and a Random scene setting. An end-to-end policy runs ten times as many episodes per configuration as an agent. Counts are per system. The primary result combines the Clean and Random configurations of each task and difficulty with equal weight.
Model
Type
Mind
Act
Mind2Act
Overall
All
Clean
Rand.
All
Clean
Rand.
All
Clean
Rand.
Agents under one fixed harness
GPT-6 Astra [ 70 ] [0.2mm] OpenAI, 2026
Agent
All
53.9/52.7
55.9/54.7
51.9/50.7
40.4/39.0
44.6/43.3
36.3/34.7
13.3/12.0
13.3/12.0
13.3/12.0
30.2/34.6
Easy
64.7/64.0
65.1/64.0
64.3/64.0
44.9/44.0
47.7/46.0
42.0/42.0
27.3/26.0
27.0/26.0
27.5/26.0
41.0/44.7
Med.
52.1/50.0
63.2/62.0
40.9/38.0
41.0/39.0
45.2/44.0
36.8/34.0
8.5/7.0
7.8/6.0
9.3/8.0
27.5/32.0
Hard
44.9/44.0
39.4/38.0
50.4/50.0
35.5/34.0
40.8/40.0
30.1/28.0
4.2/3.0
5.2/4.0
3.2/2.0
22.2/27.0
Table 4: Mind2Act World leaderboards. Each cell reports task score (0–100) / success rate (%) for one view. For each model, the first row pools the three difficulty levels, and All is the mean of the Clean and Random settings. The Overall score is 0.25 Mind +0.25 Act +0.50 Mind2Act , which gives Mind2Act as much weight as the two single views together (Section 4.3 ). The Overall success rate pools all episodes. Agents and end-to-end policies are ranked separately; within each leaderboard, the best and second-best All values of each column (by task score) are marked as Best and Second Best .
ID
GPT-6
DeepSeek
Gemini
KinRT
π0.5
G0.5
X-VLA
InternW0
FastWAM
ID
GPT-6
DeepSeek
Gemini
KinRT
π0.5
G0.5
X-VLA
InternW0
FastWAM
Home & Food Service
Manufacturing & Logistics
P1
43
0
0
0.7
1.0
0
0
0.7
0.3
E1
73
20
6.7
6.7
1.0
0
0
0
1.0
P2
0
0
0
15
14
2.0
0
1.3
0
M1
0
0
0
0
0
0
0
0
0
S1
23
0
0
0
0
0
0.3
0.7
0
DR1
77
6.7
0
19
13
25
0
23
2.0
PR4
0
0
0
18
10
17
64
6.3
0
DR3
80
27
40
79
79
51
76
73
61
CP01
60
0
0
8.0
8.7
8.0
1.3
1.7
4.3
DR4
0
0
0
0
0
0
0
0
0
Table 5: Final success rate (%) on every task, grouped by application domain. Each block lists the tasks of one domain of Table 2 and closes with the domain mean, the mean of its task rates. In each block, the first three columns are the agents and the other six the end-to-end policies; task IDs are colored by view ( Mind , Act , Mind2Act ) and named in Table 20 . Each cell pools the Clean and Random settings and the three difficulty levels: 30 episodes per task for an agent and 300 for an end-to-end policy. Shading, in the view’s color, deepens with the success rate, and a gray 0 marks a task that the system never completes. M1, DR4, PR5, PR6, CP03, CP04, CP07 and CP09 have no success in any of the 56,700 episodes. Appendix Table 13 gives task scores and the two settings separately.
Figure 7: Success rates by view, and Random versus Clean successes. (a) Success rate by view. Final success rate on Mind , Act and Mind2Act , with 95% Wilson intervals. (b) Random vs. Clean successes. For the six end-to-end policies, the ratio of successful Random episodes to successful Clean episodes on the same instances.
Model
Type
Time
Episode time (min)
API cost (USD)
/ call
All
Mind
Act
Mind2Act
/ episode
/ success
Agents under one fixed harness (900 episodes each)
GPT-6 Astra
Agent
11.0 s
22.2
19.1
16.8
30.7
25.13
72.7
DeepSeek-V4.1-Flash
Agent
35.6 s
96.1
91.9
93.0
103.2
1.24
18.9
Gemini-4-Argon
Agent
27.8 s
18.0
15.2
15.8
22.9
1.12
24.0
End-to-end policies: VLAs and WAMs (9,000 episodes each)
Table 6: Evaluation time and cost per system. Time per call is the mean duration of one API request of an agent or one action prediction of an end-to-end policy, run locally on one GPU; episode time is the mean wall-clock time of an episode, over all tasks and by view. API cost is the mean cost of an agent episode at the providers’ list prices and the cost per successful episode.
Figure 8: Failure diagnosis on an example profile card. An illustrative example, the same as in Fig. 9 and on the project website; the measured cards of three representative systems are in Appendix E.4 . (a) Three-layer diagnosis. The workflow as a funnel: in this example, the rule layer settles about a third of the failed episodes, and the rest pass to the LLM judge and, where it is uncertain, to the human check. (b) Beyond the success rate. The example end-to-end policy rarely completes these three tasks, and most of its failures on each task occur at a single early step. (c) Across all failures. The card follows the failed episodes from the step at which each first breaks to its failure code.
Failure group
Evidence in this report
Direction
Mind : M-MEMORY , M-SPATIAL , M-RULE
924 misremembered-information failures of G0.5 and 1,011 of InternW0; VLAs lose 59–72% of Mind successes under Random scenes
Task state across stages : keeping what was seen and recognizing items under a new appearance. Data : varied scenes and longer gaps between when information is shown and when it is needed
Act : A-GRASP , A-PRECISION , A-TIMING
1,324 grasp failures of G0.5 and 1,132 fine-control failures of InternW0; 202 of 300 G0.5 episodes on CP07 never grasp a piece
Grasp, contact and timing : reliable grasps and fine contact, and responses within the windows of DR2 and DR3. Data : demonstrations weighted toward these segments, with more variation in pose, contact and timing. Harness : finer motion tools
761 precondition failures of InternW0, 271 of them on CP04, which heats the pot before all ingredients are in; agents that stop with budget left (Section 6.2 )
Checking and repairing outcomes : checking a step before the next one and repairing it when it failed. Data : demonstrations that include failed steps and their repair. Harness : the outcome of each step returned, a check required before moving on, and a retry before stopping
All groups , by breakpoint
299 of 300 G0.5 episodes on CP09 end before the board is lifted
Targeted data for end-to-end policies : demonstrations that start from the states in which episodes stop and from perturbed states near them, instead of new full-length trajectories
Table 7: Directions for improvement by failure group. Grouped as on the profile card ( Mind , Act and Composition ). The directions concern the training data of end-to-end policies and the harness of agents. Evidence is from the measured cards of Appendix E.4 and the results of Section 5 .
Figure 10: Planned work beyond this report. Top: Outcome feedback , matched runs with and without it; Coverage and cost , broader and repeated evaluation with its cost; Transfer , the analysis carried to other simulators. Bottom: Diagnosis , sharper profile cards for more models and the planned loop from a card to a better model that is evaluated again. The small drawings are schematic.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Demonstration collection and quality control. A scripted expert executes each task instance at 120 Hz while the robot state, contacts and camera readbacks are logged on every physics tick. An episode is kept only if the test-time scorer gives it full score, timing-critical tasks keep the required margin, the arms make no self- or cross-arm contact, and the recording is gap-free and clock-aligned. Otherwise it is discarded and collected again. Kept episodes are stored as 30 Hz HDF5 files.
Difficulty
Frames
Duration (h)
Episodes
Avg. frames / episode
Avg. duration / episode (s)
Mind
Easy
549,776
5.09
500
1,099.55
36.65
Medium
868,075
8.04
500
1,736.15
57.87
Hard
1,216,463
11.26
500
2,432.93
81.10
Total
2,634,314
24.39
1,500
1,756.21
58.54
Act
Appendix
Table 8: Training dataset statistics by benchmark view and difficulty level. Statistics cover all 30 tasks, with 50 episodes per task at each difficulty level. Frames count synchronized samples per camera; durations use the recorded sampling frequency of 30 Hz.
Family
Input
Output
Must disclose
VLA / WAM
Instruction, 1–3 RGB views, joint state
Joint or EEF action chunk
Checkpoint, chunk, control period, data
Agent
Instruction, RGB, history, tool feedback
Motion-tool calls
Model, harness, tools, budget, latency
Hybrid
As above, split across modules
Planner calls and controller actions
Module boundaries, fallbacks, cost
Not available to any system: hidden targets and tolerances, gate timing, drift radii, depth and pixel-to-world queries.
Appendix
Table 9: System interfaces. What each system family receives and returns under the shared protocol.
Leaderboard
Physics steps
Decisions
Wall-clock safety limit
End-to-end
B
unlimited
2 hours
Agent
4B
150
6 hours
Appendix
Table 10: Run budgets. B denotes the task- and level-specific end-to-end budget in physics steps.
System
Clean Overall
Clean successes
Random Overall
Random successes
GPT-6 Astra
31.8
165/450
28.7
146/450
DeepSeek-V4.1-Flash
5.8
28/450
6.4
31/450
Gemini-4-Argon
4.8
24/450
3.9
18/450
KinRT
9.1
364/4,500
6.8
264/4,500
π0.5
8.9
340/4,500
6.3
234/4,500
G0.5
6.7
288/4,500
5.8
242/4,500
Appendix
Table 11: Outcomes by setting. Overall scores ( 0.25 Mind +0.25 Act +0.50 Mind2Act ) and numbers of successful episodes in each setting.
System
Mind
Act
Mind2Act
All episodes
Agents (300 episodes per view)
GPT-6 Astra
158 [47.0, 58.2]
117 [33.7, 44.6]
36 [8.8, 16.2]
311 [31.5, 37.7]
DeepSeek-V4.1-Flash
33 [7.9, 15.0]
26 [6.0, 12.4]
0 [0.0, 1.3]
59 [5.1, 8.4]
Gemini-4-Argon
13 [2.5, 7.3]
29 [6.8, 13.5]
0 [0.0, 1.3]
42 [3.5, 6.2]
End-to-end policies (3,000 episodes per view)
KinRT
132 [3.7, 5.2]
408 [12.4, 14.9]
88 [2.4, 3.6]
628 [6.5, 7.5]
Appendix
Table 12: Success counts and 95% confidence intervals. Each cell gives the number of successful episodes and, in brackets, the Wilson score interval of the success rate in percent. Intervals treat episodes as independent draws and therefore describe sampling over the fixed instance lists; they do not include variation between tasks or between repeated runs.
Task
GPT-6
DeepSeek
Gemini
KinRT
π0.5
G0.5
X-VLA
InternW0
FastWAM
Mind
E1
76.9 80 / 67
22.9 13 / 27
7.0 7 / 7
11.9 7 / 7
3.7 1 / 1
0.4 0 / 0
0.2 0 / 0
0.0 0 / 0
2.7 0 / 2
E2
64.7 60 / 67
24.3 20 / 20
10.0 7 / 7
2.6 1 / 1
2.2 0 / 0
2.9 0 / 0
3.0 0 / 2
2.6 0 / 1
0.9 1 / 0
P1
46.8 53 / 33
0.4 0 / 0
1.5 0 / 0
1.0 1 / 1
1.3 1 / 1
0.1 0 / 0
0.0 0 / 0
0.8 1 / 1
0.4 1 / 0
P2
0.3 0 / 0
0.0 0 / 0
0.0 0 / 0
17.8 30 / 1
17.2 27 / 1
4.6 3 / 1
0.0 0 / 0
3.1 3 / 0
0.0 0 / 0
S1
23.8 33 / 13
0.0 0 / 0
0.2 0 / 0
0.4 0 / 0
0.3 0 / 0
0.2 0 / 0
0.8 1 / 0
1.4 1 / 0
0.0 0 / 0
Appendix
Table 13: Task scores and setting-level success rates. For each task and system, the first number is the mean task score (0–100) over both settings and the gray pair the success rate (%) in the Clean / Random setting. Each setting has 15 episodes per task for an agent and 150 for an end-to-end policy.
π0.5
G0.5
ID
Task
Easy
Med.
Hard
Easy
Med.
Hard
DR1
Conveyor pickup
34
5
0
70
4
0
DR2
Light-cued response
0
0
0
37
39
34
DR3
Timed passage through cyclic gates
76
81
79
51
51
51
PR1
Resistance adjustment
23
13
13
6
3
0
PR4
Cloth folding and flat placement
28
1
1
51
0
1
Appendix
Table 14: Successful episodes by difficulty level. Counts out of 100 episodes per level (50 Clean and 50 Random ) for two end-to-end policies, on the tasks where at least one of them succeeds repeatedly. The level totals equal the task counts of the release.
Perceive Mind
Execute Act
M-PERCEIVE
misrecognized an object, color or reading
A-REACH
approach or positioning failed
Reason Mind
A-GRASP
grasp failed or not held
M-GOAL
misunderstood the goal of the task
A-PLACE
placement inaccurate or unstable
M-MEMORY
forgot or misremembered information
A-PRESS
press, turn or other interaction had no effect
M-RULE
inferred a wrong rule or mapping
A-PRECISION
error in continuous fine control
M-SPATIAL
wrong spatial mapping or layout
A-TIMING
action too early, too late or too slow
Appendix
Table 15: Failure codes used in episode diagnosis. The table groups 18 codes by the stage at which the failure occurs. Three episode-level codes cover an exhausted budget, stalled or repeated actions, and insufficient evidence.
ID
Task
GPT-6
DeepSeek
Gemini
KinRT
π0.5
G0.5
X-VLA
InternW0
FastWAM
CP01
Conveyor meal assembly
18
0
0
24
26
24
4
5
13
CP02
Rotating-cake decoration
5
0
0
1
2
0
0
1
0
CP03
Moving-rack loading
0
0
0
0
0
0
0
0
0
CP04
Temperature-controlled cooking
0
0
0
0
0
0
0
0
0
CP05
Piano replay
6
0
0
0
0
0
0
0
0
CP06
Produce weighing on a balance
3
0
0
51
62
37
2
48
1
Appendix
Table 16: Completed Mind2Act episodes by task. Each task has 30 episodes per agent and 300 per end-to-end policy, pooled over the three difficulty levels and both settings.
System
Failed
Rule layer
LLM judge
G0.5 (VLA)
8,470
3,230 (38.1%)
5,240
InternW0 (WAM)
8,220
2,760 (33.6%)
5,460
GPT-6 Astra (agent)
589
131 (22.2%)
458
Appendix
Table 17: Failures settled by each diagnosis layer. Top: failed episodes settled by the rule layer and passed to the LLM judge. Bottom: share of failures attributed to each side, among the failures the rules settle and among all failures. The remainder carry episode-level codes.
GPT-6 Astra
DeepSeek-V4.1-Flash
Gemini-4-Argon
Stop reason
Clean
Random
Clean
Random
Clean
Random
Stopped by the agent
223
237
175
210
409
420
Decision budget exhausted
15
14
160
126
0
0
Physics budget exhausted
37
38
50
44
4
2
Completion flag recorded
175
161
64
70
37
28
Adjudicated manually
0
0
1
0
0
0
Appendix
Table 18: Stop reasons of agent episodes. Number of episodes by stop reason in the Clean and Random settings, 450 episodes each.
Dimension
Letter
In MBTI, a person
In Mind2Act World, a policy
Measured on
1 Energy
E
turns outward, to action
Act-led: executes better than it reasons
Act view
I
turns inward, to thought
Mind-led: reasons better than it executes
Mind view
2 Information
S
attends to concrete facts
keeps track of what each object is
E1, E2
N
attends to the overall pattern
keeps track of where things are
S1, S2
3 Decisions
T
decides by objective criteria
meets a fixed tolerance (precision)
PR1–PR6
F
decides by the situation
acts as the scene changes (reactivity)
DR1–DR4
Appendix
Table 19: Definition of the four profile letters. Each dimension keeps the question that MBTI asks about a person and answers it with two task groups of the benchmark. The system takes the letter of the group on which it does better. Letter 1 compares the two single-capability views, and letters 2–4 each compare two task groups within one view (column Measured on ).
Figure 16: Tasks grouped by application domain. Each task sits under the domain whose real-world requirement it is built from. The color of each task ID gives its view.
Feedback-rule learning; grid size and route length
F2
Color-block arrangement by feedback
Mastermind-style trial
Feedback-rule learning (hidden arrangement); blocks and submission budget
P1
Incremental meal orders
Food-service orders
Process memory; containers, deposits, and new orders
P2
Steamer-stack order restoration
Dim-sum steamer service
Process memory (layer order); layers and distractor steamers
Appendix
Table 20: Task summary of Mind2Act World. Each view contains ten tasks, and every task has Easy, Medium, and Hard levels. Source names the real-world requirement each task is built from; Table 2 groups the tasks by application domain. The last column gives the capability category (for Mind2Act, the memory or inference demand ↔ the execution demand) and, after the semicolon, the main variables raised with difficulty. Instructions and success conditions are given on the task cards (Appendix F.2 ).
Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints. They build an understanding of the physical world that helps them convert reasoning into actions and quickly adapt to new scenes, tasks, and rules. We refer to this capability as intuitive manipulation. Existing benchmarks fail to capture this integration: they evaluate physical reasoning in isolation from execution, or measure policy performance without requiring explicit reasoning. We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution. Our tasks require models to infer task-relevant physical structure and generate feasible action sequences under explicit constraints, including contact-rich manipulation, tool use, and multi-stage dependencies. We introduce a benchmark of 35 tasks, 14K filtered trajectories, and scalable tools for generating diverse scenarios. Experiments reveal a consistent gap: vision language models show partial physical reasoning ability but fail to produce executable plans, while state-of-the-art vision-language-action models struggle to satisfy task constraints and generalize across scenarios. These results identify intuitive manipulation as a missing axis in current foundation models and generalist robot policies, and position IMBENCH as a step toward evaluating and enabling more integrated, adaptive physical intelligence.
Building robots that can perceive, reason, and act in dynamic, unstructured environments remains a central challenge. Recent embodied systems often follow a dual-system paradigm, where System 2 performs high-level reasoning and System 1 handles low-level control. We refer to System 2 as the embodied brain, the cognitive core for decision-making in manipulation. Although evaluating this embodied brain is crucial, existing benchmarks mainly measure execution success or cover only limited aspects of high-level cognition and task realism. We introduce RoboBench, a benchmark for evaluating multimodal large language models (MLLMs) as embodied brains. RoboBench covers five dimensions: Instruction Comprehension, Perception Reasoning, Generalized Planning, Affordance Prediction, and Failure Analysis. It spans 14 capabilities, 25 tasks, and 6,092 QA pairs. To improve realism, it draws from large-scale real robotic data and in-house collection across diverse embodiments, attribute-rich objects, multi-view scenes, and memory-driven navigation. For planning, RoboBench introduces an MLLM-as-world-simulator framework that assesses whether predicted plans can achieve critical object-state changes under physical and visual constraints, enabling more faithful evaluation of long-horizon reasoning than symbolic matching. Experiments on 18 state-of-the-art MLLMs reveal persistent limitations in implicit instruction understanding, spatiotemporal reasoning, cross-scenario planning, fine-grained affordance understanding, and failure diagnosis. We further analyze how embodied cognitive abilities relate to downstream robotic control. RoboBench offers a comprehensive scaffold for quantifying high-level cognition and guiding next-generation MLLMs toward more robust robotic intelligence.
Yulin Luo, Chun-Kai Fan, Menghang Dong +19
State Key Laboratory of Multimedia Information Processing, School of Computer2026 Science, Peking University · Institute for Brain and Intelligence, Fudan University · University of Science andJul Technology Beijing +2
General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities carry into the physical world. To investigate this, we introduce RobotWorld, a challenging simulation testbed for robot use: turning instructions and observations into physical task execution through robot interfaces. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving, and aerial control, with explicit interaction budgets and executable success checks. By analysing task outcomes alongside execution traces, we identify both the capabilities that transfer and the gaps that prevent reliable completion. Furthermore, we find that current agents can construct sophisticated perception and control workflows, including image segmentation, camera calibration, spatial estimation, and dynamics-based computation. These capabilities, however, do not consistently compose into successful behaviour: agents lose task-relevant object states despite reaching commanded poses, fail to correct ineffective actions, recover too late, or mistake unfinished tasks for completion. This uneven transfer also differs across models: Astra succeeds more often on spatial and constrained-contact goals, whereas Opus 5.5 succeeds more often on continuous-balance and timed-interaction goals. By linking these outcomes to execution behaviour, RobotWorld provides both a rigorous proving ground and an empirical account of the remaining capability gaps, thereby establishing concrete targets for training and designing more reliable physical-world agents.