Organizations: Michigan State University · Independent Researcher · Zhejiang University · Microsoft · University of Rochester · University of Wisconsin–Madison
Words are increasingly becoming an interface for action: large language models (LLMs) can now invoke tools, operate devices, and make decisions that affect the physical world. However, current benchmarks largely evaluate LLMs through static text responses, offline tool calls, or simulated tasks, leaving open the question of how well they can act in real physical environments. We present a benchmark for measuring LLMs in closed-loop physical interaction, where models control a harness system connected to real-world components including robotic arms, cameras, and sensors. In each task, the model must issue instructions, receive physical feedback from the environment, interpret the resulting state, and adapt its next action accordingly. This setting evaluates capabilities beyond language understanding, including grounded perception, action planning, tool selection, spatial reasoning, feedback-driven correction, and robustness to execution uncertainty. Through experiments with representative LLMs, we characterize their ability to translate linguistic reasoning into physical actions and identify common failure modes such as hallucinated state changes, poor recovery from unexpected outcomes, and unstable long-horizon plans. Our benchmark aims to provide a systematic foundation for evaluating LLMs as interactive agents that connect words, tools, and the physical world.
Figures & tables
Figure 1: Words to World (W2W-Bench): from digital task completion to physical interaction. Tool access alone does not ensure reliable task completion. a) harness systems must close the loop between device commands, observations, and verification of intended effects. b) W2W-Bench assesses this loop across four interacting dimensions—3D spatial understanding, temporal understanding, high-level planning, and low-level execution.
Figure 2: Overview of W2W-Bench. We evaluate how LLMs interact with the physical world.
Figure 3: Tool Usage by Category
Harness
LLM
3D Spatial Understanding
Temporal Understanding
Document Capture
3D Scanning
Object Recognition
Multipage QA
Avg. rank
Health Time-Series
SimuHome
Avg. rank
Codex CLI
gpt-5.6-sol
95.00
60.45
40.00
64.38
9.25
53.33
95.15
5.50
gpt-5.6-luna
95.00
62.94
45.00
52.81
8.00
50.00
89.06
9.00
gpt-5.6-terra
95.00
62.49
60.00
60.31
6.50
56.67
91.92
6.50
gpt-6-astra
90.00
62.68
80.00
67.19
5.75
44.83
92.40
8.50
Claude Code
claude-opus-5
95.00
63.11
55.00
72.19
3.75
63.33
93.61
2.75
Table 1: Main-experiment results across harness systems and LLMs. All values are percentages. Bold and underlined values indicate the best and second-best results for each task, respectively.
LLM
Object Recognition
Simu Home
LEGO Building
Functional LEGO
qwen3.8-max
80.00
87.99
0.00
0.00
claude-opus-5
55.00
91.15
0.00
70.00
gpt-5.6-sol
50.00
92.81
12.50
40.00
gpt-6-astra
75.00
99.05
10.00
80.00
Table 2: Qwen3.8 Max in Physical Environment Interaction. Harness is Pi.
Figure 6
Method
Exact P@1
Exact P@3
Exact R@3
Exact MRR
Cat. P@1
Cat. P@3
Cat. R@3
Future h
Autonomous
56.67
56.11
13.88
73.33
83.33
72.78
19.29
6.40
Fixed 9h
53.33
54.44
14.32
70.00
83.33
70.00
19.31
9.00
Δ (Fixed 9h − Autonomous)
-3.33
-1.67
+0.44
-3.33
+0.00
-2.78
+0.02
+2.60
Table 3: Ablation study of contiguous-window in Health Time-Series Analysis. Compared with the main setting, the contiguous-window is fixed as 9 hours (denoted as “Fixed 9h”), or as fixed 9h minus Autonomous (denoted as “ Δ ”). Values are percentages except future observation hours.
Figure 7: Performance versus Average Future-window Hours Used
LLM
Dual-Arm Invocation Rate
Collision Avoidance
Insufficient Coordination
Sequence Misalignment
Overall Failure Rate
GPT-6
30.0% (6/20)
100% (2/2)
0% (0/2)
0% (0/2)
33.3% (2/6)
Claude Opus 5
30.0% (9/30)
0% (0/5)
80.0% (4/5)
20.0% (1/5)
55.6% (5/9)
Overall
30.0% (15/50)
28.6% (2/7)
57.1% (4/7)
14.3% (1/7)
46.7% (7/15)
Table 4: Dual-arm usage and failure distribution by LLM. Failure-type percentages are calculated over failed dual-arm cases; the overall failure rate is calculated over all dual-arm cases; and the dual-arm invocation rate is calculated over all evaluated cases.
Configuration
Successful recovery
Unsuccessful or incomplete recovery
Undetected or misinterpreted anomaly
Codex + GPT-6
40% (4/10 cases). A support brick was displaced by 50 mm and fell off the platform. The model retrieved it, adjusted the support spacing, and then resumed assembly.
10% (1/10 cases). The lower structure drifted by 4.34 mm. The model performed a positional correction but subsequently continued using an outdated estimate as the drift persisted.
10% (1/10 cases). The two-brick structure drifted approximately 21 mm, but the model treated this as minor settling and reused the previous coordinates without adjustment.
Claude Code + Opus 5
50% (5/10 cases). After inadvertently lifting two assembled bricks by about 30.5 mm, the model checked another viewpoint, lowered them, freed the trapped gripper, and withdrew without lifting them again.
40% (4/10 cases). The base brick was displaced and inverted by 180∘ . The model retrieved and repositioned it but failed to correct its orientation, then continued assembling additional layers on the inverted base.
20% (2/10 cases). The supporting bricks were displaced by approximately 66–73 mm. The model nevertheless declared recovery successful and continued planning placement at the previous support location.
Table 5: Case-level response classification for dynamic physical anomalies in LEGO assembly.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Tool / Action (Task)
Tool Description
Device / Information Source
capture_card
Captures a document image with adjustable viewpoint and exposure.
Card scene and camera.
capture_video_scan Continuous
Acquires and fuses multiple frames along a continuous camera trajectory.
Wrist RGB-D camera, robot arm, and reconstruction system.
capture_reconstruction_view Incremental
Acquires a single RGB-D view and fuses it into the point-cloud reconstruction.
Franka arm, RGB-D camera, and point-cloud system.
capture
Captures an object from a selected camera viewpoint.
Object scene and camera.
request_page
Retrieves a selected document page for visual inspection.
A specified page of a multipage document.
zoom_region
Magnifies a selected region of a document page for closer inspection.
A specified rectangular region of a document page.
People have long dreamed of an assistant that is always there to guide them and remind what they forget, from new dishes to lost keys. Powered by multimodal large language models (MLLMs), egocentric AI assistants on wearable devices are bringing this dream within reach: they see what the user sees and give instructions for user's physical work when necessary. However, testing such assistance is challenging and usually requires real people. The expensive and rare real interaction data leaves a key question unanswered: Can today's MLLMs truly help people in physical world? To answer this question, we propose InteractiveWorld, the first verifiable environment for closed-loop interaction between human and AI assistant. Unlike existing robot-centric simulators, InteractiveWorld lets the assistant guide a human agent who observes the scene, understands instructions, and acts independently. It thus realistically simulates how AI assistant collaborates with human: the assistant instructs, the human acts, and the environment executes actions and updates what they see next. Building on it, we introduce InteractiveBench, the first dynamic benchmark for egocentric AI assistants, with 120 multi-turn tasks across six categories. In each task, the assistant guides the human in real time, and every instruction shapes the outcome, which is scored by verifiable, state-based rewards. Experiments with 12 frontier MLLMs show they are still far from reliable assistants. Even the strongest model, Claude Fable 5.1, achieves at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively. Overall, our work offers a verifiable platform and a rigorous evaluation set, laying a solid foundation for research on interactive MLLM assistants that truly help people in the physical world.
Multimodal Large Language Models (MLLMs) excel at utilizing digital APIs and increasingly serve as the "brain" of embodied AI, instructing robots to interact with the physical world. In such embodied settings, a central capability is the use of physical tools, which underpins MLLMs' ability to assist humans in real-world tasks. Despite the importance, MLLMs' proficiency in physical tool use remains largely unexplored. To address this gap, we introduce PhysTool-Bench, the first physical tool-use benchmark designed to evaluate MLLMs' ability to comprehend real-world scenarios, identify physical tools, and plan their use. PhysTool-Bench comprises 2,510 queries over 2,678 real-world physical tools spanning diverse domains, including manufacturing, electrical work, agriculture, and healthcare. Concretely, models are evaluated along two primary dimensions: 1) recognizing all physical tools present in the scene, and 2) planning the tool selection and use sequence based on the instruction and visual context. Across 13 leading MLLMs, even the strongest model (Gemini-3.1-Pro) identifies only 58.7% of tools in a scene and completes merely 21.0% of queries end-to-end. Our analysis reveals a two-level deficit: MLLMs struggle to perceive tools in realistic scenes, and the much larger drop at the planning stage further indicates a lack of functional commonsense for mapping perceived tools onto task semantics, pinpointing a critical bottleneck for the development of practical embodied AI.
Zhixin Ma, Yutong Zhou, Yongqi Li +2
Singapore Management University · The Hong Kong Polytechnic University
Multimodal Large Language Models (MLLMs) show promising results as decision-making engines for embodied agents operating in complex, physical environments. However, existing benchmarks often prioritize high-level planning or spatial reasoning, leaving the fine-grained action intelligence required for embodied physical interaction underexplored. To address this gap, we introduce CFG-Bench, a new benchmark designed to systematically evaluate this crucial capability. CFG-Bench consists of 1,368 curated videos paired with 19,562 question-answer pairs spanning three evaluation paradigms targeting four cognitive abilities: 1) Physical Interaction, 2) Temporal-Causal Relation, 3) Intentional Understanding, and 4) Evaluative Judgment. Together, these dimensions provide a systematic framework for assessing a model's ability to translate visual observations into actionable knowledge, moving beyond mere surface-level recognition. Our comprehensive evaluation on CFG-Bench reveals that leading MLLMs struggle to produce detailed instructions for physical interactions and exhibit profound limitations in the higher-order reasoning of intention and evaluation. Moreover, supervised fine-tuning (SFT) on our data demonstrates that teaching an MLLMs to articulate fine-grained actions directly translates to significant performance gains on established embodied benchmarks. Our analysis highlights these limitations and offers insights for developing more capable and grounded embodied agents. Project page: https://cfg-bench.github.io/
Dayong Liu, Chao Xu, Weihong Chen +5
Zhejiang University · IROOTECH TECHNOLOGY · Wolf 1069 b Lab, Sany Group +2