cs.AIOct 10, 2026
SaveWords to World: Benchmarking LLMs in Interactive Physical Environments
Organizations: Michigan State University · Independent Researcher · Zhejiang University · Microsoft ·
Abstract
Words are increasingly becoming an interface for action: large language models (LLMs) can now invoke tools, operate devices, and make decisions that affect the physical world. However, current benchmarks largely evaluate LLMs through static text responses, offline tool calls, or simulated tasks, leaving open the question of how well they can act in real physical environments. We present a benchmark for measuring LLMs in closed-loop physical interaction, where models control a harness system connected to real-world components including robotic arms, cameras, and sensors. In each task, the model must issue instructions, receive physical feedback from the environment, interpret the resulting state, and adapt its next action accordingly. This setting evaluates capabilities beyond language understanding, including grounded perception, action planning, tool selection, spatial reasoning, feedback-driven correction, and robustness to execution uncertainty. Through experiments with representative LLMs, we characterize their ability to translate linguistic reasoning into physical actions and identify common failure modes such as hallucinated state changes, poor recovery from unexpected outcomes, and unstable long-horizon plans. Our benchmark aims to provide a systematic foundation for evaluating LLMs as interactive agents that connect words, tools, and the physical world.
Figures & tables
Figure 1: Words to World (W2W-Bench): from digital task completion to physical interaction. Tool access alone does not ensure reliable task completion. a) harness systems must close the loop between device commands, observations, and verification of intended effects. b) W2W-Bench assesses this loop across four interacting dimensions—3D spatial understanding, temporal understanding, high-level planning, and low-level execution.
Figure 2: Overview of W2W-Bench. We evaluate how LLMs interact with the physical world.
Figure 3: Tool Usage by Category
| Harness | LLM | 3D Spatial Understanding | Temporal Understanding | ||||||
| Document Capture | 3D Scanning | Object Recognition | Multipage QA | Avg. rank | Health Time-Series | SimuHome | Avg. rank | ||
| Codex CLI | gpt-5.6-sol | 95.00 | 60.45 | 40.00 | 64.38 | 9.25 | 53.33 | 95.15 | 5.50 |
| gpt-5.6-luna | 95.00 | 62.94 | 45.00 | 52.81 | 8.00 | 50.00 | 89.06 | 9.00 | |
| gpt-5.6-terra | 95.00 | 62.49 | 60.00 | 60.31 | 6.50 | 56.67 | 91.92 | 6.50 | |
| gpt-6-astra | 90.00 | 62.68 | 80.00 | 67.19 | 5.75 | 44.83 | 92.40 | 8.50 | |
| Claude Code | claude-opus-5 | 95.00 | 63.11 | 55.00 | 72.19 | 3.75 | 63.33 | 93.61 | 2.75 |
Table 1: Main-experiment results across harness systems and LLMs. All values are percentages. Bold and underlined values indicate the best and second-best results for each task, respectively.
| LLM | Object Recognition | Simu Home | LEGO Building | Functional LEGO |
| qwen3.8-max | 80.00 | 87.99 | 0.00 | 0.00 |
| claude-opus-5 | 55.00 | 91.15 | 0.00 | 70.00 |
| gpt-5.6-sol | 50.00 | 92.81 | 12.50 | 40.00 |
| gpt-6-astra | 75.00 | 99.05 | 10.00 | 80.00 |
Table 2: Qwen3.8 Max in Physical Environment Interaction. Harness is Pi.
| Method | Exact P@1 | Exact P@3 | Exact R@3 | Exact MRR | Cat. P@1 | Cat. P@3 | Cat. R@3 | Future h |
| Autonomous | 56.67 | 56.11 | 13.88 | 73.33 | 83.33 | 72.78 | 19.29 | 6.40 |
| Fixed 9h | 53.33 | 54.44 | 14.32 | 70.00 | 83.33 | 70.00 | 19.31 | 9.00 |
| (Fixed 9h Autonomous) | -3.33 | -1.67 | +0.44 | -3.33 | +0.00 | -2.78 | +0.02 | +2.60 |
Table 3: Ablation study of contiguous-window in Health Time-Series Analysis. Compared with the main setting, the contiguous-window is fixed as 9 hours (denoted as “Fixed 9h”), or as fixed 9h minus Autonomous (denoted as “ ”). Values are percentages except future observation hours.
Figure 7: Performance versus Average Future-window Hours Used
| LLM | Dual-Arm Invocation Rate | Collision Avoidance | Insufficient Coordination | Sequence Misalignment | Overall Failure Rate |
| GPT-6 | 30.0% (6/20) | 100% (2/2) | 0% (0/2) | 0% (0/2) | 33.3% (2/6) |
| Claude Opus 5 | 30.0% (9/30) | 0% (0/5) | 80.0% (4/5) | 20.0% (1/5) | 55.6% (5/9) |
| Overall | 30.0% (15/50) | 28.6% (2/7) | 57.1% (4/7) | 14.3% (1/7) | 46.7% (7/15) |
Table 4: Dual-arm usage and failure distribution by LLM. Failure-type percentages are calculated over failed dual-arm cases; the overall failure rate is calculated over all dual-arm cases; and the dual-arm invocation rate is calculated over all evaluated cases.
| Configuration | Successful recovery | Unsuccessful or incomplete recovery | Undetected or misinterpreted anomaly |
| Codex + GPT-6 | 40% (4/10 cases). A support brick was displaced by 50 mm and fell off the platform. The model retrieved it, adjusted the support spacing, and then resumed assembly. | 10% (1/10 cases). The lower structure drifted by 4.34 mm. The model performed a positional correction but subsequently continued using an outdated estimate as the drift persisted. | 10% (1/10 cases). The two-brick structure drifted approximately 21 mm, but the model treated this as minor settling and reused the previous coordinates without adjustment. |
| Claude Code + Opus 5 | 50% (5/10 cases). After inadvertently lifting two assembled bricks by about 30.5 mm, the model checked another viewpoint, lowered them, freed the trapped gripper, and withdrew without lifting them again. | 40% (4/10 cases). The base brick was displaced and inverted by . The model retrieved and repositioned it but failed to correct its orientation, then continued assembling additional layers on the inverted base. | 20% (2/10 cases). The supporting bricks were displaced by approximately 66–73 mm. The model nevertheless declared recovery successful and continued planning placement at the previous support location. |
Table 5: Case-level response classification for dynamic physical anomalies in LEGO assembly.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Tool / Action (Task) | Tool Description | Device / Information Source |
|---|---|---|
| capture_card | Captures a document image with adjustable viewpoint and exposure. | Card scene and camera. |
| capture_video_scan Continuous | Acquires and fuses multiple frames along a continuous camera trajectory. | Wrist RGB-D camera, robot arm, and reconstruction system. |
| capture_reconstruction_view Incremental | Acquires a single RGB-D view and fuses it into the point-cloud reconstruction. | Franka arm, RGB-D camera, and point-cloud system. |
| capture | Captures an object from a selected camera viewpoint. | Object scene and camera. |
| request_page | Retrieves a selected document page for visual inspection. | A specified page of a multipage document. |
| zoom_region | Magnifies a selected region of a document page for closer inspection. | A specified rectangular region of a document page. |
Table 6: Observation Acquisition interfaces (14 entries).
| Tool / Action (Task) | Tool Description | Device / Information Source |
|---|---|---|
| execute_command | Executes device commands to control the environment. | Matter-style device interfaces. |
| write_attribute | Changes writable device attributes and configuration values. | Writable attributes of smart-home devices. |
| act_lego | Controls both robot arms through tool-center-point poses, yaw, and gripper commands. | Two robot arms, grippers, and LEGO parts. |
| home_robot | Moves the laboratory robot to its home configuration. | Laboratory robot. |
| pick_up_tip | Loads a sterile pipette tip for liquid handling. | Pipetting robot and sterile tips. |
| drop_tip | Disposes of the attached pipette tip. | Pipetting robot and waste area. |
Table 7: Action Execution interfaces (28 entries).
| Tool / Action (Task) | Tool Description | Device / Information Source |
|---|---|---|
| inspect_video_reconstruction_progress Continuous | Reports scan history, online reconstruction metrics, contact sheets, and safe candidate scans. | Reconstruction state, contact sheets, and candidate scans. |
| inspect_reconstruction_progress Incremental | Reports the current image, trajectory, online reconstruction metrics, and safe candidate viewpoints. | Reconstruction state, current RGB image, and candidate viewpoints. |
| view_home_page | Displays rooms, environmental variables, device summaries, and virtual time. | Rooms, environmental variables, and device summaries. |
| view_device_page | Displays device attributes, writable values, clusters, and supported commands. | Device attributes, clusters, writable values, and supported commands. |
| view_workflows_page | Displays scheduled workflows, execution times, steps, and status. | Workflow IDs, times, steps, and status. |
| get_state | Reports current camera settings and the remaining interaction budget. | Current camera settings and remaining budget. |
Table 8: State and Feedback Inspection interfaces (10 entries).
| Tool / Action (Task) | Tool Description | Device / Information Source |
|---|---|---|
| list_video_reconstruction_targets Continuous | Lists available reconstruction targets without exposing reference models. | Available reconstruction targets, excluding reference models. |
| list_reconstruction_targets Incremental | Lists available reconstruction target identifiers. | Available reconstruction target IDs. |
| get_clinical_context | Retrieves clinical context available before the temporal cutoff, including complaints, admission type, and started medications. | Chief complaint, admission type, and medications already started. |
| search_icd9_codes | Searches the global ICD-9 dictionary without exposing patient-diagnosis associations. | Global ICD-9 dictionary without patient-diagnosis associations. |
| view_environment_rules_page | Retrieves rules linking device actions to changes in environmental variables. | Rules linking device actions to temperature, humidity, illumination, and air quality. |
| list_parts | Lists available LEGO part types, quantities, and colors. | Part types, quantities, and colors. |
Table 9: Context and Resource Access interfaces (7 entries).
| Tool / Action (Task) | Tool Description | Device / Information Source |
|---|---|---|
| start_video_reconstruction_episode Continuous | Initializes a reconstruction episode and acquires the fixed initial keyframe. | Target configuration, wrist camera, and fixed initial keyframe. |
| start_reconstruction_episode Incremental | Initializes a reconstruction episode at the fixed initial viewpoint. | Target configuration, RGB-D camera, and fixed initial viewpoint. |
| plan_future_window | Plans a future observation window within the 72-hour post-anchor horizon. | Episode state and the 72-hour post-anchor horizon. |
| advance_time | Commits the observation window and advances the simulation clock. | Current observation window and simulation clock. |
| schedule_workflow | Schedules an action sequence at an absolute virtual time. | Workflow scheduler and absolute virtual time. |
| cancel_workflow | Cancels a workflow that has not yet executed. | Workflows that have not yet executed. |
Table 10: Planning and Tool Orchestration interfaces (8 entries).
| Tool / Action (Task) | Tool Description | Device / Information Source |
|---|---|---|
| submit_photo | Selects a captured image for submission and triggers hidden-quality evaluation. | Specified image ID and captured photographs. |
| finalize_video_reconstruction_episode Continuous | Terminates reconstruction using online evidence without revealing ground-truth scores. | Current reconstruction and online evidence; no ground-truth score disclosure. |
| finalize_reconstruction_episode Incremental | Finalizes reconstruction with a stop reason, confidence, and output artifacts. | Reconstruction artifacts and the agent’s stopping judgment. |
| guess_label | Submits the predicted object class. | Predicted object class. |
| submit_answer | Submits four answer fields and their supporting document pages. | Four answer fields and supporting document pages. |
| record_assessment | Records an intermediate differential diagnosis, uncertainty, information gain, and proposed next step. | Agent assessment, acquired evidence, and episode record. |
Table 11: Assessment and Submission interfaces (10 entries).
More in this topic
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,179 papers