Physical understanding and reasoning depend on forming compact and generalizable representations of the world. While modern vision-language models can recognize and explain diverse physical events, they often lack explicit representations of the underlying mechanisms-such as object states, physical parameters, and governing dynamics-needed for reliably reasoning how the world evolves and responds to interventions. In this work, we introduce Code-as-World, a paradigm that represents physical worlds through executable world representations. By expressing physical composition, dynamic evolution, and visual appearance as executable code, Code-as-World provides a compact, quantitatively grounded, and controllable abstraction of the physical world. To construct such representations from multimodal observations, such as natural-language descriptions or real-world videos, we develop an agentic discovery loop inspired by abductive reasoning, where an agent proposes, executes, renders, verifies, and iteratively refines executable world hypotheses. As a concrete application, we use verified executable worlds to provide scalable physical supervision for training vision-language models on quantitative physical reasoning. Experiments show that Code-as-World-VL achieves state-of-the-art performance on QuantiPhy and surpasses leading proprietary models, highlighting the potential of executable world representations as a scalable foundation for physical intelligence.
Figures & tables
Figure 1: Comparison among different physical world representations. Pixels preserve rich visual detail but lack explicit structure, leaving the underlying causes of observations ambiguous; 3D representations capture geometric structure but do not necessarily reveal physical mechanisms; and language provides compact semantic abstractions but lacks precision for continuous physical states. Code complements these representations by expressing the world as an abstract, quantitatively grounded, and executable representation.
Figure 2: Agentic discovery loop of executable world representations. Given a text prompt or real video, modality-specific processors transform the input into semantic or visual evidence. A shared propose–instantiate–execute–render–verify loop then optimizes an executable world representation over physical composition, dynamic evolution, and visual appearance.
Figure 3: Controllable resimulation with Code-as-World. In each row, the left four columns show an edited simulator rollout, and the right four show temporally aligned frames from its realistic video.
Figure 4: Agentic discovery across loop rounds. Solid curves connect one-shot to agentic loop rounds two through five. Dashed gray lines mark Best-of- 5 from independent sampling. At a matched five-evaluation budget, agentic loop outperforms on Visual Alignment, Object IoU, Traj-ADE, and Accuracy@ 2%D . Here, D denotes the frame diagonal. Traj-ADE is reported in %D , Velocity-ADE in %D /step, and accuracy at 2%D .
Figure 5: Visual results of text-driven construction : from executable simulation to realistic video. For each example, the left side shows matched frames from a simulator rollout generated from a text specification, while the right side shows the corresponding final video synthesized by the video generation model. The sim-to-real transformation introduces rich and diverse objects, materials, backgrounds, and textures while preserving the motion trajectories and physical evolution encoded by the executable world.
Figure 6: Visual results of video-driven abstraction : from real video to executable simulation. For each example, the left side shows matched frames from the input real video, while the right side shows the corresponding rollout of the abstracted executable world. The recovered simulation aligns with the observed scene composition, object motion, and physical interactions, capturing the underlying physical mechanism as a coherent, executable process.
Figure 7: Illustration of quantitative physical reasoning. A video and a question are given as input. For 3D scenes, an additional depth prior provides spatial context. The model needs to predict an answer grounded in the video. Examples adapted from [ 11 ] .
Models
Size
Kinematic Categories
Average Score
2S
2D
3S
3D
Proprietary models
Gemini-3.1 Flash [ 3 ]
–
49.4
47.5
61.4
61.1
54.8
ChatGPT-5.1 [ 58 ]
–
56.9
34.6
45.6
56.4
48.4
Gemini-2.5 Pro [ 59 ]
–
45.9
38.6
40.7
60.2
46.4
Gemini-2.5 Flash [ 59 ]
–
42.8
31.9
47.0
54.7
44.1
Table 1: Quantitative physical reasoning results. We report MRA on the 2S, 2D, 3S, and 3D subsets and their macro-average on QuantiPhy-validation. The 4B and 9B variants produce direct answers, whereas the 27B reasoning variant is scored only on the answer emitted after its reasoning trace. Rows are grouped by model family, and Code-as-World-VL variants are highlighted in purple.
Figure 8: Parameter scaling on QuantiPhy. Models with undisclosed parameter counts are grouped at the far right.
Figure 9: Reasoning process of Code-as-World-VL-27B on QuantiPhy. Two representative cases are shown vertically: an executable-world bicycle example and a real-video billiards example from internet_0027 . Red boxes mark the target in the selected frames. The abridged traces preserve the key Image-Space measurements and World-Space calibration steps.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Distributional Realism
Motion Fidelity
Video Type
JEDi MMD ↓
TRAJAN Fréchet ↓
Traj-ADE ↓ ( %D )
Velocity-ADE ↓ ( %D /step)
Accuracy@ 2%D↑ (%)
Simulator Render
3.000
406.872
1.682
0.404
78.81
Sim-to-Real Video
1.484
185.321
1.677
0.472
77.49
Appendix
Table 2: Distributional realism and motion fidelity of text-driven sim-to-real video generation. Realism metrics use held-out real videos as the reference distribution, whereas motion metrics compare CoTracker estimates against simulator ground-truth trajectories.
Figure 10: Agentic discovery with the physics engine. Solid curves report performance over five iterative discovery rounds, while dashed gray lines mark Best-of- 5 under the same five-evaluation budget. Higher is better for Visual Alignment, Object IoU, and Accuracy@ 2%D ; lower is better for Traj-ADE and Velocity-ADE.
Models
Size
RefCOCO †
RefCOCOg †
RefCOCO+ †
RefCLEF †
GOT-10K *
Open-weight models
Qwen3-VL-32B-Instruct [ 62 ]
32B
49.1
40.3
44.9
32.5
15.8
Qwen3.5-27B [ 6 ]
27B
47.0
44.8
46.8
31.6
6.1
Qwen3.5-9B [ 6 ]
9B
38.6
33.0
38.6
37.8
15.9
MiniCPM-V 4.5 [ 70 ]
8B
55.6
53.2
54.1
38.6
15.9
InternVL-3.5-8B [ 63 ]
8B
36.4
30.9
35.2
24.2
14.4
Appendix
Table 3: Pixel-level measurement evaluation. Referring-expression datasets evaluate object extent in images, while GOT-10K evaluates object extent and motion in video. Image-Space variants use only the first training phase; full Code-as-World-VL models additionally receive executable-world reinforcement learning.
Figure 11: Qualitative comparison of Image-Space grounding and measurement. The four columns evaluate target grounding, object length, speed, and acceleration, respectively. All numerical quantities are expressed in the pixel-space units specified by the question.
Training Variants
Size
Dpix
Dtext
Dvideo
2S
2D
3S
3D
Avg.
Image-Space Only
4B
✓
45.3
45.1
31.6
54.6
44.2
+ Text-Driven Worlds
4B
✓
✓
47.5
49.4
40.0
57.0
48.5
+ Video-Driven Worlds
4B
✓
✓
42.2
52.7
46.5
49.8
47.8
Full Code-as-World-VL
4B
✓
✓
✓
45.4
55.4
45.8
56.0
50.6
Image-Space Only
9B
✓
44.3
51.9
51.4
60.0
50.9
+ Text-Driven Worlds
9B
✓
✓
47.2
48.1
53.5
61.1
52.5
Appendix
Table 4: Ablation of data sources. We report MRA on the QuantiPhy subsets and their macro-average for the three data sources in our two-phase curriculum.
Figure 12: A concrete EWR code and simulation interface. A video-driven scene is encoded as a structured scene.json specification of its composition, evolution, appearance, physical parameters, relations, simulator configuration, and timeline. The simulator SDK instantiates and executes this specification in MuJoCo, while the frame-aligned evidence and simulation outputs above illustrate the resulting reconstruction.
Physical reasoning from video requires inferring latent physical properties and dynamics beyond direct observation. Direct VLM inference remains unreliable on complex physical tasks without explicit modeling and validation, while predefined tool pipelines rely on task- and domain-specific priors that limit generalization across materials, dynamics, and reasoning tasks. We introduce Asking the World (ATW), a generalist agent that constructs and interrogates task-relevant executable worlds through two adaptive stages: World Modeling calibrates a world from video, while World Probing queries, simulates, and intervenes on it to obtain question-relevant evidence. Rather than prescribing the operations in either stage, ATW determines how to model and probe according to the scene and question. We develop PolyWorld Engine, a lightweight and highly programmable Warp-based multiphysics simulator for constructing and probing worlds with rigid bodies, soft bodies, cloth, ropes, fluids, and their coupled interactions. CEM-based system identification recovers task-relevant dynamics during World Modeling. The resulting world becomes an active workspace for question-directed physical experiments rather than a predetermined downstream tool. We evaluate ATW on CLEVRER, ContPhy, and three real-world scenarios. Using Gemini-3-Flash as its base VLM, ATW achieves 80.82% overall per-question accuracy on CLEVRER, improving direct Gemini-3-Flash by 46.50 points, GPT-5.5 by 13.58 points, and PhysMind by 8.27 points. On ContPhy, it reaches 70.56% overall accuracy, surpassing Gemini-3-Flash by 28.10 points and GPT-5.5 by 3.53 points. Across the three real-world scenarios, ATW achieves 71.67% accuracy, 28.33 points above GPT-5.5. These results establish agentic world modeling and probing as an effective, execution-grounded approach to generalist physical reasoning.
Reliable physical reasoning from video requires understanding how objects move, interact, and respond to interventions. Existing vision-language models (VLMs) often struggle to interpret these dynamics and reason reliably about future and counterfactual outcomes. We introduce PhysMind, a training-free agentic framework that constructs one reusable, question-agnostic executable world per video. PhysMind recovers a temporally consistent dynamic scene through object segmentation, mesh reconstruction, and 6D pose tracking, then fits analytic continuous-time dynamics and latent physical parameters without unrolling a time-stepped simulator. Given a question, it inspects, continues, or edits the world and answers from the resulting trajectories and interactions. Relative to direct chain-of-thought (CoT) reasoning with the same VLM, PhysMind improves accuracy by 38.23 points on CLEVRER and 8.08 points on Physion++. On counterfactual questions, it exceeds the strongest evaluated VLM baseline, GPT-5.5, by 19.25 points.
Chen Yang, Shenxiang Zeng, Haoyang Zhao +6
1Tsinghua University · 2The University of Hong Kong
Coding agents are increasingly evolving beyond reproducing what a world looks like toward constructing executable worlds that capture how a world works. This shift calls for executable world modeling, which integrates two core capabilities: world understanding, reasoning about the underlying structural, mechanical, and causal principles of a world, and executable realization, translating that understanding into a code-based system whose behavior emerges from physical interactions. However, existing benchmarks provide only fragmented evaluations centered on physical reasoning, visual fidelity, or behavioral correctness, leaving the integration of these capabilities insufficiently assessed. To bridge this gap, we introduce ExeWorldBench, to our knowledge the first benchmark to comprehensively evaluate world understanding and executable realization through code-based 3D world construction. ExeWorldBench comprises three progressively challenging tasks: Mechanism Executability, which evaluates the realization of individual mechanisms from visual observations; Compositional Executability, which extends to multi-stage systems with spatial, mechanical, and causal dependencies; and Intent-Driven Executable World Synthesis, which requires constructing executable worlds from high-level goals and physical requirements. Extensive experiments on frontier models reveal substantial limitations in executable world modeling, offering insights into the development of more physically grounded and causally coherent world construction systems.
Xinge Peng, Zhen Fang, Ruibing Wang +8
University of Science and Technology of China · The Chinese University of Hong Kong