cs.ROOct 13, 2026

Code2Worlds-2: Physically Plausible Agent Harness for Embodied World Simulation

Authors: Haoyu Zhang, Siming Li, Zeyu Zhang, Yi Zhang, Tat-Seng Chua, Hao Tang

Organizations: Equal contribution. · School of Computer Science, Peking University, Beijing 100871, China. · Project lead. · School of Computing, National University of Singapore, Singapore.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent coding large language models (LLMs) can synthesize dynamic 4D worlds from natural language, but the resulting content remains a visual artifact. It lacks physical semantics such as the collision geometry, mass, friction, inertia, and joint structure, required to enable it to simulate the physical world, and its dynamics are scripted to look plausible rather than designed to execute. What is needed is an executable world: one that a physics engine can load and step, resolving contact physically. We present Code2Worlds-2, an extension of Code2Worlds that turns 4D world generation into an embodied world simulation. We keep the original dual-stream generation pipeline intact and append two modules. First, a SimReady Scene Compiler that turns a generated visual scene into a simulator-compatible representation: it assigns collision meshes via convex decomposition, estimates physical attributes with a vision-language model, infers a lightweight support graph, instantiates the engine's contact, friction, and dynamics constraints, and applies a gravity settling pass, before exporting the scene as MuJoCo MJCF. Second, a Simulator-in-the-Loop 4D Refiner that implements an automated agent harness-a compile, simulate, critique and refine loop-that converts visual dynamics into executable rigid and articulated dynamics and evolves the physical settings rather than only the code, replacing the purely visual motion critic with a simulator critic that inspects penetration depth, collision pairs, stability, contact forces, joint limits, and robot reachability. We further propose SimCode4D, a state-based benchmark with five task templates for evaluating embodied simulation. Experiments on SimCode4D and across natural, indoor, industrial, urban, and service scenes show that Code2Worlds-2 produces physically plausible, simulator-compatible, and robot-interactable scenes.

Figures & tables

Fig. 1: The feedback loops of the two systems. (a) Code2Worlds [ 1 ] corrects a scene by rewriting its code from rendered images—a visual loop, which cannot guarantee physical validity. (b) Code2Worlds-2 instead closes the loop through the simulator: Module 1 compiles a SimReady scene, Module 2 critiques the rollout, and simulator feedback guides the revisions to the physical settings and dynamics.
Fig. 2: Overview of Code2Worlds-2. Two modules are appended to the original Code2Worlds pipeline to produce a simulator-compatible, interactable 4D world, and the illustrated scenes span five domains.
Fig. 3: The Code2Worlds-2 agent harness. The generation agent closes a loop through the simulator: it emits code and a dynamics program, which are compiled, executed, and measured by physical signals; the fused verdict then revises the physical settings until validation passes.
Fig. 4: Schematic comparison of default (top) and SimReady (bottom) scene exports. A single convex hull fills concave spaces, creating phantom support and embedding the box. CoACD colliders preserve these spaces, allowing the sphere to rest on the seat after settling and the box to fit beneath the table. Dashed outlines denote collision geometry; green dots mark contact points.
Fig. 5: The stages of Module 1. Collision geometry is obtained by convex decomposition; density, friction, and restitution are estimated by a vision–language model, while mass and inertia are computed from mesh geometry; a support graph separates supported from floating bodies; and the engine’s contact model, together with a gravity settling pass, produces a simulator-compatible scene exported as MJCF.
Fig. 6: Dev15 benchmark composition and execution diagnostics. (a) The 15 cases span five domains and three realized task families (N: navigation, D: dynamic interaction, P: pick/transport). (b) All 15 cases have complete executable contracts; 12 additionally retain their authoring scenes. (c) For each domain, bars show the 95th percentile of per-episode penetration P95 values, whereas diamonds show the maximum penetration observed in any episode; both remain below the frozen 1 mm1\,\mathrm{mm} gate, and lower is better. (d) Maximum penetration, Coulomb utilization, generalized-force residual, and quaternion error are normalized by their failure thresholds; lower is better, and all values remain below one.
MeasureResultEvidence and acceptance criterion
Artifact readiness
Registered cases with complete contract quartets15/15Three cases per domain; schema and cross-references are valid
Archived and verified authoring source scenes12/15Compiled M1 worlds and source pointers are retained for all 15 cases
Independently loadable M1 worlds15/15All M1 world endpoints load under the frozen MuJoCo build
Executable runtime packages15/15All task-runtime endpoints load and execute
Case-specific probe–validator pairs15/15Unique hash-bound paths are frozen for every case
TABLE I: Implementation and execution results on the frozen SimCode4D-Dev15-RC1 suite. The registered matrix comprises 225 executions over 75 unique case–physics pairs.
Method / stageBuildTaskPhysicalRTSRepairs
Fixed-source progression
Default export33/3663/18075/18045/1800
++ SimReady scene compilation33/3662/18093/18050/1800
++ SimReady scene compilation ++ feedback repair34/3669/180108/18060/1801.61
Matched-prompt comparison
Code2Worlds [ 1 ]3/35/150/150/15–
TABLE II: Generated-world acceptance under two non-pooled protocols.
Fig. 7: Representative task rollouts from the five evaluation domains. Rows show one case per domain, and columns show six temporally ordered keyframes.
Task familyCasesDefaultEnrichedRepaired
Navigation445/6035/6035/60
Pick/transport20/3014/3014/30
Dynamic60/901/9011/90
TABLE III: Task-family RTS for the 12 fixed-source cases. Each case contributes three generated packages and five evaluation physics seeds.
Ablated variantMeasureFullAblated
Uniform attributesRTS69/7554/75
 Physical pass69/7555/75
Single convex hullRTS15/1510/15
 Physical pass15/1511/15
No pose stabilizationStable8/121/12
 Penetration-safe11/120/12
TABLE IV: Module 1 ablations on paired eligible inputs. The first two interventions report held-out physics trials; pose stabilization reports initial object states. Higher is better for every measure.
Feedback ruleBuildTaskPhysicalRTS
No repair33/3662/18093/18050/180
Visual only36/3684/180112/18067/180
State only36/3696/180116/18081/180
Fused, fixed budget35/3681/18099/18062/180
TABLE V: Module 2 feedback ablation from 36 matched enriched starts. Build is measured over 36 worlds; task, physical, and RTS outcomes are measured over 180 held-out physics trials. Failed builds remain in the denominator.
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

CaseDomainRobotTask familyGoal (frozen prompt)
Indoor-1 vacuum living roomIndoormobile vacuumObstacle nav.Drive the vacuum east through the living-room corridor, enter the frozen goal, and remain stopped for 0.5 seconds.
Indoor-2 go2 living roomIndoorUnitree Go2 quadrupedObstacle nav.Traverse the living room east with the Go2, stay in the frozen corridor, enter the goal, and stop.
Indoor-3 vacuum westIndoormobile vacuumObstacle nav.Drive the vacuum west through the indoor corridor, enter the frozen goal, and remain stopped for 0.5 seconds.
Industrial-1 agv aisleIndustrialfour-wheel AGVObstacle nav.Drive the four-wheel AGV east along the industrial aisle, hold the lane, enter the goal, and stop.
Industrial-2 agv cargoIndustrialfour-wheel AGV with welded cargoDynamic interactionTransport the physically modeled cargo east with the AGV, retain the declared weld attachment, enter the cargo goal, and stop.
Industrial-3 weldIndustrialUR5e welding armDynamic interactionTrace the frozen weld seam continuously with the UR5e tool tip and finish at rest.
Appendix
TABLE B.1: The frozen SimCode4D-Dev15-RC1 case index. Cases are listed per domain in the frozen suite order; each row gives the scene, the benchmark-owned robot, the realized task family, and the natural-language goal frozen in case_spec.json .
Case IDRoom, target, and supportTask familyGoal criterion
E2E-LIVING-PUSHLiving room; ceramic mug on wooden side tableContact push0.15 m0.15\,\mathrm{m} tabletop displacement
E2E-DINING-TRANSPORTDining room; ceramic mug on dining tablePick and transport0.15 m0.15\,\mathrm{m} tabletop transfer and 0.05 m0.05\,\mathrm{m} lift
E2E-KITCHEN-PUSHKitchen; ceramic bowl on kitchen worktopContact push0.15 m0.15\,\mathrm{m} worktop displacement
Appendix
TABLE E.1: Frozen matched-prompt cases. The task family and numeric goal are post-generation evaluation metadata, not additional generator instructions.
CaseTarget/supportC2W-A+M1/M2SS
LivingMug/side table−x-x−x-x−y-y
DiningMug/dining table+x+x+x+x−x-x
KitchenBowl/worktop−x-x+x+x+x+x
Appendix
TABLE E.2: Frozen source-relative target goals. C2W-A denotes the Code2Worlds uniform MuJoCo import; SS denotes SceneSmith’s official export. Each goal is 0.15 m0.15\,\mathrm{m} from the initial target center along the listed direction.
SeedMass scaleFriction scale
4011.019340.97693
4090.981030.97091
4191.047531.05625
4210.968230.94074
4310.955791.06771
Appendix
TABLE E.3: Held-out target-body mass and target-contact friction multipliers for the three matched-prompt cases.
CaseConditionTaskPhysicalRTS
LivingC2W-A5/50/50/5
 +M1/M25/55/55/5
 SS0/55/50/5
DiningC2W-A0/50/50/5
 +M1/M25/55/55/5
 SS0/55/50/5
Appendix
TABLE E.4: Per-case disaggregation of the three matched-prompt SimCode4D experimental cases. Every row has five held-out physics trials; all nine packages build. C2W-A is Code2Worlds with the uniform MuJoCo import, +M1/M2 reuses its visual output, and SS is SceneSmith’s official export.
More in this topic

Physics-Based Simulation

Physics-based simulation models physical systems with equations and numerical solvers, increasingly augmented by learned models for speed and control.

See all 227 papers

Explore similar work

CardsList
May 14, 2026

Coding Agent Is Good As World Simulator

World models have emerged as a powerful paradigm for building interactive simulation environments, with recent video-based approaches demonstrating impressive progress in generating visually plausible dynamics. However, because these models typically infer dynamics from video and represent them in latent states, they do not explicitly enforce physical constraints. As a result, the generated video rollouts are not physically plausible, exhibiting unstable contacts, distorted shapes, or inconsistent motion. In this paper, we present an agentic framework constructing physics-based world models through executable simulation code. The framework coordinates planning, code generation, visual review, and physics analysis agents. The planning agent converts the natural language prompt into a structured scene plan, the code agent implements it as executable simulation code, and the visual review agent provide visual feedback while the physics analysis agent checks physical consistency. The code is iteratively revised based on the feedback until the simulation matches the prompt reqirements and physical constraints. Experimental results show that our framework outperforms advanced video-based models in physical accuracy, instruction fidelity and visual quality, which could be applied to various scenarios including driving simulation and embodied robot tasks.

Hongyu Wang, Jingquan Wang, Bocheng Zou +2

Department of Mechanical & Aerospace Engineering, University of Wisconsin-Madison · School of Computer, Data, and Information Sciences, University of Wisconsin-Madison

Robotics SimulationWorld ModelsAgentic Code GenerationPhysics-Based SimulationEmbodied World Models
AskPDF
Oct 6, 2026

WorldSolver: Can LLM Agents Simulate the Physical Dynamics via Solver Generation?

LLM-based agents are increasingly advancing scientific and engineering problem solving, with physics simulation emerging as a challenging yet practical testbed for reproducing complex physical phenomena with application in embodied AI, games and films. As the workhorse of such simulation, a solver computes how the state of a dynamic system evolves over time. Building such solvers requires physical understanding to identify appropriate models, mathematical reasoning to formulate the underlying dynamics, and software engineering to implement them as executable code, yet this capability of LLM agents remains underexplored. To this end, we introduce WorldSolver, a benchmark of 168 simulation tasks derived from physical phenomena in 61 classic computer graphics papers, spanning 7 physical domains. Each task contains a code scaffold that provides a fixed simulation environment for the scene, with the solver implementation left for the agent to complete. Specifically, we evaluate them along three dimensions: Execution Checks for successful execution, Visual Fidelity for reproducing the intended dynamic behavior in the rendered simulation, and Physical Plausibility for physics-grounded verification of the generated dynamics. Experiments on frontier agents reveal that producing executable solvers is difficult itself, and satisfying visual and physical correctness is even harder. GPT-5.6-Sol and Claude-Opus-5 perform comparatively better than the other evaluated agents, yet achieve overall scores of only 48.7% and 46.7%, respectively. WorldSolver is an early step toward agentic solver generation, and we hope it helps drive progress toward agents that can faithfully simulate the dynamic physical world. Code is available at https://github.com/sirujiang/WorldSolver.

Siru Jiang, Yongzhe Lyu, Shuo Lu +6

NLPR & MAIS, CASIA · PKU · Huawei Noah’s Ark Lab

Physics-Based SimulationCode GenerationAI Agent BenchmarksLLM Agent Evaluation
AskPDF
Oct 3, 2026

Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos

Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on manual environment construction and calibration. We study whether frontier foundation models and coding agents can automate this process end to end. We formulate autonomous video-to-simulation as a software engineering task in which an agent observes an embodied video, constructs the corresponding simulated environment and robot behavior, and iteratively refines the result through execution feedback. To evaluate this capability, we introduce Video2World, a benchmark comprising 222 reconstruction instances derived from 189 robot and human demonstration videos. Video2World measures reconstructed worlds along geometric fidelity, dynamic fidelity, and functional correctness, capturing spatial perception, physical reasoning, and executable interaction. Evaluating 9 frontier coding-agent systems reveals a sharp improvement in Task success beginning with Claude Opus 5, rising from below 5% to over 15%, while substantial gaps to human-assisted reconstruction remain. We further find that worlds that look better could work worse: better visual fidelity does not always lead to higher task success. This echoes the broader gap between perceptual realism and factual correctness observed in generative models.

Jinzhou Tang, Zijun Zhang, Jing Yang +10

Aether AI · University of California, San Diego

Interactive World ModelsCoding AgentsWorld Model EvaluationAI Agent Benchmarks
AskPDF