Code2Worlds-2: Physically Plausible Agent Harness for Embodied World Simulation

Haoyu Zhang; Siming Li; Zeyu Zhang; Yi Zhang; Tat-Seng Chua; Hao Tang

cs.ROOct 13, 2026

Code2Worlds-2: Physically Plausible Agent Harness for Embodied World Simulation

Authors: Haoyu Zhang, Siming Li, Zeyu Zhang, Yi Zhang, Tat-Seng Chua, Hao Tang

Organizations: Equal contribution. · School of Computer Science, Peking University, Beijing 100871, China. · Project lead. · School of Computing, National University of Singapore, Singapore.

Abstract

Recent coding large language models (LLMs) can synthesize dynamic 4D worlds from natural language, but the resulting content remains a visual artifact. It lacks physical semantics such as the collision geometry, mass, friction, inertia, and joint structure, required to enable it to simulate the physical world, and its dynamics are scripted to look plausible rather than designed to execute. What is needed is an executable world: one that a physics engine can load and step, resolving contact physically. We present Code2Worlds-2, an extension of Code2Worlds that turns 4D world generation into an embodied world simulation. We keep the original dual-stream generation pipeline intact and append two modules. First, a SimReady Scene Compiler that turns a generated visual scene into a simulator-compatible representation: it assigns collision meshes via convex decomposition, estimates physical attributes with a vision-language model, infers a lightweight support graph, instantiates the engine's contact, friction, and dynamics constraints, and applies a gravity settling pass, before exporting the scene as MuJoCo MJCF. Second, a Simulator-in-the-Loop 4D Refiner that implements an automated agent harness-a compile, simulate, critique and refine loop-that converts visual dynamics into executable rigid and articulated dynamics and evolves the physical settings rather than only the code, replacing the purely visual motion critic with a simulator critic that inspects penetration depth, collision pairs, stability, contact forces, joint limits, and robot reachability. We further propose SimCode4D, a state-based benchmark with five task templates for evaluating embodied simulation. Experiments on SimCode4D and across natural, indoor, industrial, urban, and service scenes show that Code2Worlds-2 produces physically plausible, simulator-compatible, and robot-interactable scenes.

Figures & tables

Fig. 1: The feedback loops of the two systems. (a) Code2Worlds [ 1 ] corrects a scene by rewriting its code from rendered images—a visual loop, which cannot guarantee physical validity. (b) Code2Worlds-2 instead closes the loop through the simulator: Module 1 compiles a SimReady scene, Module 2 critiques the rollout, and simulator feedback guides the revisions to the physical settings and dynamics.

Fig. 2: Overview of Code2Worlds-2. Two modules are appended to the original Code2Worlds pipeline to produce a simulator-compatible, interactable 4D world, and the illustrated scenes span five domains.

Fig. 3: The Code2Worlds-2 agent harness. The generation agent closes a loop through the simulator: it emits code and a dynamics program, which are compiled, executed, and measured by physical signals; the fused verdict then revises the physical settings until validation passes.

Fig. 4: Schematic comparison of default (top) and SimReady (bottom) scene exports. A single convex hull fills concave spaces, creating phantom support and embedding the box. CoACD colliders preserve these spaces, allowing the sphere to rest on the seat after settling and the box to fit beneath the table. Dashed outlines denote collision geometry; green dots mark contact points.

Fig. 5: The stages of Module 1. Collision geometry is obtained by convex decomposition; density, friction, and restitution are estimated by a vision–language model, while mass and inertia are computed from mesh geometry; a support graph separates supported from floating bodies; and the engine’s contact model, together with a gravity settling pass, produces a simulator-compatible scene exported as MJCF.

Fig. 6: Dev15 benchmark composition and execution diagnostics. (a) The 15 cases span five domains and three realized task families (N: navigation, D: dynamic interaction, P: pick/transport). (b) All 15 cases have complete executable contracts; 12 additionally retain their authoring scenes. (c) For each domain, bars show the 95th percentile of per-episode penetration P95 values, whereas diamonds show the maximum penetration observed in any episode; both remain below the frozen $1\,\mathrm{mm}$ gate, and lower is better. (d) Maximum penetration, Coulomb utilization, generalized-force residual, and quaternion error are normalized by their failure thresholds; lower is better, and all values remain below one.

Measure	Result	Evidence and acceptance criterion
Artifact readiness
Registered cases with complete contract quartets	15/15	Three cases per domain; schema and cross-references are valid
Archived and verified authoring source scenes	12/15	Compiled M1 worlds and source pointers are retained for all 15 cases
Independently loadable M1 worlds	15/15	All M1 world endpoints load under the frozen MuJoCo build
Executable runtime packages	15/15	All task-runtime endpoints load and execute
Case-specific probe–validator pairs	15/15	Unique hash-bound paths are frozen for every case

TABLE I: Implementation and execution results on the frozen SimCode4D-Dev15-RC1 suite. The registered matrix comprises 225 executions over 75 unique case–physics pairs.

Method / stage	Build	Task	Physical	RTS	Repairs
Fixed-source progression
Default export	33/36	63/180	75/180	45/180	0
$+$ SimReady scene compilation	33/36	62/180	93/180	50/180	0
$+$ SimReady scene compilation $+$ feedback repair	34/36	69/180	108/180	60/180	1.61
Matched-prompt comparison
Code2Worlds [ 1 ]	3/3	5/15	0/15	0/15	–

TABLE II: Generated-world acceptance under two non-pooled protocols.

Fig. 7: Representative task rollouts from the five evaluation domains. Rows show one case per domain, and columns show six temporally ordered keyframes.

Task family	Cases	Default	Enriched	Repaired
Navigation	4	45/60	35/60	35/60
Pick/transport	2	0/30	14/30	14/30
Dynamic	6	0/90	1/90	11/90

TABLE III: Task-family RTS for the 12 fixed-source cases. Each case contributes three generated packages and five evaluation physics seeds.

Ablated variant	Measure	Full	Ablated
Uniform attributes	RTS	69/75	54/75
	Physical pass	69/75	55/75
Single convex hull	RTS	15/15	10/15
	Physical pass	15/15	11/15
No pose stabilization	Stable	8/12	1/12
	Penetration-safe	11/12	0/12

TABLE IV: Module 1 ablations on paired eligible inputs. The first two interventions report held-out physics trials; pose stabilization reports initial object states. Higher is better for every measure.

Feedback rule	Build	Task	Physical	RTS
No repair	33/36	62/180	93/180	50/180
Visual only	36/36	84/180	112/180	67/180
State only	36/36	96/180	116/180	81/180
Fused, fixed budget	35/36	81/180	99/180	62/180

TABLE V: Module 2 feedback ablation from 36 matched enriched starts. Build is measured over 36 worlds; task, physical, and RTS outcomes are measured over 180 held-out physics trials. Failed builds remain in the denominator.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Case	Domain	Robot	Task family	Goal (frozen prompt)
Indoor-1 vacuum living room	Indoor	mobile vacuum	Obstacle nav.	Drive the vacuum east through the living-room corridor, enter the frozen goal, and remain stopped for 0.5 seconds.
Indoor-2 go2 living room	Indoor	Unitree Go2 quadruped	Obstacle nav.	Traverse the living room east with the Go2, stay in the frozen corridor, enter the goal, and stop.
Indoor-3 vacuum west	Indoor	mobile vacuum	Obstacle nav.	Drive the vacuum west through the indoor corridor, enter the frozen goal, and remain stopped for 0.5 seconds.
Industrial-1 agv aisle	Industrial	four-wheel AGV	Obstacle nav.	Drive the four-wheel AGV east along the industrial aisle, hold the lane, enter the goal, and stop.
Industrial-2 agv cargo	Industrial	four-wheel AGV with welded cargo	Dynamic interaction	Transport the physically modeled cargo east with the AGV, retain the declared weld attachment, enter the cargo goal, and stop.
Industrial-3 weld	Industrial	UR5e welding arm	Dynamic interaction	Trace the frozen weld seam continuously with the UR5e tool tip and finish at rest.

Appendix

TABLE B.1: The frozen SimCode4D-Dev15-RC1 case index. Cases are listed per domain in the frozen suite order; each row gives the scene, the benchmark-owned robot, the realized task family, and the natural-language goal frozen in case_spec.json .

Case ID	Room, target, and support	Task family	Goal criterion
E2E-LIVING-PUSH	Living room; ceramic mug on wooden side table	Contact push	$0.15\,\mathrm{m}$ tabletop displacement
E2E-DINING-TRANSPORT	Dining room; ceramic mug on dining table	Pick and transport	$0.15\,\mathrm{m}$ tabletop transfer and $0.05\,\mathrm{m}$ lift
E2E-KITCHEN-PUSH	Kitchen; ceramic bowl on kitchen worktop	Contact push	$0.15\,\mathrm{m}$ worktop displacement

Appendix

TABLE E.1: Frozen matched-prompt cases. The task family and numeric goal are post-generation evaluation metadata, not additional generator instructions.

Case	Target/support	C2W-A	+M1/M2	SS
Living	Mug/side table	$-x$	$-x$	$-y$
Dining	Mug/dining table	$+x$	$+x$	$-x$
Kitchen	Bowl/worktop	$-x$	$+x$	$+x$

Appendix

TABLE E.2: Frozen source-relative target goals. C2W-A denotes the Code2Worlds uniform MuJoCo import; SS denotes SceneSmith’s official export. Each goal is

0.15\,\mathrm{m}

from the initial target center along the listed direction.

Seed	Mass scale	Friction scale
401	1.01934	0.97693
409	0.98103	0.97091
419	1.04753	1.05625
421	0.96823	0.94074
431	0.95579	1.06771

Appendix

TABLE E.3: Held-out target-body mass and target-contact friction multipliers for the three matched-prompt cases.

Case	Condition	Task	Physical	RTS
Living	C2W-A	5/5	0/5	0/5
	+M1/M2	5/5	5/5	5/5
	SS	0/5	5/5	0/5
Dining	C2W-A	0/5	0/5	0/5
	+M1/M2	5/5	5/5	5/5
	SS	0/5	5/5	0/5

Appendix

TABLE E.4: Per-case disaggregation of the three matched-prompt SimCode4D experimental cases. Every row has five held-out physics trials; all nine packages build. C2W-A is Code2Worlds with the uniform MuJoCo import, +M1/M2 reuses its visual output, and SS is SceneSmith’s official export.

Physics-Based Simulation

Physics-based simulation models physical systems with equations and numerical solvers, increasingly augmented by learned models for speed and control.

See all 227 papers

Coding Agent Is Good As World Simulator
May 14, 2026Hongyu Wang, Jingquan Wang, Bocheng Zou +2Robotics Simulation World Models
Department of Mechanical & Aerospace Engineering, University of Wisconsin-Madison · School of Computer, Data, and Information Sciences, University of Wisconsin-Madison
WorldSolver: Can LLM Agents Simulate the Physical Dynamics via Solver Generation?
Oct 6, 2026Siru Jiang, Yongzhe Lyu, Shuo Lu +6Physics-Based Simulation Code Generation
NLPR & MAIS, CASIA · PKU · Huawei Noah’s Ark Lab
Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
Oct 3, 2026Jinzhou Tang, Zijun Zhang, Jing Yang +10Interactive World Models Coding Agents
Aether AI · University of California, San Diego

May 14, 2026J/K move · Enter open · S save

Coding Agent Is Good As World Simulator

Hongyu Wang, Jingquan Wang, Bocheng Zou, Radu Serban+1

Department of Mechanical & Aerospace Engineering, University of Wisconsin-Madison · School of Computer, Data, and Information Sciences, University of Wisconsin-Madison

Robotics Simulation World Models Agentic Code Generation Physics-Based Simulation Embodied World Models

Open paper AskPDF arXiv

World models have emerged as a powerful paradigm for building interactive simulation environments, with recent video-based approaches demonstrating impressive progress in generating visually plausible dynamics. However, because these models typically infer dynamics from video and represent them in latent states, they do not explicitly enforce physical constraints. As a result, the generated video rollouts are not physically plausible, exhibiting unstable contacts, distorted shapes, or inconsistent motion. In this paper, we present an agentic framework constructing physics-based world models through executable simulation code. The framework coordinates planning, code generation, visual review, and physics analysis agents. The planning agent converts the natural language prompt into a structured scene plan, the code agent implements it as executable simulation code, and the visual review agent provide visual feedback while the physics analysis agent checks physical consistency. The code is iteratively revised based on the feedback until the simulation matches the prompt reqirements and physical constraints. Experimental results show that our framework outperforms advanced video-based models in physical accuracy, instruction fidelity and visual quality, which could be applied to various scenarios including driving simulation and embodied robot tasks.

Abstract

Physics-Based Simulation

Explore similar work

Coding Agent Is Good As World Simulator

WorldSolver: Can LLM Agents Simulate the Physical Dynamics via Solver Generation?

Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos