NeoWorld-3: From Pixels to Code-Defined Physical Worlds

Eachen Soong; Zijian Niu; Haoran Xu; Yihong Dong; Yunbo Wang; Xiaokang Yang

cs.CVOct 20, 2026

NeoWorld-3: From Pixels to Code-Defined Physical Worlds

Authors: Eachen Soong, Zijian Niu, Haoran Xu, Yihong Dong, Yunbo Wang, Xiaokang Yang

Organizations: Shanghai Jiao Tong University · Doxi AI

Abstract

Simulation-ready world reconstruction demands physical interactivity beyond visual realism, which existing black-box 3D generation fails to deliver. We present NeoWorld-3, a coding-agent framework that reconstructs interactive 3D worlds from visual observations via tunable, executable programs. At its core, NeoSDK provides the underlying computational engine for low-level geometries, part composition, differentiable parameter refinement, and physics validation. At the same time, NeoMCP exposes these capabilities as structured, tool-callable APIs for vision-language agents to visually inspect, edit code, and iteratively refine programs. Through a hierarchical coordinator-worker architecture, the framework scales seamlessly from standalone articulated objects to complex compositional scenes. Evaluations across object-level and scene-level reconstruction benchmarks demonstrate that NeoWorld-3 achieves state-of-the-art geometric fidelity and physical readiness. Crucially, manipulation trajectories planned within our reconstructed twins achieve successful zero-shot execution on a physical dual-arm robot, completing the Real-Sim-Real loop.

Figures & tables

	Representation			Geometry		Simulation-Ready		Self-Improvement
Method	Scene	Tunable	Parts	Watertight	Diff.	Articulation	Physics	RSI
Generative image-to-3D ( Pixal3D [ 12 ] )	$\checkmark$	$\times$	$\times$	$\triangle$	$\times$	$\times$	$\times$	$\times$
VLM-based ( PhysX-Anything [ 2 ] )	$\triangle$	$\times$	$\checkmark$	$\triangle$	$\times$	$\triangle$	$\checkmark$	$\times$
Coding-agent-based ( Articraft [ 30 ] )	$\checkmark$	$\triangle$	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$	$\times$
NeoWorld-3	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Capabilities of different simulation-ready reconstruction routes. Diff denotes differentiable optimization, and RSI denotes recursive self-improvement.

\checkmark

\triangle

, and

\times

denote full, partial, and absent support. We compare our approach with three representative methods from distinct reconstruction approaches.

Figure 2: Overview of the NeoWorld-3 framework. At the object level, vision-language agents construct tunable geometry programs, revising program structure based on real-sim feedback and optimizing geometric parameters through differentiable refinement, with parts and joints added for articulation. At the scene level, object subagents reconstruct individual assets, while a scene agent coordinates object integration and manages global layout using multi-view observations. Object- and scene-level feedback loops guide reconstruction and revision, producing simulation-ready scenes for sim-based policy optimization and sim-to-real deployment.

Figure 3: Qualitative comparison of object- and scene-level reconstructions. We compare NeoWorld-3 with VIGA and Codex+BlenderMCP. All methods use GPT-5.6 Terra for object-level reconstruction and GPT-6 Sol for scene-level reconstruction. NeoWorld-3 achieves higher geometric and visual fidelity at the object level, along with more accurate spatial layouts, object correspondence, proportions, materials, and colors at the scene level.

Method	Exec (%) $\uparrow$	SigLIP-2 $\uparrow$	DINOv3 $\uparrow$	Framed IoU $\uparrow$	CD $\downarrow$	Uni3D 3D–3D $\uparrow$	Uni3D Image–3D $\uparrow$
Codex+BlenderMCP [ 1 ]	97.64	0.8571	0.6245	0.5736	0.0747	0.5658	0.3064
VIGA [ 28 ]	99.52	0.8219	0.6351	0.6547	0.0680	0.5803	0.3084
Articraft [ 30 ]	100.00	0.8100	0.5692	0.5692	0.0719	0.5618	0.3087
Ours	100.00	0.8605	0.6606	0.7550	0.0365	0.6229	0.3202

Table 2: Quantitative comparison of object-level reconstruction on 3DCodeBench. We evaluate program execution rate (Exec), 2D visual representation similarity (SigLIP-2, DINOv3), aligned silhouette overlap (Framed IoU), reference-geometry agreement (Chamfer Distance, CD), and multimodal semantic alignment (Uni3D). NeoWorld-3 achieves the best overall performance across all criteria.

Method	AS-gIoU $\downarrow$	AS-CD $\downarrow$	AOR $\downarrow$	Type (%) $\uparrow$	Axis Error (rad) $\downarrow$	Pivot Error (norm.) $\downarrow$
Codex+BlenderMCP [ 1 ]	0.9126	0.2557	0.0035	87.16	0.3078	0.2846
Articraft [ 30 ]	0.8904	0.2031	0.0001	87.22	0.3522	0.1532
Ours	0.7792	0.1297	0.0000	90.13	0.2687	0.0962

Table 3: Quantitative evaluation of articulated object reconstruction on PartNet-Mobility. We evaluate geometric fidelity under kinematic configurations via Articulated-State generalized IoU error (AS-gIoU) and Chamfer Distance (AS-CD), self-collision via Articulated Overlap Ratio (AOR), alongside joint parameter accuracy (Joint Type Accuracy, Axis Error, and Pivot Position Error).

Method	DINOv3 $\uparrow$	Complete $\uparrow$	Layout $\uparrow$	Function $\uparrow$	P-Pen (%) $\downarrow$	OCD (cm) $\downarrow$
Codex+BlenderMCP [ 1 ]	0.7661	4.3000	4.1667	4.7000	9.21	53.0194
VIGA [ 28 ]	0.7888	4.5333	4.8000	4.6333	–	41.6275
Ours	0.8142	4.3667	4.9333	4.8667	0.67	25.9445

Table 4: Quantitative evaluation of multi-object scene reconstruction on RoboCasa365. We evaluate visual similarity (DINOv3), MLLM-assessed scene completeness (Complete), spatial arrangement (Layout), and functional plausibility (Function), alongside part-level penetration rate (P-Pen) and object center distance (OCD). P-Pen is not applicable to VIGA, which does not support articulation.

Configuration	DINOv3 $\uparrow$	Complete $\uparrow$	Layout $\uparrow$	Function $\uparrow$	P-Pen (%) $\downarrow$	OCD (cm) $\downarrow$
Direct GPT	0.6817	3.4583	3.5000	3.4583	45.68	107.8875
Codex	0.7513	4.2333	4.1333	4.2667	11.61	47.6677
NeoWorld-3 w/ BlenderMCP	0.7644	4.4667	4.2333	4.6333	45.41	47.4549
NeoWorld-3 (full)	0.8142	4.3667	4.9333	4.8667	0.67	25.9445

Table 5: Agent and tooling analysis on RoboCasa365. We compare direct generation, general-purpose coding agents, and NeoWorld-3 w/ BlenderMCP, which replaces NeoMCP and NeoSDK with BlenderMCP, against the full system. Complete, Layout, and Function are MLLM-assessed scores. OCD denotes object center distance in centimeters.

Figure 4: Real-Sim-Real manipulation. Left: wrist scanning of the physical scene. Middle: the plan executed in the reconstructed MuJoCo twin. Right: the same plan executed on the physical robot. The top row shows the initial scene and the bottom row the bowl being placed into the microwave.

Figure 5: Chained bimanual manipulation on the physical robot. (a) Initial scene. (b) One arm picks up the cube. (c) The cube is placed into the bowl. (d) The other arm opens the microwave door. (e) The bowl is placed inside the microwave. (f) The microwave door is closed.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Observation and measurement
Verb	What the call returns
read_observation	one photograph at full upright resolution with its raw camera and depth
measure	text-prompted masks with their metric lift: bounds, centroid, principal normal, floor footprint
sample_depth	world points under named pixels, with per-pixel validity
scene_planes	measured planes of the capture: normal, offset, extent, support
triangulate †	a world point from 2–64 calibrated views of one author-supplied landmark, with residuals

Appendix

Table 6: The scene author’s tools: the NeoMCP live-scene surface. Each verb carries a strict schema and returns a typed record (Section C.2 ). Verbs marked

\dagger

are mounted only when their option is enabled; each run records the tools it actually mounted. Diagnostic labels and admission reports are not independent reconstruction-quality verdicts.

Route	Input and Output	Checks and Configuration
Visual kinematic export	Declared links and joints $\rightarrow$ visual URDF	Joint sweeps and native/URDF playback agreement; visual meshes and kinematic frames.
Benchmark physics export	Delivered scene $\rightarrow$ USD with UsdPhysics	Rest contact, collision decomposition, body properties, and native joints. Indexed boundary checks and seam repair are enabled by --watertight (default off).
Robot twin	Reconstructed parts $\rightarrow$ configured simulator world	Collision proxies, body properties, and robot-frame alignment, as specified in Section B.1 .
SDK solid adaptation	Metric indexed mesh and adaptation plan $\rightarrow$ certified physical carriers	Closure, manifoldness, self-intersection, orientation, and positive volume; an independent SDK interface separate from scene export.

Appendix

Table 7: Physical-asset interfaces and validation checks. Summary of the four physical-asset routes, their input/output representations, and the corresponding validation procedures and configuration options.

Figure 6: Geometry validity and declared articulation. Left: SDK boundary inspection and explicit surface-preserving repair, distinct from metric adaptation and certification. Right: source-owned joint declarations, geometric refutation, and visual kinematic export checked against native playback. Sweep records distinguish measured checks from unavailable measurements. Bottom: scalar joint-position fitting on a declared axis has synthetic validation, with real second-configuration observations still pending.

Class	Finding	Description
Refuses the probe	rest_overlap	Overlapping geometry at rest.
	collision	Collision during the per-joint sweep.
	closure_gap	Closure gap at $q_{0}$ .
	combined_collision	Collision between links moved apart by two or more joints.
Needs a decision	cover_without_cavity	A covering link does not open onto a cavity.
	travel_leaves_parent	A sliding link leaves its parent at full travel.

Appendix

Table 8: Articulation probe findings. Classification of findings produced by kinematic validation.

Figure 7: Bounded tool development from completed tasks. Original advice and recorded traces inform a separate reviser. Candidate source helpers undergo native/twin validation and a matched task experiment before provisional development adoption. Each task retains a fixed library snapshot for reproducible execution.

Figure 8: Re-posing a finished reconstruction from a wrist scan. Outlines of the MuJoCo twin’s bodies before (thin) and after (thick) repose , projected into four of the eight undistorted wrist RGB-D views in raw sensor orientation (magenta: bowl; yellow: cube; no occlusion between bodies). The gripper fingers at the bottom of each view are the declared occluder. Both twins use the board-anchored base, so the outlines differ only by the admitted moves of Table 9 .

Body	Proposal $(x,y;\psi)$	Mean IoU	Weakest-view IoU	Outcome
Bowl	$-15.6,\,-6.2$ mm	$0.814\rightarrow 0.861$	$0.691\rightarrow 0.742$	admitted
Cube	$-14.4,\,+3.4$ mm; $-4.05^{\circ}$	$0.706\rightarrow 0.858$	$0.590\rightarrow 0.737$	admitted
Microwave	—	—	—	refused (unstable proposal)

Appendix

Table 9: Re-posing three bodies from one eight-view wrist scan. Proposal: the estimated planar pose update from masked depth observations, expressed in the scene frame. IoU: silhouette IoU against the assigned SAM 3 masks over the views used for each body, before

\rightarrow

after. Yaw is not fitted for the bowl and is folded modulo

90^{\circ}

for the cube. The estimates from the two view subsets for the bowl and cube agree within 2.9 mm and

1.4^{\circ}

, respectively, whereas those for the microwave differ by 9.0 mm and

9.3^{\circ}

Joint	Status	Gain (counts/N m)	Friction (N m)	Noise (N m)	$R^{2}$
J1	borrowed	223	0.29	0.31	–
J2	fitted	223	0.50	0.37	0.92
J3	fitted	238	0.50	0.32	0.80
J4	fitted	1505	0.06	0.05	0.92
J5	fitted	1530	0.05	0.05	0.91
J6	borrowed	1530	0.04	0.05	–

Appendix

Table 10: Current-to-torque model of the right arm , fitted on free-space motion. Noise is the single-frame residual; J1 and J6 borrow the gain of the motor of the same type.

Panel of Figure 9 — Figure 9: Compliant execution. Top: constraint force while opening a twin door to $30^{\circ}$ with the hinge believed up to 42 mm away, following the believed circle (left) and yielding to the measured force (right). Middle left: pre-grasp wrist inspection in the chained task (raw wrist frame, 640 $\times$ 480, not rotated); red is the planned cube, green the fitted cube. Middle right: a door-opening plan executed in the twin with the hardware’s loop and guards. Bottom: force estimated from motor currents in the table-contact test and its negative control.

Physics-Aware 3D Scene Reconstruction

See all 13 papers

NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation
Aug 25, 2026Yumeng He, Eachen Soong, Xiaotian Yang +53D Scene Generation Monocular 3D Reconstruction
Shanghai Jiao Tong University · Huazhong University of Science and Technology
Coding Agent Is Good As World Simulator
May 14, 2026Hongyu Wang, Jingquan Wang, Bocheng Zou +2Robotics Simulation World Models
Department of Mechanical & Aerospace Engineering, University of Wisconsin-Madison · School of Computer, Data, and Information Sciences, University of Wisconsin-Madison
WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations
Oct 22, 2026Tianling Xu, Weijie Wan, Honghao Su +9Robotics Simulation Digital Twins
Xspark AI · University College London · Tsinghua University +2

Aug 25, 2026J/K move · Enter open · S save

NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation

Yumeng He, Eachen Soong, Xiaotian Yang, Weijia Zhang+4

Shanghai Jiao Tong University · Huazhong University of Science and Technology

3D Scene Generation Monocular 3D Reconstruction

Open paper AskPDF arXiv

The advancement of Embodied AI necessitates high-quality simulation assets that faithfully mirror the real world. However, transforming raw visual observations into simulation-ready scenes remains challenging due to the lack of physical grounding and scene-level interactivity in current image-to-URDF methods. We propose NeoWorld-Pro, a framework that reformulates monocular scene reconstruction as procedural programming for interactive 3D environments. Leveraging the zero-shot reasoning and code synthesis capabilities of MLLMs, NeoWorld-Pro converts a single RGB image into executable programs specifying object geometry, articulation, and physical properties. A physics-in-the-loop mechanism then iteratively refines the generated programs by validating their execution in a physics engine, enforcing physically plausible articulations, valid object compositions and interactions, and accurate spatial relationships. Experiments show that NeoWorld-Pro outperforms open-loop and prior monocular reconstruction methods, while enabling complex downstream tasks such as stable stacking and fine-grained manipulation.

Abstract

Physics-Aware 3D Scene Reconstruction

Explore similar work

NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation

Coding Agent Is Good As World Simulator

WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations