NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation

Yumeng He; Eachen Soong; Xiaotian Yang; Weijia Zhang; Zanwei Zhou; Junru Gong; Xiaokang Yang; Yunbo Wang

cs.CVAug 25, 2026

NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation

Authors: Yumeng He, Eachen Soong, Xiaotian Yang, Weijia Zhang, Zanwei Zhou, Junru Gong, Xiaokang Yang, Yunbo Wang

Organizations: Shanghai Jiao Tong University · Huazhong University of Science and Technology

Abstract

The advancement of Embodied AI necessitates high-quality simulation assets that faithfully mirror the real world. However, transforming raw visual observations into simulation-ready scenes remains challenging due to the lack of physical grounding and scene-level interactivity in current image-to-URDF methods. We propose NeoWorld-Pro, a framework that reformulates monocular scene reconstruction as procedural programming for interactive 3D environments. Leveraging the zero-shot reasoning and code synthesis capabilities of MLLMs, NeoWorld-Pro converts a single RGB image into executable programs specifying object geometry, articulation, and physical properties. A physics-in-the-loop mechanism then iteratively refines the generated programs by validating their execution in a physics engine, enforcing physically plausible articulations, valid object compositions and interactions, and accurate spatial relationships. Experiments show that NeoWorld-Pro outperforms open-loop and prior monocular reconstruction methods, while enabling complex downstream tasks such as stable stacking and fine-grained manipulation.

Figures & tables

Figure 1: Demonstration of NeoWorld-Pro. Our framework ensures simulation-readiness by optimizing: (a-b) physically plausible intra-object articulations under external forces, (c-d) stable inter-object spatial configurations, and (e-f) scene viability for assembly and manipulation tasks.

Figure 2: Overview of the pipeline. Given a monocular RGB image, NeoWorld-Pro first performs scene parsing and reasoning to establish a scene tree and layout. Each foreground object is then programmed into an executable Blender script to generate geometry and articulation. A two-level physics-in-the-loop mechanism ensures simulation-readiness: (1) an object-level loop for resting stability and joint executability, and (2) a CEM-based scene-level loop that optimizes scene plausibility, such as relative object scale and placement, collision-free spatial consistency, and stability.

Figure 3: Physics-in-the-loop refinement improves resting stability and joint executability. (a) Increasing static friction torque $\tau_{s}$ , dynamic friction torque $\tau_{d}$ , and passive viscous damping $b$ reduces the maximum angular drift of the easel braces under gravity from $37.24^{\circ}$ to $0.16^{\circ}$ . (b) Reducing $\tau_{s}$ , $\tau_{d}$ , and flap mass $m$ increases joint range coverage from $5.56\%$ to $99.91\%$ under force limit.

Figure 4: Object-level reconstruction results of articulated assets. Compared with existing methods, NeoWorld-Pro recovers more faithful geometry, appearance, and articulation structure.

Figure 5: Qualitative scene reconstruction on our synthetic scene dataset. Each row compares reconstructions from the same monocular input. NeoWorld-Pro more faithfully preserves object identity, relative scale and orientation, and inter-object layout across both tabletop and larger workspace scenes. “N/A” denotes examples outside TabletopGen’s tabletop setting.

Figure 6: Reconstruction from real-world images. The results illustrate generalization to real-world appearance and clutter while retaining functional articulations.

Method	Total	#Pred	#Hit	Miss $\downarrow$	Type $\downarrow$	Axis $\downarrow$	Pivot $\downarrow$
Articulate-Anything [ 10 ]	245	170	95	61.22	69.39	63.25	0.32
PhysX-Anything [ 2 ]	245	192	126	48.57	58.78	45.80	0.37
URDF-Anything+ [ 30 ]	245	96	74	69.80	82.86	50.41	0.40
Articraft [ 40 ]	245	304	190	22.45	29.39	30.14	0.23
Ours	245	268	216	11.83	18.78	28.52	0.19

Table 1: Object-level evaluation of joint articulation and kinematics. #Total , #Pred , and #Hit are the number of ground-truth joints, predicted joints, and matched hits. We report the joint miss rate ( Miss ), joint type identification error ( Type ), orientation error ( Axis ), and positional error ( Pivot ).

Full benchmark
Method	S-Pen $\downarrow$	P-Pen $\downarrow$	S-Stab $\uparrow$	O-Stab $\uparrow$	CLIP $\uparrow$	DINOv2 $\uparrow$	Complete $\uparrow$	Layout $\uparrow$	Function $\uparrow$
VIGA [ 36 ]	56.67	63.37	23.33	27.33	0.8831	0.6087	4.70	3.95	3.45
SAGE [ 31 ]	6.67	2.41	70.00	91.60	0.8729	0.5379	3.80	2.60	2.90
Ours	0.00	0.00	100.00	100.00	0.9128	0.7855	6.20	5.35	5.58
Tabletop subset
VIGA [ 36 ]	54.55	67.26	18.18	25.66	0.8764	0.5950	4.55	3.91	3.00

Table 2: Scene-level results on the full benchmark and the 11-scene tabletop subset. S-Pen, P-Pen, S-Stab, and O-Stab denote scene-level (inter-object), part-level (inner-object) penetration rate, and scene-/object-level stability, respectively. These metrics are percentages. MLLM scores (the last three columns) use a

1

–

7

scale.

3D Scene Generation

3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.

See all 190 papers

NeoWorld-3: From Pixels to Code-Defined Physical Worlds
Oct 20, 2026Eachen Soong, Zijian Niu, Haoran Xu +3Physics-Aware 3D Scene Reconstruction 3D Reconstruction
Shanghai Jiao Tong University · Doxi AI
SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images
Oct 20, 2026Rundong Luo, Yunong Liu, Di Cao +4Physics-Aware 3D Scene Reconstruction 3D Reconstruction
Luma AI · Cornell University
SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects
May 19, 2026Puyi Wang, Yuhao Wang, Linjie Li +43D Asset Generation LLM-Based Program Synthesis
1The Chinese University of Hong Kong · 2Shanghai Jiao Tong University · 3Shanghai AI Laboratory +2

Oct 20, 2026J/K move · Enter open · S save

NeoWorld-3: From Pixels to Code-Defined Physical Worlds

Eachen Soong, Zijian Niu, Haoran Xu, Yihong Dong+2

Shanghai Jiao Tong University · Doxi AI

Physics-Aware 3D Scene Reconstruction 3D Reconstruction Object-Centric 3D Reconstruction

Open paper AskPDF arXiv

Simulation-ready world reconstruction demands physical interactivity beyond visual realism, which existing black-box 3D generation fails to deliver. We present NeoWorld-3, a coding-agent framework that reconstructs interactive 3D worlds from visual observations via tunable, executable programs. At its core, NeoSDK provides the underlying computational engine for low-level geometries, part composition, differentiable parameter refinement, and physics validation. At the same time, NeoMCP exposes these capabilities as structured, tool-callable APIs for vision-language agents to visually inspect, edit code, and iteratively refine programs. Through a hierarchical coordinator-worker architecture, the framework scales seamlessly from standalone articulated objects to complex compositional scenes. Evaluations across object-level and scene-level reconstruction benchmarks demonstrate that NeoWorld-3 achieves state-of-the-art geometric fidelity and physical readiness. Crucially, manipulation trajectories planned within our reconstructed twins achieve successful zero-shot execution on a physical dual-arm robot, completing the Real-Sim-Real loop.

Abstract

3D Scene Generation

Explore similar work

NeoWorld-3: From Pixels to Code-Defined Physical Worlds

SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images

SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects