The advancement of Embodied AI necessitates high-quality simulation assets that faithfully mirror the real world. However, transforming raw visual observations into simulation-ready scenes remains challenging due to the lack of physical grounding and scene-level interactivity in current image-to-URDF methods. We propose NeoWorld-Pro, a framework that reformulates monocular scene reconstruction as procedural programming for interactive 3D environments. Leveraging the zero-shot reasoning and code synthesis capabilities of MLLMs, NeoWorld-Pro converts a single RGB image into executable programs specifying object geometry, articulation, and physical properties. A physics-in-the-loop mechanism then iteratively refines the generated programs by validating their execution in a physics engine, enforcing physically plausible articulations, valid object compositions and interactions, and accurate spatial relationships. Experiments show that NeoWorld-Pro outperforms open-loop and prior monocular reconstruction methods, while enabling complex downstream tasks such as stable stacking and fine-grained manipulation.
Figures & tables
Figure 1: Demonstration of NeoWorld-Pro. Our framework ensures simulation-readiness by optimizing: (a-b) physically plausible intra-object articulations under external forces, (c-d) stable inter-object spatial configurations, and (e-f) scene viability for assembly and manipulation tasks.
Figure 2: Overview of the pipeline. Given a monocular RGB image, NeoWorld-Pro first performs scene parsing and reasoning to establish a scene tree and layout. Each foreground object is then programmed into an executable Blender script to generate geometry and articulation. A two-level physics-in-the-loop mechanism ensures simulation-readiness: (1) an object-level loop for resting stability and joint executability, and (2) a CEM-based scene-level loop that optimizes scene plausibility, such as relative object scale and placement, collision-free spatial consistency, and stability.
Figure 3: Physics-in-the-loop refinement improves resting stability and joint executability. (a) Increasing static friction torque τs , dynamic friction torque τd , and passive viscous damping b reduces the maximum angular drift of the easel braces under gravity from 37.24∘ to 0.16∘ . (b) Reducing τs , τd , and flap mass m increases joint range coverage from 5.56% to 99.91% under force limit.
Figure 4: Object-level reconstruction results of articulated assets. Compared with existing methods, NeoWorld-Pro recovers more faithful geometry, appearance, and articulation structure.
Figure 5: Qualitative scene reconstruction on our synthetic scene dataset. Each row compares reconstructions from the same monocular input. NeoWorld-Pro more faithfully preserves object identity, relative scale and orientation, and inter-object layout across both tabletop and larger workspace scenes. “N/A” denotes examples outside TabletopGen’s tabletop setting.
Figure 6: Reconstruction from real-world images. The results illustrate generalization to real-world appearance and clutter while retaining functional articulations.
Method
Total
#Pred
#Hit
Miss ↓
Type ↓
Axis ↓
Pivot ↓
Articulate-Anything [ 10 ]
245
170
95
61.22
69.39
63.25
0.32
PhysX-Anything [ 2 ]
245
192
126
48.57
58.78
45.80
0.37
URDF-Anything+ [ 30 ]
245
96
74
69.80
82.86
50.41
0.40
Articraft [ 40 ]
245
304
190
22.45
29.39
30.14
0.23
Ours
245
268
216
11.83
18.78
28.52
0.19
Table 1: Object-level evaluation of joint articulation and kinematics. #Total , #Pred , and #Hit are the number of ground-truth joints, predicted joints, and matched hits. We report the joint miss rate ( Miss ), joint type identification error ( Type ), orientation error ( Axis ), and positional error ( Pivot ).
Method
S-Pen ↓
P-Pen ↓
S-Stab ↑
O-Stab ↑
CLIP ↑
DINOv2 ↑
Complete ↑
Layout ↑
Function ↑
Full benchmark
VIGA [ 36 ]
56.67
63.37
23.33
27.33
0.8831
0.6087
4.70
3.95
3.45
SAGE [ 31 ]
6.67
2.41
70.00
91.60
0.8729
0.5379
3.80
2.60
2.90
Ours
0.00
0.00
100.00
100.00
0.9128
0.7855
6.20
5.35
5.58
Tabletop subset
VIGA [ 36 ]
54.55
67.26
18.18
25.66
0.8764
0.5950
4.55
3.91
3.00
Table 2: Scene-level results on the full benchmark and the 11-scene tabletop subset. S-Pen, P-Pen, S-Stab, and O-Stab denote scene-level (inter-object), part-level (inner-object) penetration rate, and scene-/object-level stability, respectively. These metrics are percentages. MLLM scores (the last three columns) use a 1 – 7 scale.
Simulation-ready world reconstruction demands physical interactivity beyond visual realism, which existing black-box 3D generation fails to deliver. We present NeoWorld-3, a coding-agent framework that reconstructs interactive 3D worlds from visual observations via tunable, executable programs. At its core, NeoSDK provides the underlying computational engine for low-level geometries, part composition, differentiable parameter refinement, and physics validation. At the same time, NeoMCP exposes these capabilities as structured, tool-callable APIs for vision-language agents to visually inspect, edit code, and iteratively refine programs. Through a hierarchical coordinator-worker architecture, the framework scales seamlessly from standalone articulated objects to complex compositional scenes. Evaluations across object-level and scene-level reconstruction benchmarks demonstrate that NeoWorld-3 achieves state-of-the-art geometric fidelity and physical readiness. Crucially, manipulation trajectories planned within our reconstructed twins achieve successful zero-shot execution on a physical dual-arm robot, completing the Real-Sim-Real loop.
Simulation is a powerful tool for robotics, but building diverse, simulation-ready scenes at scale remains difficult. Reconstructing scenes from single images is a promising alternative, but visual similarity alone is insufficient: objects must also be recovered, accurately placed, and physically grounded. Existing methods that rely on post-hoc physics settling often misplace objects, producing scenes that look correct but behave poorly under manipulation. We present SceneRig, an agentic system that turns a single image into an editable, simulation-ready scene. SceneRig combines geometric, visual, and physical tools to reconstruct a faithful 3D scene. An agent then performs edit-and-validate loops inside a physics simulator to achieve physical stability. Across diverse inputs, SceneRig improves instance segmentation, object-centric appearance, and 3D placement while keeping the reconstructed scenes physically stable. Furthermore, SceneRig is effective for downstream robotics: its scenes predict real-robot policy outcomes more accurately than the baseline method and more reliably support replay of recorded manipulations without trajectory adaptation.
Indoor scene synthesis underpins embodied AI, robotic manipulation, and simulation-based policy evaluation, where a useful scene must specify not only what the environment looks like, but also how its objects are structured. Existing pipelines, however, typically represent generated content as static meshes and inherit articulation only from curated asset libraries, which limits object-level controllability and prevents new interactable assets from being produced on demand. We address this gap by formulating physically interactable indoor scene synthesis as programmatic world generation, and present SceneCode, a framework that compiles a natural language prompt into an executable, code-driven indoor world rather than a collection of opaque meshes. A room-level agentic backbone first turns the prompt into a structured house layout and emits per-object AssetRequests through a planner--designer--critic loop. Each request is then routed to one of five code-generation strategies and converted into a synthesized part-wise Blender Python programs that are validated through an execution-guided repair-and-refine loop. The resulting programs are compiled into simulation-ready assets, and exported as SDF for physics simulation. A persistent scene-state registry links object requests, executable programs, rendered geometry, and simulation assets, turning scene assembly into a traceable and locally editable world-building process. We evaluate SceneCode across scene-level synthesis, object-level asset quality, human judgment, and downstream robot interaction. Results show that executable world programs improve prompt-faithful indoor scene generation and produce assets with cleaner mesh structure, and simulator-loadable articulation metadata. Project page: https://scene-code.github.io/.
Puyi Wang, Yuhao Wang, Linjie Li +4
1The Chinese University of Hong Kong · 2Shanghai Jiao Tong University · 3Shanghai AI Laboratory +2