SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images

Rundong Luo; Yunong Liu; Di Cao; Matthew Tancik; Shyamal Buch; Colton Stearns; Wenqi Xian

cs.ROOct 20, 2026

SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images

Authors: Rundong Luo, Yunong Liu, Di Cao, Matthew Tancik, Shyamal Buch, Colton Stearns, Wenqi Xian

Organizations: Luma AI · Cornell University

Abstract

Simulation is a powerful tool for robotics, but building diverse, simulation-ready scenes at scale remains difficult. Reconstructing scenes from single images is a promising alternative, but visual similarity alone is insufficient: objects must also be recovered, accurately placed, and physically grounded. Existing methods that rely on post-hoc physics settling often misplace objects, producing scenes that look correct but behave poorly under manipulation. We present SceneRig, an agentic system that turns a single image into an editable, simulation-ready scene. SceneRig combines geometric, visual, and physical tools to reconstruct a faithful 3D scene. An agent then performs edit-and-validate loops inside a physics simulator to achieve physical stability. Across diverse inputs, SceneRig improves instance segmentation, object-centric appearance, and 3D placement while keeping the reconstructed scenes physically stable. Furthermore, SceneRig is effective for downstream robotics: its scenes predict real-robot policy outcomes more accurately than the baseline method and more reliably support replay of recorded manipulations without trajectory adaptation.

Figures & tables

Figure 1: Single-image simulation-ready scene reconstruction for robotics. SceneRig combines vision tools, agentic reasoning, and physical simulation to create an editable, simulation-ready scene from a single image (top). Its reconstructions support real-to-sim policy evaluation and open-loop trajectory replay, reproducing real-world outcomes more faithfully than SimFoundry (bottom).

Figure 2: Overview of SceneRig . From a single image, SceneRig recovers objects, supporting surfaces, and their relations, and initializes a gravity-aligned metric scene through physical settling. Geometry, material, and lighting agents construct the surrounding scene. Finally, a pose-refinement agent inspects paired reference/render crops and invokes geometry- and physics-grounded tools to correct object placement.

Figure 3: Qualitative scene reconstruction results. Compared with the baselines, SceneRig more consistently preserves object identity, scale, and relative placement while reconstructing the surrounding scene.

	Source-view reconstruction				Pairwise VLM evaluation			Physical stability
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	nCLIP $\downarrow$	Completeness $\uparrow$	Layout $\uparrow$	Appearance $\uparrow$	Scene (%) $\uparrow$	Object (%) $\uparrow$
VIGA	11.414	0.529	0.653	0.180	0.225	-0.237	-0.249	10.0	27.1
REST3D	11.092	0.590	0.729	0.214	-1.074	-0.840	-1.625	31.3	76.6
SceneConductor	10.140	0.503	0.727	0.172	0.385	-0.837	-0.473	8.8	36.5
SimFoundry	12.060	0.582	0.720	0.194	-0.432	-0.683	-0.792	98.8	99.9
SceneRig (ours)	13.277	0.544	0.518	0.122	0.895	2.597	3.139	100.0	100.0

Table 1: Scene-level reconstruction, pairwise preference, and physical stability. VLM columns report mean-centered Bradley–Terry (BT) scores from pairwise judgments of source and nearby views. Within each criterion, a score advantage of

\Delta

implies a modeled pairwise win probability of

1/(1+e^{-\Delta})

. An object passes the stability test if its displacement after 5 s is below 50 mm and its final velocity is zero; a scene passes only if every object does. All methods use Claude Opus 5 as the underlying VLM.

	Object-centric metrics							3D object layout
Method	ARI $\uparrow$	FG-IoU $\uparrow$	Count MAE $\downarrow$	PSNR ∗ $\uparrow$	SSIM ∗ $\uparrow$	LPIPS ∗ $\downarrow$	nCLIP ∗ $\downarrow$	OSPA@5 $\downarrow$	OSPA@10 $\downarrow$	F1@10 $\uparrow$
SAM3D ∗	0.784	0.705	–	12.810	0.403	0.195	0.078	2.7	3.7	0.88
VIGA	0.283	0.260	3.36	8.987	0.315	0.325	0.149	4.9	8.9	0.29
REST3D	0.210	0.204	3.20	9.636	0.378	0.296	0.148	4.7	8.7	0.29
SceneConductor	0.197	0.187	1.14	9.493	0.329	0.317	0.152	5.0	9.5	0.14
SimFoundry	0.676	0.589	1.05	11.497	0.379	0.240	0.103	3.3	4.8	0.82

Table 2: Object recovery and 3D layout. Results on 44 simulator-grounded scenes. PSNR ∗ /SSIM ∗ use object pixels (table excluded); LPIPS ∗ /nCLIP ∗ average per-object crop scores with non-object pixels white. Layout metrics use global scale alignment; OSPA distances and cutoffs are in centimeters.

Table 6

Environment		Everything $\to$ bin	Fruits $\to$ plate	Fruits $\to$ bowl	Marker $\to$ cup	Mustard $\to$ bin	All	Cohen’s $\kappa\uparrow$	Pearson $r\uparrow$
Real-world	success	7/10	7/10	7/10	5/10	7/10	33/50	–	–
SimFoundry	success	4/10	6/10	5/10	6/10	6/10	27/50	–	-0.38
	agreement	5/10	7/10	2/10	5/10	5/10	24/50	-0.07	–
SceneRig	success	9/10	8/10	5/10	4/10	7/10	33/50	–	0.70
	agreement	8/10	7/10	6/10	7/10	8/10	36/50	0.38	–

Table 6: Real-to-sim policy evaluation. Results for

\pi_{0.5}

on 50 real episodes and their reconstructions. Entries count successful episodes (success) or matching real and simulated outcomes (agreement). Cohen’s

\kappa

uses these 50 episodes; Pearson’s

r

correlates the five real and simulated task success rates.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: \par Relational perception and mask recovery. A VLM identifies entities and proposes relations. MolmoPoint locations select SAM3 masks; subsequent checks recover missed objects, merge duplicate or split masks, complete occluded objects, and revise the root-surface records.

Figure 7: \par Agentic scene construction. Scene structure, root material, and lighting are refined in a shared Blender scene before object-pose refinement. Each stage pairs a generator with a verifier. The initialization stage’s generator agent additionally has a rules check tool that is required to pass before sending the results to the verifier.

Figure 8: \par Pose refinement and export. The agent investigates objects and requests edits. The Move tool searches and physically validates bounded moves; the Edit tool simulates direct edits for agent review. After a final gravity pass, object poses and physical properties are exported to a simulation-ready USD file.

Figure 9: \par Prompt reuse with fresh scene evidence. In this example, tool call #11 reuses the unchanged prefix of tool call #10 (green), appends the cup/saucer investigation results (amber), and refreshes the object state and budget after the cache boundary (purple). The crops come from the recorded run; the layout illustrates the context policy.

Component	Role and conditioning
Configurable VLM	Category/instance proposals, semantic reviews, physical-property estimates, and stage agents; launcher default: claude-opus-5 .
MolmoPoint-8B / SAM3	Instance-description localization / category, synonym, crop, and point-conditioned masks.
MoGe-2 / SAM3D	Metric camera and point map / textured per-object mesh.
LanPaint + Qwen-Image-Edit-2509	Masked occluder removal and one-sided outpainting.
LingBot-Depth	Edited RGB, surviving metric depth, and intrinsics; ViT-L/14 v0.5 pretrained checkpoint.
SHARP + GPT-Image-2	Gaussian novel-view rendering followed by image repair; point-splat/masked-edit fallback.

Appendix

Table 7: \par Models and their roles. The two image-editing paths have different inputs and purposes; generated references never replace the real source photograph.

Setting	Default
Input / image-edit resolution	RGB input long edge 1536 pixels; GPT image-edit input long edge at most 2048 pixels.
Root plane fitting	200 RANSAC trials, at most 5000 points, 0.01 inlier distance; reliable root planes require inlier fraction $\geq 0.6$ and image coverage $\geq 2\%\par$ .
Occluder image editing	20 diffusion steps, guidance 4, seed 0; one LanPaint Langevin step per diffusion step.
Ordinary / support CoACD	Concavity 0.05 / 0.02; at most 32 / 64 parts; a capped support can be retried with 128 parts. Ordinary resolution 50; support resolution $\operatorname{clip\par}(\lceil\ell_{\max}/0.003\rceil,100,200)$ , where $\ell_{\max}$ is mesh maximum extent.
Physics material fallback	Density 250 kg/m 3 ; contact offset 0.002, rest offset 0.
Settlement	120 Hz; 600-step ordinary or 240-step local budget, with a further 600 steps for an unconverged drop.

Appendix

Table 8: \par Key implementation defaults. Physical lengths are in meters unless otherwise stated.

Environment		Everything $\to$ bin	Fruits $\to$ plate	Fruits $\to$ bowl	Marker $\to$ cup	Mustard $\to$ bin	All	Cohen’s $\kappa\uparrow$	Pearson $r\uparrow$
Real-world	success	5/10	7/10	2/10	3/10	3/10	20/50	–	–
SimFoundry	success	4/10	4/10	0/10	0/10	2/10	10/50	–	0.88
	agreement	5/10	5/10	8/10	7/10	5/10	30/50	0.09	–
SceneRig	success	5/10	6/10	1/10	1/10	0/10	13/50	–	0.93
	agreement	6/10	7/10	7/10	6/10	7/10	33/50	0.25	–

Appendix

Table 9: \par Real-to-sim evaluation with LingBot-VLA 2.0. The same fine-tuned checkpoint is evaluated on 50 real episodes and both reconstructions. Success counts successful episodes; agreement counts matching real and simulated verdicts. Cohen’s

\kappa

uses all 50 episodes; Pearson’s

r

uses the five task-level success rates.

Policy	Method	Agreement $\uparrow$	Cohen’s $\kappa\uparrow$	Pearson $r\uparrow$	MMRV $\downarrow$	Rate error $\downarrow$	McNemar
$\pi_{0.5}$	SimFoundry	48% [36, 62]	$-0.07$ [ $-0.32$ , 0.18]	$-0.38$	0.12	0.16	18 vs. 6, $p=0.023$
$\pi_{0.5}$	SceneRig	72% [60, 84]	0.38 [0.11, 0.63]	0.70	0.00	0.12	18 vs. 6, $p=0.023$
LingBot-VLA 2.0	SimFoundry	60% [46, 74]	0.09 [ $-0.16$ , 0.35]	0.88	0.06	0.20	7 vs. 4, $p=0.55$
LingBot-VLA 2.0	SceneRig	66% [52, 78]	0.25 [0.00, 0.50]	0.93	0.04	0.14	7 vs. 4, $p=0.55$
Pooled	SimFoundry	54% [45, 63]	0.09 [ $-0.08$ , 0.27]	0.84	0.11	0.18	25 vs. 10, $p=0.017$
Pooled	SceneRig	69% [60, 78]	0.38 [0.20, 0.56]	0.92	0.04	0.13	25 vs. 10, $p=0.017$

Appendix

Table 10: \par Statistical analysis of real-to-sim agreement. Agreement and Cohen’s

\kappa

use per-episode verdicts, with 95% paired-bootstrap intervals. McNemar counts the episodes on which only SceneRig or only SimFoundry agrees with the real verdict. Pearson’s

r

, MMRV, and rate error (the mean absolute difference between real and simulated task success rates) use task-level success rates and are descriptive.

Task	Code edits only	Full SceneRig
Everything $\to$ bin	1/5	3/5
Cup $\to$ bowl	1/5	4/5
Fruits $\to$ bowl	2/5	4/5
Marker $\to$ cup	2/5	4/5
Mustard $\to$ bin	5/5	5/5
All	11/25 (44%)	20/25 (80%)

Appendix

Table 11: \par Object placement ablation for open-loop replay. Successful replays out of five real demonstrations per task. Code edits only removes Move while retaining Edit ; both variants use the same replay protocol.

	Source-view reconstruction				Object-centric appearance				Instance structure		3D object layout
VLM backbone	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	nCLIP $\downarrow$	PSNR ∗ $\uparrow$	SSIM ∗ $\uparrow$	LPIPS ∗ $\downarrow$	nCLIP ∗ $\downarrow$	ARI $\uparrow$	FG-IoU $\uparrow$	OSPA@5 $\downarrow$	F1@10 $\uparrow$
Claude Opus 5	13.277	0.544	0.518	0.122	13.609	0.440	0.170	0.062	0.878	0.819	2.4	0.93
Claude Fable 5	12.837	0.535	0.535	0.126	13.082	0.432	0.170	0.064	0.879	0.820	2.4	0.94
GPT-5.6-terra	13.918	0.577	0.527	0.128	13.338	0.437	0.178	0.067	0.850	0.781	2.6	0.92
Kimi K3	12.830	0.542	0.557	0.121	13.196	0.439	0.184	0.066	0.865	0.802	2.5	0.93
GPT-6 Astra	16.512	0.592	0.369	0.100	14.535	0.462	0.151	0.060	0.891	0.838	2.3	0.95

Appendix

Table 12: \par Reconstruction with different VLM backbones. Every row uses SceneRig with the listed backbone for all configurable VLM roles across preprocessing and the agent stages.

Figure 10: \par Qualitative comparison across VLM backbones.

	Source-view reconstruction			Object-centric appearance			Instance structure		3D object layout
Placement	PSNR $\uparrow$	LPIPS $\downarrow$	nCLIP $\downarrow$	PSNR ∗ $\uparrow$	LPIPS ∗ $\downarrow$	nCLIP ∗ $\downarrow$	ARI $\uparrow$	FG-IoU $\uparrow$	OSPA@5 $\downarrow$	F1@10 $\uparrow$
SAM3D raw poses	12.915	0.538	0.131	12.810	0.195	0.078	0.784	0.705	2.7	0.88
SAM3D raw poses + physics settle	12.768	0.546	0.129	12.408	0.198	0.079	0.748	0.664	3.0	0.84
SceneRig , code edits only	13.272	0.523	0.124	13.415	0.178	0.066	0.858	0.792	2.4	0.92
SceneRig	13.277	0.518	0.122	13.609	0.170	0.062	0.878	0.819	2.4	0.93

Appendix

Table 13: \par Object placement ablation. SAM3D’s raw poses share every other component (segmentation, assets, camera, root surfaces) with SceneRig . “Code edits only” removes Move ; the agent specifies pose changes through Edit , which still runs physical simulation on the edited scene.

		Full-scene appearance			Object-centric appearance			Instance structure		3D object layout		Physics	API cost
VLM backbone	Method	PSNR $\uparrow$	LPIPS $\downarrow$	nCLIP $\downarrow$	PSNR ∗ $\uparrow$	LPIPS ∗ $\downarrow$	nCLIP ∗ $\downarrow$	ARI $\uparrow$	FG-IoU $\uparrow$	OSPA@5 $\downarrow$	F1@10 $\uparrow$	Stable scenes $\uparrow$	( $/scene)$ \downarrow$
Claude Opus 5	VIGA	11.042	0.671	0.191	8.725	0.337	0.154	0.261	0.239	4.9	0.20	8/80	16.13
Claude Opus 5	SceneRig	13.277	0.518	0.122	13.609	0.170	0.062	0.878	0.819	2.4	0.93	80/80	4.22
GPT-6 Astra	VIGA	15.316	0.433	0.125	14.077	0.187	0.082	0.721	0.648	4.6	0.37	24/80	7.74
	Codex harness	17.007	0.370	0.100	14.773	0.178	0.073	0.653	0.594	4.2	0.60	16/80	3.99
	SceneRig	16.512	0.369	0.100	14.535	0.151	0.060	0.891	0.838	2.3	0.95	80/80	6.24

Appendix

Table 14: \par Reconstruction with different harnesses and shared VLM backbones. We compare VIGA, SceneRig , and the direct Codex harness. VIGA and Codex appearance and layout metrics are computed after an additional 5 s settling pass. Note that results from Tables and evaluate each method’s native reconstruction output. Stability is measured over the settling pass. API cost includes both VLM and image generation model costs.

Figure 11: \par Scene reconstruction with a shared GPT-6 Astra backbone.

	Stage-1 segmentation			Final scene
Method	ARI $\uparrow$	FG-IoU $\uparrow$	Count MAE $\downarrow$	ARI $\uparrow$	FG-IoU $\uparrow$	Count MAE $\downarrow$
VIGA	0.450	0.394	4.43	0.283	0.260	3.36
REST3D	0.864	0.809	3.18	0.210	0.204	3.20
SceneConductor	0.824	0.800	1.11	0.197	0.187	1.14
SimFoundry	0.780	0.732	1.84	0.676	0.589	1.05
SceneRig (ours)	0.981	0.970	0.27	0.878	0.819	0.27

Appendix

Table 15: \par Instance segmentation at the segmentation stage and in the final scene. Results on the 44 simulator-grounded scenes. Stage-1 metrics use visible instance masks on the original input image, whereas final-scene counts include only objects visible in the source-view render. Changes in visibility can therefore change the count even without a change in the reconstructed object inventory.

Figure 12: \par Additional qualitative reconstruction results.

Physics-Aware 3D Scene Reconstruction

See all 13 papers

Jun 2, 2026

SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image

Reconstructing interactive, simulation-ready 3D scenes from a single image is a critical bottleneck for robotic manipulation. While recent single-image lifters recover plausible per-object shapes, composing them yields scenes that collapse under physical simulation due to interpenetrating, hovering, or sinking objects. Existing physics-aware methods address this strictly as a post-hoc layout correction, leaving the underlying geometric errors unresolved. To address this, we introduce SimuScene, a compositional 3D reconstruction pipeline that puts physics in the loop of shape and layout estimation. Rather than using physics merely for layout cleanup, we utilize the physics engine as a diagnostic measurement tool during the generative process itself. By diagnostically simulating reconstructed objects under gravity, we convert penetration and support failures into quantitative correction signals that drive gravity-axis stretching and amodal shape resampling. This physics-informed feedback loop mitigates accumulated reconstruction errors and produces a stable, simulation-ready compositional 3D scene. Extensive experiments demonstrate state-of-the-art performance on physical stability and geometric alignment benchmarks. We further highlight SimuScene's utility by deploying reconstructed environments in humanoid control and robot-arm manipulation tasks.

Inhee Lee, Sangwon Baik, Sungjoo Kim +3

Seoul National University

3D Reconstruction Robotics Simulation Robotic Manipulation 3D Scene Layout Generation Single-View 3D Reconstruction Physics-Aware 3D Scene Reconstruction

Ask PDF

Oct 26, 2026

SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments

Visual captures provide detailed scene appearance, but converting them into interactive simulation environments requires object reconstruction, physical modeling, and correction of intermediate errors. We present SceneAgent, an agentic pipeline that coordinates these steps through simulator-based visual inspection and repair. The pipeline processes 3D Gaussian Splatting (3DGS), photogrammetry, LiDAR captures, AI-generated 3DGS scenes, and 4D videos of dynamic scenes, and combines object segmentation, physical parameter estimation, part decomposition, articulation, and deformable simulation. To support policy learning, SceneAgent generates diverse object variants ("digital sisters") and scripted demonstrations. On a cluttered bin scene from GraspClutter6D, SceneAgent raises F1 at 1 cm for every coding agent with scorable outputs both with and without its skills, and its reconstructions are better than SimFoundry: all agent configurations exceed SimFoundry's F1 score, with a best of 60.75% versus 6.9%. On three dynamic scenes from 4DCodeBench, SceneAgent raises the mean dynamic IoU of Gemini 4 Argon from 0.86% to 99.09%. We further demonstrate diverse simulated interactions and preliminary policy transfer to a real robot.

Luke Hollis, Tianxing Fan, Heng Yang

Computational Robotics Group, School of Engineering and Applied Sciences, Harvard University

Robotics Simulation Physics-Aware 3D Scene Reconstruction 3D Reconstruction Dynamic Scene Reconstruction

Ask PDF

Sep 21, 2026

Fysiverse-3D-SimReady Technical Report: Agentic Physical Simulation for Pragmatic 3D World Reconstruction

Agentic recognition requires visual perception to move beyond static scene understanding and produce structured scene representations that support the perception--reasoning--action loop. Existing single-image 3D generation methods, however, mainly produce visually plausible object assets rather than simulation-ready scene states. When independently generated meshes are composed in a shared space, they may fail to align with the input camera, violate gravity, interpenetrate nearby objects, or become unstable under physics simulation. We present Fysiverse-3D-SimReady, a grounded refinement framework for reconstructing simulation-ready multi-object scenes from a single RGB image with instance and ground prompts. The method places generated object meshes into a shared gravity-aligned scene, refines their camera-space poses through differentiable rendering, and corrects scene-level supports and contacts for stable physical execution. The scene is then used by an agentic simulation workflow, which converts a scene-specific task goal into an executable physics rollout rendered from the original camera view. Experiments show that Fysiverse-3D-SimReady improves input-view alignment, contact plausibility, and physical stability over existing single-image reconstruction and scene generation baselines, while enabling goal-conditioned physical interactions from a single image.

Lintao Wang, Mingyang Sun, Yang Liu +2

Physical Superintelligence Lab, Fysics AI · College of Intelligent Robotics and Advanced Manufacturing, Fudan University · Multimedia Laboratory (MMLab), The Chinese University of Hong Kong +1

3D Reconstruction 3D Scene Generation Physics-Based Simulation Single-View 3D Reconstruction

Ask PDF

Abstract

Physics-Aware 3D Scene Reconstruction

Explore similar work

SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image

SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments

Fysiverse-3D-SimReady Technical Report: Agentic Physical Simulation for Pragmatic 3D World Reconstruction