Organizations: Siebel School of Computing and Data Science, University of Illinois at Urbana-Champaign · Department of Computer Science, The University of Hong Kong · Meta Reality Labs
Estimating simulation-ready scenes from real-world observations is crucial for downstream planning and policy learning tasks. Regretfully, existing methods struggle in cluttered environments, often exhibiting prohibitive computational cost, poor robustness, and restricted generality when scaling to multiple interacting objects. We propose a unified optimization-based formulation for real-to-sim scene estimation that jointly recovers the shapes and poses of multiple rigid objects under physical constraints. Our method is built on two key technical innovations. First, we leverage the recently introduced shape-differentiable contact model, whose global differentiability permits joint optimization over object geometry and pose while modeling inter-object contacts. Second, we exploit the structured sparsity of the augmented Lagrangian Hessian to derive an efficient linear system solver whose computational cost scales favorably with scene complexity. Building on this formulation, we develop an end-to-end Simulation-ready Physics-Aware Reconstruction for Cluttered Scenes (SPARCS) pipeline, which integrates learning-based object initialization, physics-constrained joint shape-pose optimization, and differentiable texture refinement. Experiments on cluttered scenes with up to 5 objects and 22 convex hulls demonstrate that our approach robustly reconstructs physically valid, simulation-ready object shapes and poses. Project webpage: https://rory-weicheng.github.io/SPARCS/.
Figures & tables
Fig. 2: We illustrate the three types of objectives in Equation 4 , regularizing the distance between convex hull vertex Xijk (red), the SAM3D-identified mesh vertex pi∈Mi (blue), and the point cloud pi∈Pi (yellow). Further, we highlight a case (a) where objective value can increase. Suppose our rigid body (light brown) consists of two disjoint convex hulls (b), the closest point to the blue vertex is the bottom surface of the top hull. After an update to hull vertices (c), the two hulls merge and the closest point is moved to the right boundary.
Fig. 3: Suppose we would like to model a box (light brown) put on a chair (dark brown). The box and the chair are the i th and i′ th rigid bodies respectively, where the box is modeled as a single convex hull and the chair is modeled as the union of 4 convex hulls. Each convex hull is a polytope spanned by a set of vertices Xijk (red). Between the ij th convex hull on the box and the i′j′ th convex hull on the chair modeling the back support, we introduce a separating plane \left(\begin{array}[]{cc}{n},&{d}\end{array}\right)_{iji^{\prime}j^{\prime}} (blue) as a proxy for the contact model.
Fig. 4: An illustration of our friction model, between the ij th and i′j′ th convex hull. On each vertex, e.g. Xijk , the normal force fijk,i′j′⊥ is a function of x and q (Equation 10 ) and the friction force fijk,i′j′∥ is additional decision variables to be optimized. We follow the idea of SDRS contact model and use the separating plane as the proxy for contact modeling and each force fijk,i′j′∥ applied on the ij th convex hull is counter-acted by −fijk,i′j′∥ applied on the separating plane \left(\begin{array}[]{cc}{n},&{d}\end{array}\right)_{iji^{\prime}j^{\prime}} , and the case is the same for the i′j′ th convex hull. We then model the separating plane as a physical object with zero mass, so that all the forces applied on it must be balanced.
Fig. 5: We illustrate the structure of matrix H (left) and matrix A (right). A is block-diagonal and each block is small. H is factored using the Woodbury matrix identity.
Fig. 6: A visualization of our benchmarking scenarios. Top Row: The input single-view image observation. Middle Row: The estimated simulation-ready rigid bodies models. Bottom Row: Our estimated scenes achieve physical force equilibrium in MuJoCo [ 34 ] , showing simulated contact forces after settling.
Scenario
Max Kinetic Energy (J) ↓
Max Drift Distance (cm) ↓
Ours
SAM3D
Ours
SAM3D
1
7.19×10−4
4.77×100
1.62
59.41
2
2.24×10−3
5.68×100
0.83
87.56
3
1.12×10−2
2.08×100
3.10
31.06
4
3.32×10−3
2.32×100
0.73
172.55
5
4.36×10−3
5.73×100
2.28
51.69
TABLE I: We summarize the simulator stability by the amount of kinetic energy gain (left) during the first 1 second and drift distance (right) over the first 1 minute of simulation time.
Scenario
PSNR ↑
Ours vs. RGB
SAM3D vs. RGB
Ours vs. SAM3D
1
17.92
18.11
20.16
2
19.99
18.99
22.32
3
18.37
17.34
18.70
4
21.43
21.11
23.95
5
20.15
20.32
20.17
TABLE II: We profile the PSNR between the images rendered using our estimated shape and pose (Ours), SAM3D estimated initial guess, which is further adjusted by FoundationPose (SAM3D), and the Groudtruth RGBD image (RGB).
Scenario
#Hull
#Vertex
#ALM
#LM
Wall Time (min)
1
6
299
7
2536
46.1
2
16
795
6
2322
259.7
3
10
487
9
4529
224.2
4
12
597
7
3106
202.15
5
22
1099
7
2640
539.92
TABLE III: We summarize the statistics of our benchmarking scenarios. From left to right: the total number of convex hulls, the total number of vertices, number of ALM outer iterations, number of LM iterations, and the total computational time.
Fig. 7: The convergence history of a typical optimization procedure, which converges within 9 ALM iterations.
Fig. 8: Our method involves Visual (SAM3D, FoundationPose) inference & geometry process, constraint & Jacobian evaluation, linear solve, and other operations. This figure shows the performance breakdown in percentage.
Fig. 9: Two demonstrations of conducting robotic manipulation tasks in MuJoCo simulation on the reconstructed scene.
Scenario
#Params
Woodbury (Alg. 1 )
Direct LU
Speedup
1
2751
0.475
0.681
1.43 ×
2
12045
3.162
19.713
6.23 ×
3
7425
1.318
7.226
5.48 ×
4
9075
1.814
8.085
4.45 ×
5
19932
7.342
63.909
8.71 ×
TABLE IV: The average cost comparison of solving the structured linear system (in seconds) using our method (Algorithm 1 ) and direct LU factorization.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 10: Two non-convex objects separated using SDRS contact model, each consisting of two convex hulls. Left: Without aggregation, we need 4 separating planes. Right: With aggregation, we only introduce a single separating plane.
Fig. 11: We illustrate various violations to physical constraints induced by visual inference. Top: penetrations (red) and floating objects (blue) always occur as SAM3D and FoundationPose do not explicitly enforce physics constraints. Bottom: These violations always lead to simulation blow up in MuJoCo.
Fig. 12: An illustration of our visual inference and geometry process pipeline. Top Left: Initial estimation of each object mesh; Top Right: Filtered point cloud; Bottom Left: Penetrating free mesh after shrinkage; Bottom Right: Convex decomposition.
Fig. 13: A visualization of our additional examples on scenes from the YCB-V dataset. Top Row: The input single-view image observation. Middle Row: The estimated simulation-ready rigid bodies models. Bottom Row: Our estimated scenes achieve physical force equilibrium in MuJoCo [ 34 ] , where we visualized the contact forces.
Fig. 14: A visualization of the baseline methods. Top row: Gen3DSR [ 4 ] , middle row: SceneComplete [ 1 ] , bottom row: MIDI [ 16 ] (with different reconstructed objects visualized in different colors)
Scen.
#Hull
#Vertex
#Params
#ALM
#LM
Wall Time
6
14
698
10590
7
2844
200.11
7
14
695
10545
7
1643
148.48
8
14
694
10485
7
1936
140.58
9
6
299
3678
6
2480
36.13
10
15
746
11310
6
1083
104.48
Appendix
TABLE V: The statistics of the additional examples. From left to right: the total number of convex hulls, the total number of vertices, number of parameters in linear solve, number of ALM outer iterations, number of LM iterations, and the total computational time in minutes.
Reconstructing interactive, simulation-ready 3D scenes from a single image is a critical bottleneck for robotic manipulation. While recent single-image lifters recover plausible per-object shapes, composing them yields scenes that collapse under physical simulation due to interpenetrating, hovering, or sinking objects. Existing physics-aware methods address this strictly as a post-hoc layout correction, leaving the underlying geometric errors unresolved. To address this, we introduce SimuScene, a compositional 3D reconstruction pipeline that puts physics in the loop of shape and layout estimation. Rather than using physics merely for layout cleanup, we utilize the physics engine as a diagnostic measurement tool during the generative process itself. By diagnostically simulating reconstructed objects under gravity, we convert penetration and support failures into quantitative correction signals that drive gravity-axis stretching and amodal shape resampling. This physics-informed feedback loop mitigates accumulated reconstruction errors and produces a stable, simulation-ready compositional 3D scene. Extensive experiments demonstrate state-of-the-art performance on physical stability and geometric alignment benchmarks. We further highlight SimuScene's utility by deploying reconstructed environments in humanoid control and robot-arm manipulation tasks.
Simulation is a powerful tool for robotics, but building diverse, simulation-ready scenes at scale remains difficult. Reconstructing scenes from single images is a promising alternative, but visual similarity alone is insufficient: objects must also be recovered, accurately placed, and physically grounded. Existing methods that rely on post-hoc physics settling often misplace objects, producing scenes that look correct but behave poorly under manipulation. We present SceneRig, an agentic system that turns a single image into an editable, simulation-ready scene. SceneRig combines geometric, visual, and physical tools to reconstruct a faithful 3D scene. An agent then performs edit-and-validate loops inside a physics simulator to achieve physical stability. Across diverse inputs, SceneRig improves instance segmentation, object-centric appearance, and 3D placement while keeping the reconstructed scenes physically stable. Furthermore, SceneRig is effective for downstream robotics: its scenes predict real-robot policy outcomes more accurately than the baseline method and more reliably support replay of recorded manipulations without trajectory adaptation.
Compositional scene reconstruction seeks to create object-centric representations rather than holistic scenes from real-world videos, which is natively applicable for simulation and interaction. Conventional compositional reconstruction approaches primarily emphasize on visual appearance and show limited generalization ability to real-world scenarios. In this paper, we propose SimRecon, a framework that realizes a "Perception-Generation-Simulation" pipeline towards cluttered scene reconstruction, which first conducts scene-level semantic reconstruction from video input, then performs single-object generation, and finally assembles these assets in the simulator. However, naively combining these three stages leads to visual infidelity of generated assets and physical implausibility of the final scene, a problem particularly severe for complex scenes. Thus, we further propose two bridging modules between the three stages to address this problem. To be specific, for the transition from Perception to Generation, critical for visual fidelity, we introduce Active Viewpoint Optimization, which actively searches in 3D space to acquire optimal projected images as conditions for single-object completion. Moreover, for the transition from Generation to Simulation, essential for physical plausibility, we propose a Scene Graph Synthesizer, which guides the construction from scratch in 3D simulators, mirroring the native, constructive principle of the real world. Extensive experiments on the ScanNet dataset validate our method's superior performance over previous state-of-the-art approaches.