Organizations: Siebel School of Computing and Data Science, University of Illinois at Urbana-Champaign · Department of Computer Science, The University of Hong Kong · Meta Reality Labs
Estimating simulation-ready scenes from real-world observations is crucial for downstream planning and policy learning tasks. Regretfully, existing methods struggle in cluttered environments, often exhibiting prohibitive computational cost, poor robustness, and restricted generality when scaling to multiple interacting objects. We propose a unified optimization-based formulation for real-to-sim scene estimation that jointly recovers the shapes and poses of multiple rigid objects under physical constraints. Our method is built on two key technical innovations. First, we leverage the recently introduced shape-differentiable contact model, whose global differentiability permits joint optimization over object geometry and pose while modeling inter-object contacts. Second, we exploit the structured sparsity of the augmented Lagrangian Hessian to derive an efficient linear system solver whose computational cost scales favorably with scene complexity. Building on this formulation, we develop an end-to-end Simulation-ready Physics-Aware Reconstruction for Cluttered Scenes (SPARCS) pipeline, which integrates learning-based object initialization, physics-constrained joint shape-pose optimization, and differentiable texture refinement. Experiments on cluttered scenes with up to 5 objects and 22 convex hulls demonstrate that our approach robustly reconstructs physically valid, simulation-ready object shapes and poses. Project webpage: https://rory-weicheng.github.io/SPARCS/.
Figures & tables
Fig. 2: We illustrate the three types of objectives in Equation 4 , regularizing the distance between convex hull vertex Xijk (red), the SAM3D-identified mesh vertex pi∈Mi (blue), and the point cloud pi∈Pi (yellow). Further, we highlight a case (a) where objective value can increase. Suppose our rigid body (light brown) consists of two disjoint convex hulls (b), the closest point to the blue vertex is the bottom surface of the top hull. After an update to hull vertices (c), the two hulls merge and the closest point is moved to the right boundary.
Fig. 3: Suppose we would like to model a box (light brown) put on a chair (dark brown). The box and the chair are the i th and i′ th rigid bodies respectively, where the box is modeled as a single convex hull and the chair is modeled as the union of 4 convex hulls. Each convex hull is a polytope spanned by a set of vertices Xijk (red). Between the ij th convex hull on the box and the i′j′ th convex hull on the chair modeling the back support, we introduce a separating plane \left(\begin{array}[]{cc}{n},&{d}\end{array}\right)_{iji^{\prime}j^{\prime}} (blue) as a proxy for the contact model.
Fig. 4: An illustration of our friction model, between the ij th and i′j′ th convex hull. On each vertex, e.g. Xijk , the normal force fijk,i′j′⊥ is a function of x and q (Equation 10 ) and the friction force fijk,i′j′∥ is additional decision variables to be optimized. We follow the idea of SDRS contact model and use the separating plane as the proxy for contact modeling and each force fijk,i′j′∥ applied on the ij th convex hull is counter-acted by −fijk,i′j′∥ applied on the separating plane \left(\begin{array}[]{cc}{n},&{d}\end{array}\right)_{iji^{\prime}j^{\prime}} , and the case is the same for the i′j′ th convex hull. We then model the separating plane as a physical object with zero mass, so that all the forces applied on it must be balanced.
Fig. 5: We illustrate the structure of matrix H (left) and matrix A (right). A is block-diagonal and each block is small. H is factored using the Woodbury matrix identity.
Fig. 6: A visualization of our benchmarking scenarios. Top Row: The input single-view image observation. Middle Row: The estimated simulation-ready rigid bodies models. Bottom Row: Our estimated scenes achieve physical force equilibrium in MuJoCo [ 34 ] , showing simulated contact forces after settling.
Scenario
Max Kinetic Energy (J) ↓
Max Drift Distance (cm) ↓
Ours
SAM3D
Ours
SAM3D
1
7.19×10−4
4.77×100
1.62
59.41
2
2.24×10−3
5.68×100
0.83
87.56
3
1.12×10−2
2.08×100
3.10
31.06
4
3.32×10−3
2.32×100
0.73
172.55
5
4.36×10−3
5.73×100
2.28
51.69
TABLE I: We summarize the simulator stability by the amount of kinetic energy gain (left) during the first 1 second and drift distance (right) over the first 1 minute of simulation time.
Scenario
PSNR ↑
Ours vs. RGB
SAM3D vs. RGB
Ours vs. SAM3D
1
17.92
18.11
20.16
2
19.99
18.99
22.32
3
18.37
17.34
18.70
4
21.43
21.11
23.95
5
20.15
20.32
20.17
TABLE II: We profile the PSNR between the images rendered using our estimated shape and pose (Ours), SAM3D estimated initial guess, which is further adjusted by FoundationPose (SAM3D), and the Groudtruth RGBD image (RGB).
Scenario
#Hull
#Vertex
#ALM
#LM
Wall Time (min)
1
6
299
7
2536
46.1
2
16
795
6
2322
259.7
3
10
487
9
4529
224.2
4
12
597
7
3106
202.15
5
22
1099
7
2640
539.92
TABLE III: We summarize the statistics of our benchmarking scenarios. From left to right: the total number of convex hulls, the total number of vertices, number of ALM outer iterations, number of LM iterations, and the total computational time.
Fig. 7: The convergence history of a typical optimization procedure, which converges within 9 ALM iterations.
Fig. 8: Our method involves Visual (SAM3D, FoundationPose) inference & geometry process, constraint & Jacobian evaluation, linear solve, and other operations. This figure shows the performance breakdown in percentage.
Fig. 9: Two demonstrations of conducting robotic manipulation tasks in MuJoCo simulation on the reconstructed scene.
Scenario
#Params
Woodbury (Alg. 1 )
Direct LU
Speedup
1
2751
0.475
0.681
1.43 ×
2
12045
3.162
19.713
6.23 ×
3
7425
1.318
7.226
5.48 ×
4
9075
1.814
8.085
4.45 ×
5
19932
7.342
63.909
8.71 ×
TABLE IV: The average cost comparison of solving the structured linear system (in seconds) using our method (Algorithm 1 ) and direct LU factorization.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 10: Two non-convex objects separated using SDRS contact model, each consisting of two convex hulls. Left: Without aggregation, we need 4 separating planes. Right: With aggregation, we only introduce a single separating plane.
Fig. 11: We illustrate various violations to physical constraints induced by visual inference. Top: penetrations (red) and floating objects (blue) always occur as SAM3D and FoundationPose do not explicitly enforce physics constraints. Bottom: These violations always lead to simulation blow up in MuJoCo.
Fig. 12: An illustration of our visual inference and geometry process pipeline. Top Left: Initial estimation of each object mesh; Top Right: Filtered point cloud; Bottom Left: Penetrating free mesh after shrinkage; Bottom Right: Convex decomposition.
Fig. 13: A visualization of our additional examples on scenes from the YCB-V dataset. Top Row: The input single-view image observation. Middle Row: The estimated simulation-ready rigid bodies models. Bottom Row: Our estimated scenes achieve physical force equilibrium in MuJoCo [ 34 ] , where we visualized the contact forces.
Fig. 14: A visualization of the baseline methods. Top row: Gen3DSR [ 4 ] , middle row: SceneComplete [ 1 ] , bottom row: MIDI [ 16 ] (with different reconstructed objects visualized in different colors)
Scen.
#Hull
#Vertex
#Params
#ALM
#LM
Wall Time
6
14
698
10590
7
2844
200.11
7
14
695
10545
7
1643
148.48
8
14
694
10485
7
1936
140.58
9
6
299
3678
6
2480
36.13
10
15
746
11310
6
1083
104.48
Appendix
TABLE V: The statistics of the additional examples. From left to right: the total number of convex hulls, the total number of vertices, number of parameters in linear solve, number of ALM outer iterations, number of LM iterations, and the total computational time in minutes.