We present HSImul3R, a unified framework for simulation-ready 3D reconstruction of human-scene interactions (HSI) from casual captures, including sparse-view images and monocular videos. Existing methods suffer from a perception-simulation gap: visually plausible reconstructions often violate physical constraints, leading to instability in physics engines and failure in embodied AI applications. To bridge this gap, we introduce a physically-grounded bi-directional optimization pipeline that treats the physics simulator as an active supervisor to jointly refine human dynamics and scene geometry. In the forward direction, we employ Scene-targeted Reinforcement Learning to optimize human motion under dual supervision of motion fidelity and contact stability. In the reverse direction, we propose Direct Simulation Reward Optimization, which leverages simulation feedback on gravitational stability and interaction success to refine scene geometry. We further present HSIBench, a new benchmark with diverse objects and interaction scenarios. Extensive experiments demonstrate that HSImul3R produces the first stable, simulation-ready HSI reconstructions and can be directly deployed to real-world humanoid robots.
Figures & tables
Figure 1: Examples of real-world humanoid robotic deployment. Given casual captures, our approach achieves simulation-ready 3D reconstruction of human–scene interactions by refining the human motions and scene geometry via a physically-grounded bi-directional optimization pipeline. Our optimized human motions can be seamlessly transferred and deployed in humanoid robotics.
Figure 2: Examples of HSIBench and corresponding simulation results by HSImul3R. Our approach enables simulation-ready 3D reconstruction of human–scene interactions from casual captures. In addition, we collect HSIBench, a dataset comprising 16-view synchronized captures of diverse human–scene interactions, covering a wide range of scene objects, human subjects, and motions.
Figure 3: Overview of HSImul3R. Given casual captures as inputs, we achieve simulation-ready reconstruction of human–scene interactions via a physics-in-the-loop optimization pipeline. We first propose to inject an 3D explicit generative prior into the reconstruction pipeline to achieve better alignment between human and scene. Then, (1) in the forward-pass, we propose a scene-targeted reinforcement learning that optimize the human motion to achieve interaction stability within the simulator, (2) in the reverse-pass, we introduce a direct simulation reward optimization (DSRO) to refine the scene geometry via simulation feedback regarding the stability. Specifically, we define the 4 types regarding the feedback. Type 1: objects not stabilizing under gravity; Type 2: objects failing to stabilize during human interaction; Type 3: objects stabilizing but without meaningful interaction; Type 4: objects with stable interaction.
Method
Stability-HSI ( % ) ↑
Scene Penetration
Human Motion Quality
Easy
Medium
Hard
SP-3D ( % ) ↓
W-MPJPE ↓
PA-MPJPE ↓
HSfM [ 52 ]
10.52
4.50
2.66
69.51
5.02
2.79
V1
13.96
8.81
4.17
77.12
6.18
3.20
V2
39.56
22.71
7.05
-
4.91
2.71
V3
42.57
23.84
10.18
-
4.60
2.42
V4
29.56
16.62
5.17
-
4.57
2.39
Table 1: Quantitative comparison regarding human-scene-interaction reconstruction and simulation. We assess (1) post-simulation interaction stability, (2) human–scene penetration in the 3D reconstruction, and (3) changes in the human motions after simulation. Our method consistently outperforms the baseline method (HSfM) and different variants.
Method
Stability-HSI ( % ) ↑
SG ( % ) ↑
CD ↓
F-Score ↑
Easy
Medium
Hard
MIDI [ 23 ]
29.56
16.62
5.17
79.19
0.198
81.95
DSO* [ 31 ]
38.75
25.91
7.88
87.23
0.191
86.26
Ours
53.68
30.56
13.92
91.50
0.173
88.25
Table 2: Quantitative comparison regarding image-to-3D generation quality. Stability-HSI’ denotes the ratio of simulations in which the human–scene interaction remains stable, while “Stability-Gravity”(SG) refers to placing the object alone in the simulator and evaluating whether it can stand stably under gravity. Note that we fine-tune DSO on pre-trained MIDI model for fair comparison.
Figure 4: Qualitative comparison regarding image-to-3D object reconstruction. Our method enhances the object’s geometric structure while reducing surface “bumps” that may negatively impact human interaction.
Figure 5: Qualitative comparisons with HSfM [ 52 ] . Due to challenges such as (1) penetration issues and (2) inaccurate scene-object structures with geometric distortions, HSfM often struggles to achieve stable interactions in the simulator, frequently leading to unintended object displacement.
Method
Stability-HSI ( % ) ↑
Scene Penetration
Human Motion Quality
Easy
Medium
Hard
SP-3D ( % ) ↓
W-MPJPE ↓
PA-MPJPE ↓
16-view
55.16
29.51
13.59
21.81
4.01
1.99
10-view
52.93
32.17
13.03
21.00
4.06
2.05
4-view
53.68
30.56
13.92
22.90
4.09
2.17
Table 3: Quantitative analysis of number of input views. We quantitatively evaluate reconstruction quality and simulation stability under different numbers of input views. While additional views slightly improve motion quality, they don’t have obvious impact over the interaction stability and increase scene penetration.
Figure 6: Sim-to-Real results deployed on Unitree G1 humanoid robotics. The refined human motions by our method could be successfully transferred and deployed on a real-world humanoid robot to conduct various human-scene-interaction scenarios.
Figure 7: Ablation studies on Eq. 5 . Without the proposed scene-targeted RL, the simulation often results in unintended object displacement and fails to maintain stable interactions.