Reconstructing non-rigid objects with physical plausibility remains a significant challenge. Existing approaches leverage differentiable rendering for per-scene optimization, recovering geometry and dynamics but requiring expensive tuning or manual annotation, which limits practicality and generalizability. To address this, we propose ReconPhys, the first feedforward framework that jointly learns physical attribute estimation and 3D Gaussian Splatting reconstruction from a single monocular video. Our method employs a dual-branch architecture trained via a self-supervised strategy, eliminating the need for ground-truth physics labels. Given a video sequence, ReconPhys simultaneously infers geometry, appearance, and physical attributes. Experiments on a large-scale synthetic dataset demonstrate superior performance: our method achieves 21.64 PSNR in future prediction compared to 13.27 by state-of-the-art optimization baselines, while reducing Chamfer Distance from 0.349 to 0.004. Crucially, ReconPhys enables fast inference (<1 second) versus hours required by existing methods, facilitating rapid generation of simulation-ready assets for robotics and graphics.
Figures & tables
Figure 1: ReconPhys predicts physical attributes using a feedforward neural network (FFN). This approach obviates the need for per-scene optimization and requires only a monocular video of the object as input. Within one second, it produces 3DGS assets with binded physical attributes. This capability offers a promising and efficient pathway for generating simulation-ready data for non-rigid manipulation, enabling the rapid transfer of real-world objects into simulation environments without human intervention.
Figure 2: Overview of our framework. Given an input image, a 3DGS predictor reconstructs the 3D object and samples anchor points to build a spring–mass system. Meanwhile, a physics predictor estimates physical attributes from videos to parameterize the system, producing simulatable 3D objects.
Figure 3: Training pipeline of ReconPhys with self-forcing. The physics predictor estimates physical parameters from input videos to drive a spring–mass system for motion prediction. The predicted state is rendered and supervised by a reconstruction loss, with a self-forcing feedback loop for temporal prediction.
Figure 4: Visualization of different methods. Our method produces more stable dynamics and realistic future states across different objects, while baseline methods either generate unstable deformations or cannot perform future prediction.
Method
Dynamic Reconstruction
Future Prediction
Time
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
EMD ↓
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
EMD ↓
4DGS Wu et al. (2024)
30.33
0.983
0.0392
0.593
1.5010
-
-
-
-
-
> 1h
Spring-Gaus Zhong et al. (2024)
22.26
0.874
0.1517
0.466
0.582
13.27
0.745
0.2856
0.349
0.424
> 1h
Ours
33.84
0.953
0.0366
0.001
0.007
21.64
0.907
0.0876
0.004
0.017
< 1s
Table 1: Cross-object generalization on 46 unseen objects. Our method achieves superior future prediction through physics-aware generalization, while maintaining reconstruction quality. Best results in bold .
Figure 5: Visualization of real-world non-rigid assets. Two objects are shown dropping and deforming over time T, with real-world input (top) and corresponding simulation results (bottom).
Reconstruction
Method
Cylinder-1
Cylinder-2
Headset-1
Headset-2
Hamburger-1
Hamburger-2
Lantern-1
Lantern-2
PSNR ↑
Spring-Gaus Zhong et al. (2024)
20.89
20.77
23.87
21.02
26.003
21.51
20.37
19.09
Ours
36.86
36.35
33.13
31.79
33.63
34.07
33.14
30.86
SSIM ↑
Spring-Gaus
0.8637
0.8744
0.8993
0.8744
0.9326
0.9104
0.8275
0.8169
Ours
0.9833
0.9822
0.9517
0.9435
0.9614
0.9670
0.9473
0.9327
LPIPS ↓
Spring-Gaus
0.1880
0.1616
0.1306
0.1424
0.0906
0.1077
0.1840
0.1884
Ours
0.0102
0.0233
0.0297
0.0425
0.0362
0.0339
0.0257
0.0610
Table 2: Physical disentanglement on objects with two distinct attributes sets. Our method preserves geometric fidelity while maximizing trajectory divergence between physical states.
Method
Stiffness
Damp
Mass
Friction
Spring-Gaus
827.67
2.546
2.276
1.082
Ours
297.3
1.151
1.337
1.508
Table 3: Comparison of physical attributes errors.
Figure 6: For the same object with different physical attributes, our method produces more accurate deformation predictions compared to Spring-Gaus.
Figure 7: Our method can be utilized in non-rigid manipulation. Four manipulation scenarios are demonstrated: (a) stretching the pencil bag, (b) squeezing the pillow, (c) stretching the hamburger, and (d) squashing and stretching the toy, showcasing the versatility of our approach across diverse deformable objects.