Reconstructing non-rigid objects with physical plausibility remains a significant challenge. Existing approaches leverage differentiable rendering for per-scene optimization, recovering geometry and dynamics but requiring expensive tuning or manual annotation, which limits practicality and generalizability. To address this, we propose ReconPhys, the first feedforward framework that jointly learns physical attribute estimation and 3D Gaussian Splatting reconstruction from a single monocular video. Our method employs a dual-branch architecture trained via a self-supervised strategy, eliminating the need for ground-truth physics labels. Given a video sequence, ReconPhys simultaneously infers geometry, appearance, and physical attributes. Experiments on a large-scale synthetic dataset demonstrate superior performance: our method achieves 21.64 PSNR in future prediction compared to 13.27 by state-of-the-art optimization baselines, while reducing Chamfer Distance from 0.349 to 0.004. Crucially, ReconPhys enables fast inference (<1 second) versus hours required by existing methods, facilitating rapid generation of simulation-ready assets for robotics and graphics.
Figures & tables
Figure 1: ReconPhys predicts physical attributes using a feedforward neural network (FFN). This approach obviates the need for per-scene optimization and requires only a monocular video of the object as input. Within one second, it produces 3DGS assets with binded physical attributes. This capability offers a promising and efficient pathway for generating simulation-ready data for non-rigid manipulation, enabling the rapid transfer of real-world objects into simulation environments without human intervention.
Figure 2: Overview of our framework. Given an input image, a 3DGS predictor reconstructs the 3D object and samples anchor points to build a spring–mass system. Meanwhile, a physics predictor estimates physical attributes from videos to parameterize the system, producing simulatable 3D objects.
Figure 3: Training pipeline of ReconPhys with self-forcing. The physics predictor estimates physical parameters from input videos to drive a spring–mass system for motion prediction. The predicted state is rendered and supervised by a reconstruction loss, with a self-forcing feedback loop for temporal prediction.
Figure 4: Visualization of different methods. Our method produces more stable dynamics and realistic future states across different objects, while baseline methods either generate unstable deformations or cannot perform future prediction.
Method
Dynamic Reconstruction
Future Prediction
Time
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
EMD ↓
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
EMD ↓
4DGS Wu et al. (2024)
30.33
0.983
0.0392
0.593
1.5010
-
-
-
-
-
> 1h
Spring-Gaus Zhong et al. (2024)
22.26
0.874
0.1517
0.466
0.582
13.27
0.745
0.2856
0.349
0.424
> 1h
Ours
33.84
0.953
0.0366
0.001
0.007
21.64
0.907
0.0876
0.004
0.017
< 1s
Table 1: Cross-object generalization on 46 unseen objects. Our method achieves superior future prediction through physics-aware generalization, while maintaining reconstruction quality. Best results in bold .
Figure 5: Visualization of real-world non-rigid assets. Two objects are shown dropping and deforming over time T, with real-world input (top) and corresponding simulation results (bottom).
Reconstruction
Method
Cylinder-1
Cylinder-2
Headset-1
Headset-2
Hamburger-1
Hamburger-2
Lantern-1
Lantern-2
PSNR ↑
Spring-Gaus Zhong et al. (2024)
20.89
20.77
23.87
21.02
26.003
21.51
20.37
19.09
Ours
36.86
36.35
33.13
31.79
33.63
34.07
33.14
30.86
SSIM ↑
Spring-Gaus
0.8637
0.8744
0.8993
0.8744
0.9326
0.9104
0.8275
0.8169
Ours
0.9833
0.9822
0.9517
0.9435
0.9614
0.9670
0.9473
0.9327
LPIPS ↓
Spring-Gaus
0.1880
0.1616
0.1306
0.1424
0.0906
0.1077
0.1840
0.1884
Ours
0.0102
0.0233
0.0297
0.0425
0.0362
0.0339
0.0257
0.0610
Table 2: Physical disentanglement on objects with two distinct attributes sets. Our method preserves geometric fidelity while maximizing trajectory divergence between physical states.
Method
Stiffness
Damp
Mass
Friction
Spring-Gaus
827.67
2.546
2.276
1.082
Ours
297.3
1.151
1.337
1.508
Table 3: Comparison of physical attributes errors.
Figure 6: For the same object with different physical attributes, our method produces more accurate deformation predictions compared to Spring-Gaus.
Figure 7: Our method can be utilized in non-rigid manipulation. Four manipulation scenarios are demonstrated: (a) stretching the pencil bag, (b) squeezing the pillow, (c) stretching the hamburger, and (d) squashing and stretching the toy, showcasing the versatility of our approach across diverse deformable objects.
Reconstructing simulation-ready deformable objects is important for vision, graphics, and robotics. Existing physics-driven methods can recover physical digital twins from videos, but they suffer from two fundamental limitations: they typically assume a homogeneous material across the whole object, and their scene-specific inverse optimization, combined with the inherent ambiguity of monocular observation, yields inconsistent parameters for the same material across different scenes or interactions. We propose MatPhys, a material-aware feed-forward framework that predicts spring-mass parameters from a single-view video, addressing these two issues with two coupled designs. To relax the homogeneous material assumption, we use DINO features to decompose the object into semantically meaningful parts and to query a part-level material prior, assigning each part its own physical behavior. To enforce cross-scene consistency, we introduce a learned material codebook of shared material embeddings as the bridge between appearance and physics, and further use the part-level prior as a reference distribution that constrains the decoder so that the same material yields consistent parameters across scenes and interactions. Together, these designs turn an under-constrained monocular problem into feed-forward inference grounded on shared, reusable material concepts. Experiments show that our method matches per-scene optimization baselines in reconstruction and future prediction, while achieving stronger generalization to unseen interactions and objects with more consistent physical parameters.
Yang Yang, Yiyan Wang, Zheming Liu +1
1The University of Osaka · 3Huawei Technologies Japan K.K. · 2The University of Tokyo
Video generative models have made remarkable progress, yet they often yield visual artifacts that violate grounding in physical dynamics. Recent works such as PhysGen3D tackle single image-to-3D physics through mesh reconstruction and Physically-Based Rendering, but challenges remain in modeling fluid dynamics, multi-object interactions and photorealism. This work introduces 3DPhysVideo, a novel training-free pipeline that generates physically realistic videos from a single image. We repurpose an off-the-shelf video model for two stages. First, we use it as a novel view synthesizer to reconstruct complete 360-degree 3D scene geometry by guiding the image-to-video (I2V) flow model with rendered point clouds. Second, after applying physics solvers to this geometry, the physically simulated point cloud is used to guide the same I2V flow model to synthesize final, high-quality videos. Consistency-Guided Flow SDE, which decomposes the predicted velocity of the I2V flow model into denoising and consistency bias, enforces consistency to the conditional inputs, allowing us to effectively repurpose the model for both 3D reconstruction and simulation-guided video generation. In the diverse experiments including multi-objects, and fluid interaction scenes, our method successfully bridges the gap from single-images to physically plausible videos, while remaining efficient to run on a single consumer GPU. It outperforms state-of-the-art baselines on GPT-based scores, VideoPhy benchmark and human evaluation.
Existing inverse physics methods recover physical parameters from multi-view videos, where geometric constraints across views resolve scale and 3D structure. In monocular settings, however, such constraints are absent, leading to severe scale ambiguity, inaccurate geometry, and weak coupling between appearance optimization and physical simulation. To address these challenges, we propose MonoPhysics, a framework for monocular inverse physics estimation of deformable objects that jointly optimizes geometry, appearance, and physical parameters using a differentiable simulator and 3D Gaussian Splatting. Our key contribution is removing the multi-view capture requirement of existing methods, a necessary step toward handling in-the-wild video. MonoPhysics introduces three visual-physical bridges: scene re-parameterization, physics-aware geometry refinement, and a differentiable position map. We evaluate on Vid2Sim, real-world captures, and a new dataset of elastic and plasticine objects that we introduce. MonoPhysics outperforms monocular baselines in future prediction and recovers Young's modulus on Vid2Sim with accuracy comparable to a multi-view baseline. Code and data are available at https://daniel03c1.github.io/MonoPhysics/.
Daniel Rho, Jun Myeong Choi, Matthew Thornton +2
University of North Carolina at Chapel Hill · Meta