PhyMo: Learning Physical Dynamics with Accurate and Continuous Motion from Multi-View Videos

Shangjia Liu; Jinxi Li; Siyuan Zhou; Bo Yang; Bing Wang

cs.CVOct 21, 2026

PhyMo: Learning Physical Dynamics with Accurate and Continuous Motion from Multi-View Videos

Authors: Shangjia Liu, Jinxi Li, Siyuan Zhou, Bo Yang, Bing Wang

Organizations: Spatial Intelligence Group, The Hong Kong Polytechnic University, Hong Kong SAR, China · vLAR Group, The Hong Kong Polytechnic University, Hong Kong SAR, China

Abstract

Recovering dynamic 3D scene geometry, appearance, and the physical motion states that govern scene evolution from multi-view videos is important yet challenging. A key difficulty is that appearance can be deceptive. Under rendering supervision alone, a model may fit the observed images by changing color or opacity to compensate for inaccurate deformation, rather than recovering the true evolving geometry and motion. This issue is especially severe in scenes with fast or locally complex motion, where inaccurate trajectories can still produce plausible images over the observed frames, but lead to unreliable reconstruction and discontinuous future prediction. In this paper, we propose PhyMo, a framework for learning evolving dynamics from multi-view videos. We introduce a Motion-Aware Trajectory Representation that drives the model to recover accurate motion from deformation, rather than relying on appearance shortcuts to explain observed motion. On top of this, we propose a Continuous Motion Evolution module with bidirectional kinematic continuity constraint to promote temporally consistent evolution of displacement, velocity, and acceleration. Together, these designs yield geometrically faithful interpolation and continuous motion extrapolation. Experiments on four dynamic scene benchmarks demonstrate state-of-the-art performance.

Figures & tables

Figure 1: Given video of a dynamic scene, our PhyMo can achieve geometrically faithful reconstruction and continuous motion prediction, comparing to other methods. In (a), white points denote the canonical point cloud, while colored points denote the dynamic point cloud at $t=0.5$ .

Figure 2: Overview of PhyMo. The left module, Motion-Aware Trajectory , starts from a canonical 3D representation and uses deformation network $F_{defo}$ to derive a motion-confidence gate $q_{i}$ that guides canonical appearance learning, encouraging motion to be explained by deformation rather than appearance shortcuts. The right module, Continuous Motion Evolution , bi-directionally sampling from the trajectory, Motion Network $M_{\theta}$ predicts velocity and acceleration and performs RK2 evolution , where bidirectional kinematic continuity loss and render loss are applied.

Figure 3: Two Failure Modes of 4D Reconstruction. 1. Static-background appearance shortcut: Adjusting static-region appearance $c_{i}$ improves PSNR but doesn’t recover evolving geometry. 2. Dynamic primitives disappearance shortcut: For fast motion, the optimizer makes primitives transparent by reducing $\sigma_{i}$ instead of recovering the correct trajectory, leading to a local minimum.

	NVIDIA Dynamic Scene			PhysInOne
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CD (m) $\downarrow$
TiNeuVox [ 12 ]	30.304	0.868	0.216	30.923	0.957	0.123	0.641
4DGS [ 52 ]	24.411	0.462	0.532	28.940	0.941	0.141	0.544
DefGS [ 55 ]	26.662	0.893	0.127	30.165	0.944	0.137	0.877
FreeGave [ 26 ]	30.276	0.891	0.096	30.850	0.952	0.124	0.496
ModecGS [ 22 ]	30.080	0.914	0.081	30.110	0.938	0.113	0.678

Table 1: Geometrically Faithful Interpolation Evaluation on NVIDIA Dynamic and PhysInOne Datasets. The best metric is highlight in red, the second best in orange.

Figure 4: Qualitative Results of Geometrically Faithful Interpolation Task.

	NVIDIA Dynamic Scene			Dynamic Multipart			FreeGave-GoPro			PhysInOne
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CD (m) $\downarrow$
TiNeuVox [ 12 ]	24.556	0.863	0.215	20.804	0.923	0.090	20.600	0.760	0.292	19.264	0.884	0.254	0.925
NVFi [ 24 ]	28.462	0.876	0.214	25.235	0.955	0.046	22.747	0.756	0.393	25.571	0.902	0.131	0.610
DefGS [ 55 ]	24.240	0.895	0.140	20.664	0.930	0.067	21.344	0.858	0.170	20.384	0.867	0.213	1.029
4DGS [ 52 ]	22.510	0.703	0.408	20.564	0.935	0.067	22.035	0.871	0.198	20.687	0.876	0.283	0.947
ODEGS [ 49 ]	25.296	0.916	0.117	33.371	0.982	0.009	25.561	0.899	0.124	24.023	0.916	0.121	0.854

Table 2: Quantitative Results for Continuous Motion Extrapolation on All Four Datasets. The best metric is highlight in red, while the second best highlight in orange.

Figure 5: Qualitative Results of Continuous Motion Extrapolation on Four Datasets.

					Interp.			Extrap.
	$q_{i}$	$\ell_{\text{cont1}}$	$\ell_{\text{cont2}}$	$M_{\theta}$	PSNR $\uparrow$	SSIM $\uparrow$	CD (m) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	CD (m) $\downarrow$
(1)	✗	✓	✓	✓	30.164	0.909	0.540	29.016	0.924	0.358
(2)	✓	✗	✓	✓	31.927	0.965	0.066	28.945	0.921	0.079
(2)	✓	✓	✗	✓	32.628	0.934	0.047	27.468	0.914	0.151
(3)	✓	✗	✗	✗	32.521	0.970	0.046	23.297	0.901	0.547
$1\delta t$	✓	✓	✓	✓	32.498	0.956	0.059	29.411	0.947	0.075

Table 3: Quantitative Results of Ablation Studies. The best metric is highlight in red, while the second best highlight in orange.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: RGB Rendering and the Evolving Geometry of Sandwich Scene.

Figure 7: Training Dynamics of Motion-Aware Trajectory Representation on the Sandwich Scene.

	Design			Interp.
Variant	$q_{i}$	Per-G.	Local Motion	PSNR $\uparrow$	SSIM $\uparrow$	CD (m) $\downarrow$
w/o $q_{i}$	✗	✗	✗	30.164	0.909	0.540
Freeze-then-release	✗	✗	✗	31.285	0.902	0.267
Disp.-based $q_{i}$	✓	✓	✗	30.874	0.924	0.351
Local-motion $q_{i}$	✓	✓	✓	32.631	0.971	0.046

Appendix

Table 4: Ablation Study of Motion-Guided Appearance on PhysInOne.

Figure 8: Qualitative Results of Ablation about Motion Guidance and Continuity Loss.

	Dynamic MultiPart			FreeGave-GoPro
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
TiNeuVox [ 12 ]	29.951	0.968	0.051	19.123	0.740	0.328
4DGS [ 52 ]	37.025	0.992	0.016	28.940	0.941	0.141
DefGS [ 55 ]	34.459	0.983	0.014	28.465	0.924	0.097
FreeGave [ 26 ]	35.776	0.986	0.016	28.457	0.925	0.091
ModecGS [ 22 ]	35.768	0.991	0.011	27.946	0.928	0.083

Appendix

Table 5: Interpolation Evaluation on Dynamic MultiPart and FreeGave-GoPro Dataset.

Figure 9: Qualitative Results for Extrapolation of Bus Scene on PhysInOne Dataset.

Figure 10: Qualitative Results for Extrapolation of Freefall Scene on PhysInOne Dataset.

Figure 11: Qualitative Results for Extrapolation of Wristrest Scene on FreeGave-GoPro Dataset.

Figure 12: Qualitative Results for Interpolation of Laptop Scene on PhysInOne Dataset.

Figure 13: Qualitative Results for Interpolation of Skating Scene on NVIDIA Dynamic Scene Dataset.

Figure 14: Qualitative Results of Longer Extrapolation on Four Datasets.

Dynamic Scene Reconstruction

Dynamic scene reconstruction builds time-varying 3D models of moving objects and environments from video or multi-view images.

See all 289 papers

Analytic Dynamics: Learning Physics-Grounded Representation for Fast Intrinsic Dynamics Inference from Monocular Videos
Aug 31, 2026Jailing Lin, Jikuan Zhang, Jianhua SunMaterials Property Prediction Video Representation Learning
School of Artificial Intelligence, Shanghai Jiao Tong University
VeloGauss: Learning Physically Consistent Gaussian Velocity Fields from Videos
May 11, 2026Nengbo Lu, Bin Zhao3D Gaussian Splatting Dynamic Scene Reconstruction
Guangxi Key Laboratory of Robot Intelligent Perception and Control, School of Artificial Intelligence, Guilin University of Electronic Technology, Guilin, 541004, China
PhyMAGIC: Physical Motion-Aware Generative Inference with Confidence-guided LLM
May 22, 2025Siwei Meng, Yawei Luo, Ping LiuPhysics-Informed Generative Modeling Physical Consistency in Video Generation
Department of Computer Science & Engineering, University of Nevada, Reno, USA · School of Software Technology, Zhejiang University, China

Aug 31, 2026J/K move · Enter open · S save

Analytic Dynamics: Learning Physics-Grounded Representation for Fast Intrinsic Dynamics Inference from Monocular Videos

Jailing Lin, Jikuan Zhang, Jianhua Sun

School of Artificial Intelligence, Shanghai Jiao Tong University

Materials Property Prediction Video Representation Learning System Identification Physics-Informed ML Inverse Dynamics

Open paper AskPDF arXiv

Inferring object dynamics from visual observations is essential for intelligent agents to reason about and interact with the physical world, yet remains challenging due to the fundamental gap between visual evidence and intrinsic dynamics. Existing methods either rely on costly per-scene optimization, limiting efficiency and scalability, or directly map visual evidence to intrinsic dynamics without intermediate physical abstractions, making them prone to appearance and geometry shortcuts. To bridge this gap, we propose Analytic Dynamics, a feed-forward dynamics inference framework that introduces an intermediate physics-grounded dynamics representation between visual observations and intrinsic dynamics. Specifically, we leverage privileged physical states, including position, displacement, and deformation gradient fields, which are available in simulation, to learn a structured dynamics representation that is difficult to discover from visual observations alone. By aligning visual representations with this space, we equip visual models with a physics-grounded inductive bias, guiding them to capture dynamics-relevant patterns for material model classification and parameter regression. To facilitate this research, we develop a dynamics data generation pipeline and benchmark containing paired physical state trajectories, rendered videos, and ground-truth material models and parameters. Extensive experiments demonstrate that Analytic Dynamics achieves efficient, accurate, and generalizable dynamics inference from monocular videos.

Abstract

Dynamic Scene Reconstruction

Explore similar work

Analytic Dynamics: Learning Physics-Grounded Representation for Fast Intrinsic Dynamics Inference from Monocular Videos

VeloGauss: Learning Physically Consistent Gaussian Velocity Fields from Videos

PhyMAGIC: Physical Motion-Aware Generative Inference with Confidence-guided LLM