To reconstruct a predictive 4D representation of the physical world from monocular video captured by a moving camera, we jointly reconstruct dynamic Gaussian radiance fields and infer their underlying physical dynamics. This paper proposes Physical SLAM, a physics-aware method for constructing 4D Gaussian fields of deformable objects and predicting their future state when subjected to external actions, given a sequence of RGB-D images. Our method separates the scene into a static background Gaussian field for camera localization and a dynamic object Gaussian field driven by a spring--mass physical model of elastic deformation, inertia, and damping. To integrate visual observations with physical dynamics, we introduce virtual observation forces that gradually pull simulated surface nodes toward observed depth points, avoiding abrupt replacement of simulated node positions. Meanwhile, the physical parameters of the spring--mass model are updated from simulation errors aggregated over the full reconstruction observation window, enabling the reconstructed dynamics to align with observations. To couple appearance, geometry, and physics, we further design a physics-aware Gaussian transformation and surface alignment strategy, where dynamic Gaussians are transformed along the deforming physical surface of the spring--mass model and constrained to remain aligned with the surface during rendering-based optimization. The resulting representation supports observed-frame rendering and action-conditioned simulation, extending SLAM maps to describe how deformable objects respond to manipulation. On the PhysTwin benchmark, our method improves reconstruction and prediction PSNR over PhysTwin by 10.4% and 11.0%, respectively. On the TUM RGB-D benchmark, it reduces average camera tracking error by 20.6%, improves SSIM by 3.6%, and reduces LPIPS by 8.3% compared with 4DGS-SLAM.
Figures & tables
Fig. 1: Physics-aware SLAM with 4D Gaussian field for reconstruction and prediction. Existing 4D Gaussian SLAM methods mainly reconstruct time-varying geometry and appearance. Our method further reconstructs the physical dynamics of deformable objects, enabling online reconstruction and physically grounded action-conditioned prediction from RGB-D observations. For prediction, our method simulates the dynamics of the spring–mass model under the supplied actions and renders the resulting object states from a query camera pose.
Fig. 2: Pipeline of Physical SLAM. We decouple the static and dynamic mapping using semantics. For the static part, we build a globally consistent map and use the map to track the camera pose. For the dynamics part, we use a spring–mass system with external actions and observation forces to track the trajectory of surface nodes and use the geometry loss to progressively optimize the physical parameters; in parallel, we use a surface alignment method to align the 3D Gaussians to the mesh of the object to realise the tightly coupled optimization of physical dynamics, geometry, and appearance.
Fig. 3: Qualitative Results Compared with PhysTwin. Ours produces renderings that better align with the observations, with sharper appearance and more accurate silhouettes. Green boxes and connecting lines indicate magnified regions; blue and orange borders distinguish reconstruction and prediction, respectively.
Method
Reconstruction
Prediction
CD ↓
Track ↓
IoU(%) ↑
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
Track ↓
IoU(%) ↑
PSNR ↑
SSIM ↑
LPIPS ↓
Spring-Gau [ 27 ]
0.041
0.050
57.6
23.445
0.928
0.102
0.062
0.094
46.4
22.488
0.924
0.113
GS-Dynamics [ 41 ]
0.014
0.022
72.1
26.260
0.940
0.052
0.041
0.070
49.8
22.540
0.924
0.097
PhysTwin [ 3 ]
0.005
0.009
84.4
28.214
0.945
0.034
0.012
0.022
72.5
25.617
0.941
0.055
Ours
0.005
0.009
91.9
31.140
0.980
0.024
0.011
0.022
75.9
28.430
0.961
0.049
TABLE I: Comparison with state-of-the-art methods on the PhysTwin benchmark.
Fig. 4: Visual comparison of rendering results on the TUM RGB-D dataset.
Method
fr3/sit_st
fr3/sit_xyz
fr3/sit_rpy
fr3/walk_st
fr3/walk_xyz
fr3/walk_rpy
Avg.
RoDyn-SLAM [ 28 ]
1.5
5.6
5.7
1.7
8.3
8.1
5.1
MonoGS [ 2 ]
0.48
1.7
6.1
21.9
30.7
34.2
15.8
SplaTAM [ 6 ]
0.52
1.5
11.8
83.2
134.2
142.3
62.2
4DGS-SLAM [ 10 ]
0.58
2.9
2.6
0.52
2.1
2.6
1.8
Ours
0.53
0.87
2.29
0.50
1.23
3.13
1.43
TABLE II: Trajectory ATE RMSE [cm] ↓ on the TUM RGB-D sequences.
Method
Metric
fr3/sit_st
fr3/sit_xyz
fr3/sit_rpy
fr3/walk_st
fr3/walk_xyz
fr3/walk_rpy
Avg.
MonoGS [ 2 ]
PSNR ↑
19.95
23.92
16.99
16.47
14.02
15.12
17.74
SSIM ↑
0.739
0.803
0.572
0.604
0.436
0.497
0.608
LPIPS ↓
0.213
0.182
0.405
0.355
0.581
0.560
0.382
Gaussian-SLAM [ 8 ]
PSNR ↑
18.57
19.22
16.75
14.91
14.67
14.50
16.43
SSIM ↑
0.848
0.796
0.652
0.607
0.483
0.467
0.642
LPIPS ↓
0.291
0.326
0.521
0.489
0.626
0.630
0.480
TABLE III: Rendering quality on the TUM RGB-D FR3 sequences.
Method
Reconstruction
Prediction
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
4DGS-SLAM [ 10 ]
22.65
0.774
0.251
–
–
–
Ours
24.20
0.817
0.295
24.09
0.831
0.263
TABLE IV: Quantitative results on our moving-camera manipulation dataset.
Variant
Reconstruction
Prediction
CD ↓
Track ↓
IoU(%) ↑
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
Track ↓
IoU(%) ↑
PSNR ↑
SSIM ↑
LPIPS ↓
Full
0.005
0.009
91.9
31.140
0.980
0.024
0.011
0.022
75.9
28.430
0.961
0.049
w/o obs-force
0.005
0.010
83.5
29.650
0.958
0.044
0.012
0.022
68.2
25.120
0.945
0.064
w/o surface align loss
0.005
0.009
86.0
30.353
0.964
0.026
0.011
0.022
69.7
25.965
0.947
0.051
TABLE V: Ablation study on the PhysTwin benchmark.
Combining 3D Gaussian splatting with Simultaneous Localization and Mapping (SLAM) has gained popularity as it enables continuous 3D environment reconstruction during motion. However, existing methods struggle in dynamic environments, particularly moving objects complicate 3D reconstruction and, in turn, hinder reliable tracking. The emergence of 4D reconstruction, especially 4D Gaussian splatting, offers a promising direction for addressing these challenges, yet its potential for 4D-aware SLAM remains largely underexplored. Along this direction, we propose a robust and efficient framework, namely Reweighting Uncertainty in Gaussian Splatting SLAM (RU4D-SLAM) for 4D scene reconstruction, that introduces temporal factors into spatial 3D representation while incorporating uncertainty-aware perception of scene changes, blurred image synthesis, and dynamic scene reconstruction. We enhance dynamic scene representation by integrating motion blur rendering, and improve uncertainty-aware tracking by extending per-pixel uncertainty modeling, which is originally designed for static scenarios, to handle blurred images. Furthermore, we propose a semantic-guided reweighting mechanism for per-pixel uncertainty estimation in dynamic scenes, and introduce a learnable opacity weight to support adaptive 4D mapping. Extensive experiments on standard benchmarks demonstrate that our method substantially outperforms state-of-the-art approaches in both trajectory accuracy and 4D scene reconstruction, particularly in dynamic environments with moving objects and low-quality inputs. Code available: https://ru4d-slam.github.io
Yangfan Zhao, Hanwei Zhang, Ke Huang +3
Capital Normal University · Saarland University · Xi’an Jiaotong-Liverpool University +1
Real-time dense SLAM is a core capability for robotics applications that require robust localization and high- quality mapping in dynamic or fast-changing environments. Recent 3D Gaussian Splatting (3DGS)-based SLAM methods have shown promising performance, but most are designed for narrow-FoV pinhole cameras, where limited angular coverage weakens pose observability and often leads to unstable photo- metric optimization under rapid motion and large viewpoint changes. We present PanoGS-SLAM, the first panoramic dense SLAM system built on 3D Gaussian Splatting. Our method per- forms differentiable rendering and pose optimization directly in the spherical domain, enabling omnidirectional photometric constraints for more stable tracking. To improve geometric consistency and robustness, we introduce (1) a sphere-consistent photometric loss that compensates for the area distortion of equirectangular projection, and (2) a depth-guided Gaussian initialization strategy that stabilizes incremental mapping in newly observed regions. Extensive experiments on both real and synthetic panoramic benchmarks (PALVIO and SynPano) show that PanoGS-SLAM consistently outperforms geometric and GS-based baselines in tracking accuracy and rendering quality, while achieving fast front-end convergence and real-time perfor- mance. In addition, controlled field-of-view experiments reveal a clear monotonic improvement in optimization conditioning and convergence stability as angular coverage increases, high- lighting the fundamental role of sensing geometry in shaping the optimization landscape of differentiable Gaussian-based SLAM. The source code will be made publicly available.
Yongqi Mao, Hao Shi, Yufan Zhang +3
Zhejiang University · Ant Group · National University of Defense Technology
Handling the dynamic environments is a significant research challenge in Visual Simultaneous Localization and Mapping (SLAM). Recent research combines 3D Gaussian Splatting (3DGS) with SLAM to achieve both robust camera pose estimation and photorealistic renderings. However, using SLAM to efficiently reconstruct both static and dynamic regions remains challenging. In this work, we propose an efficient framework for dynamic 3DGS SLAM guided by optical flow. Using the input depth and prior optical flow, we first propose a category-agnostic motion mask generation strategy by fitting a camera ego-motion model to decompose the optical flow. This module separates dynamic and static Gaussians and simultaneously provides flow-guided camera pose initialization. We boost the training speed of dynamic 3DGS by explicitly modeling their temporal centers at keyframes. These centers are propagated using 3D scene flow priors and are dynamically initialized with an adaptive insertion strategy. Alongside this, we model the temporal opacity and rotation using a Gaussian Mixture Model (GMM) to adaptively learn the complex dynamics. The empirical results demonstrate our state-of-the-art performance in tracking, dynamic reconstruction, and training efficiency.
Yunsong Wang, Gim Hee Lee
School of Computing, National University of Singapore