Organizations: Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · School of Automation and Intelligent Sensing, the State Key Laboratory of Avionics Integration and Aviation System of-Systems Synthesis, and Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University, Shanghai 200240, China
Visual simultaneous localization and mapping (V-SLAM) is a fundamental capability for autonomous perception and navigation. However, endoscopic scenes violate the rigidity assumption due to persistent soft-tissue deformations, creating a strong coupling ambiguity between camera ego-motion and intrinsic deformation. Although recent monocular non-rigid SLAM methods have made notable progress, they often lack effective decoupling mechanisms and rely on sparse or low-fidelity scene representations, which leads to tracking drift and limited reconstruction quality. To address these limitations, we propose NRGS-SLAM, a monocular non-rigid SLAM system for endoscopy based on 3D Gaussian Splatting. To resolve the coupling ambiguity, we introduce a deformation-aware 3D Gaussian map that augments each Gaussian primitive with a learnable deformation probability, optimized via a Bayesian self-supervision strategy without requiring external non-rigidity labels. Building on this representation, we design a deformable tracking module that performs robust coarse-to-fine pose estimation by prioritizing low-deformation regions, followed by efficient per-frame deformation updates. A carefully designed deformable mapping module progressively expands and refines the map, balancing representational capacity and computational efficiency. In addition, a unified robust geometric loss incorporates external geometric priors to mitigate the inherent ill-posedness of monocular non-rigid SLAM. Extensive experiments on multiple public endoscopic datasets demonstrate that NRGS-SLAM achieves more accurate camera pose estimation (up to 50% reduction in RMSE) and higher-quality photo-realistic reconstructions than state-of-the-art methods. Comprehensive ablation studies further validate the effectiveness of our key design choices. Source code will be publicly available upon paper acceptance.
Figures & tables
Figure 1: (a) Illustration of the Coupling Ambiguity: In monocular non-rigid scenarios, pixel displacement arise from a complex combination of camera ego-motion and intrinsic tissue deformation. This creates a fundamental coupling ambiguity, making it difficult to distinguish rigid motion from non-rigid dynamics or their combination. (b) Existing methods often struggle to effectively decouple these factors during joint optimization and rely on sparse representations (e.g., points or meshes), resulting in tracking drift and low visual fidelity. (c) Our proposed NRGS-SLAM introduces a deformation-aware 3D Gaussian map. By learning a per-primitive deformation probability (visualized by the color spectrum ranging from blue (rigid) to red (non-rigid)), our system explicitly decouples camera tracking from deformation updates, enabling both accurate pose estimation and high-fidelity photo-realistic reconstruction.
Figure 2: Overview of our proposed NRGS-SLAM. Central to our approach is the deformation-aware 3D Gaussian map (see Sec. IV ), which represents the scene using canonical 3D Gaussians augmented by deformation probabilities and models temporal deformations via Gaussian Basis Functions. Given sequential RGB captured in a deformable environment, the measurement preprocessing module (see Sec. V ) extracts geometric priors and generates valid and co-visibility masks. Subsequently, deformable tracking (see Sec. VI ) performs frame-by-frame estimation of camera poses and scene deformations. Upon the insertion of a new keyframe, the deformable mapping module (see Sec. VII ) is triggered to expand and globally optimize the map.
Figure 3: Left : The canonical space is populated with 3D Gaussian primitives augmented by a learnable deformation probability wd . Primitives in rigid regions (blue, wd→0 ) remain static, while those in deformable tissues (red, wd→1 ) are allowed to deform over time. Middle : These probabilistic attributes are projected via differentiable rasterization. Right : The resulting 2D deformation confidence map Mdef visualizes the pixel-wise confidence of scene non-rigidity. It serves as a crucial weighting mask to decouple camera ego-motion from intrinsic scene deformation during tracking and mapping.
Figure 4: Overview of the measurement preprocessing pipeline. The module processes the incoming frame It to extract geometric cues via a geometric foundation model (left) and determines valid and co-visibility masks (right).
Figure 5: Overview of the deformable tracking pipeline. Top: Deformation-Aware Camera Tracking estimates the camera pose in a coarse-to-fine manner. The first stage computes an initial pose using deformation-filtered 3D-2D correspondences. The second stage refines this pose by aligning rendered frame and geometries with current observations and geometric priors; crucially, this optimization is weighted by the rendered deformation probability map to prioritize reliable rigid regions. Bottom: Per-frame Deformation Estimation updates the deformation field to match the input frame. Guided by the deformation probabilities, we employ an efficient residual-based optimization to selectively capture non-rigid variations.
Figure 6: Overview of the deformable mapping. When a new keyframe is inserted, the system operates within a sliding window to optimize camera poses and extend the map with newly observed regions. A global deformable bundle adjustment jointly refines camera states and scene parameters. Within this optimization, the Deformation Probability Estimation module (Sec. VII-D ) infers posterior deformation probabilities to supervise their optimization, while the Dynamic Deformation Field Management module (Sec. VII-E ) adaptively adjusts the number of temporal basis functions.
Methods
P2-2
P2-3
P2-4
P2-5
Avg.
Clip
Full
Clip
Full
Clip
Full
Clip
Full
Clip
Full
RMSE
S.D.
RMSE
S.D.
RMSE
S.D.
RMSE
S.D.
RMSE
S.D.
RMSE
S.D.
RMSE
S.D.
RMSE
S.D.
RMSE
S.D.
RMSE
S.D.
Traditional Monocular Non-rigid SLAM
DefSLAM
48.45
19.22
28.96
21.99
0.026
0.018
3.93
3.75
33.73
16.34
54.54
23.64
28.46
13.19
60.85
17.97
27.67
12.19
37.07
16.84
NR-SLAM
38.62
16.85
28.22
20.60
0.026
0.018
2.69
2.43
23.97
10.70
40.24
18.87
×
×
×
×
20.87 †
9.19 †
23.72 †
13.97 †
General-Purpose GS-based SLAM
Table I : Quantitative comparison of camera localization accuracy (ATE) on the StereoMIS dataset. Values are reported as RMSE and Standard Deviation (S.D.) in millimeters (mm). “Clip” denotes evaluation on a continuous 1,000-frame segment. × indicates tracking failure. † denotes that the average is computed over successfully tracked sequences only.
Methods
c1_descending_t4_v4
c1_sigmoid1_t4_v4
c1_sigmoid2_t4_v4
c2_transverse1_t1_v4
Avg.
RMSE
S.D.
RMSE
S.D.
RMSE
S.D.
RMSE
S.D.
RMSE
S.D.
Traditional Monocular non-rigid SLAM
DefSLAM
21.35
11.27
10.59
4.31
10.35
3.71
13.49
5.99
13.94
6.32
NR-SLAM
16.84
10.74
×
×
6.09
2.87
10.93
6.29
11.29 †
6.63 †
General-Purpose GS-based SLAM
MonoGS
34.56
16.70
27.44
11.80
29.07
15.95
35.65
18.25
31.67
15.68
Table II : Quantitative comparison of camera localization accuracy (ATE) on the C3VDv2 dataset. Values are reported as RMSE and Standard Deviation (S.D.) in millimeters (mm). × indicates tracking failure. † denotes that the average is computed over successfully tracked sequences only.
Figure 7: Qualitative comparison of camera trajectory estimation on the StereoMIS dataset. The top row presents representative image frames in temporal order. The subsequent rows show 3D trajectory comparisons of different methods. Estimated trajectories are shown in blue and ground truth in black. The star marker denotes the starting point.
Methods
P2-2 (Clip)
P2-3 (Clip)
P2-4 (Clip)
P2-5 (Clip)
Avg. (Clip)
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
General-Purpose GS-based SLAM
MonoGS
16.59
0.620
0.671
23.09
0.660
0.321
16.99
0.478
0.565
15.40
0.382
0.693
18.02
0.535
0.563
S3PO
15.84
0.659
0.713
23.21
0.591
0.399
19.71
0.572
0.729
20.05
0.522
0.749
19.70
0.586
0.648
4DTAM
17.78
0.652
0.569
23.39
0.649
0.252
21.79
0.582
0.471
19.78
0.577
0.597
20.68
0.615
0.472
Endoscopy-Specific Differentiable Rendering SLAM
Table III : Quantitative comparison of rendering quality on the StereoMIS dataset under Clip (1000 frames) and Full sequence settings. PSNR (dB) and SSIM are higher-is-better, while LPIPS is lower-is-better.
Figure 8: Qualitative comparison of camera trajectory estimation on the C3VDv2 dataset [ 44 ] . The top row presents representative image frames in temporal order. The subsequent rows show 3D trajectory comparisons of different methods. Estimated trajectories are shown in blue and ground truth in black. The star marker denotes the starting point.
Methods
rectified18
rectified19
rectified23
Avg.
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
General-Purpose GS-based SLAM
MonoGS
25.69
0.848
0.457
29.30
0.889
0.455
23.01
0.723
0.489
26.00
0.820
0.467
S3PO
17.93
0.751
0.465
26.56
0.880
0.496
18.90
0.678
0.489
21.13
0.770
0.484
4DTAM
23.04
0.863
0.428
29.07
0.901
0.470
23.04
0.747
0.458
25.05
0.837
0.452
Endoscopy-Specific Differentiable Rendering SLAM
Table IV : Quantitative comparison of rendering quality on the Hamlyn dataset.
Methods
c1_descending_t4_v4
c1_sigmoid1_t4_v4
c1_sigmoid2_t4_v4
c2_transverse1_t1_v4
Avg.
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
General-Purpose GS-based SLAM
MonoGS
17.04
0.406
0.592
16.61
0.510
0.720
17.20
0.640
0.761
15.14
0.538
0.659
16.50
0.523
0.683
S3PO
15.75
0.572
0.790
19.11
0.690
0.729
17.76
0.600
0.768
18.93
0.746
0.638
17.89
0.652
0.731
4DTAM
22.61
0.618
0.673
26.48
0.728
0.647
24.57
0.667
0.688
22.18
0.777
0.580
23.96
0.698
0.647
Endoscopy-Specific Differentiable Rendering SLAM
Table V : Quantitative comparison of rendering quality on the C3VDv2 dataset.
Figure 9: Qualitative comparison of rendering results across different methods at multiple time steps on StereoMIS and C3VDv2 datasets. For each method, the RGB rendering is shown with the corresponding depth map inset at the bottom right. On both datasets, the depth maps are included for reference only. For C3VDv2, the depth maps are inferred offline using [ 38 ] , while for StereoMIS, they are computed using the stereo model provided by the dataset [ 44 ] .
Figure 10: Qualitative comparison of environment mapping results on Sequence P2-5 of the StereoMIS dataset. The top row shows the results of 4DTAM and the bottom row shows the results of NRGS-SLAM. The reconstructed camera trajectory is visualized in blue. Cyan ellipses highlight regions for visual comparison. NRGS-SLAM demonstrates improved structural completeness and higher-fidelity reconstruction with fewer artifacts compared to 4DTAM.
Configuration
StereoMIS
C3VDv2
RMSE ↓
S.D. ↓
RMSE ↓
S.D. ↓
A1. w/o Geo. Priors
13.85
6.42
13.55
6.65
A2. w/o Robust Opt.
8.95
4.36
9.36
4.25
A3. w/o Val. Mask
7.92
3.32
8.42
3.91
A4. w/o Co-vis. Mask
8.02
3.84
8.13
3.75
Ours (Full)
7.41
3.03
7.96
3.66
Table VI : Ablation Study on Measurement Preprocessing and Geometric Supervision.
Configuration
StereoMIS
C3VDv2
RMSE ↓
S.D. ↓
RMSE ↓
S.D. ↓
B1. w/o Stage 1 (PnP Init.)
8.75
3.94
8.84
4.12
B2. w/o Stage 2 (Refinement)
18.45
6.82
21.26
9.41
B3. w/o Def. Weighting
10.84
4.24
13.43
6.45
Ours (Full Model)
7.41
3.03
7.96
3.66
Table VII : Ablation Study on Deformable Camera Tracking.
Configuration
StereoMIS
C3VDv2
RMSE ↓
S.D. ↓
FPS ↑
RMSE ↓
S.D. ↓
FPS ↑
C1. Full Param. Update
7.32
2.98
0.652
7.66
3.24
0.77
Ours (Residual-based)
7.41
3.03
0.924
7.96
3.66
1.05
Table VIII : Ablation Study on Per-frame Deformation Estimation. FPS: Frames Per Second.
Figure 11: Ablation study of the Deformation-Aware Camera Tracking. (a) ATE RMSE versus optimization iterations for the full model and the variant without Stage 1. (b) Comparison between the variant without deformation-aware weighting (left) and the full model (right). The top row shows the estimated camera trajectories colored by RMSE. The bottom row presents the observed image sequence over a time interval with large tracking errors. The rightmost image shows the 2D deformation confidence map of the final frame.
Configuration
StereoMIS
C3VDv2
RMSE ↓
S.D. ↓
FPS ↑
RMSE ↓
S.D. ↓
FPS ↑
D1. w/o Pose Opt.
8.04
3.56
0.96
8.24
3.71
1.09
D2. w/o Def. Prob. Est.
11.95
5.42
1.09
11.49
4.54
1.26
D3. w/o Dyn. Mgmt.
7.84
3.26
0.81
8.62
3.97
0.95
D4. w/o wd -Gating
8.15
3.62
0.94
8.54
3.98
1.05
Ours (Full Model)
7.41
3.03
0.92
7.96
3.66
1.05
Table IX : Ablation Study on Deformable Mapping.
Figure 12: Impact of dynamic deformation management on the evolution of active deformation parameters. The number of active deformation parameters over frames for (a) without dynamic management and (b) with dynamic management. Without management, parameters grow steadily over time, whereas dynamic management maintains a bounded and stable model complexity.
Dataset-Scene
Frame Number
Total Runtime
Tracking /Frame
FPS
StereoMIS-P2_4
4281
4896s
232ms
0.874
C3VDv2-C1_descending_t4_v4
361
383s
209ms
0.943
Table X : Runtime Analysis.
Figure 13: Speed–accuracy trade-off on two datasets. Higher FPS indicates faster processing, while lower ATE RMSE indicates better accuracy.
Dense 3D reconstruction is critical for clinical endoscopic navigation and documentation. While Gaussian Splatting SLAM systems show promise in this domain, they fundamentally rely on strict multi-view photometric consistency. In routine procedures, this assumption is severely violated by intermittent optical degradations like moving debris and water flushing. Standard systems erroneously fuse these cameraattached artifacts into the persistent 3D geometry, causing severe tracking drift and irreversible map corruption. To address this limitation, we propose EndoMD-SLAM, a framework designed to maintain stability under optical degradation through specialized tracking and mapping mechanisms. On the tracking side, a memory-driven gating mechanism detects unreliable observations to suspend map updates and utilizes historical keyframes for drift-aware relocalization. On the mapping side, a self-supervised static-transient decomposition isolates visual contaminants into a dedicated transient field. This explicit separation prevents artifacts from structurally entangling with the persistent anatomical map. We curate a degradationfocused benchmark from colonoscopy videos to systematically evaluate these failure modes. Extensive experiments show that while standard baselines fail under severe optical degradation, EndoMD-SLAM preserves geometric integrity, reducing absolute trajectory error by 91% and improving rendering fidelity by 9.9 dB PSNR.
Nuo Chen, Kangqi Ni, Lulin Liu +5
Texas A&M University, College Station, TX, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA · University of Minnesota, Minneapolis, MN, USA +1
Gaussian splatting is the current state-of-the-art for dense, deformable 3D anatomy reconstruction in robot-assisted minimally invasive surgery (RAMIS); however, most pipelines are offline and depend on accurate camera trajectory priors (often from robotic kinematics), limiting applicability when priors are missing or noisy. To address these limitations, we propose Track2Map, an online 3D Gaussian Splatting pipeline that jointly optimizes camera trajectory and 3D deformable scene representation directly from surgical video. Track2Map is therefore capable of robust 3D reconstructions when camera trajectory priors are either absent or noisy, and due to its online nature it effectively works as a Simultaneous Localisation and Mapping (SLAM) method. To stabilize optimization in the presence of tissue motion and ambiguous visual cues, we introduce a track-anchored deformation initialization using dense 2D point tracks. Track statistics are further utilized to disentangle camera motion from scene deformation by detecting static camera periods and reducing drift during incremental mapping. Experiments on StereoMIS show improved reconstruction quality and camera trajectory against competing SLAM methods, as well as compared to non-SLAM methods that utilize camera trajectory priors. The code is available at https://track2map.github.io/.
Tianyi Song, Sierra Bonilla, Xinwei Ju +6
Department of Computer Science and UCL Hawkes Institute, University College London, London WC1E 6BT, UK · Intuitive Surgical, Inc., Sunnyvale, CA 94086, USA
Gaussian splatting has recently gained traction as a compelling map representation for SLAM systems, enabling dense and photo-realistic scene modeling. However, its application to monocular SLAM remains challenging due to the lack of reliable geometric cues from monocular input. Without geometric supervision, mapping or tracking could fall in local-minima, resulting in structural degeneracies and inaccuracies. To address this challenge, we propose GaussianFlow SLAM, a monocular 3DGS-SLAM that leverages optical flow as a geometry-aware cue to guide the optimization of both the scene structure and camera poses. By encouraging the projected motion of Gaussians, termed GaussianFlow, to align with the optical flow, our method introduces consistent structural cues to regularize both map reconstruction and pose estimation. Furthermore, we introduce normalized error-based densification and pruning modules to refine inactive and unstable Gaussians, thereby contributing to improved map quality and pose accuracy. Experiments conducted on public datasets demonstrate that our method achieves superior rendering quality and tracking accuracy compared with state-of-the-art algorithms. The source code is available at: https://github.com/url-kaist/gaussianflow-slam.
Dong-Uk Seo, Jinwoo Jeon, Eungchang Mason Lee +1
School of Electrical Engineering, KAIST (Korea Advanced Institute of Science and Technology), Daejeon 34141, Republic of Korea