NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting

Jiwei Shan; Zeyu Cai; Yirui Li; Yongbo Chen; Lijun Han; Yun-hui Liu; Hesheng Wang; Shing Shin Cheng

cs.CVFeb 19, 2026

NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting

Authors: Jiwei Shan, Zeyu Cai, Yirui Li, Yongbo Chen, Lijun Han, Yun-hui Liu, Hesheng Wang, Shing Shin Cheng

Organizations: Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · School of Automation and Intelligent Sensing, the State Key Laboratory of Avionics Integration and Aviation System of-Systems Synthesis, and Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University, Shanghai 200240, China

Abstract

Visual simultaneous localization and mapping (V-SLAM) is a fundamental capability for autonomous perception and navigation. However, endoscopic scenes violate the rigidity assumption due to persistent soft-tissue deformations, creating a strong coupling ambiguity between camera ego-motion and intrinsic deformation. Although recent monocular non-rigid SLAM methods have made notable progress, they often lack effective decoupling mechanisms and rely on sparse or low-fidelity scene representations, which leads to tracking drift and limited reconstruction quality. To address these limitations, we propose NRGS-SLAM, a monocular non-rigid SLAM system for endoscopy based on 3D Gaussian Splatting. To resolve the coupling ambiguity, we introduce a deformation-aware 3D Gaussian map that augments each Gaussian primitive with a learnable deformation probability, optimized via a Bayesian self-supervision strategy without requiring external non-rigidity labels. Building on this representation, we design a deformable tracking module that performs robust coarse-to-fine pose estimation by prioritizing low-deformation regions, followed by efficient per-frame deformation updates. A carefully designed deformable mapping module progressively expands and refines the map, balancing representational capacity and computational efficiency. In addition, a unified robust geometric loss incorporates external geometric priors to mitigate the inherent ill-posedness of monocular non-rigid SLAM. Extensive experiments on multiple public endoscopic datasets demonstrate that NRGS-SLAM achieves more accurate camera pose estimation (up to 50% reduction in RMSE) and higher-quality photo-realistic reconstructions than state-of-the-art methods. Comprehensive ablation studies further validate the effectiveness of our key design choices. Source code will be publicly available upon paper acceptance.

Figures & tables

Figure 1: (a) Illustration of the Coupling Ambiguity: In monocular non-rigid scenarios, pixel displacement arise from a complex combination of camera ego-motion and intrinsic tissue deformation. This creates a fundamental coupling ambiguity, making it difficult to distinguish rigid motion from non-rigid dynamics or their combination. (b) Existing methods often struggle to effectively decouple these factors during joint optimization and rely on sparse representations (e.g., points or meshes), resulting in tracking drift and low visual fidelity. (c) Our proposed NRGS-SLAM introduces a deformation-aware 3D Gaussian map. By learning a per-primitive deformation probability (visualized by the color spectrum ranging from blue (rigid) to red (non-rigid)), our system explicitly decouples camera tracking from deformation updates, enabling both accurate pose estimation and high-fidelity photo-realistic reconstruction.

Figure 2: Overview of our proposed NRGS-SLAM. Central to our approach is the deformation-aware 3D Gaussian map (see Sec. IV ), which represents the scene using canonical 3D Gaussians augmented by deformation probabilities and models temporal deformations via Gaussian Basis Functions. Given sequential RGB captured in a deformable environment, the measurement preprocessing module (see Sec. V ) extracts geometric priors and generates valid and co-visibility masks. Subsequently, deformable tracking (see Sec. VI ) performs frame-by-frame estimation of camera poses and scene deformations. Upon the insertion of a new keyframe, the deformable mapping module (see Sec. VII ) is triggered to expand and globally optimize the map.

Figure 3: Left : The canonical space is populated with 3D Gaussian primitives augmented by a learnable deformation probability $w_{d}$ . Primitives in rigid regions (blue, $w_{d}\to 0$ ) remain static, while those in deformable tissues (red, $w_{d}\to 1$ ) are allowed to deform over time. Middle : These probabilistic attributes are projected via differentiable rasterization. Right : The resulting 2D deformation confidence map $\mathbf{M}_{\text{def}}$ visualizes the pixel-wise confidence of scene non-rigidity. It serves as a crucial weighting mask to decouple camera ego-motion from intrinsic scene deformation during tracking and mapping.

Figure 4: Overview of the measurement preprocessing pipeline. The module processes the incoming frame $I_{t}$ to extract geometric cues via a geometric foundation model (left) and determines valid and co-visibility masks (right).

Figure 5: Overview of the deformable tracking pipeline. Top: Deformation-Aware Camera Tracking estimates the camera pose in a coarse-to-fine manner. The first stage computes an initial pose using deformation-filtered 3D-2D correspondences. The second stage refines this pose by aligning rendered frame and geometries with current observations and geometric priors; crucially, this optimization is weighted by the rendered deformation probability map to prioritize reliable rigid regions. Bottom: Per-frame Deformation Estimation updates the deformation field to match the input frame. Guided by the deformation probabilities, we employ an efficient residual-based optimization to selectively capture non-rigid variations.

Figure 6: Overview of the deformable mapping. When a new keyframe is inserted, the system operates within a sliding window to optimize camera poses and extend the map with newly observed regions. A global deformable bundle adjustment jointly refines camera states and scene parameters. Within this optimization, the Deformation Probability Estimation module (Sec. VII-D ) infers posterior deformation probabilities to supervise their optimization, while the Dynamic Deformation Field Management module (Sec. VII-E ) adaptively adjusts the number of temporal basis functions.

Methods	P2-2				P2-3				P2-4				P2-5				Avg.
	Clip		Full		Clip		Full		Clip		Full		Clip		Full		Clip		Full
	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.
Traditional Monocular Non-rigid SLAM
DefSLAM	48.45	19.22	28.96	21.99	0.026	0.018	3.93	3.75	33.73	16.34	54.54	23.64	28.46	13.19	60.85	17.97	27.67	12.19	37.07	16.84
NR-SLAM	38.62	16.85	28.22	20.60	0.026	0.018	2.69	2.43	23.97	10.70	40.24	18.87	$\times$	$\times$	$\times$	$\times$	20.87 †	9.19 †	23.72 †	13.97 †
General-Purpose GS-based SLAM

Table I : Quantitative comparison of camera localization accuracy (ATE) on the StereoMIS dataset. Values are reported as RMSE and Standard Deviation (S.D.) in millimeters (mm). “Clip” denotes evaluation on a continuous 1,000-frame segment.

\times

indicates tracking failure.

\dagger

denotes that the average is computed over successfully tracked sequences only.

Methods	c1_descending_t4_v4		c1_sigmoid1_t4_v4		c1_sigmoid2_t4_v4		c2_transverse1_t1_v4		Avg.
Methods	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.
Traditional Monocular non-rigid SLAM
DefSLAM	21.35	11.27	10.59	4.31	10.35	3.71	13.49	5.99	13.94	6.32
NR-SLAM	16.84	10.74	$\times$	$\times$	6.09	2.87	10.93	6.29	11.29 †	6.63 †
General-Purpose GS-based SLAM
MonoGS	34.56	16.70	27.44	11.80	29.07	15.95	35.65	18.25	31.67	15.68

Table II : Quantitative comparison of camera localization accuracy (ATE) on the C3VDv2 dataset. Values are reported as RMSE and Standard Deviation (S.D.) in millimeters (mm).

\times

indicates tracking failure.

\dagger

denotes that the average is computed over successfully tracked sequences only.

Figure 7: Qualitative comparison of camera trajectory estimation on the StereoMIS dataset. The top row presents representative image frames in temporal order. The subsequent rows show 3D trajectory comparisons of different methods. Estimated trajectories are shown in blue and ground truth in black. The star marker denotes the starting point.

Methods	P2-2 (Clip)			P2-3 (Clip)			P2-4 (Clip)			P2-5 (Clip)			Avg. (Clip)
Methods	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS
General-Purpose GS-based SLAM
MonoGS	16.59	0.620	0.671	23.09	0.660	0.321	16.99	0.478	0.565	15.40	0.382	0.693	18.02	0.535	0.563
S3PO	15.84	0.659	0.713	23.21	0.591	0.399	19.71	0.572	0.729	20.05	0.522	0.749	19.70	0.586	0.648
4DTAM	17.78	0.652	0.569	23.39	0.649	0.252	21.79	0.582	0.471	19.78	0.577	0.597	20.68	0.615	0.472
Endoscopy-Specific Differentiable Rendering SLAM

Table III : Quantitative comparison of rendering quality on the StereoMIS dataset under Clip (1000 frames) and Full sequence settings. PSNR (dB) and SSIM are higher-is-better, while LPIPS is lower-is-better.

Figure 8: Qualitative comparison of camera trajectory estimation on the C3VDv2 dataset [ 44 ] . The top row presents representative image frames in temporal order. The subsequent rows show 3D trajectory comparisons of different methods. Estimated trajectories are shown in blue and ground truth in black. The star marker denotes the starting point.

Methods	rectified18			rectified19			rectified23			Avg.
Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
General-Purpose GS-based SLAM
MonoGS	25.69	0.848	0.457	29.30	0.889	0.455	23.01	0.723	0.489	26.00	0.820	0.467
S3PO	17.93	0.751	0.465	26.56	0.880	0.496	18.90	0.678	0.489	21.13	0.770	0.484
4DTAM	23.04	0.863	0.428	29.07	0.901	0.470	23.04	0.747	0.458	25.05	0.837	0.452
Endoscopy-Specific Differentiable Rendering SLAM

Table IV : Quantitative comparison of rendering quality on the Hamlyn dataset.

Methods	c1_descending_t4_v4			c1_sigmoid1_t4_v4			c1_sigmoid2_t4_v4			c2_transverse1_t1_v4			Avg.
Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
General-Purpose GS-based SLAM
MonoGS	17.04	0.406	0.592	16.61	0.510	0.720	17.20	0.640	0.761	15.14	0.538	0.659	16.50	0.523	0.683
S3PO	15.75	0.572	0.790	19.11	0.690	0.729	17.76	0.600	0.768	18.93	0.746	0.638	17.89	0.652	0.731
4DTAM	22.61	0.618	0.673	26.48	0.728	0.647	24.57	0.667	0.688	22.18	0.777	0.580	23.96	0.698	0.647
Endoscopy-Specific Differentiable Rendering SLAM

Table V : Quantitative comparison of rendering quality on the C3VDv2 dataset.

Figure 9: Qualitative comparison of rendering results across different methods at multiple time steps on StereoMIS and C3VDv2 datasets. For each method, the RGB rendering is shown with the corresponding depth map inset at the bottom right. On both datasets, the depth maps are included for reference only. For C3VDv2, the depth maps are inferred offline using [ 38 ] , while for StereoMIS, they are computed using the stereo model provided by the dataset [ 44 ] .

Figure 10: Qualitative comparison of environment mapping results on Sequence P2-5 of the StereoMIS dataset. The top row shows the results of 4DTAM and the bottom row shows the results of NRGS-SLAM. The reconstructed camera trajectory is visualized in blue. Cyan ellipses highlight regions for visual comparison. NRGS-SLAM demonstrates improved structural completeness and higher-fidelity reconstruction with fewer artifacts compared to 4DTAM.

Configuration	StereoMIS		C3VDv2
Configuration	RMSE $\downarrow$	S.D. $\downarrow$	RMSE $\downarrow$	S.D. $\downarrow$
A1. w/o Geo. Priors	13.85	6.42	13.55	6.65
A2. w/o Robust Opt.	8.95	4.36	9.36	4.25
A3. w/o Val. Mask	7.92	3.32	8.42	3.91
A4. w/o Co-vis. Mask	8.02	3.84	8.13	3.75
Ours (Full)	7.41	3.03	7.96	3.66

Table VI : Ablation Study on Measurement Preprocessing and Geometric Supervision.

Configuration	StereoMIS		C3VDv2
Configuration	RMSE $\downarrow$	S.D. $\downarrow$	RMSE $\downarrow$	S.D. $\downarrow$
B1. w/o Stage 1 (PnP Init.)	8.75	3.94	8.84	4.12
B2. w/o Stage 2 (Refinement)	18.45	6.82	21.26	9.41
B3. w/o Def. Weighting	10.84	4.24	13.43	6.45
Ours (Full Model)	7.41	3.03	7.96	3.66

Table VII : Ablation Study on Deformable Camera Tracking.

Configuration	StereoMIS			C3VDv2
Configuration	RMSE $\downarrow$	S.D. $\downarrow$	FPS $\uparrow$	RMSE $\downarrow$	S.D. $\downarrow$	FPS $\uparrow$
C1. Full Param. Update	7.32	2.98	0.652	7.66	3.24	0.77
Ours (Residual-based)	7.41	3.03	0.924	7.96	3.66	1.05

Table VIII : Ablation Study on Per-frame Deformation Estimation. FPS: Frames Per Second.

Figure 11: Ablation study of the Deformation-Aware Camera Tracking. (a) ATE RMSE versus optimization iterations for the full model and the variant without Stage 1. (b) Comparison between the variant without deformation-aware weighting (left) and the full model (right). The top row shows the estimated camera trajectories colored by RMSE. The bottom row presents the observed image sequence over a time interval with large tracking errors. The rightmost image shows the 2D deformation confidence map of the final frame.

Configuration	StereoMIS			C3VDv2
Configuration	RMSE $\downarrow$	S.D. $\downarrow$	FPS $\uparrow$	RMSE $\downarrow$	S.D. $\downarrow$	FPS $\uparrow$
D1. w/o Pose Opt.	8.04	3.56	0.96	8.24	3.71	1.09
D2. w/o Def. Prob. Est.	11.95	5.42	1.09	11.49	4.54	1.26
D3. w/o Dyn. Mgmt.	7.84	3.26	0.81	8.62	3.97	0.95
D4. w/o $w_{d}$ -Gating	8.15	3.62	0.94	8.54	3.98	1.05
Ours (Full Model)	7.41	3.03	0.92	7.96	3.66	1.05

Table IX : Ablation Study on Deformable Mapping.

Figure 12: Impact of dynamic deformation management on the evolution of active deformation parameters. The number of active deformation parameters over frames for (a) without dynamic management and (b) with dynamic management. Without management, parameters grow steadily over time, whereas dynamic management maintains a bounded and stable model complexity.

Dataset-Scene	Frame Number	Total Runtime	Tracking /Frame	FPS
StereoMIS-P2_4	4281	4896s	232ms	0.874
C3VDv2-C1_descending_t4_v4	361	383s	209ms	0.943

Table X : Runtime Analysis.

Figure 13: Speed–accuracy trade-off on two datasets. Higher FPS indicates faster processing, while lower ATE RMSE indicates better accuracy.

Explore similar work

EndoMD-SLAM: Endoscopic Gaussian Splatting SLAM under Optical Degradation with Memory and Static-Transient Decomposition
Aug 9, 2026Nuo Chen, Kangqi Ni, Lulin Liu +5Simultaneous Localization and Mapping 3DGS SLAM
Texas A&M University, College Station, TX, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA · University of Minnesota, Minneapolis, MN, USA +1
Track2Map: Online Deformable SLAM with Motion-Aware Pose Optimization in Robotic Surgery
Jul 9, 2026Tianyi Song, Sierra Bonilla, Xinwei Ju +6Simultaneous Localization and Mapping 3D Gaussian Splatting
Department of Computer Science and UCL Hawkes Institute, University College London, London WC1E 6BT, UK · Intuitive Surgical, Inc., Sunnyvale, CA 94086, USA
GaussianFlow SLAM: Monocular Gaussian Splatting SLAM Guided by GaussianFlow
Apr 17, 2026Dong-Uk Seo, Jinwoo Jeon, Eungchang Mason Lee +13D Gaussian Splatting 3DGS SLAM
School of Electrical Engineering, KAIST (Korea Advanced Institute of Science and Technology), Daejeon 34141, Republic of Korea

Aug 9, 2026J/K move · Enter open · S save

EndoMD-SLAM: Endoscopic Gaussian Splatting SLAM under Optical Degradation with Memory and Static-Transient Decomposition

Nuo Chen, Kangqi Ni, Lulin Liu, Joga Ivatury+4

Texas A&M University, College Station, TX, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA · University of Minnesota, Minneapolis, MN, USA+1

Simultaneous Localization and Mapping 3DGS SLAM Visual SLAM

Open paper AskPDF arXiv

Dense 3D reconstruction is critical for clinical endoscopic navigation and documentation. While Gaussian Splatting SLAM systems show promise in this domain, they fundamentally rely on strict multi-view photometric consistency. In routine procedures, this assumption is severely violated by intermittent optical degradations like moving debris and water flushing. Standard systems erroneously fuse these cameraattached artifacts into the persistent 3D geometry, causing severe tracking drift and irreversible map corruption. To address this limitation, we propose EndoMD-SLAM, a framework designed to maintain stability under optical degradation through specialized tracking and mapping mechanisms. On the tracking side, a memory-driven gating mechanism detects unreliable observations to suspend map updates and utilizes historical keyframes for drift-aware relocalization. On the mapping side, a self-supervised static-transient decomposition isolates visual contaminants into a dedicated transient field. This explicit separation prevents artifacts from structurally entangling with the persistent anatomical map. We curate a degradationfocused benchmark from colonoscopy videos to systematically evaluate these failure modes. Extensive experiments show that while standard baselines fail under severe optical degradation, EndoMD-SLAM preserves geometric integrity, reducing absolute trajectory error by 91% and improving rendering fidelity by 9.9 dB PSNR.

Abstract

Explore similar work

EndoMD-SLAM: Endoscopic Gaussian Splatting SLAM under Optical Degradation with Memory and Static-Transient Decomposition

Track2Map: Online Deformable SLAM with Motion-Aware Pose Optimization in Robotic Surgery

GaussianFlow SLAM: Monocular Gaussian Splatting SLAM Guided by GaussianFlow