EndoS3:Endoscopic Semantic Splatting SLAM

Mona Sheikh Zeinoddin; Benjamin Wright; Zafer Tandogdu; Greg Shaw; Evangelos Mazomenos; Danail Stoyanov

cs.CVNov 3, 2026

EndoS3:Endoscopic Semantic Splatting SLAM

Authors: Mona Sheikh Zeinoddin, Benjamin Wright, Zafer Tandogdu, Greg Shaw, Evangelos Mazomenos, Danail Stoyanov

Organizations: Hawkes Institute, University College London, UK. · Dept of Urology, University College London Hospitals, UK.

Abstract

Purpose: Despite advances in 3D surgical scene understanding, semantic simultaneous localisation and mapping (SLAM) remains under-explored for two key reasons. First, it requires solving three challenging tasks in parallel: camera pose tracking, high-fidelity deformable reconstruction, and semantic mapping, making progress particularly demanding. Second, the scarcity of datasets providing the necessary ground truth to evaluate all three tasks comprehensively under conditions that closely reflect real surgical procedures environments. Method: We propose EndoS3, a Gaussian Splatting-based semantic SLAM framework that uses instance masks to guide all three tasks. An instance-level motion decomposition module separates static and dynamic instances by evaluating residual flow after compensating for camera motion, accounting for flow noise and comparing motion across neighbouring instances. The camera is tracked against the static Gaussians, while an instance-aware warp field gives each dynamic instance its own control nodes and deformation network. A semantic node allocation module redistributes a fixed node budget across instances according to their size and non-rigidity. We also introduce CRCD++, an ex-vivo Cholecystectomy dataset with RGB-D frames, camera pose and instance annotations for full semantic SLAM evaluation. Results: EndoS3 surpasses all state-of-the-art general-purpose and surgical semantic SLAM methods in tracking, reconstruction and semantic mapping on the CRCD++ and SemSup datasets. Conclusion: Semantic-aware motion and deformation modelling is vital to surgical semantic SLAM. EndoS3 and CRCD++ offer a method and benchmark for improved geometric and semantic understanding in endoscopy.

Figures & tables

	Cholec- Seg8k Hong et al. (2020)	Endo- Vis18 Allan and others (2020)	Endo- scapes Mascagni and others (2025)	Semantic- SuPer Lin et al. (2023)	Stereo- MIS Hayoz et al. (2023)	D4D Docea and others (2026)	eCRCD Oh and others (2025)	CRCD++
Poses	No	No	No	No	Yes	Yes	Yes	Yes
Depth	No	No	No	No	No	GT	No	Pseudo-GT
Sem.	13	12 s	6 s	3	1 t	1 t	2 nt	4
Seq. len.	80	Sparse	Sparse	150	1.4–9.6k	$\sim$ 3k	120	0.2–2.1k
Frames	8,080	2,384	493	600	$\sim$ 58.7k	300k+	–	14,688

Table 1: Overview of Semantic datasets. Sem.: semantic classes ( s sparse; t tools only; nt no tools); Seq. len.: sequence length (frames); GT/Pseudo-GT: measured/stereo-estimated depth. Green: met; red: not met.

Figure 1: Overview of our framework - EndoS 3 uses semantics in both tracking and mapping.

Figure 2: Semantic annotations. Input — eCRCD Track Anything — Our manual annotations.

Figure 3: Our camera tracking results on CRCD++ C1 and C3 - our Instance-level motion decomposition allows EndoS 3 to produce much smoother trajectories.

Method	Metric	CRCD++						Sem.-SuPer
Method	Metric	C1	C2	C3	G3	E3	Avg.	Pins
DDS-SLAM Shan et al. (2024)	ATE $\downarrow$	1.78	3.36	6.59	4.83	5.13	4.34	9.21
DDS-SLAM Shan et al. (2024)	Depth $\downarrow$	11.1	19.9	13.4	8.0	14.8	13.45	9.21
SNI-SLAM Zhu et al. (2024)	ATE $\downarrow$	1.56	1.48	6.96	2.60	5.75	3.67	9.70
	Depth $\downarrow$	5.1	17.9	14.3	8.8	17.6	12.72
	mIoU $\uparrow$	63.6	52.1	60.2	32.6	41.5	50.0

Table 2: Tracking, depth and semantics. ATE-RMSE and depth error in mm; mIoU in %; SemSuP: pin reprojection error in px

\downarrow

. “–”: not available.

Method	Metric	CRCD++						Sem.-SuPer
Method	Metric	C1	C2	C3	G3	E3	Avg.	Avg.
DDS-SLAM Shan et al. (2024)	PSNR $\uparrow$	22.19	19.52	21.68	22.90	21.47	21.55	30.81
	SSIM $\uparrow$	0.667	0.583	0.596	0.580	0.639	0.613	0.792
	LPIPS $\downarrow$	0.669	0.720	0.717	0.734	0.644	0.697	0.248
SNI-SLAM Zhu et al. (2024)	PSNR $\uparrow$	19.32	16.09	16.60	16.39	18.61	17.40	22.47
	SSIM $\uparrow$	0.694	0.552	0.631	0.703	0.577	0.631	0.660

Table 3: Rendering quality on CRCD++ and Semantic-SuPer. PERSEUS has no renderer.

Figure 4: Semantic renders on CRCD++ and Semantic-SuPer.

MD	IW	NA	PSNR $\uparrow$	Depth $\downarrow$	ATE $\downarrow$	mIoU $\uparrow$
$\times$	$\checkmark$	$\checkmark$	22.18	5.85	2.91	52.8
$\checkmark$	$\times$	$\checkmark$	23.28	4.62	2.55	71.3
$\checkmark$	$\checkmark$	$\times$	23.08	4.43	2.83	70.9
$\checkmark$	$\checkmark$	$\checkmark$	24.45	4.35	2.75	73.4

Table 4: Ablation on CRCD++ - seq C4. MD: motion decomposition (otherwise only instruments are dynamic); IW: instance-aware warp field (otherwise one shared MLP); NA: semantic node allocation (otherwise a fixed node set).

$z^{A}$	$z^{B}$	PSNR $\uparrow$	ATE $\downarrow$	mIoU $\uparrow$
$\checkmark$	$\times$	24.39	3.03	71.9
$\times$	$\checkmark$	24.29	2.80	72.6
$\checkmark$	$\checkmark$	24.45	2.75	73.4

Table 5: Motion-decomposition CRCD++ - seq C4.

z^{A}

: instance-level test;

z^{B}

: boundary-band test. Removing either leaves the other as the sole criterion.

Method	PSNR (dB) $\uparrow$	Depth (mm) $\downarrow$	ATE (mm) $\downarrow$	mIoU (%) $\uparrow$
DDS-SLAM (GT)	21.55	13.45	4.34	–
Ours (GT)	23.55	2.96	3.03	84.2
Ours (Pred.)	24.28	3.54	3.30	65.8

Table 6: Predicted segmentation Avg. performance on CRCD++.

Figure 5: CRCD++ and SemSuP: Rendered colour map, CRCD++: Rendered depth map, SemSuP: Depth rendered as a pointcloud error heatmap.

Simultaneous Localization and Mapping

Simultaneous localization and mapping estimates a robot’s motion while building a map of an unknown environment.

See all 203 papers

NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting
Feb 19, 2026Jiwei Shan, Zeyu Cai, Yirui Li +5Visual SLAM 3DGS SLAM
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · School of Automation and Intelligent Sensing, the State Key Laboratory of Avionics Integration and Aviation System of-Systems Synthesis, and Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University, Shanghai 200240, China
EndoMD-SLAM: Endoscopic Gaussian Splatting SLAM under Optical Degradation with Memory and Static-Transient Decomposition
Aug 9, 2026Nuo Chen, Kangqi Ni, Lulin Liu +5Simultaneous Localization and Mapping 3DGS SLAM
Texas A&M University, College Station, TX, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA · University of Minnesota, Minneapolis, MN, USA +1
Track2Map: Online Deformable SLAM with Motion-Aware Pose Optimization in Robotic Surgery
Jul 9, 2026Tianyi Song, Sierra Bonilla, Xinwei Ju +6Simultaneous Localization and Mapping 3D Gaussian Splatting
Department of Computer Science and UCL Hawkes Institute, University College London, London WC1E 6BT, UK · Intuitive Surgical, Inc., Sunnyvale, CA 94086, USA

Feb 19, 2026J/K move · Enter open · S save

NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting

Jiwei Shan, Zeyu Cai, Yirui Li, Yongbo Chen+4

Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · School of Automation and Intelligent Sensing, the State Key Laboratory of Avionics Integration and Aviation System of-Systems Synthesis, and Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University, Shanghai 200240, China

Visual SLAM 3DGS SLAM Camera Pose Estimation Dynamic Scene Reconstruction Medical Imaging

Open paper AskPDF arXiv

Visual simultaneous localization and mapping (V-SLAM) is a fundamental capability for autonomous perception and navigation. However, endoscopic scenes violate the rigidity assumption due to persistent soft-tissue deformations, creating a strong coupling ambiguity between camera ego-motion and intrinsic deformation. Although recent monocular non-rigid SLAM methods have made notable progress, they often lack effective decoupling mechanisms and rely on sparse or low-fidelity scene representations, which leads to tracking drift and limited reconstruction quality. To address these limitations, we propose NRGS-SLAM, a monocular non-rigid SLAM system for endoscopy based on 3D Gaussian Splatting. To resolve the coupling ambiguity, we introduce a deformation-aware 3D Gaussian map that augments each Gaussian primitive with a learnable deformation probability, optimized via a Bayesian self-supervision strategy without requiring external non-rigidity labels. Building on this representation, we design a deformable tracking module that performs robust coarse-to-fine pose estimation by prioritizing low-deformation regions, followed by efficient per-frame deformation updates. A carefully designed deformable mapping module progressively expands and refines the map, balancing representational capacity and computational efficiency. In addition, a unified robust geometric loss incorporates external geometric priors to mitigate the inherent ill-posedness of monocular non-rigid SLAM. Extensive experiments on multiple public endoscopic datasets demonstrate that NRGS-SLAM achieves more accurate camera pose estimation (up to 50% reduction in RMSE) and higher-quality photo-realistic reconstructions than state-of-the-art methods. Comprehensive ablation studies further validate the effectiveness of our key design choices. Source code will be publicly available upon paper acceptance.

Abstract

Simultaneous Localization and Mapping

Explore similar work

NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting

EndoMD-SLAM: Endoscopic Gaussian Splatting SLAM under Optical Degradation with Memory and Static-Transient Decomposition

Track2Map: Online Deformable SLAM with Motion-Aware Pose Optimization in Robotic Surgery