4D Triangle Splatting Reconstruction of Dynamic Endoscopic Scenes from Monocular Videos

Laura Salort-Benejam; Stamatia Giannarou; Antonio Agudo

cs.CVNov 1, 2026

4D Triangle Splatting Reconstruction of Dynamic Endoscopic Scenes from Monocular Videos

Authors: Laura Salort-Benejam, Stamatia Giannarou, Antonio Agudo

Organizations: Institut de Robòtica i Informàtica Industrial, CSIC-UPC, Barcelona, Spain · Imperial College London, London, United Kingdom

Abstract

Robust 4D reconstruction of deformable endoscopic scenes from monocular videos remains challenging due to non-rigid tissue deformation, illumination changes, occlusions, and unknown camera trajectories. Reliable dynamic reconstruction of endoscopies could significantly enhance visualization, diagnostic accuracy and intraoperative guidance of minimally invasive procedures. We propose SurgicalTS, a self-supervised triangle-splatting framework for dynamic endoscopic scene reconstruction from static monocular cameras. The scene is represented by a canonical set of triangles jointly optimized with a deformation network. We introduce a quaternion-based deformation model, an improved color parametrization, spatio-temporal priors, and a redefined initialization strategy that improves completeness under occlusions. Extensive evaluation on challenging in-vivo endoscopy videos from diverse procedures shows that SurgicalTS successfully models soft-tissue deformations and tool-tissue interactions over time, while consistently outperforming state-of-the-art methods. Our method also reduces training times while maintaining competitive geometric and photometric accuracy results, highlighting the effectiveness of quaternion-based deformation modelling with physical consistency for robust and efficient 4D reconstruction of endoscopic scenes.

Figures & tables

Figure 1: Overview of our SurgicalTS pipeline for triangle splatting 4D reconstruction of dynamic monocular endoscopy videos. We optimize a canonical space with a deformation network to model the scene geometry and appearance over time.

	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$
SurgicalTS	42.339	0.977	0.014	1.825	6.703
w/o $\mathcal{L}_{\text{edge}}^{D}$	42.062	0.977	0.015	1.818	6.607
w/o $\mathcal{L}_{\text{grad}}^{D}$	42.371	0.978	0.014	1.957	7.753
w/o $\mathcal{L}_{\text{grad}}^{C}$	42.238	0.977	0.015	1.876	7.115
w/o $\mathcal{L}_{\text{depth}}^{D}$	43.183	0.980	0.011	3.585	10.482
w/o $\mathcal{L}_{\text{depth}}^{C}$	42.767	0.979	0.012	7.128	13.199

Table 1: Quantitative results of the ablation study on the pulling sequence. Best results in bold , second best underlined .

Panel of Figure 2 — Figure 2: Qualitative results of the ablation study on a frame from the pulling sequence. Row 1: RGB images. Row 2: Surface depth. Row 3: Surface normal.

	Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	FPS $\uparrow$	${T_{t}}\downarrow$
Pulling	EndoNeRF	34.217	0.938	0.160	–	–	–	–
	EndoGaussian	37.308	0.958	0.070	–	–	–	–
	Endo-4DTS	40.39	0.971	0.026	7.171	16.803	16.50	329
	SurgicalTS (Ours)	39.890	0.969	0.027	2.259	6.539	14.95	221
Cutting	EndoNeRF	34.186	0.932	0.151	–	–	–	–
	EndoGaussian	38.287	0.962	0.058	–	–	–	–

Table 2: Quantitative comparison of our SurgicalTS with EndoNeRF [ 30 ] , EndoGaussian [ 17 ] , and Endo-4DTS [ 25 ] on the EndoNeRF dataset. Best results in bold .

Panel of Figure 3 — Figure 3: Qualitative evaluation on EndoNeRF dataset [ 30 ] . On both sides rows show from top to bottom: RGB, Depth, and Normals images. Left: Pulling. Right: Cutting.

	Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	FPS $\uparrow$	${T_{t}}\downarrow$
TECAB1	EndoNeRF	25.791	0.742	0.255	–	–	–	–
	NeRFscopy	25.811	0.750	0.255	–	–	–	–
	Endo-4DTS	28.260	0.863	0.136	4.762	15.358	41.30	158
	SurgicalTS (Ours)	29.473	0.894	0.096	4.721	14.805	48.88	80
TECAB2	EndoNeRF	24.954	0.685	0.419	–	–	–	–
	NeRFscopy	24.864	0.689	0.429	–	–	–	–

Table 3: Quantitative comparison of our SurgicalTS with EndoNeRF [ 30 ] , NeRFscopy [ 26 ] , and Endo-4DTS [ 25 ] on the NeRFscopy dataset [ 26 ] . Best results in bold .

Panel of Figure 4 — Figure 4: Qualitative evaluation on NeRFscopy dataset [ 26 ] . On both sides columns show from left to right: input and rendered RGB, input and rendered depth. Left: TECAB1 and TECAB2 images. Right: Bronchoscopy and lung lobectomy images.

4D Reconstruction

4D reconstruction recovers changing 3D scenes, shapes, and motion over time from images, videos, or other sparse measurements.

See all 144 papers

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions
Feb 19, 2026Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh +54D Reconstruction 3D Reconstruction
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China · State Key Laboratory of Robotics and Intelligent Systems, Shenyang, China +3
NeRFscopy: Neural Radiance Fields for in-vivo Time-Varying Tissues from Endoscopy
Feb 17, 2026Laura Salort-Benejam, Antonio AgudoEndoscopic 3D Reconstruction 3D Reconstruction
Institut de Rob`otica i Inform`atica Industrial, CSIC-UPC, Spain
SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation
Mar 7, 2026Kaiyuan Xu, Fangzhou Hong, Daniel Elson +13D Reconstruction Endoscopic 3D Reconstruction
The Hamlyn Centre for Robotic Surgery, Imperial College London, SW7 2AZ, UK. · S-Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798. · Department of Computer Science, University of Liverpool, L69 7ZX, UK.

Feb 19, 2026J/K move · Enter open · S save

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions

Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh, Yirui Li+4

Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China · State Key Laboratory of Robotics and Intelligent Systems, Shenyang, China+3

4D Reconstruction 3D Reconstruction Endoscopic 3D Reconstruction Dynamic Scene Reconstruction 3D Gaussian Splatting

Open paper AskPDF arXiv

Reconstructing deformable surgical scenes from endoscopic videos is challenging and clinically important. Recent state-of-the-art methods based on implicit neural representations or 3D Gaussian splatting have made notable progress. However, most are designed for deformable scenes with fixed endoscope viewpoints and rely on stereo depth priors or accurate structure-from-motion for initialization and optimization, limiting their ability to handle monocular sequences with large camera motion in real clinical settings. To address this, we propose Local-EndoGS, a high-quality 4D reconstruction framework for monocular endoscopic sequences with arbitrary camera motion. Local-EndoGS introduces a progressive, window-based global representation that allocates local deformable scene models to each observed window, enabling scalability to long sequences with substantial motion. To overcome unreliable initialization without stereo depth or accurate structure-from-motion, we design a coarse-to-fine strategy integrating multi-view geometry, cross-window information, and monocular depth priors, providing a robust foundation for optimization. We further incorporate long-range 2D pixel trajectory constraints and physical motion priors to improve deformation plausibility. Experiments on three public endoscopic datasets with deformable scenes and varying camera motions show that Local-EndoGS consistently outperforms state-of-the-art methods in appearance quality and geometry. Ablation studies validate the effectiveness of our key designs. Code will be released upon acceptance at: https://github.com/IRMVLab/Local-EndoGS.

Abstract

4D Reconstruction

Explore similar work

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions

NeRFscopy: Neural Radiance Fields for in-vivo Time-Varying Tissues from Endoscopy

SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation