4D Monocular Surgical Reconstruction under Arbitrary Camera Motions

Jiwei Shan; Zeyu Cai; Cheng-Tai Hsieh; Yirui Li; Hao Liu; Lijun Han; Hesheng Wang; Shing Shin Cheng

cs.CVFeb 19, 2026

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions

Authors: Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh, Yirui Li, Hao Liu, Lijun Han, Hesheng Wang, Shing Shin Cheng

Organizations: Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China · State Key Laboratory of Robotics and Intelligent Systems, Shenyang, China · School of Integrated Circuits, Shanghai Jiao Tong University, Shanghai, China · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China

Abstract

Reconstructing deformable surgical scenes from endoscopic videos is challenging and clinically important. Recent state-of-the-art methods based on implicit neural representations or 3D Gaussian splatting have made notable progress. However, most are designed for deformable scenes with fixed endoscope viewpoints and rely on stereo depth priors or accurate structure-from-motion for initialization and optimization, limiting their ability to handle monocular sequences with large camera motion in real clinical settings. To address this, we propose Local-EndoGS, a high-quality 4D reconstruction framework for monocular endoscopic sequences with arbitrary camera motion. Local-EndoGS introduces a progressive, window-based global representation that allocates local deformable scene models to each observed window, enabling scalability to long sequences with substantial motion. To overcome unreliable initialization without stereo depth or accurate structure-from-motion, we design a coarse-to-fine strategy integrating multi-view geometry, cross-window information, and monocular depth priors, providing a robust foundation for optimization. We further incorporate long-range 2D pixel trajectory constraints and physical motion priors to improve deformation plausibility. Experiments on three public endoscopic datasets with deformable scenes and varying camera motions show that Local-EndoGS consistently outperforms state-of-the-art methods in appearance quality and geometry. Ablation studies validate the effectiveness of our key designs. Code will be released upon acceptance at: https://github.com/IRMVLab/Local-EndoGS.

Figures & tables

Figure 1: (a)–(c) Illustrations of three typical types of camera motion in surgical scenes: (a) fixed camera, (b) camera moving around the tissue, and (c) camera moving forward. (d) Monocular reconstruction results of different methods under camera motion. State-of-the-art 4D surgical reconstruction algorithms experience significant degradation in reconstruction quality when the camera moves, while our method maintains superior performance.

Figure 2: Overview of Local-EndoGS. Given a long monocular endoscopic sequence with arbitrary camera motion, Local-EndoGS reconstructs the entire deformable scene using a progressive window-based global scene representation ( 3.2 ). Specifically, the sequence is first divided into multiple local windows based on its dynamic characteristics. For each local window, the scene structure is initialized using a local canonical space initialization strategy ( 3.4 ). Then, a local deformable scene representation ( 3.3 ) is used to model each region. The parameters of each local scene representation are then optimized using carefully designed loss functions ( 3.5 ) in a progressive manner until all local models are fully optimized.

Figure 3: Comparison of Feature Matching and Point Cloud Results Using Traditional Methods and Track-Any-Point (TAP) model [ 6 ] . (a) Correspondences obtained using SIFT keypoints with brute-force matching. (b) Sparse point cloud reconstructed from the correspondences shown in (a). (c) Correspondences from the TAP model. (d) Dense point cloud from TAP-based correspondences. Green and red lines indicate correct and incorrect feature matches, respectively.

Figure 4: Visualization of RGB images rendered by 3DGS $\phi^{C}_{i}$ initialized from the coarse stage and their corresponding reconstruction error maps with respect to the ground truth. Top row : rendered RGB images. Bottom row : pixel-wise reconstruction error maps, where higher values indicate greater reconstruction errors, especially near tissue boundaries and regions with deformation.

Dataset	Sequence	Frames	Resolution	Camera Motion	Windows
EndoNeRF [ 52 ]	Pulling	63	$512\times 640$	Fixed camera	1
EndoNeRF [ 52 ]	Cutting	156	$512\times 640$	Fixed camera	1
StereMIS [ 13 ]	Sequence1	1000	$512\times 640$	Moving around tissue	49
StereMIS [ 13 ]	Sequence2	1500	$512\times 640$	Moving around tissue	69
EndoMapper [ 1 ]	Sequence1	267	$512\times 640$	Moving forward	15
	Sequence2	267	$512\times 640$	Moving forward	16

Table 1: Summary of the datasets and sequences used in our study. Each sequence’s number of frames, image resolution, camera motion type, and window count are listed.

Metric	Definition
Abs Rel	$\frac{1}{\|D\|}\sum_{d\in D}\frac{\|d-d^{}\|}{d^{}}$
Sq Rel	$\frac{1}{\|D\|}\sum_{d\in D}\frac{(d-d^{})^{2}}{d^{}}$
RMSE	$\sqrt{\frac{1}{\|D\|}\sum_{d\in D}(d-d^{*})^{2}}$
RMSE log	$\sqrt{\frac{1}{\|D\|}\sum_{d\in D}(\log d-\log d^{*})^{2}}$
Accuracy ( $\delta<1.25$ )	$\frac{1}{\|D\|}\sum_{d\in D}\mathbb{I}\left(\max\left(\frac{d}{d^{}},\ \frac{d^{}}{d}\right)<1.25\right)$
Accuracy ( $\delta<1.25^{2}$ )	$\frac{1}{\|D\|}\sum_{d\in D}\mathbb{I}\left(\max\left(\frac{d}{d^{}},\ \frac{d^{}}{d}\right)<1.25^{2}\right)$

Table 2: Depth error and accuracy metrics used for evaluation. Here,

d

and

d^{*}

denote the predicted and ground-truth depth values, respectively, and

D

represents the set of predicted depth values.

Pulling
Method	PSNR	SSIM	LPIPS	Abs Rel	Sq Rel	RMSE	RMSE log	$\delta<1.25$	$\delta<1.25^{2}$
EndoNeRF	37.723	0.949	0.088	0.354	8.500	22.607	0.390	0.421	0.746
EndoSurf	37.117	0.950	0.108	9.854	943.824	39.685	1.049	0.325	0.583
ForPlane	30.555	0.896	0.104	0.313	8.802	26.267	0.344	0.425	0.787
DDS-SLAM	25.508	0.797	0.371	0.431	9.363	18.062	0.448	0.425	0.671
EndoGS	25.663	0.852	0.355	0.126	4.272	15.771	0.176	0.807	0.927

Table 3: Quantitative comparison of different methods on the EndoNeRF dataset. Columns highlighted in xx indicate that higher values are better, while those in xx indicate that lower values are better. The best results are highlighted in bold , and the second best results are underlined .

StereoMIS-Sequence1
Method	PSNR	SSIM	LPIPS	Abs Rel	Sq Rel	RMSE	RMSE log	$\delta<1.25$	$\delta<1.25^{2}$
EndoNeRF	25.913	0.746	0.481	0.420	23.226	58.555	0.478	0.310	0.590
EndoSurf	24.883	0.715	0.520	3.219	472.041	51.691	0.736	0.319	0.578
ForPlane	16.389	0.651	0.792	0.640	36.502	62.023	0.613	0.299	0.544
DDS-SLAM	16.585	0.569	0.673	0.355	20.031	40.680	0.402	0.462	0.726
EndoGS	20.412	0.691	0.643	0.572	10.705	29.252	0.522	0.454	0.740

Table 4: Quantitative comparison of different methods on the StereoMIS dataset. Columns highlighted in xx indicate that higher values are better, while those highlighted in xx indicate that lower values are better. The best results are shown in bold , and the second-best results are underlined .

Figure 5: Qualitative comparison of image rendering and depth prediction on deformable scenes at different time points from the StereoMIS dataset. Each pair of rows represents a specific time point: the first row shows the rendered RGB images, and the second row shows the predicted depth maps. The figure presents results from three time points (from top to bottom), illustrating how the observed scene changes as the camera moves. Compared to existing methods, Local-EndoGS (Ours) consistently provides finer reconstruction details, while the baseline methods show limitations in both image quality and depth accuracy.

EndoMapper-Sequence1
Method	PSNR	SSIM	LPIPS	Abs Rel	Sq Rel	RMSE	RMSE log	$\delta<1.25$	$\delta<1.25^{2}$
EndoNeRF	11.192	0.626	0.824	0.401	30.472	64.341	0.602	0.301	0.589
EndoSurf	23.767	0.689	0.714	7.182	740.954	62.653	0.962	0.258	0.494
ForPlane	27.026	0.811	0.552	0.825	24.607	49.890	0.622	0.283	0.603
DDS-SLAM	22.315	0.739	0.643	0.443	28.178	40.461	0.594	0.364	0.623
ENeRF-SLAM	19.661	0.757	0.609	0.499	31.652	44.688	0.700	0.298	0.582

Table 5: Quantitative comparison of various methods on the EndoMapper dataset. Columns highlighted in xx indicate that higher values are better, while those highlighted in xx indicate that lower values are better. The best results are shown in bold , and the second-best results are underlined .

Figure 6: Qualitative comparison of appearance (RGB) and geometric (depth) reconstruction results produced by Local-EndoGS and existing methods on deformable scenes from the EndoMapper dataset. Our method produces more accurate and visually consistent results for both appearance and depth, with fewer artifacts and better delineation of anatomical structures compared to existing approaches.

Method	EndoNeRF dataset		StereoMIS dataset
Method	Train Time	Rendering Speed	Train Time	Rendering Speed
EndoNeRF	$\sim$ 8 h	0.18 fps	$\sim$ 18 h	0.13 fps
EndoSurf	$\sim$ 8 h	0.18 fps	$\sim$ 10 h	0.26 fps
Forplane	3.00 min	1.40 fps	2.80 min	1.30 fps
Endo-GS	4.43 min	118.50 fps	6.55 min	85.33 fps
Deform3DGS	1.39 min	354.80 fps	3.28 min	202.17 fps

Table 6: Training Time and Rendering Speed on Different Datasets

Progressive Window-based Global Scene Representation (PWGSR)
Method	PSNR	SSIM	Abs Rel	Sq Rel	RMSE	RMSE log	$\delta<1.25$	$\delta<1.25^{2}$	FPS
w/o windows	21.76	0.713	1.105	38.888	32.925	0.692	0.354	0.617	306
w/o AWP	31.09	0.910	0.137	4.621	9.674	0.170	0.891	0.949	325
Local Canonical Space Initialization (LCSI)
w SD	31.77	0.923	0.124	3.765	9.249	0.153	0.917	0.975	325
w/o LCSI	28.38	0.885	0.296	9.858	18.794	0.426	0.657	0.814	325

Table 7: Ablation Studies of Different Model Components on the StereoMIS dataset.

Figure 7: Qualitative comparison of ablation results for RGB reconstruction (top row) and depth estimation (bottom row). (a) Without local windows; (b) Without local canonical space initialization (LCSI); (c) Full model; (d) Reference.

Figure 8: Visual ablation study on the impact of Physics-Based Regularization (PBR). The top row shows the rendering results of the model without PBR ( w/o PBR ), while the bottom row presents the outputs of the full model . Regions highlighted in yellow indicate areas where the absence of PBR leads to visual artifacts, demonstrating that PBR helps preserve structural consistency and anatomical fidelity.

Method	PSNR	Abs Rel	RMSE	Train Time
w/o Depth Sup.	31.49	0.129	9.271	8.46 min
w/ Depth Sup.	31.41	0.128	9.293	9.92 min

Table 8: Effect of Monocular Depth Supervision

Figure 9: Illustrations of the performance comparison with different hyperparameter configurations on the StereoMIS dataset.

Explore similar work

SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation
Mar 7, 2026Kaiyuan Xu, Fangzhou Hong, Daniel Elson +13D Reconstruction Monocular 3D Reconstruction
The Hamlyn Centre for Robotic Surgery, Imperial College London, SW7 2AZ, UK. · S-Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798. · Department of Computer Science, University of Liverpool, L69 7ZX, UK.
EndoPrior-GS: Dynamic Endoscopic Reconstruction with a Joint Texture Prior
Sep 29, 2026Jiaqi Huang, Shidong Wang, Tong Xin +13D Gaussian Splatting Medical Imaging
School of Engineering, Newcastle University, Newcastle upon Tyne NE1 7RU, UK · School of Computing, Newcastle University, Newcastle upon Tyne NE4 5TG, UK
NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting
Feb 19, 2026Jiwei Shan, Zeyu Cai, Yirui Li +5Visual SLAM 3DGS SLAM
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · School of Automation and Intelligent Sensing, the State Key Laboratory of Avionics Integration and Aviation System of-Systems Synthesis, and Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University, Shanghai 200240, China

Mar 7, 2026J/K move · Enter open · S save

SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation

Kaiyuan Xu, Fangzhou Hong, Daniel Elson, Baoru Huang

The Hamlyn Centre for Robotic Surgery, Imperial College London, SW7 2AZ, UK. · S-Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798. · Department of Computer Science, University of Liverpool, L69 7ZX, UK.

3D Reconstruction Monocular 3D Reconstruction Surgical Robotics Camera Pose Estimation

Open paper AskPDF arXiv

Reconstructing surgical scenes from monocular endoscopic video is critical for advancing robotic-assisted surgery. However, the application of state-of-the-art general-purpose reconstruction models is constrained by two key challenges: the lack of supervised training data and performance degradation over long video sequences. To overcome these limitations, we propose SurgCUT3R, a systematic framework that adapts unified 3D reconstruction models to the surgical domain. Our contributions are threefold. First, we develop a data generation pipeline that exploits public stereo surgical datasets to produce large-scale, metric-scale pseudo-ground-truth depth maps, effectively bridging the data gap. Second, we propose a hybrid supervision strategy that couples our pseudo-ground-truth with geometric self-correction to enhance robustness against inherent data imperfections. Third, we introduce a hierarchical inference framework that employs two specialized models to effectively mitigate accumulated pose drift over long surgical videos: one for global stability and one for local accuracy. Experiments on the SCARED and StereoMIS datasets demonstrate that our method achieves a competitive balance between accuracy and efficiency, delivering near state-of-the-art but substantially faster pose estimation and offering a practical and effective solution for robust reconstruction in surgical environments. Project page: https://chumo-xu.github.io/SurgCUT3R-ICRA26/.

Abstract

Explore similar work

SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation

EndoPrior-GS: Dynamic Endoscopic Reconstruction with a Joint Texture Prior

NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting