SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation

Kaiyuan Xu; Fangzhou Hong; Daniel Elson; Baoru Huang

cs.CVMar 7, 2026

SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation

Authors: Kaiyuan Xu, Fangzhou Hong, Daniel Elson, Baoru Huang

Organizations: The Hamlyn Centre for Robotic Surgery, Imperial College London, SW7 2AZ, UK. · S-Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798. · Department of Computer Science, University of Liverpool, L69 7ZX, UK.

Abstract

Reconstructing surgical scenes from monocular endoscopic video is critical for advancing robotic-assisted surgery. However, the application of state-of-the-art general-purpose reconstruction models is constrained by two key challenges: the lack of supervised training data and performance degradation over long video sequences. To overcome these limitations, we propose SurgCUT3R, a systematic framework that adapts unified 3D reconstruction models to the surgical domain. Our contributions are threefold. First, we develop a data generation pipeline that exploits public stereo surgical datasets to produce large-scale, metric-scale pseudo-ground-truth depth maps, effectively bridging the data gap. Second, we propose a hybrid supervision strategy that couples our pseudo-ground-truth with geometric self-correction to enhance robustness against inherent data imperfections. Third, we introduce a hierarchical inference framework that employs two specialized models to effectively mitigate accumulated pose drift over long surgical videos: one for global stability and one for local accuracy. Experiments on the SCARED and StereoMIS datasets demonstrate that our method achieves a competitive balance between accuracy and efficiency, delivering near state-of-the-art but substantially faster pose estimation and offering a practical and effective solution for robust reconstruction in surgical environments. Project page: https://chumo-xu.github.io/SurgCUT3R-ICRA26/.

Figures & tables

Fig. 1: Qualitative results of 3D reconstruction. With videos (small images) as input, this figure shows the reconstruction from the first frame (large images left) and the accumulated 3D model from multiple frames (large images right). This alignment between the single-frame and multi-frame reconstruction results highlights the geometric consistency of our method.

Fig. 2: Overview of SurgCUT3R. Left: The unified reconstruction pipeline. Streaming video frames are encoded via a ViT encoder and interact with a persistent state, which is continuously updated to sequentially output the pointmap and camera parameter for each frame. Right: Our hierarchical framework for long-sequence inference. The pink lines represent camera trajectories. A sparse but globally stable trajectory from a global model ( $M_{global}$ ) provides anchor points to correct and stitch the dense but locally drifting trajectories from a local model ( $M_{local}$ ), producing a final, drift-corrected trajectory.

Fig. 3: Pipeline of pseudo GT depth generation. The process rectifies the raw stereo pair before feeding it into FoundationStereo [ 35 ] to generate a geometrically correct and metric-scale depth map.

Dataset	Category	Method	Abs Rel $\downarrow$	Sq Rel $\downarrow$	RMSE $\downarrow$	RMSE LOG $\downarrow$	$\boldsymbol{\delta<1.25}\uparrow$	ATE $\downarrow$	RTE $\downarrow$	FPS $\uparrow$
SCARED [ 1 ]	Optimization-based	MonST3R(w/ Opt) [ 39 ]	0.098	1.237	7.979	0.132	0.904	21.774	1.582	0.3
	Optimization-based	MegaSaM [ 18 ]	0.056	0.392	4.586	0.074	0.978	2.002	0.315	0.7
	Feed-forward	Spann3R [ 30 ]	0.119	2.524	10.218	0.148	0.867	10.258	1.260	19.2
		AF-SfMLearner [ 28 ]	0.073	0.534	5.028	0.081	0.964	10.312	0.971	3.6
		EndoDAC [ 6 ]	0.059	0.443	4.833	0.079	0.973	10.225	0.963	36.3
		SurgCUT3R (Ours)	0.057	0.410	4.647	0.077	0.977	5.514	0.752	19.7

TABLE I: QUANTITATIVE EVALUATION OF OUR SURGCUT3R METHOD AGAINST EXISTING METHODS IN ENDOSCOPIC SCENE RECONSTRUCTION. THE OPTIMAL AND SUBOPTIMAL RESULTS ARE SHOWN IN BOLD AND UNDERLINED RESPECTIVELY.

Fig. 4: Qualitative results of monocular depth estimation. We compare our method with MonST3R [ 39 ] , Spann3R [ 30 ] , AF-SfMLearner [ 28 ] , EndoDAC [ 6 ] and MegaSaM [ 18 ] on SCARED [ 1 ] and StereoMIS [ 9 ] datasets. Our method achieves the best qualitative results in feed-forward methods.

Configuration	Sq Rel $\downarrow$	RMSE $\downarrow$	$\boldsymbol{\delta<1.25}\uparrow$
w/o $L_{\text{consistency}}$	0.423	4.763	0.975
w/ $L_{\text{consistency}}$ (Ours)	0.410	4.647	0.977

TABLE II: ABLATION STUDY ON THE LOSS FUNCTION

Configuration	ATE $\downarrow$
CUT3R [ 31 ] Only	9.361
Dual-Arch (Ours)	5.514

TABLE III: ABLATION STUDY ON THE ARCHITECTURE

Fig. 5: Qualitative comparison of camera trajectories. Left: Without the hierarchical inference framework. Right: With our hierarchical inference framework (Ours).

Explore similar work

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions
Feb 19, 2026Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh +54D Reconstruction 3D Reconstruction
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China · State Key Laboratory of Robotics and Intelligent Systems, Shenyang, China +3
Learning Surgical Robotic Manipulation with 3D Spatial Priors
Mar 4, 2026Yu Sheng, Lidian Wang, Xiaomeng Chu +6Robotic Manipulation Surgical Robotics
University of Science and Technology of China · Tuodao Medical · Yale University
EndoVGGT: GNN-Enhanced Depth Estimation for Surgical 3D Reconstruction
Mar 25, 2026Falong Fan, Yi Xie, Arnis Lektauers +23D Reconstruction Depth Estimation
University of Arizona, Tucson, AZ, USA

Feb 19, 2026J/K move · Enter open · S save

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions

Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh, Yirui Li+4

Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China · State Key Laboratory of Robotics and Intelligent Systems, Shenyang, China+3

4D Reconstruction 3D Reconstruction Dynamic Scene Reconstruction 3D Gaussian Splatting

Open paper AskPDF arXiv

Reconstructing deformable surgical scenes from endoscopic videos is challenging and clinically important. Recent state-of-the-art methods based on implicit neural representations or 3D Gaussian splatting have made notable progress. However, most are designed for deformable scenes with fixed endoscope viewpoints and rely on stereo depth priors or accurate structure-from-motion for initialization and optimization, limiting their ability to handle monocular sequences with large camera motion in real clinical settings. To address this, we propose Local-EndoGS, a high-quality 4D reconstruction framework for monocular endoscopic sequences with arbitrary camera motion. Local-EndoGS introduces a progressive, window-based global representation that allocates local deformable scene models to each observed window, enabling scalability to long sequences with substantial motion. To overcome unreliable initialization without stereo depth or accurate structure-from-motion, we design a coarse-to-fine strategy integrating multi-view geometry, cross-window information, and monocular depth priors, providing a robust foundation for optimization. We further incorporate long-range 2D pixel trajectory constraints and physical motion priors to improve deformation plausibility. Experiments on three public endoscopic datasets with deformable scenes and varying camera motions show that Local-EndoGS consistently outperforms state-of-the-art methods in appearance quality and geometry. Ablation studies validate the effectiveness of our key designs. Code will be released upon acceptance at: https://github.com/IRMVLab/Local-EndoGS.

Abstract

Explore similar work

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions

Learning Surgical Robotic Manipulation with 3D Spatial Priors

EndoVGGT: GNN-Enhanced Depth Estimation for Surgical 3D Reconstruction