Accurate depth estimation plays a critical role in the navigation of endoscopic surgical robots, forming the foundation for 3D reconstruction and safe instrument guidance. Fine-tuning pretrained models heavily relies on endoscopic surgical datasets with precise depth annotations. While existing self-supervised depth estimation techniques eliminate the need for accurate depth annotations, their performance degrades in environments with weak textures and variable lighting, leading to sparse reconstruction with invalid depth estimation. Depth completion using sparse depth maps can mitigate these issues and improve accuracy. Despite the advances in depth completion techniques in general fields, their application in endoscopy remains limited. To overcome these limitations, we propose EndoDDC, an endoscopy depth completion method that integrates images, sparse depth information with depth gradient features, and optimizes depth maps through a diffusion model, addressing the issues of weak texture and light reflection in endoscopic environments. Extensive experiments on two publicly available endoscopy datasets show that our approach outperforms state-of-the-art models in both depth accuracy and robustness. This demonstrates the potential of our method to reduce visual errors in complex endoscopic environments. Our code will be released at https://github.com/yinheng-lin/EndoDDC.
Figures & tables
Fig. 1: Compared with fine-tuning the SOTA foundational model, our method generates more robust and accurate depth from sparse depth and RGB input, achieving superior sparse to dense reconstruction.
Fig. 2: Overview of EndoDDC : After feature extraction from the RGB image and sparse depth map, the Depth Grad Fusion module iteratively updates the state hidden network based on depth and gradient features. This output is then fed into the Depth Diffusion guidance model to conditionally optimize the initial depth. Finally, the optimized coarse depth map undergoes up-sampling and SPN refinement module to produce the final depth map.
Fig. 3: Details of the proposed model. (a) Details of the Depth Grad Fusion module; (b) The architectural design of the Depth Diffusion module.
Fig. 4: Qualitative comparison on C3VD and StereoMIS datasets. We compare EndoDDC with SOTA depth estimate and depth completion methods; our method generates less error in terms of tissue details.
Models
C3VD [ 40 ]
StereoMIS [ 41 ]
RMSE (mm) ↓
MAE (mm) ↓
REL (mm) ↓
δ ↑
RMSE (mm) ↓
MAE (mm) ↓
REL (mm) ↓
δ ↑
EndoDAC [ 22 ]
9.7476
7.5541
0.1081
0.9162
11.8435
8.7066
0.2084
0.6765
DepthAnything-v2 [ 10 ]
5.2202
3.6901
0.0671
0.9892
2.2465
1.63765
0.0277
0.8857
Marigold-DC [ 42 ]
0.8294
0.3275
0.0106
0.9985
2.6755
1.0395
0.0122
0.9968
CompletionFormer [ 31 ]
0.6875
0.2320
0.0071
0.9988
1.6536
0.6261
0.0069
0.9986
OGNI-DC [ 9 ]
0.6770
0.2283
0.0067
0.9988
1.5857
0.6114
0.0063
0.9986
TABLE I: Quantitative depth comparison with SOTA methods on C3VD [ 40 ] and StereoMIS [ 41 ] dataset.
Fig. 5: Results of EndoDDC at different levels of sparsity . Our method generates robust dense depth across different levels of sparsity.
Methods
50
500
5000
50000
RMSE (mm) ↓
REL (mm) ↓
RMSE (mm) ↓
REL (mm) ↓
RMSE (mm) ↓
REL (mm) ↓
RMSE (mm) ↓
REL (mm) ↓
CompletionFormer [ 31 ]
4.9533
0.0717
0.6875
0.0071
0.5848
0.0064
5.7541
0.1629
Marigold-DC [ 42 ]
1.8955
0.0302
0.8294
0.0106
0.5596
0.0091
0.4569
0.0088
OGNI-DC [ 9 ]
3.6632
0.0767
0.6770
0.0067
0.3757
0.0029
0.1403
0.0007
Ours
3.6772
0.0703
0.6591
0.0064
0.3442
0.0025
0.1177
0.0004
TABLE II: Quantitative Comparison Results of Depth Completion Models at Different Sparsity Levels on C3VD [ 40 ] dataset.
Methods
C3VD [ 40 ]
StereoMIS [ 41 ]
RMSE (mm) ↓
MAE (mm) ↓
REL (mm) ↓
δ ↑
RMSE (mm) ↓
MAE (mm) ↓
REL (mm) ↓
δ ↑
Baseline
0.6883
0.2329
0.0069
0.9987
1.6159
0.6163
0.0065
0.9985
Baseline+DDI [ 9 ]
0.6770
0.2283
0.0067
0.9988
1.5857
0.6114
0.0063
0.9986
Ours (w/o Depth Grad)
0.6669
0.2304
0.0068
0.9988
1.5729
0.5893
0.0064
0.9985
Ours (w/o Init Depth)
0.6453
0.2261
0.0063
0.9983
1.5604
0.5675
0.0062
0.9985
Ours Full
0.6412
0.2104
0.0060
0.9990
1.4691
0.5515
0.0061
0.9988
TABLE III: Ablation Study on C3VD [ 40 ] and StereoMIS [ 41 ] dataset. ”Baseline” indicates using [ 31 , 38 ] without the Depth Grad Fusion and Depth Diffusion. ”w/o” denotes ”without”. DDI indicates Depth-Depth Iteration [ 9 ] .
Depth estimation is a foundational component for 3D reconstruction in minimally invasive endoscopic surgeries. However, existing monocular depth estimation techniques often exhibit limited performance to the varying illumination and complex textures of the surgical environment. While applying foundation models offers a promising approach to enhance the depth estimation performance, the domain gap between the natural images used for pre-training and the target endoscopic images leads to significant semantic perception deficiencies. In this study, EndoUFM is introduced as an unsupervised monocular depth estimation framework that innovatively Utilizes dual Foundation Models for Endoscopic images, thereby enhancing the depth estimation performance by leveraging the powerful pre-learned priors. The framework features a novel adaptive fine-tuning strategy that incorporates Random Vector Low-Rank Adaptation (RVLoRA) to enhance model adaptability, and a Residual block based on Depthwise Separable Convolution (Res-DSC) to improve the capture of fine-grained local features. A mask-guided smoothness loss is also introduced to enforce depth consistency within anatomical structures. Extensive experiments on the SCARED, Hamlyn, SERV-CT, and EndoNeRF datasets confirm that our method achieves state-of-the-art performance while maintaining an efficient model size. This work contributes to augmenting surgeons' spatial perception during minimally invasive procedures, thereby enhancing surgical precision and safety, with crucial implications for augmented reality and navigation systems. Our code is available at https://github.com/RealMindyY/EndoUFM.
Xinning Yao, Bo Liu, Bojian Li +3
Image Processing Center, Beihang University, Beijing, 100191, China. · State Key Laboratory of High-Efficiency Reusable Aerospace Transportation Technology, Beijing 102206, China.
Depth estimation is a significant task for 3D perception in endoscopic surgeries. However, illumination interference and feature diversity in various endoscopic scenes are still challenges for generalizable depth estimation and ego-motion estimation. Based on this, a novel self-supervised framework, EndoMINI, is proposed for depth estimation in endoscopic scenes. Specifically, mixture of low-rank experts (MiLoRE) is proposed to perform parameter-efficient fine-tuning, which can also boost the model adaptation to scenes with different characteristics. Meanwhile, an intrinsic image alignment (IIA) is introduced into the training loss to alleviate the influence of light reflectance in endoscopy with a novel intrinsic image decomposition network. The proposed method is evaluated on SCARED datasets for supervised depth estimation, and two endoscopic datasets, Hamlyn and SERV-CT, for zero-shot depth estimation, compared with state-of-the-art works as well. The experimental results demonstrate outstanding performance of the proposed model and the effects of the main contributions.
Liangjing Shao, Beilei Cui, Yiming Huang +2
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · Shenzhen Loop Area Institute, China
Accurate, dense depth estimation is crucial for robotic perception, but commodity sensors often yield sparse or incomplete measurements due to hardware limitations. Existing RGBD-fused depth completion methods learn priors jointly conditioned on training RGB distribution and specific depth patterns, limiting domain generalization and robustness to various depth patterns. Recent efforts leverage monocular depth estimation (MDE) models to introduce domain-general geometric priors, but current two-stage integration strategies relying on explicit relative-to-metric alignment incur additional computation and introduce structured distortions. To this end, we present Any2Full, a one-stage, domain-general, and pattern-agnostic framework that reformulates completion as a scale-prompting adaptation of a pretrained MDE model. To address varying depth sparsity levels and irregular spatial distributions, we design a Scale-Aware Prompt Encoder. It distills scale cues from sparse inputs into unified scale prompts, guiding the MDE model toward globally scale-consistent predictions while preserving its geometric priors. Extensive experiments demonstrate that Any2Full achieves superior robustness and efficiency. It outperforms OMNI-DC by 32.2% in average AbsREL and delivers a 1.4× speedup over PriorDA with the same MDE backbone, establishing a new paradigm for universal depth completion. Codes and checkpoints are available at https://github.com/zhiyuandaily/Any2Full.
Zhiyuan Zhou, Ruofeng Liu, Taichi Liu +4
Rutgers University, USA · Michigan State University, USA · JD Logistics, China +1