Confidence-aware Monocular Depth Estimation for Minimally Invasive Surgery
Authors: Muhammad Asad, Emanuele Colleoni, Pritesh Mehta, Nicolas Toussaint, Ricardo Sanchez-Matilla, Maria Robu, Faisal Bashir, Rahim Mohammadi, +2 more
Organizations: Medtronic Surgical, London, UK. · UCL Hawkes Institute, University College London, London,UK.
Purpose: Monocular depth estimation (MDE) is vital for scene understanding in minimally invasive surgery (MIS). However, endoscopic video sequences are often contaminated by smoke, specular reflections, blur, and occlusions, limiting the accuracy of MDE models. In addition, current MDE models do not output depth confidence, which could be a valuable tool for improving their clinical reliability. Methods: We propose a novel confidence-aware MDE framework featuring three significant contributions: (i) Calibrated confidence targets: an ensemble of fine-tuned stereo matching models is used to capture disparity variance into pixel-wise confidence probabilities; (ii) Confidence-aware loss: Baseline MDE models are optimized with confidence-aware loss functions, utilizing pixel-wise confidence probabilities such that reliable pixels dominate training; and (iii) Inference-time confidence: a confidence estimation head is proposed with two convolution layers to predict per-pixel confidence at inference, enabling assessment of depth reliability. Results: Comprehensive experimental validation across internal and public datasets demonstrates that our framework improves depth estimation accuracy and can robustly quantify the prediction's confidence. On the internal clinical endoscopic dataset (StereoKP), we improve dense depth estimation accuracy by ~8% as compared to the baseline model. Conclusion: Our confidence-aware framework enables improved accuracy of MDE models in MIS, addressing challenges posed by noise and artifacts in pre-clinical and clinical data, and allows MDE models to provide confidence maps that may be used to improve their reliability for clinical applications.
Figures & tables
Figure 1: Factors such as image acquisition noise, lens contamination/smudges, blur, occluded views, and lighting issues in MIS datasets contribute towards unreliable depth data that impacts MDE models trained on MIS datasets. Depth confidence can be used to identify noise-free regions in confidence-aware training. Examples from the StereoKP dataset.
Figure 2: The proposed confidence-aware training methods. Our proposed blocks are shown in blue. Method (a) is used to add depth confidence to a depth dataset, whereas (b) shows our confidence-aware MDE training framework.
Dataset
Description
Setting
Modalities
Total Frames
StereoKP
Internal, 57 pre-clinical + 6 clinical videos
Pre-clinical + Clinical
Metric depth using Xu et al. (2023) + manually annotated instrument keypoints
10,491
MicroCT-SE
Internal, surrogate and two ex-vivo liver
Lab
Metric depth from MicroCT
16,775
MicroCT-PK
Internal, ex-vivo porcine kidney
Lab
Metric depth using from MicroCT
4046
Hamlyn Recasens et al. (2021)
Public in-vivo stereo endoscope videos Recasens et al. (2021)
Clinical
Metric depth using stereo-matching Recasens et al. (2021)
92,673
DaVinci Ye et al. (2017)
Public in-vivo stereo endoscope videos Ye et al. (2017)
Clinical
Metric depth using Xu et al. (2023)
41,433
Table 1: Summary of stereo endoscopic datasets utilized for experimental validation of proposed confidence-aware MDE methods.
Figure 3: Comparison of frames from different datasets used in this work. StereoKP, Hamlyn Recasens et al. (2021) and DaVinci Ye et al. (2017) are from clinical settings whereas both MicroCT-SE and MicroCT-PK are from lab settings. StereoKP and MicroCT datasets are internal wheres Hamlyn and DaVinci are public datasets.
σ
ARE (%) ↓
δ1<1.25 (%) ↑
0.2
9.58
92.68
0.5
8.75
94.00
0.7
8.86
94.14
1.0
9.04
93.40
Table 2: Sensitivity analysis of the hyperparameter σ in the confidence probability equation (Eq. 1 ), showing the effect on ARE and δ1<1.25 metrics using the StereoKP dataset.
CH
CAL
MAE (mm) ↓
Acc @ 2mm (%) ↑
ARE (%) ↓
δ1<1.25 (%) ↑
✗
✗
2.04
72.4
12.41
85.83
✗
✓
1.76
78.2
8.99
93.62
✓
✗
1.81
76.5
9.20
93.63
✓
✓
1.79
77.9
8.86
94.14
Table 3: Ablation study showing the impact of Confidence Head (CH) and Confidence-aware Loss (CAL) on model performance across four configurations. Results demonstrate that both components, individually and combined, improve MAE, accuracy at 2mm, ARE, and δ1<1.25 over the baseline. Best results are shown in bold .
Dataset
DAv1-B Baseline
DAv1-B CA (proposed)
ARE ↓
δ1↑
ARE ↓
δ1↑
StereoKP
12.41
85.83
8.86
94.14
MicroCT-SE
5.69
99.62
5.04
99.60
MicroCT-PK
18.64
96.98
14.74
97.27
Hamlyn Recasens et al. (2021)
13.97
80.00
13.74
80.10
DaVinci Ye et al. (2017)
8.28
96.60
8.03
96.80
Table 4: Evaluation metrics for baseline and proposed methods across datasets. Bold values shows best results for a given metric and dataset.
Figure 4: Qualitative results on StereoKP dataset, shows ground truth and predicted depth from DAv1-B baseline and DAv1-B CA (proposed) models, additionally showing predicted confidence from CA model. Arrows show regions where CA model performed particularly better as compared to baseline.
Depth estimation is a foundational component for 3D reconstruction in minimally invasive endoscopic surgeries. However, existing monocular depth estimation techniques often exhibit limited performance to the varying illumination and complex textures of the surgical environment. While applying foundation models offers a promising approach to enhance the depth estimation performance, the domain gap between the natural images used for pre-training and the target endoscopic images leads to significant semantic perception deficiencies. In this study, EndoUFM is introduced as an unsupervised monocular depth estimation framework that innovatively Utilizes dual Foundation Models for Endoscopic images, thereby enhancing the depth estimation performance by leveraging the powerful pre-learned priors. The framework features a novel adaptive fine-tuning strategy that incorporates Random Vector Low-Rank Adaptation (RVLoRA) to enhance model adaptability, and a Residual block based on Depthwise Separable Convolution (Res-DSC) to improve the capture of fine-grained local features. A mask-guided smoothness loss is also introduced to enforce depth consistency within anatomical structures. Extensive experiments on the SCARED, Hamlyn, SERV-CT, and EndoNeRF datasets confirm that our method achieves state-of-the-art performance while maintaining an efficient model size. This work contributes to augmenting surgeons' spatial perception during minimally invasive procedures, thereby enhancing surgical precision and safety, with crucial implications for augmented reality and navigation systems. Our code is available at https://github.com/RealMindyY/EndoUFM.
Xinning Yao, Bo Liu, Bojian Li +3
Image Processing Center, Beihang University, Beijing, 100191, China. · State Key Laboratory of High-Efficiency Reusable Aerospace Transportation Technology, Beijing 102206, China.
Depth estimation is a significant task for 3D perception in endoscopic surgeries. However, illumination interference and feature diversity in various endoscopic scenes are still challenges for generalizable depth estimation and ego-motion estimation. Based on this, a novel self-supervised framework, EndoMINI, is proposed for depth estimation in endoscopic scenes. Specifically, mixture of low-rank experts (MiLoRE) is proposed to perform parameter-efficient fine-tuning, which can also boost the model adaptation to scenes with different characteristics. Meanwhile, an intrinsic image alignment (IIA) is introduced into the training loss to alleviate the influence of light reflectance in endoscopy with a novel intrinsic image decomposition network. The proposed method is evaluated on SCARED datasets for supervised depth estimation, and two endoscopic datasets, Hamlyn and SERV-CT, for zero-shot depth estimation, compared with state-of-the-art works as well. The experimental results demonstrate outstanding performance of the proposed model and the effects of the main contributions.
Liangjing Shao, Beilei Cui, Yiming Huang +2
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · Shenzhen Loop Area Institute, China
We present a real-time, hybrid framework for occlusion handling and depth-aware tracking in minimally invasive surgeries (MIS). Occlusions caused by surgical instruments often obscure the region of interest (ROI), affecting surgical precision. To address this, our approach integrates enhanced You Only Look Once (YOLO) based detection, Particle Filtering (PF), and monocular depth estimation using Multiple Depth Estimation Accuracy with Single Network (MiDaS). Occlusion is detected using a distance-based threshold between instruments and ROI. Experimental results demonstrate robust instrument detection with a confidence score of 0.85, effective occlusion event handling, and accurate depth prediction with an MAE of 0.34 cm and RMSE of 0.53 cm. Independent phantom-based validation against stereo depth measurements yielded an MAE of 0.65 cm and RMSE of 0.66 cm. The framework further achieved an occlusion detection accuracy of 81.95% while maintaining real-time performance at 38.7 FPS. Preliminary surgeon feedback indicates that the proposed visualization framework is clinically relevant and holds promise for future intraoperative image-guided navigation and decision support in MIS.
Anuska Sarkar, Partha Acharya, Vishnu Nandury +2
School of Medical Science and Technology, Indian Institute of Technology Kharagpur, Kharagpur 721302 India · Computer Science Department, Arizona State University, Tempe, Arizona 85287 USA · Narayana Hospital, Kolkata 700107 India