DUNEDepth: Bronchoscopic Depth and Pose Estimation via Hybrid Adaptation of a 3D-Vision Foundation Model
Authors: Qi Chang, Austin Kao, Ange Lou, William E. Higgins
Organizations: School of Electrical Engineering and Computer Science, The Pennsylvania State University, University Park, PA 16802, USA · Vanderbilt University, Nashville, TN 37235, USA
State-of-the-art bronchoscopy guidance systems and computer-aided diagnosis methods have emerged as important tools for early lung cancer detection. Effective use of these tools depends on being able to accurately link a patient's bronchoscopic airway exam video to their pre-operative 3D chest CT scan. Unfortunately, the derivation of such links, which include bronchoscope depth and camera motion, from the exam video remains an open problem. Building on recently proposed 3D-vision foundation models, we propose DUNEDepth, a hybrid architecture that combines a shared DUNE ViT encoder, a twin MASt3R cross-attention decoder structure, and a prediction head that includes elements of the MASt3R and Reloc3r models. To fine-tune the architecture to our application, we utilize an integrated hybrid synthetic-supervised/self-supervised training strategy that draws on both bronchoscopic video and CT-derived virtual bronchoscopic views. Experiments drawing on bronchoscopic video sequences demonstrate that DUNEDepth more accurately derives depth information and camera motion trajectories than previously proposed self-supervised methods.
Figures & tables
Figure 1: Fusion of CT and bronchoscopic video for lung cancer management.
Figure 2: DUNEDepth architecture.
Median-scaled ↓
Non-mapped ↓
Method
L1
L2
SSIM
L1
L2
DA-V2 †
0.090
0.023
0.269
–
–
VGGT-1B †
0.088
0.020
0.254
–
–
Monodepth2
0.122
0.057
0.369
0.169
0.078
SC-SfMLearner
0.067
0.013
0.230
0.196
0.080
DUNEDepth
0.074
0.013
0.283
0.130
0.034
Table 1: Depth accuracy results on the 42-point test set ( (X2,Y2) , (X3,Y3) ).
Sequence Name
Method
τA (mm) ↓
τE (mm) ↓
ρtrans↑
SC-SfMLearner
4.14
5.34
0.46
A
VGGT-1B †
5.21
8.22
0.25
DUNEDepth
2.22
2.43
0.76
SC-SfMLearner
5.61
6.04
0.75
B
VGGT-1B †
12.49
13.21
0.26
DUNEDepth
5.22
6.92
0.78
Table 2: Trajectory tracking accuracy for four video sequences. Sequences A, B, C, and D consist of 72, 71, 240, and 331 contiguous frames, respectively. (Note that CT-video registration introduces <3 mm uncertainty in ground-truth poses [ 21 ] .)
Method
τA (mm) ↓
τE (mm) ↓
DUNEDepth
4.32
7.71
w/o synthetic, depth images
5.42
8.49
w/o rotation pseudo-labels
5.47
11.80
Table 3: Ablation study of DUNEDepth’s trajectory tracking accuracy. Results are based on video sequence D.
Figure 3: Depth-map comparison for three video frames from (X3,Y3) (median-scaled relative depth; magma colormap; darker values are deeper).
Figure 4: Trajectory tracking results for two video sequences. Black = ground truth trajectory, RED = DUNEDepth, Orange = SC-SfMLearner. The green dot is the starting point, while a colored square corresponds to an end point.
Bronchoscopy inserts a camera through the airways to diagnose endoluminal disease. Locating the camera along the airway usually requires additional equipment, such as an electromagnetic tracker and a preoperative CT scan. In this paper, we investigate the use of the monocular camera stream to localise the bronchoscope without requiring additional sensing. We introduce a monocular SLAM system that localises the tip of a bronchoscope and reconstructs the airway surface around it from RGB video alone. A Depth Anything 3 backbone, fine-tuned on over a million photorealistic airway renderings, returns metric per-pixel depth and is the system's only source of scale. DINOv3 features routed through a custom cross-attention matcher supply sub-pixel correspondences on mucosa where classical descriptors fail. Both enter a factor graph alongside a dense constraint from projective iterative closest point (ICP) registration against the system's own running reconstruction, so surface evidence stabilises the directions the visual factors leave weakly observable. On six airways rendered from patient CT scans, the system reaches 26.1 mm mean absolute trajectory error and 77.5% mean surface F-score. Its mean surface accuracy exceeds that of five recent monocular SLAM and feed-forward baselines. MASt3R-SLAM, which fits a free scale to the ground truth, has a lower mean trajectory error (22.0 mm). ORB-SLAM3 completes only two of the six airways, whereas our system completes all six and also reconstructs two ex vivo specimens.
Salvatore Esposito, Matias Mattamala, Daniel Rebain +4
University of Edinburgh · University of British Columbia
Recent breakthroughs in self-supervised learning (SSL), such as the Latent-Euclidean Joint-Embedding Predictive Architecture (LeJEPA), alongside successes in integrating visual encoders with language models, have driven the demand for adaptable, high-capacity vision encoders in Computed Tomography (CT). In this work, we explore 2D slice-based architectures as a flexible alternative to native 3D models for processing volumetric CT data. Using the CT-RATE dataset, we trained DALE-CT (Depth-Aware Latent-Euclidean Computed Tomography), a 2D model family built entirely from scratch using LeJEPA, and compared its performance against a continually pre-trained DINOv2 baseline. To enhance representation quality, we developed a novel 3D depth-aware pre-training strategy anchored by dense auxiliary supervision from both automated anatomical masks and human-annotated abnormalities. Under linear probe evaluation with Multiple Instance Learning (MIL) for multi-abnormality detection, the frozen backbone of this dual-supervised model (DALE-CT-2S) achieves a Macro AUROC of 0.833. This performance demonstrates near-parity with state-of-the-art 3D vision-language models, achieved entirely from scratch with significantly less data and no textual supervision. To ensure reproducibility, all training code, evaluation scripts, and model weights have been made publicly available.
Evan W. Damron, Mahmut S. Gokmen, Mitchell A. Klusty +3
College of Medicine Office of Research, University of Kentucky, Lexington, KY 40506 USA
Accurate localization of the bronchoscope within the bronchial tree is essential for clinicians to be able to reach target lesions, perform biopsies and avoid misidentification of airway segments during diagnostic and therapeutic procedures. However, existing navigation systems typically rely on patient-specific CT scans or additional external sensors, increasing cost, setup time and patient radiation exposure. This work presents BronchoTop, a real-time, RGB-only framework for topological bronchoscopy localization that eliminates the need for patient-specific data. BronchoTop estimates scope location relative to a generic airway model through four modules: lumen detection and tracking, lumen-branch label association, probabilistic scope location estimation, and switch verification. By using only standard bronchoscopy video input, BronchoTop provides practical, real-time navigational assistance to physicians. Evaluation on phantom, simulated and real data demonstrates state-of-the-art accuracy, improving existing approaches performance by over 20% on real bronchoscopy sequences. BronchoTop is the first published framework including both the localization algorithms as well as all the real data used, together with code to generate additional simulations, encouraging and facilitating further developments and benchmarking. The results highlight BronchoTop's potential to enhance procedural safety, efficiency and accessibility in clinical and robotic bronchoscopy.
Clara Tomasini, Ana Cristina Murillo, Luis Riazuelo