Dense Endoluminal Mapping and Localisation from Monocular Bronchoscopy

Salvatore Esposito; Matias Mattamala; Daniel Rebain; Liang Zhao; Kev Dhaliwal; Mohsen Khadem; Subramanian Ramamoorthy

Fig. 1: From bronchoscope video to an endoluminal map. A robotic bronchoscope (left) streams RGB video of the airway interior. This video feds our system, which estimates the camera pose and reconstructs the airway wall, using learnt modules trained on synthetic bronchoscopy data (top right). The robot image is taken from the Ion system page of Intuitive [ 6 ] .

Fig. 2: System overview. For every frame, the front end (Sec. IV ) predicts metric depth and dense descriptors. Camera pose tracking (Sec. VI ) matches descriptors across keyframes and solves PnP for an initial pose. The back end (Sec. VII ) refines that pose by pose-graph optimisation, registers the depth against the running map by ICP, adds the resulting constraint to the pose graph, and integrates the fused pose into a dense TSDF reconstruction. Map management (Sec. V ) maintains the resulting landmark and TSDF maps.

Symbol	Description
$\mathbf{I}_{t}$ , $\mathbf{K}$	RGB frame at time $t$ ; camera intrinsics
$\mathbf{D}_{t}$ , $\mathbf{S}_{t}$	predicted metric depth and its standard deviation
$\mathbf{Q}_{t}$	per-pixel depth reliability, $\in[0,1]$
$\mathbf{T}_{\mathtt{W}\mathtt{C}}^{t}$ , $\mathbf{T}_{\mathtt{W}\mathtt{C}}^{t,\star}$	camera pose; best camera pose estimate (after pass two)
$\boldsymbol{\ell}_{\mathtt{W}}^{j}$	position of landmark $j$ in the world frame $\mathtt{W}$
$\mathbf{u}_{tj}$ , $z_{tj}$	measured patch coordinate and optical-axis depth

TABLE I: Notation used across the article.

Fig. 3: Keypoint correspondences on textureless mucosa using DINO features. The rows show two ex vivo sequences and the columns the same keypoints tracked over four forward displacements. The second row overlays the cosine similarity map for the red keypoint. The same regions are tracked despite the displacement. The front end therefore uses the pre-trained DINOv3 features without fine-tuning.

Fig. 4: Two-pass factor graph. White circles are keyframe poses

\mathbf{T}_{\mathtt{W}\mathtt{C}}^{t}

and landmarks

\boldsymbol{\ell}_{\mathtt{W}}^{j}

; coloured circles are factors, one colour per type (legend, bottom). (1) The first pass optimises an anchor factor on the first pose, a motion factor between consecutive poses, a reprojection factor for each observation of a landmark, and one depth factor between a landmark and the keyframe that created it. It returns the first-pass pose

\mathbf{T}_{\mathtt{W}\mathtt{C}}^{t}

. (2) That pose initialises the point-to-plane ICP registration of Sec. VII-B against the TSDF submap. (3) The accepted correspondences add one ICP factor on the current pose, alongside the ICP factors of earlier poses, and re-optimising propagates the correction to earlier poses and landmarks, returning the fused pose

\mathbf{T}_{\mathtt{W}\mathtt{C}}^{t,\star}

.

Fig. 5: Camera paths on the six CT-derived airways. Each panel is a synthetic airway rendered from the CT scan of one patient (all six are ATM’22 cases [ 17 ] ; Sec. IX-A ). Open circles mark the start; filled diamonds mark the end.

Anatomy	AbsRel $\downarrow$	RMSE (mm) $\downarrow$	$\boldsymbol{\delta_{1}}$ (%) $\uparrow$	$\boldsymbol{s^{\star}}$ ( $\to 1$ )
Patient 1	$0.174$	$5.22$	$73.5$	$0.883$
Patient 2	$0.186$	$4.80$	$75.3$	$0.882$
Patient 3	$0.159$	$5.13$	$79.3$	$0.954$
Patient 4	$0.116$	$3.65$	$88.3$	$1.058$
Patient 5	$0.171$	$4.98$	$75.4$	$0.931$
Patient 6	$0.148$	$4.50$	$85.7$	$0.920$

TABLE II: Metric depth accuracy of the bronchoscopy-adapted DA 3 predictor on the six CT-derived airways (Patients 1 to 6, Sec. IX-A ), per valid pixel against simulator ground truth. The scale factor

s^{\star}=\mathrm{median}(z/\hat{z})

tests absolute scale. Lower AbsRel/RMSE, higher

\delta_{1}

, and

s^{\star}

nearer

1.00

are better.

Fig. 6: Estimated trajectories on the six CT-derived airways (rows: synthetic airways rendered from the CT scans of Patients 1 to 6, all ATM’22 cases [ 17 ] ; Sec. IX-A ) for ours, Endo3R, MASt3R-SLAM, VGGT-SLAM 2.0 and ViSTA-SLAM (columns). Dashed grey is ground truth. Colour shows per-frame ATE after alignment (blue

\to

red, common scale capped at

100

mm). Each trajectory is aligned with Sim(3) on its first

40\,\%

of poses and then left unaltered; Table III uses SE(3) for ours. Each row uses the projection plane that the airway’s ground truth moves in; a baseline whose estimate leaves the airway also leaves the panel. Numerical ATE values are in Table III .

	Method	Patient 1	Patient 2	Patient 3	Patient 4	Patient 5	Patient 6	Mean †
ATE (mm) $\downarrow$	Ours	26.42	34.19	28.26	34.53	22.57	10.83	26.13
	ORB-SLAM3 [ 18 ]	$\times$	$\times$	$\times$	20.84	$\times$	6.98	13.91
	Endo3R [ 33 ]	52.36	52.10	163.42	107.05	62.00	84.39	86.89
	MASt3R-SLAM [ 30 ]	15.17	12.35	23.62	23.07	29.12	28.77	22.02
	VGGT-SLAM 2.0 [ 31 ]	321.39	46.73	111.57	53.46	72.66	158.75	127.43
	ViSTA-SLAM [ 32 ]	33.41	17.64	47.60	40.81	18.28	30.27	31.34

TABLE III: Tracking and reconstruction metrics on the six CT-derived airways (Patients 1 to 6, Sec. IX-A ). Our method is aligned with SE(3) and keeps its metric scale; every other method is aligned with Sim(3). The alignments are computed using the first

40\,\%

of poses. F-score at

10

mm. “

\times

” marks tracking failure or under

80\,\%

temporal coverage; “–” marks a sparse map, which has no surface to evaluate. † The mean is over the entries of the row that are not “

\times

”.

Fig. 7: Qualitative comparison of airway-surface reconstruction. Ground truth (GT) surface points and reconstructions from our method and four dense SLAM baselines, on the six CT-derived airways (rows: synthetic airways rendered from the CT scans of Patients 1 to 6, all ATM’22 cases [ 17 ] ; Sec. IX-A ). We register predicted clouds to GT rigidly, and every method keeps the same projection and scale. We crop each panel to the ground truth depth range and mark a method with nothing left inside it accordingly. We omit ORB-SLAM3 for the reconstruction evaluation because of its sparse map.

Configuration	ATE $\downarrow$	F-score $\uparrow$	CD $\downarrow$
Ours (full)	6.12	95.3	4.10
ORB instead of DINO matcher	$\times$	$\times$	$\times$
w/o TSDF map and ICP factor	16.34	64.9	13.97
single-pass fusion	18.88	70.2	15.95
w/o GNC (initial scale)	27.47	55.3	23.55

TABLE IV: Ablation study on the six synthetic phantom sequences (means over B00, B46, B71, B96, B97 and B118; Sec. IX-A ; not the CT-derived airways of Table III ), each row removing or replacing one component. ATE and CD in mm, F-score in %; best per column in bold . “

\times

” indicates tracking failure.

Fig. 8: Reconstruction comparison on ex vivo tissue. Each panel is framed on its own extent because real sequences carry no ground truth pose and monocular scale is free. The comparison therefore judges structural coherence. An empty panel is itself the result.

Fig. 9: From bronchoscope frame to endoluminal surface. Left: input RGB frame from an ex vivo bronchoscopy sequence. Centre: dense point cloud obtained by back-projecting the DA 3 depth at the SLAM-estimated pose. Right: corresponding region of the fused TSDF surface after marching-cubes extraction and light smoothing.

Fig. 10: Estimated trajectory against electromagnetic-tracking ground truth, evaluated as odometry. Five acquisition windows, each projected onto its dominant plane

(u,v)

. Dashed grey is the EMT reference over its longest continuous valid stretch. We align the estimate with a similarity transform fitted on the start of each window (Sec. IX-E ), and the colour (shared bar, mm) shows the drift accumulated since the start.

Fig. 11: Endoluminal reconstruction on ex vivo tissue. Graph-optimised sparse landmark maps for four of the five EMT windows of Fig. 10 , metrically scaled and rendered as depth-ordered surfels; colour is the observed mucosal appearance and the scale bar is

10

mm. The maps recover a continuous lumen with visible cartilage ridging and bifurcation openings.

Fig. 12: Novel-view synthesis from the estimated trajectory. For each of the four sequences, an observed frame above the rendering of that same held-out view; the yellow box marks a detailed region, magnified in the inset. See the supplementary video for a closer look.

Sequence	Views	Segments	PSNR train	PSNR held out
1	$647$	$21$	$35.18$	$33.86$
2	$270$	$9$	$35.33$	$34.70$
3	$270$	$9$	$35.43$	$34.53$
4	$337$	$11$	$35.55$	$33.98$
Mean			$\mathbf{35.37}$	$\mathbf{34.27}$

TABLE V: Novel-view synthesis, per sequence. A

3

D Gaussian Splatting model trained from the estimated poses of each EMT sequence. Held-out views are excluded from training on a global index.

Dense Endoluminal Mapping and Localisation from Monocular Bronchoscopy

Abstract

Figures & tables

Explore similar work

BronchoTop: Bronchoscopy Navigation via RGB-Only Topological Localization

Geometry-Aware Camera Localization for Bronchoscopy

Geometry-Aware Visual Odometry for Bronchoscopic Navigation via High-Gain Observer Fusion