Robot-assisted endoluminal procedures are increasingly used for early cancer intervention. However, the intricate, narrow and tortuous pathways within the luminal anatomy pose substantial difficulties for robot navigation. Vision-based navigation offers a promising solution, but existing localization approaches are error-prone due to tissue deformation, in vivo artifacts and a lack of distinctive landmarks for consistent localization. This paper presents a novel EndoSERV localization method to address these challenges. It includes two main parts, i.e., SEgment-to-structure and Real-to-Virtual mapping, and hence the name. For long-range and complex luminal structures, we divide them into smaller sub-segments and estimate the odometry independently. To cater for label insufficiency, an efficient transfer technique maps real image features to the virtual domain to use virtual pose ground truth. The training phases of EndoSERV include an offline pretraining to extract texture-agnostic features, and an online phase that adapts to real-world conditions. Extensive experiments based on both public and clinical datasets have been performed to demonstrate the effectiveness of the method even without any real pose labels.
Figures & tables
Fig. 1: (a). Scale ambiguity in monocular SLAM : During the testing phase, the results of monocular SLAM require alignment with ground truth trajectories, which is impractical in clinical applications due to the lack of absolute scale information. (b). Appearance similarity in endoscopic images : Different bronchial branches often exhibit nearly identical geometric and topological structures, which can lead to ambiguous feature matching and incorrect associations with virtual frames.
Fig. 2: Motivations of this work. Segment-to-Structure : Due to the complex and long-term structure in the luminal path, a divide-and-conquer strategy is proposed for long-term pose estimation. Real-to-Virtual : Due to the lack of real pose labels in clinical scenarios, in this work, pre-operative CT data are used as the structure prior for intra-operative odometry estimation.
Fig. 3: System overview of EndoSERV. (a). A sliding windows strategy for long-term pose estimation. Black windows denote the training images, while pink windows represent the testing images. The sliding windows move along the temporal axis, enabling the system to alternate between training buffers and testing buffers. (b). The detail pipeline within a sub-segment, consisting of offline training, online training, and testing phase.
Fig. 4: Training pipeline overview. (a). Offline training pipeline. Virtual images are augmented using the pretrained diffusion model, generating texture-diverse augmented images. A novel aligner is designed to constrain the feature encoder to extract features into a unified feature space. Scene coordinate head is designed to generate the scene coordinate map. (b). All virtual images are first compressed to the virtual buffer during a retrieval process, which is used to fine-tune the transfer model quickly with the real buffer. An augmentation-then-recovery strategy is proposed to refine the distortion and deformation issue. After aligning everything to the virtual domain, a scene coordinate head is trained to estimate the camera pose.
Fig. 5: DDAug framework. The real image is generated from the virtual image using the pretrained transfer model. Three augmentations are applied: Color jitter, mixup with the noisy image, and camera parameter perturbation.
Method
C3VD dataset
Clinical data
ATE( mm )
ATE( mm )
SfM
AF-SfMLearner [ 3 ]
4.69 ± 2.18
12.49 ± 5.06
EndoDAC [ 6 ]
3.89 ± 2.50
11.10 ± 5.63
LightMono [ 18 ]
4.81 ± 3.40
12.26 ± 5.97
NeRF-SLAM
EndoGSLAM [ 11 ]
4.35 ± 1.90
12.04 ± 5.32
MonoGS [ 39 ]
4.08 ± 1.75
11.61 ± 5.70
TABLE I: Pose estimation results of C3VD and clinical experiments
Fig. 6: Two examples of pose estimation from the C3VD dataset are provided. Visualizations for all methods are first centralized, followed by the presentation of 3D images along with their projections on the three coordinate axes.
Method
C3VD dataset
Clinical data
rRPE(deg)
rRPE(deg)
AI-copilot [ 40 ]
2.72 ± 0.56
4.16 ± 1.93
CycleGAN [ 14 ]
1.87 ± 0.83
3.14 ± 1.72
UNSB [ 29 ]
1.80 ± 1.11
3.69 ± 1.17
EndoSERV(Ours)
0.50 ± 0.26
1.05 ± 0.44
TABLE II: rRPE results of Real-Virtual alignment methods
Fig. 7: (a). Robotics system overview. The robotic system is equipped with the continuum joints allowing 4DoF operations including translation, rotation and X/Y bending. During surgery, the surgeon controls the robot system, capturing intra-operative images from the monocular camera. EndoSERV module estimates the absolute pose and generates the trajectory for intra-operative guidance. (b). Two examples of pose estimation from the clinical dataset are provided. All Real-Virtual Alignment Methods overcome the need of 7-DoF alignment with pose ground truth. EndoSERV achieves the best performance, whose trajectory crosses through the airway less frequently.
Fig. 8: Surgical Guidance with Confidence-Aware Camera Localization. The process is composed of two refining phases and three testing phases. During the refining phase, the confidence progressively increases, while in the testing phase, the confidence experiences a decline at a specific time point.
Different modules in EndoSERV
Result
Style Transfer module
6.70ms
Refinement Module
15.60ms
Scene Coordinate Regression
2.94ms
PnP & RANSAC
12.48ms
Total time
37.73ms
TABLE III: Computational Time Evaluation
Fig. 9: Real-to-Virtual feature similarity for different configurations. It measures the feature similarity using different feature encoders and Real-Virtual Alignment methods. The similarity is measured using cosine similarity.
Fig. 10: Texture-agnostic features for real-virtual domain gap and discriminative features for appearance-similar endoscopic scenarios . (a) and (b) show the feature maps extracted from virtual and real images on the C3VD dataset and the clinical dataset, respectively, with and without offline pretraining. It can be observed that offline pretraining significantly increases the similarity between the extracted virtual and real features. (c) illustrates that although images from different bronchial branches may exhibit anatomical similarity, EndoSERV is still capable of extracting distinctive features. Specifically, for images from the same location but different domains, EndoSERV extracts highly similar features. In contrast, for anatomically similar images from different locations, it is able to capture distinct features.
Augmentations
ATE
Texture diverse
DDAug
✗
✗
9.03 ± 4.68
✓
✗
8.16 ± 4.37
✗
✓
8.80 ± 4.69
✓
✓
6.22 ± 2.83
Rand-Aug
7.59 ± 4.31
TABLE IV: Ablation Study on each augmentation strategy in EndoSERV
Uncertainty Filter Threshold
ATE (↓)
No Filter
6.22±2.83
Top 90% confidence (10% filtered)
5.81±2.50
Top 80% confidence (20% filtered)
5.58±2.41
Top 70% confidence (30% filtered)
5.44±2.26
Top 60% confidence (40% filtered)
5.37±2.19
TABLE V: Effect of Uncertainty-Based Filtering on Pose Estimation Accuracy (ATE).