Robot-assisted endoluminal procedures are increasingly used for early cancer intervention. However, the intricate, narrow and tortuous pathways within the luminal anatomy pose substantial difficulties for robot navigation. Vision-based navigation offers a promising solution, but existing localization approaches are error-prone due to tissue deformation, in vivo artifacts and a lack of distinctive landmarks for consistent localization. This paper presents a novel EndoSERV localization method to address these challenges. It includes two main parts, i.e., SEgment-to-structure and Real-to-Virtual mapping, and hence the name. For long-range and complex luminal structures, we divide them into smaller sub-segments and estimate the odometry independently. To cater for label insufficiency, an efficient transfer technique maps real image features to the virtual domain to use virtual pose ground truth. The training phases of EndoSERV include an offline pretraining to extract texture-agnostic features, and an online phase that adapts to real-world conditions. Extensive experiments based on both public and clinical datasets have been performed to demonstrate the effectiveness of the method even without any real pose labels.
Figures & tables
Fig. 1: (a). Scale ambiguity in monocular SLAM : During the testing phase, the results of monocular SLAM require alignment with ground truth trajectories, which is impractical in clinical applications due to the lack of absolute scale information. (b). Appearance similarity in endoscopic images : Different bronchial branches often exhibit nearly identical geometric and topological structures, which can lead to ambiguous feature matching and incorrect associations with virtual frames.
Fig. 2: Motivations of this work. Segment-to-Structure : Due to the complex and long-term structure in the luminal path, a divide-and-conquer strategy is proposed for long-term pose estimation. Real-to-Virtual : Due to the lack of real pose labels in clinical scenarios, in this work, pre-operative CT data are used as the structure prior for intra-operative odometry estimation.
Fig. 3: System overview of EndoSERV. (a). A sliding windows strategy for long-term pose estimation. Black windows denote the training images, while pink windows represent the testing images. The sliding windows move along the temporal axis, enabling the system to alternate between training buffers and testing buffers. (b). The detail pipeline within a sub-segment, consisting of offline training, online training, and testing phase.
Fig. 4: Training pipeline overview. (a). Offline training pipeline. Virtual images are augmented using the pretrained diffusion model, generating texture-diverse augmented images. A novel aligner is designed to constrain the feature encoder to extract features into a unified feature space. Scene coordinate head is designed to generate the scene coordinate map. (b). All virtual images are first compressed to the virtual buffer during a retrieval process, which is used to fine-tune the transfer model quickly with the real buffer. An augmentation-then-recovery strategy is proposed to refine the distortion and deformation issue. After aligning everything to the virtual domain, a scene coordinate head is trained to estimate the camera pose.
Fig. 5: DDAug framework. The real image is generated from the virtual image using the pretrained transfer model. Three augmentations are applied: Color jitter, mixup with the noisy image, and camera parameter perturbation.
Method
C3VD dataset
Clinical data
ATE( mm )
ATE( mm )
SfM
AF-SfMLearner [ 3 ]
4.69 ± 2.18
12.49 ± 5.06
EndoDAC [ 6 ]
3.89 ± 2.50
11.10 ± 5.63
LightMono [ 18 ]
4.81 ± 3.40
12.26 ± 5.97
NeRF-SLAM
EndoGSLAM [ 11 ]
4.35 ± 1.90
12.04 ± 5.32
MonoGS [ 39 ]
4.08 ± 1.75
11.61 ± 5.70
TABLE I: Pose estimation results of C3VD and clinical experiments
Fig. 6: Two examples of pose estimation from the C3VD dataset are provided. Visualizations for all methods are first centralized, followed by the presentation of 3D images along with their projections on the three coordinate axes.
Method
C3VD dataset
Clinical data
rRPE(deg)
rRPE(deg)
AI-copilot [ 40 ]
2.72 ± 0.56
4.16 ± 1.93
CycleGAN [ 14 ]
1.87 ± 0.83
3.14 ± 1.72
UNSB [ 29 ]
1.80 ± 1.11
3.69 ± 1.17
EndoSERV(Ours)
0.50 ± 0.26
1.05 ± 0.44
TABLE II: rRPE results of Real-Virtual alignment methods
Fig. 7: (a). Robotics system overview. The robotic system is equipped with the continuum joints allowing 4DoF operations including translation, rotation and X/Y bending. During surgery, the surgeon controls the robot system, capturing intra-operative images from the monocular camera. EndoSERV module estimates the absolute pose and generates the trajectory for intra-operative guidance. (b). Two examples of pose estimation from the clinical dataset are provided. All Real-Virtual Alignment Methods overcome the need of 7-DoF alignment with pose ground truth. EndoSERV achieves the best performance, whose trajectory crosses through the airway less frequently.
Fig. 8: Surgical Guidance with Confidence-Aware Camera Localization. The process is composed of two refining phases and three testing phases. During the refining phase, the confidence progressively increases, while in the testing phase, the confidence experiences a decline at a specific time point.
Different modules in EndoSERV
Result
Style Transfer module
6.70ms
Refinement Module
15.60ms
Scene Coordinate Regression
2.94ms
PnP & RANSAC
12.48ms
Total time
37.73ms
TABLE III: Computational Time Evaluation
Fig. 9: Real-to-Virtual feature similarity for different configurations. It measures the feature similarity using different feature encoders and Real-Virtual Alignment methods. The similarity is measured using cosine similarity.
Fig. 10: Texture-agnostic features for real-virtual domain gap and discriminative features for appearance-similar endoscopic scenarios . (a) and (b) show the feature maps extracted from virtual and real images on the C3VD dataset and the clinical dataset, respectively, with and without offline pretraining. It can be observed that offline pretraining significantly increases the similarity between the extracted virtual and real features. (c) illustrates that although images from different bronchial branches may exhibit anatomical similarity, EndoSERV is still capable of extracting distinctive features. Specifically, for images from the same location but different domains, EndoSERV extracts highly similar features. In contrast, for anatomically similar images from different locations, it is able to capture distinct features.
Augmentations
ATE
Texture diverse
DDAug
✗
✗
9.03 ± 4.68
✓
✗
8.16 ± 4.37
✗
✓
8.80 ± 4.69
✓
✓
6.22 ± 2.83
Rand-Aug
7.59 ± 4.31
TABLE IV: Ablation Study on each augmentation strategy in EndoSERV
Uncertainty Filter Threshold
ATE (↓)
No Filter
6.22±2.83
Top 90% confidence (10% filtered)
5.81±2.50
Top 80% confidence (20% filtered)
5.58±2.41
Top 70% confidence (30% filtered)
5.44±2.26
Top 60% confidence (40% filtered)
5.37±2.19
TABLE V: Effect of Uncertainty-Based Filtering on Pose Estimation Accuracy (ATE).
Minimally invasive procedures performed within confined anatomical spaces depend on continuous endoscopic visualization. Current robotic endoscope systems can stabilize or reposition an endoscope, but they do not possess relevant context to provide effective visualization assistance. We present EndoNav, an anatomy-grounded natural-language framework that translates high-level surgeon commands into autonomous endoscopic visualization behaviors within patient-specific sinonasal anatomy. Spoken surgeon commands are transcribed and interpreted by an endoscopic viewpoint agent conditioned on a patient-specific anatomical scene representation. Rather than generating robot motion directly, the viewpoint agent generates structured visualization objectives that are converted into target viewpoints and inspection trajectories, which are then executed through geometry-constrained endoscope motion planning and joint-space control. We evaluate EndoNav using a structured three-pass sinus examination across three CT-derived anatomical models. For one cadaveric specimen, autonomous visualization is compared with sinus examinations performed by two resident surgeons. EndoNav achieved mean visualization IoUs of 87.04% and 84.37% relative to the two surgeon examinations, compared with an inter-surgeon IoU of 87.44%, while recovering 92.91% and 93.20% of surgeon-observed anatomical surfaces, respectively. These results demonstrate the feasibility of grounding high-level anatomical commands into patient-specific geometric objectives and translating them into anatomically constrained robotic visualization behaviors.
Accurate intraoperative navigation is essential for robot-assisted endoluminal intervention, but remains difficult because of limited endoscopic field of view and dynamic artifacts. Existing navigation platforms often rely on external localization technologies, such as electromagnetic tracking or shape sensing, which increase hardware complexity and remain vulnerable to intraoperative anatomical mismatch. We present a vision-only autonomy framework that performs long-horizon bronchoscopic navigation using preoperative CT-derived virtual targets and live endoscopic video, without external tracking during navigation. The framework uses hierarchical long-short agents: a short-term reactive agent for continuous low-latency motion control, and a long-term strategic agent for decision support at anatomically ambiguous points. When their recommendations conflict, a world-model critic predicts future visual states for candidate actions and selects the action whose predicted state best matches the target view. We evaluated the system in a high-fidelity airway phantom, three ex vivo porcine lungs, and a live porcine model. The system reached all planned segmental targets in the phantom, maintained 80% success to the eighth generation ex vivo, and achieved in vivo navigation performance comparable to the expert bronchoscopist. These results support the preclinical feasibility of sensor-free autonomous bronchoscopic navigation.
Junyang Wu, Mingyi Luo, Fangfang Xie +7
Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, 200240, China. · Shanghai Chest Hospital, Shanghai, 10587, China.
We present LoTIS, a model for visual navigation that provides robot-agnostic image-space guidance by localizing a reference RGB trajectory in the robot's current view, without requiring camera calibration, poses, or robot-specific training. Instead of predicting actions tied to specific robots, we predict the image-space coordinates of the reference trajectory as they would appear in the robot's current view. This creates robot-agnostic visual guidance that easily integrates with local planning. Consequently, our model's predictions provide guidance zero-shot across diverse embodiments. By decoupling perception from action and learning to localize trajectory points rather than imitate behavioral priors, we enable a cross-trajectory training strategy for robustness to viewpoint and camera changes. We outperform state-of-the-art methods by 20-50 percentage points in success rate on conventional forward navigation, achieving 94-98% success rate across diverse sim and real environments. Furthermore, we achieve over 5x improvements on challenging tasks where baselines fail, such as backward traversal. The system is straightforward to use: we show how even a video from a phone camera directly enables different robots to navigate to any point on the trajectory. Videos, demo, and code are available at https://finnbusch.com/lotis.
Finn Lukas Busch, Matti Vahs, Quantao Yang +4
Division of Robotics, Perception, and Learning, KTH Royal Institute of Technology, Sweden, and also affiliated with Digital Futures