Accurate intraoperative navigation is essential for robot-assisted endoluminal intervention, but remains difficult because of limited endoscopic field of view and dynamic artifacts. Existing navigation platforms often rely on external localization technologies, such as electromagnetic tracking or shape sensing, which increase hardware complexity and remain vulnerable to intraoperative anatomical mismatch. We present a vision-only autonomy framework that performs long-horizon bronchoscopic navigation using preoperative CT-derived virtual targets and live endoscopic video, without external tracking during navigation. The framework uses hierarchical long-short agents: a short-term reactive agent for continuous low-latency motion control, and a long-term strategic agent for decision support at anatomically ambiguous points. When their recommendations conflict, a world-model critic predicts future visual states for candidate actions and selects the action whose predicted state best matches the target view. We evaluated the system in a high-fidelity airway phantom, three ex vivo porcine lungs, and a live porcine model. The system reached all planned segmental targets in the phantom, maintained 80% success to the eighth generation ex vivo, and achieved in vivo navigation performance comparable to the expert bronchoscopist. These results support the preclinical feasibility of sensor-free autonomous bronchoscopic navigation.
Figures & tables
Figure 1: Conceptual overview of the autonomous robotic navigation framework. Preoperatively, following a patient CT scan, fully automated algorithms are deployed to segment the bronchial tree and target lesions, and to plan the optimal intraoperative trajectory. This planned path is formulated as a sequential series of virtual image targets. Intraoperatively, the intelligent agent autonomously navigates through these consecutive sub-targets to ultimately access the bronchial segment nearest to the lesion. To achieve this, the agent continuously executes a dual-state decision policy: if it determines that the robot has successfully reached the current sub-target, it updates the objective to the subsequent waypoint; otherwise, it generates specific kinematic control commands to actuate the 6-degree-of-freedom (6-DoF) flexible robot toward the current target.
Figure 2: Architecture of the hierarchical multi-agent autonomous navigation framework. The system decomposes the navigation task across two temporal scales: a short-term reactivate agent that compensates for immediate endoluminal dynamics, and a long-term strategic agent that acts as a high-level supervisor. Inter-agent coordination is governed by an interaction and consensus mechanism. If the strategic agent’s proposed action aligns with the top-K predicted logits of the reactive agent, a consensus is achieved and the action is executed. In the event of a conflict, resolution is source-dependent: preoperative guidance commands will not be executed. Conversely, if a conflict originates from the LLM guidance, a world model serves as a critic to simulate potential downstream outcomes and deduce the optimal control signal.
Figure 3: Results of phantom experiments. A. Bronchial segmentation of the experimental phantom displaying 17 planned trajectories (five of which were utilized for artifact experiments). B. A comparison between artifact-degraded and clean images. C. A successful trajectory of clean phantom. D. A successful trajectory of artifact phantom. E. Frequency distribution of the levels reached by different methods. F. Comparison of time and number of actions across different methods on the clean phantom. G. Comparison of time and number of actions across different methods on the artifact phantom. H. Details of the maximum bronchial generation reached by different methods for each of the 17 trajectories. I. Quantitative analysis of the bronchial generations reached by different methods. J. SSIM between the expert’s view and the final images obtained by our method and ViNT upon reaching the endpoint. K. SSIM comparison between the final images of our method and the expert’s view in both clean and artifact phantom scenarios.
Figure 4: Ex vivo evaluation on diverse porcine lungs. (A) 3D segmentation of three distinct porcine lungs showing morphological variability. (B) Endoscopic views at identical anatomical landmarks across different lungs, highlighting visual domain gaps. (C) System response to specific challenges: navigating through mucus occlusion, autonomous target switching upon visual matching, and adaptive view adjustment to avoid obstructions. (D) Success rates of 59 navigation trajectories across bronchial generations. (E) Distribution of procedure time and action steps. (F) Failure modes caused by lens fouling and complete bubble occlusion.
Figure 5: Results of in-vivo experiments. A. Experimental setup for in vivo studies. B. Two examples by the automated system and the human expert. C. End-effector distance between the expert and the automated system. D. Structural Similarity Index Measure (SSIM) of the endpoints between the expert and the automated system. E. Distance to the nodule upon reaching the endpoint across different methods. F. Intraoperative CBCT images displaying preoperative bronchial and nodule segmentations, as well as intraoperative bronchoscope segmentation. G. Quantitative analysis of time and number of actions across different methods. H. Analysis of the lateral adjustment rate and the forward progression rate.
Figure S1 : Representative trajectories from clean phantom experiments. Sixteen representative cases are shown. For each case, the planned navigation trajectory within the bronchial phantom is overlaid, together with the endoscopic view at the onset of the experiment (start frame) and the corresponding image upon reaching the target location (end frame).
Figure S2 : Representative trajectories from in vivo experiments. Seven representative cases are shown. For each case, the planned navigation trajectory within the bronchial anatomy is overlaid, together with the endoscopic view at the onset of the experiment (start frame) and the corresponding image upon reaching the target location (end frame). In addition, a composite visualization summarizing all seven trajectories is provided, along with the terminal bronchial generation reached by each trajectory.
Figure S3 : Image sequences along phantom navigation trajectories for the baseline and our method. For each trajectory, five representative endoscopic images sampled from distinct bronchial branches are shown. Blue trajectories denote the centerline, directing toward the target lumen. Our method consistently follows the target navigation path and successfully reaches the intended branches. In contrast, the baseline (VinT) demonstrates reliable navigation in the early stages but frequently deviates from the target trajectory and becomes disoriented at deeper branches.
Figure S4 : Image sequences along exvivo navigation trajectories for our method. For each trajectory, ten representative endoscopic images sampled from distinct bronchial branches are shown. Blue trajectories denote the centerline, directing toward the target lumen. Our method consistently follows the target navigation path and successfully reaches the intended branches.
Figure S5 : Image sequences along invivo navigation trajectories for our method. For each trajectory, ten representative endoscopic images sampled from distinct bronchial branches are shown. Blue trajectories denote the centerline, directing toward the target lumen. Our method consistently follows the target navigation path and successfully reaches the intended branches.
Figure S6 : In vivo intraoperative CBCT visualization. Left: CBCT-derived results from six in vivo experiments, including bronchial tree segmentation, centerline visualization, nodule localization, and the intraoperative bronchoscope position at the target site. Right: Raw intraoperative CBCT images, where three distinct colors indicate the bronchoscope positions achieved by the proposed autonomous navigation and by two expert operators.
Figure S7 : Virtual image generation pipeline. (A) CT segmentation pipeline: raw CT volumes are processed by dedicated airway and nodule segmentation networks to reconstruct 3D meshes of the bronchial tree and target nodules. (B) Path planning via VMTK skeletonization: using the trachea as the start point and an intraluminal point near the target nodule as the endpoint, a navigation centerline is generated. (C) Virtual rendering: the segmented airway mesh and planned centerline are imported into a virtual engine to synthesize virtual images serving as virtual targets. (D) Post-processing: action candidates and prompts are generated. Action candidates are obtained by computing the relative pose between adjacent virtual images and mapping it to one of six discrete actions. Prompts indicate the lumen aligned with the centerline vanishing direction, with the target lumen highlighted by a red arrow.
Figure S8 : In vivo experimental settings for expert teleoperation and autonomous navigation. (A) Teleoperated control. An expert remotely navigates the robot using a keyboard-based interface, with discrete commands for forward/backward motion, lateral and vertical adjustments, and target switching. (B) Autonomous control. An autonomous agent executes navigation actions by jointly leveraging real-time observations and a virtual target representation, without human intervention.
Figure S9 : The role of world model in the navigation framework . (A) Future state predictions generated by the world model under different candidate actions, together with the corresponding LPIPS distances to the goal image. (B) When the short-term agent and long-term agent yield inconsistent action estimates, the world model predicts the resulting future states for each candidate action and evaluates their similarity to the virtual target. The action associated with the highest similarity is selected as the optimal action.
Figure S10 : Prompts for the LLM-guided agent. The prompt design comprises a fixed system prompt and a dynamically generated user prompt instantiated at each inference step.
Figure S11 : Prompts for the target switch agent. The prompt design comprises a fixed system prompt and a dynamically generated user prompt instantiated at each inference step.
Figure S12 : User interface for action annotation. The interface presents real observations alongside corresponding virtual targets. Experts use the real image as the observation and the virtual target as the navigation goal, selecting one to four plausible actions as annotations, which are recorded as ground-truth labels.
Figure S13 : Navigation process in two phantoms. Illuminated paths indicate the navigation trajectories. Two distinct trajectories are recorded across two different phantoms.
Figure S14 : Ablation study on in vivo phantom and retrospective in vivo data. We evaluate model performance on a gold-standard retrospective in phantom and in vivo dataset using precision, recall, and F1-score under increasing training data scales. When trained solely on expert-annotated phantom data, the model exhibits poor generalization. Incorporating CycleGAN-based synthetic data leads to a substantial performance improvement, highlighting the importance of data scaling via style transfer. Further inclusion of ex vivo porcine lung data yields a modest additional gain.
Figure S15 : Working mechanism of LLM-based guidance. The LLM-based guidance framework is illustrated with representative examples from phantom, ex vivo, and in vivo settings. In the virtual view, arrows pointing toward the target lumen provide high-level semantic cues that prompt the LLM to propose an action candidate. During the procedure, the agent evaluates real-time observations against the virtual target and produces probability logits over the action space. If the LLM-proposed action falls within the top-5 logits, it is directly accepted. Otherwise, a world model is invoked to simulate future states under alternative actions, and the action whose predicted future state exhibits the highest similarity to the virtual target is selected as the final decision.
Figure S16 : Working mechanism of preoperative guidance. The preoperative guidance strategy is illustrated with representative examples from phantom, ex vivo, and in vivo settings. Action candidates are derived from preoperative centerline coordinate variations along the planned path. During the intra-operative phase, the agent evaluates each candidate using real-time observations and the corresponding virtual target, producing probability logits over the action space. An action is executed only if the candidate falls within the top-3 logits; otherwise, the system outputs None and no action is performed, ensuring conservative and robust decision-making under uncertainty.
Figure S17 : Future prediction of the world model . The world model receives the first one frame of an unseen video as context input and the predicts next 15 steps into the future given the action sequence, without access to intermediate images
Agent Level
Sub-process
Trigger Type
Triggers / Trial
Per-call Latency (s)
Short-Term
Model Inference
Continuous
76.6 ± 20.206
0.0065 ± 0.0012
Robot Execution
Continuous
76.6 ± 20.206
3 ± 0
Long-Term
Preoperative Guidance
Event-driven
22.6 ± 5.177
0.715 ± 0.502
VLM Reasoning
Event-driven
1 ± 0
42.339 ± 7.773
Table S1 : Timing analysis of the proposed hierarchical agent architecture. The system is explicitly decoupled across temporal abstractions. The short-term agent operates continuously for reactive control and low-level decision-making, whereas the long-term agent is event-driven and invoked only at sparse, task-dependent milestones for high-level reasoning and planning. Trigger frequency and per-call latency are reported as mean ± standard deviation over N=5 independent trials. Reported latencies correspond to algorithmic computation or physical execution time per trigger. For short-term robot execution, a fixed waiting period of 3 s is enforced after each action before the next control cycle begins, ensuring completion of the current motion; therefore, the robot execution latency is reported as 3 s per trigger. Decision actions without physical execution (e.g., subgoal switching) are omitted, as they do not include robot execution time.
Accurate localization of the bronchoscope within the bronchial tree is essential for clinicians to be able to reach target lesions, perform biopsies and avoid misidentification of airway segments during diagnostic and therapeutic procedures. However, existing navigation systems typically rely on patient-specific CT scans or additional external sensors, increasing cost, setup time and patient radiation exposure. This work presents BronchoTop, a real-time, RGB-only framework for topological bronchoscopy localization that eliminates the need for patient-specific data. BronchoTop estimates scope location relative to a generic airway model through four modules: lumen detection and tracking, lumen-branch label association, probabilistic scope location estimation, and switch verification. By using only standard bronchoscopy video input, BronchoTop provides practical, real-time navigational assistance to physicians. Evaluation on phantom, simulated and real data demonstrates state-of-the-art accuracy, improving existing approaches performance by over 20% on real bronchoscopy sequences. BronchoTop is the first published framework including both the localization algorithms as well as all the real data used, together with code to generate additional simulations, encouraging and facilitating further developments and benchmarking. The results highlight BronchoTop's potential to enhance procedural safety, efficiency and accessibility in clinical and robotic bronchoscopy.
Clara Tomasini, Ana Cristina Murillo, Luis Riazuelo
Navigational bronchoscopy is critical for pulmonary interventions, yet current platforms depend heavily on pre-operative CT or external sensors, limiting their use in critical care and resource-constrained settings. Vision-only navigation offers a scalable alternative, but conventional visual odometry (VO) struggles with texture-poor airway images, specularities, and the vanishing-point singularities of tubular anatomy, leading to frequent tracking failures and drift. We present a geometry-aware VO framework that explicitly leverages vanishing-point cues from airway lumens. Detected lumens are back-projected to 3D rays, whose weighted fusion yields a stable forward heading even when parallax cues are absent. This heading, together with looming-based velocity estimates, is fused with noisy VO outputs using a bespoke high-gain observer that enforces airway-following priors and rejects drift. We validate the method on ex-vivo mechanically ventilated human lungs with electromagnetic tracking ground truth. Compared to state-of-the-art pipelines (ORB-SLAM2, LoFTR-VO, DPVO), our approach reduces absolute trajectory error by more than 50% and achieves the lowest relative pose error across all test sequences.
Mohammadreza Kasaei, Francis Xiatian Zhang, Feng Li +3
School of Informatics, University of Edinburgh, UK · Baillie Gifford Pandemic Science Hub, Institute for Regeneration and Repair, University of Edinburgh, UK · Walker Department of Mechanical Engineering and Texas Robotics, The University of Texas at Austin, USA
Reliable three-dimensional (3D) navigation of magnetically actuated untethered microrobots remains a major barrier to clinical translation. X-ray fluoroscopy is the standard real-time imaging modality for endovascular procedures, but single-plane fluoroscopy provides only a two-dimensional (2D) projection, eliminating depth information and complicating autonomous navigation. Recovering this information through biplane imaging or repeated contrast-enhanced angiography increases procedural complexity, radiation exposure, or contrast burden. Here, we introduce VISTA (Virtual Integration for Spatial Tracking and Autonomy), a digital twin framework enabling contrast-free autonomous navigation under single-plane fluoroscopy. VISTA reconstructs vascular anatomy as a 3D digital twin, discretizes the vessel centerline into navigation milestones, and assigns the detected 2D robot position to the nearest projected milestone. Consecutive milestones define the local vessel orientation used to generate magnetic actuation commands, converting single-plane fluoroscopic observations into topology-constrained navigation states without requiring contrast injection during navigation. VISTA is demonstrated across anatomically distinct vascular phantoms under continuous flow and within the inferior vena cava of a live rat in vivo. Compared with conventional fluoroscopic human-in-the-loop control, VISTA reduced navigation time by up to 62%, corrective actuation commands by up to 98%, and radiation exposure by up to 57%. These results establish VISTA as a digital twin-guided framework for contrast-free autonomous navigation of untethered endovascular microrobots using widely available single-plane fluoroscopy.
Husnu Halid Alabay, Tuan-Anh Le, Ping Wang +1
Department of Physiology and Biomedical Engineering, Mayo Clinic, Scottsdale, Arizona, USA · School of Biological and Health Systems Engineering, Arizona State University, Tempe, AZ, USA