Sample-Efficient Learning with Online Expert Correction for Autonomous Catheter Steering in Endovascular Bifurcation Navigation
Authors: Hao Wang, Tianliang Yao, Bo Lu, Zhiqiang Pei, Liu Dong, Lei Ma, Peng Qi
Organizations: Department of Control Science and Engineering, College of Electronics and Information Engineering, and Shanghai Institute of Intelligent Science and Technology, Tongji University, Shanghai 200092, China · Department of Electronic Engineering, Faculty of Engineering, The Chinese University of Hong Kong, Hong Kong SAR 999077, China · Robotics and Microsystems Center, School of Mechanical and Electric Engineering, Soochow University, Suzhou, Jiangsu 215131, China · School of Oriental Pan-Vascular Devices Innovation College, University of Shanghai for Science and Technology, Shanghai 200093, China · Shanghai Operation Robot Co., Ltd., Shanghai 201318, China · State Key Laboratory of Cardiovascular Diseases and Medical Innovation Center, Shanghai East Hospital, School of Medicine, Tongji University 200092, Shanghai, China
Robot-assisted endovascular intervention offers a safe and effective solution for remote catheter manipulation, reducing radiation exposure while enabling precise navigation. Reinforcement learning (RL) has recently emerged as a promising approach for autonomous catheter steering; however, conventional methods suffer from sparse reward design and reliance on static vascular models, limiting their sample efficiency and generalization to intraoperative variations. To overcome these challenges, this paper introduces a sample-efficient RL framework with online expert correction for autonomous catheter steering in endovascular bifurcation navigation. The proposed framework integrates three key components: (1) A segmentation-based pose estimation module for accurate real-time state feedback, (2) A fuzzy controller for bifurcation-aware orientation adjustment, and (3) A structured reward generator incorporating expert priors to guide policy learning. By leveraging online expert correction, the framework reduces exploration inefficiency and enhances policy robustness in complex vascular structures. Experimental validation on a robotic platform using a transparent vascular phantom demonstrates that the proposed approach achieves convergence in 123 training episodes -- a 25.9% reduction compared to the baseline Soft Actor-Critic (SAC) algorithm -- while reducing average positional error to 83.8% of the baseline. These results indicate that combining sample-efficient RL with online expert correction enables reliable and accurate catheter steering, particularly in anatomically challenging bifurcation scenarios critical for endovascular navigation.
Figures & tables
Fig. 1: Illustration of endovascular navigation enhanced by expert knowledge and behavior modeling. The framework utilizes intraoperative imaging to detect the real-time position and orientation of the catheter tip. By combining expert procedural patterns with reinforcement learning strategies, the system dynamically adjusts navigation to ensure safe and efficient traversal through vascular bifurcations. This hybrid approach improves accuracy and reliability in reaching target anatomical sites during complex interventions. The schematic was created using BioRender ( https://biorender.com ).
Fig. 2: Expert-in-the-loop catheter navigation framework integrating reinforcement learning and fuzzy control. (a) Robotic catheterization setup and agent–environment interaction: the agent observes the vascular state st and outputs action at to command catheter translation, rotation, and gripping. (b) Policy learning that combines Soft Actor–Critic (SAC) with Generative Adversarial Imitation Learning (GAIL): mini-batches from a replay buffer train an actor–critic with twin critics ( Q1 , Q2 ). A discriminator supplies an expert reward shaped by prior knowledge, and the policy is optimized using a weighted sum of SAC and GAIL rewards ( wSAC , wGAIL ). Bifurcation detection triggers the expert-in-the-loop module. (c) Online fuzzy pose correction at bifurcations: an expert selects a target pose; a U-Net-based segmentation extracts the catheter mask, followed by smoothing/erosion and geometry fitting for real-time pose estimation. Translation and rotation errors feed a fuzzy controller that adjusts robot commands to reach the expert-corrected pose.
Algorithm
Episodes
Avg. Steps
Success Rate
Avg. Time (s)
Avg. Error (px)
TD3
300
9.21
41.67% (125/300)
78.09
208.76
SAC
300
8.32
44.67% (134/300)
60.39
98.55
SAC-GAIL
300
7.57
53.67% (161/300)
59.04
149.53
SAC-EIL
300
7.66
51.33% (154/300)
61.05
86.90
SAC-EIL-GAIL
300
7.61
59.00% (177/300)
59.41
82.58
TABLE I: Comparison of Autonomous Navigation Across RL Algorithms for Bifurcation Steering
Fig. 3: Autonomous catheter navigation and constant-curvature modeling. (a) After training, the learned policy autonomously steers the catheter to the predefined target region in a silicone renal artery phantom by continuously localizing the tip in fluoroscopic frames. (b) Upon bifurcation detection, the catheter centerline (skeleton) is extracted and fitted with a constant-curvature model. Directional consistency is assessed by the sign/magnitude of the cross product between the fitted tangent and the skeleton tangent; the resulting geometric errors drive an expert-in-the-loop fuzzy controller that updates robot commands to achieve the expert-specified pose. Insets report the tip–target distance (pixels) and cross-product values over time.
Fig. 4: Comparison of algorithm performance metrics. (a) Normalized Comparison of Key Performance Indicators Across Different RL Algorithms. (b) Average reward per episode during SAC-EIL-GAIL training. As the number of episodes increases, the reward gradually converges. This indicates stable and effective learning.
Fig. 5: This figure illustrates the error distributions of TD3, SAC, SAC-GAIL, SAC-EIL, and SAC-EIL-GAIL. The boxplots highlight the central tendency and variability of each method, while the red asterisks indicate statistically significant differences with SAC-EIL-GAIL ( ∗p<0.05,∗∗p<0.01,∗∗∗p<0.001 ).
Robotic endovascular intervention uses robotic systems to navigate vascular instruments and perform procedures with greater precision, control, or autonomy.
Department of Control Science and Engineering, College of Electronics and Information Engineering, and Shanghai Institute of Intelligent Science and Technology, Tongji University, Shanghai 200092, China · School of Mechanical Engineering, Tongji University, Shanghai 200092, China · Department of Electronic Engineering, Faculty of Engineering, The Chinese University of Hong Kong, Hong Kong SAR 999077, China +2
Department of Control Science and Engineering, College of Electronic and Information Engineering, and Shanghai Institute of Intelligent Science and Technology, Tongji University, Shanghai 200092, China. · Department of Electronic Engineering, Faculty of Engineering, The Chinese University of Hong Kong, Hong Kong SAR 999077, China. · Shanghai Operation Robot Co., Ltd., Shanghai 201318, China.
Surgical & Interventional Engineering, School of Biomedical Engineering & Imaging Sciences, King’s College London, UK · Department of Radiology, National Taiwan University Hospital, Taiwan · Department of Neuroradiology, King’s College Hospital, UK