RAG-RUSS: A Retrieval-Augmented Robotic Ultrasound for Autonomous Carotid Examination

Dianye Huang; Ziping Cong; Nassir Navab; Zhongliang Jiang

cs.ROMar 1, 2026

RAG-RUSS: A Retrieval-Augmented Robotic Ultrasound for Autonomous Carotid Examination

Authors: Dianye Huang, Ziping Cong, Nassir Navab, Zhongliang Jiang

Organizations: Computer-Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich (TUM), Germany · Munich Center for Machine Learning (MCML), Germany

Abstract

Robotic ultrasound (US) has recently attracted increasing attention as a means to overcome the limitations of conventional US examinations, such as the strong operator dependence. However, the decision-making process of existing methods is often either rule-based or relies on end-to-end learning models that operate as black boxes. This has been seen as a main limit for clinical acceptance and raises safety concerns for widespread adoption in routine practice. To tackle this challenge, we introduce the RAG-RUSS, an interpretable framework capable of performing a full carotid examination in accordance with the clinical workflow while explicitly explaining both the current stage and the next planned action. Furthermore, given the scarcity of medical data, we incorporate retrieval-augmented generation to enhance generalization and reduce dependence on large-scale training datasets. The method was trained on data acquired from 28 volunteers, while an additional four volumetric scans recorded from previously unseen volunteers were reserved for testing. The results demonstrate that the method can explain the current scanning stage and autonomously plan probe motions to complete the carotid examination, encompassing both transverse and longitudinal planes.

Figures & tables

Fig. 1: An illustration showing a representative use case of the RAG-RUSS system. Given a clinician’s query, RAG-RUSS retrieves similar annotated scans via RAG, then identifies the stage , generates an explanation , and predicts the next_API . This API is performed by the robotic arm, and the acquired US images are fed back into the system, forming a closed loop that enables autonomous, interpretable carotid artery scanning.

Fig. 2: Overview of the carotid ultrasound scanning workflow and data acquisition/processing pipeline. Left: presents eight predefined scanning stages of the carotid artery examination. Right: US images and the probe’s pose trajectory are recorded and subsequently compounded into a 3D volumetric data for use in the simulation environment.

Age (years)	Height (m)	Weight (kg)	BMI
27.97 $\pm$ 3.91	1.74 $\pm$ 0.09	70.22 $\pm$ 12.24	23.06 $\pm$ 2.86

TABLE I: statistics of volunteers (N=32).

Fig. 3: Architecture and runtime inference of the proposed RAG-RUSS for carotid artery US scanning. Left: System architecture comprising an LLM backbone (Vicuna v1.5, 7B [ 26 ] ), the medical vision foundation model (PubMedCLIP-ViT-B/32 [ 27 ] ), and a RAG module that retrieves similar scanning contexts to support decision-making. Right: Inputs/outputs of RAG-RUSS and the signal flow when deploying it for carotid US scanning. For more details on inputs/outputs and architecture of RAG-RUSS, refer to Fig. 4 and Section III -B , respectively.

Fig. 4: An illustrutive multi-turn QA example of RAG-RUSS. The inputs are: i). System prompt that specifies the task description along with the predefined executable APIs and scanning stages. ii). Two retrieved scanning contexts via the RAG component. iii). The current query involves the two input US images and the previously predicted stage, and then poses three questions sequentially. RAG-RUSS then outputs: i). the current stage , ii). a short explanation , and iii). the next_API to execute.

Fig. 5: Scheme for building an RAG database. The scanning context includes two US frames, the previous stage, and associated VQA annotations.

Stage	16-dim. Embedding Vector
Examine CCA proximal	[0.1, 0.1, 0.1, $\cdots$ , 0.1]
Examine CCA distal	[0.2, 0.2, 0.2, $\cdots$ , 0.2]
Examine bifurcation	[0.4, 0.4, 0.4, $\cdots$ , 0.4]
Transverse completed	[0.5, 0.5, 0.5, $\cdots$ , 0.5]
Return to carotid bulb	[0.6, 0.6, 0.6, $\cdots$ , 0.6]
Return completed	[0.7, 0.7, 0.7, $\cdots$ , 0.7]

TABLE II: Textual Stage to embedding vector mapping.

Stage	Top@1 Acc. $\uparrow$	Top@2 Acc. $\uparrow$
Examine CCA proximal	95.3%	96.2%
Examine CCA distal	85.5%	89.3%
Examine bifurcation	66.7%	74.9%
Transverse scan completed	30.5%	37.3%
Return to carotid bulb	48.0%	59.7%
Return completed	67.0%	76.9%

TABLE III: Context Example Retrieval Accuracy

Method	CCA proximal	CCA distal	Bifurcation	Trans. completed	Return completed	Long. completed	Avg acc.
RAG only	96.5%	82.3%	84.3%	50.0%	25.0%	50.0%	64.7%
VLM only	92.6%	93.3%	69.2%	75.0%	50.0%	50.0%	71.7%
RAG-RUSS@1	69.9%	92.1%	53.6%	50.0%	75.0%	75.0%	69.3%
RAG-RUSS@2	97.8%	95.0%	81.7%	100.0%	25.0%	75.0%	79.1%
RAG-RUSS@1 and RAG-RUSS@2 denote configurations in which the RAG component retrieves one or two in-context learning examples, respectively, for the RAG-RUSS input.
Trans. completed: Transverse scan completed. Long. completed: Longitudinal scan complete. Avg. acc.: Average accuracy.

TABLE IV: Stage-level recognition accuracy under different ablation settings.

Fig. 6: Visualization example of the closed-loop evaluation results of RAG-RUSS@2. The upper panel presents US images corresponding to eight stages predicted by the model. The lower panel illustrates the complete scanning trajectory together with the annotated scanning regions, providing a visualization of the model’s recognized progression through different stages.

Robotic Ultrasound

Robotic ultrasound uses robots to acquire and interpret ultrasound images for autonomous scanning and image-guided procedures.

See all 27 papers

From Scanning Guidelines to Action: A Robotic Ultrasound Agent with LLM-Based Reasoning
Mar 15, 2026Yuan Bi, Yiping Zhou, Pei Liu +3Robotic Ultrasound Large Language Model-Based Robot Planning
Chair for Computer-Aided Medical Procedures and Augmented Reality, Technical University of Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Medical Intelligence and Robotic Cognition Lab, The University of Hong Kong, Hong Kong, China
EmbodiedUS-FS: Fast Slow Intelligence for Ultrasound Robotics
Jun 21, 2026Fangzhuo Zhang, Xinyu Wang, Xiao Yang +1Robotic Control Robotics
independent researcher · University of Wyoming · Shanghai World Foreign Language Academy +1
US-VLA: An Ultrasound Vision-Language-Action Model for Embodied Abdomina
Aug 17, 2026Cheng Zhang, Xingzheng Wu, Guihao Yan +4Robotic Ultrasound Vision-Language-Action Models
Innovation School of Artificial Intelligence, Hefei University of Technology Hefei, China

Mar 15, 2026J/K move · Enter open · S save

From Scanning Guidelines to Action: A Robotic Ultrasound Agent with LLM-Based Reasoning

Yuan Bi, Yiping Zhou, Pei Liu, Feng Li+2

Chair for Computer-Aided Medical Procedures and Augmented Reality, Technical University of Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Medical Intelligence and Robotic Cognition Lab, The University of Hong Kong, Hong Kong, China

Robotic Ultrasound Large Language Model-Based Robot Planning Robotic Tool Use

Open paper AskPDF arXiv

Robotic ultrasound offers advantages over free-hand scanning, including improved reproducibility and reduced operator dependency. In clinical practice, US acquisition relies heavily on the sonographer's experience and situational judgment. When transferring this process to robotic systems, such expertise is often encoded explicitly through fixed procedures and task-specific models, yielding pipelines that can be difficult to adapt to new scanning tasks. In this work, we propose a unified framework for autonomous robotic US scanning that leverages a LLM-based agent to interpret US scanning guidelines and execute scans by dynamically invoking a set of provided software tools. Instead of encoding fixed scanning procedures, the LLM agent retrieves and reasons over guideline steps from scanning handbooks and adapts its planning decisions based on observations and the current scanning state. This enables the system to handle variable and decision-dependent workflows, such as adjusting scanning strategies, repeating steps, or selecting the appropriate next tool call in response to image quality or anatomical findings. Because the reasoning underlying tool selection is also critical for transparent and trustworthy planning, we further fine tune the LLM agent using a RL based strategy to improve both its reasoning quality and the correctness of tool selection and parameterization, while maintaining robust generalization to unseen guidelines and related tasks. We first validate the approach via verbal execution on 10 US scanning guidelines, assessing reasoning as well as tool selection and parameterization, and showing the benefit of RL fine tuning. We then demonstrate real world feasibility on robotic scanning of the gallbladder, spine, and kidney. Overall, the framework follows diverse guidelines and enables reliable autonomous scanning across multiple anatomical targets within a unified system.

Abstract

Robotic Ultrasound

Explore similar work

From Scanning Guidelines to Action: A Robotic Ultrasound Agent with LLM-Based Reasoning

EmbodiedUS-FS: Fast Slow Intelligence for Ultrasound Robotics

US-VLA: An Ultrasound Vision-Language-Action Model for Embodied Abdomina