From Freehand Demonstrations to Autonomous Robotic Ultrasound: An Anatomy-Anchored Learning Framework

Yuan Bi; Fangxu Han; Ye Su; Yiping Zhou; Zhongliang Jiang; Nassir Navab

cs.ROOct 23, 2026

From Freehand Demonstrations to Autonomous Robotic Ultrasound: An Anatomy-Anchored Learning Framework

Authors: Yuan Bi, Fangxu Han, Ye Su, Yiping Zhou, Zhongliang Jiang, Nassir Navab

Organizations: Chair for Computer Aided Medical Procedures (CAMP), Technical University of Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China

Abstract

Ultrasound imaging is highly operator-dependent, motivating robotic systems that reproduce expert scanning skills consistently. These skills involve coordinated interpretation of anatomical landmarks, ultrasound images, and probe-tissue interactions that are difficult to encode in hand-designed control strategies. Imitation learning offers an alternative, but its observation and action representations should reflect the anatomically structured nature of scanning. We propose an anatomy-anchored imitation-learning framework that combines anatomy-guided probe placement with learned ultrasound navigation. The placement module identifies a task-specific anatomical starting point from external visual observations and establishes the initial probe pose as the reference frame for scanning. A flow-based policy then generates scanning motions from anchor-relative probe poses, ultrasound images, probe-mounted camera views, and contact forces, while a termination head predicts task completion. This avoids continuous reliance on third-person vision and reduces dependence on the robot-patient configuration. A handheld acquisition platform enables multimodal demonstrations to be collected during natural freehand scanning without a robot. On 16 unseen participants, task success reaches 96%, 94%, 88%, and 94% for kidney, gallbladder, and liver standard-plane acquisition and radial artery sweeping, respectively. Compared with a fine-tuned pi_0 model, success on the abdominal tasks improves by 10-42 percentage points, with lower time-to-target, peak contact force, and trajectory jerk. Ablations support the complementary roles of anatomical anchoring, egocentric vision, and force feedback. End-to-end trials with automatic placement achieve 90-100% success, and the policies transfer without retraining to a second robot with 83-100% success.

Figures & tables

Fig. 1: From freehand demonstrations to autonomous robotic ultrasound. Top: Robot-dependent demonstration collection requires teleoperation and a dedicated robotic setup. Bottom: Our framework enables robot-free collection of expert scanning demonstrations and transfers the acquired skills to autonomous robotic execution through anatomy-anchored imitation learning.

Fig. 2: Overview of the proposed anatomy-anchored imitation-learning framework. A handheld acquisition platform records synchronized probe poses, contact forces, ultrasound images, and wrist-camera views during freehand demonstrations. These multimodal observations, together with an anchor-relative pose representation, enable learning scanning policies for autonomous robotic ultrasound.

Fig. 3: Separation of probe placement and ultrasound navigation. Rule-based placement uses external anatomical cues to establish the initial probe pose and anatomical anchor. The learned policy then performs local scanning relative to this anchor using multimodal feedback.

Fig. 4: Portable ultrasound demonstration acquisition platform. Assembled (left) and exploded (right) views show the integrated sensing components, handheld housing, probe holder, and robot mounting interface, supporting multimodal freehand demonstration collection and subsequent robotic deployment.

Fig. 5: Anatomy-anchored formulation of ultrasound scanning. (a) Probe motion is modeled as a continuous velocity field in an anchor-relative pose space. (b) Landmark-guided probe placement establishes the initial anchor frame for subsequent ultrasound-guided navigation, illustrated for liver, gallbladder, left-kidney, and radial artery scanning.

Fig. 6: Architecture of the anchor-relative Streaming Flow Policy. Ultrasound images, wrist-camera images, and anchor-relative probe states are fused by a shared Transformer encoder into $\mathbf{c}_{\mathrm{base}}$ , and a separate force branch adds the residual $\mathbf{c}_{\mathrm{force}}$ to form the trajectory condition $\mathbf{c}_{\mathrm{traj}}$ . Conditioned on $\mathbf{c}_{\mathrm{traj}}$ , the velocity field $v_{\theta}$ is integrated from the current pose $\mathbf{x}^{A}_{t}$ over $\tau\in[0,1]$ and composed with the episode anchor ${}^{G}\mathbf{T}_{A}$ to produce the probe-tip trajectory. A termination head $g_{\psi}$ predicts $p_{\mathrm{stop}}$ from the stop-gradient copy $\mathrm{sg}(\mathbf{c}_{\mathrm{base}})$ .

Fig. 7: Anatomical landmark localization for automatic probe placement. (a) The scene camera captures two overlapping views of the participant. (b) The views are stitched into a whole-body image. (c) HSMR reconstructs the underlying skeleton to identify the target landmark, illustrated here by the inferior xiphoid tip, which guides initial probe placement and subsequent anatomical anchor definition.

Fig. 8: Experimental setup for autonomous robotic ultrasound scanning. A 7-DoF robot manipulates the ultrasound probe, with a wrist camera and force/torque sensor providing visual and contact feedback. The participant holds an emergency-stop button throughout the procedure.

Region	Task	Demos.	Volunteers	Starting condition $\rightarrow$	Ending condition
Abdomen	Kidney	69	5	T12 surface landmark $\rightarrow$	Left kidney, longitudinal view
	Gallbladder	60	5	Inferior xiphoid tip $\rightarrow$	Gallbladder, longitudinal view
	Liver	82	5	Inferior xiphoid tip $\rightarrow$	Left portal vein with segmental branches
Forearm	Radial artery	83	2	Wrist $\rightarrow$	Bifurcation of radial and ulnar arteries

TABLE I: Demonstration collection across scanning tasks. For abdominal tasks, the long axis of the probe footprint is initially aligned with the craniocaudal direction; the forearm task is a cross-sectional sweep. Volunteer counts are reported per task.

Region	Task	Method	Rollouts	Task success (%) $\uparrow$	ROI quality $\uparrow$	Autonomous stop (%) $\uparrow$	Time-to-target (s) $\downarrow$	Peak force (N)	Jerk $\mathrm{(m/s^{3})}\downarrow$
Abdomen	Kidney	Fine-tuned $\pi_{0}$	50	86.0	$2.78\pm 1.22$	23.3	$11.2\pm 4.0$	$7.2\pm 8.3$	$0.444\pm 0.228$
	Kidney	Anatomy-anchored SFP	50	96.0	$3.47\pm 1.09$	97.9	$5.7\pm 2.7$	$4.3\pm 2.0$	$0.292\pm 0.078$
	Gallbladder	Fine-tuned $\pi_{0}$	50	52.0	$2.19\pm 1.28$	19.2	$18.1\pm 8.7$	$4.4\pm 2.1$	$0.376\pm 0.067$
	Gallbladder	Anatomy-anchored SFP	50	94.0	$3.97\pm 1.04$	93.6	$9.7\pm 5.5$	$3.3\pm 1.8$	$0.262\pm 0.047$
	Liver	Fine-tuned $\pi_{0}$	50	58.0	$2.64\pm 1.39$	17.2	$12.8\pm 7.7$	$4.7\pm 2.2$	$0.420\pm 0.330$
	Liver	Anatomy-anchored SFP	50	88.0	$3.78\pm 1.00$	95.5	$9.5\pm 5.3$	$3.8\pm 2.0$	$0.217\pm 0.031$

TABLE II: Comparison of the proposed pipeline with fine-tuned

\pi_{0}

across scanning tasks on unseen participants.

Fig. 9: Representative ultrasound images from human demonstrations (top) and autonomous robotic rollouts (bottom). From left to right: kidney, gallbladder, and liver standard views, followed by images sampled along the radial artery sweep. White circles indicate the radial artery cross-section.

Fig. 10: Representative autonomous rollouts of the Anatomy-anchored SFP on unseen participants for the four scanning tasks: left-kidney, gallbladder, and liver standard-plane acquisition (abdomen) and radial artery sweeping (forearm). Each row shows four snapshots of one rollout. Each snapshot pairs the ultrasound image (left) with a schematic of the corresponding probe pose on the body (right): anterior view for the gallbladder and liver, posterior view for the kidney, and palmar view of the right forearm for the radial artery. Each episode starts at the task-specific anatomical landmark, whose probe pose defines the anchor frame $\{A\}$ ( $t=0$ ), and ends when the termination head predicts $p_{\mathrm{stop}}\geq\delta_{\mathrm{stop}}$ ( $t=T$ ). The time axis is schematic, and intermediate snapshots ( $t_{1}$ , $t_{2}$ ) are not equally spaced across tasks. Start and end conditions follow Tab. I . GB: gallbladder; PV: portal vein; LPV: left portal vein; RA: radial artery; UA: ulnar artery.

Task	Variant	Configuration	Rollouts	Task success (%) $\uparrow$	ROI quality $\uparrow$	Autonomous stop (%) $\uparrow$	Time-to-target (s) $\downarrow$	Peak force (N)	Jerk $\mathrm{(m/s^{3})}\downarrow$
Kidney	w/o wrist camera	Nominal	30	76.7	$2.79\pm 1.17$	78.3	$6.2\pm 2.1$	$5.3\pm 2.5$	$0.295\pm 0.087$
	w/o force input	Nominal	30	86.7	$3.07\pm 1.08$	92.3	$7.3\pm 3.4$	$5.8\pm 5.5$	$0.360\pm 0.307$
	Absolute pose	Nominal	30	76.7	$2.59\pm 1.29$	73.9	$7.0\pm 2.5$	$5.6\pm 2.8$	$0.372\pm 0.153$
	Absolute pose	Rotated $45^{\circ}$	30	70.0	$2.27\pm 1.16$	71.4	$6.9\pm 2.7$	$5.2\pm 2.9$	$0.443\pm 0.189$
	Anatomy-anchored SFP	Nominal	50	96.0	$3.47\pm 1.09$	97.9	$5.7\pm 2.7$	$4.3\pm 2.0$	$0.292\pm 0.078$
	Anatomy-anchored SFP	Rotated $45^{\circ}$	30	93.3	$3.54\pm 1.15$	96.4	$6.7\pm 2.7$	$4.8\pm 2.0$	$0.305\pm 0.197$

TABLE III: Ablation study of wrist-camera input, force input, and pose representation on unseen participants.

Fig. 11: Normal contact force over scan progress for Anatomy-anchored SFP and the w/o force input variant in (a) kidney navigation and (b) radial artery sweeping. Lines show the mean and shaded regions the interquartile range across $n$ rollouts.

Task	Rollouts	Success (%) $\uparrow$	Total time (s) $\downarrow$	Peak force (N)	Jerk (m/s 3 ) $\downarrow$
Kidney	30	100.0	$107.3\pm 14.6$	$10.2\pm 6.2$	$0.371\pm 0.023$
Gallbladder	30	93.3	$116.2\pm 24.1$	$7.0\pm 3.5$	$0.361\pm 0.030$
Liver	30	90.0	$116.3\pm 27.1$	$5.0\pm 1.4$	$0.356\pm 0.056$
Radial artery	30	90.0	$33.0\pm 4.3$	$8.2\pm 3.6$	$0.537\pm 0.054$

TABLE IV: End-to-end performance of the complete system (placement and scanning) on unseen participants.

Fig. 12: Experimental setup for autonomous robotic ultrasound scanning using KUKA iiwa7.

Task	Rollouts	Success (%) $\uparrow$	Time-to-Target (s) $\downarrow$	Peak force (N)	Jerk (m/s 3 ) $\downarrow$
Kidney	30	100.0	$12.7\pm 3.7$	$5.7\pm 2.8$	$0.431\pm 0.140$
Gallbladder	30	86.7	$18.2\pm 3.5$	$4.4\pm 2.0$	$0.345\pm 0.053$
Liver	30	93.3	$10.5\pm 3.3$	$4.6\pm 2.5$	$0.478\pm 0.119$
Radial artery	30	83.3	$24.6\pm 5.1$	$6.3\pm 3.9$	$0.597\pm 0.115$

TABLE V: Cross-embodiment validation of autonomous scanning using a KUKA robot on unseen participants.

Robotic Ultrasound

Robotic ultrasound uses robots to acquire and interpret ultrasound images for autonomous scanning and image-guided procedures.

See all 27 papers

Apr 28, 2026

SAMe: A Semantic Anatomy Mapping Engine for Robotic Ultrasound

Robotic ultrasound has advanced local image-driven control, contact regulation, and view optimization, yet current systems lack the anatomical understanding needed to determine what to scan, where to begin, and how to adapt to individual patient anatomy. These gaps make systems still reliant on expert intervention to initiate scanning. Here we present SAMe, a semantic anatomy mapping engine that provides robotic ultrasound with an explicit anatomical prior layer. SAMe addresses scan initiation as a target-to-anatomy-to-action process: it grounds under-specified clinical complaints into structured target organs, instantiates a patient-specific anatomical representation for the grounded targets from a single external body image, and translates this representation into control-facing 6-DoF probe initialization states without any additional registration using preoperative CT or MRI. The anatomical representation maintained by SAMe is explicit, lightweight (single-organ inference in 0.08s), and compatible with downstream control by design. Across semantic grounding, anatomical instantiation, and real-robot evaluation, SAMe shows strong performance across the full initialization pipeline. In real-robot experiments, centroid-based SAMe initialization outperformed the body-keypoint-based heuristic baseline under a budget-matched single-target setting for both liver (86.7% versus 46.7%) and kidney (80.0% versus 73.3%) initialization. Furthermore, The trial-level organ-hit rate reached 97.3% for liver and 83.3% for kidney when multiple candidate targets were available. These results establish an explicit anatomical prior layer that addresses scan initialization and is designed to support broader downstream autonomous scanning pipelines, providing the anatomical foundation for complaint-driven, anatomically informed robotic ultrasonography.

Jing Zhang, Duojie Chen, Wentao Jiang +7

School of Computer Science, Wuhan University, China · Hubei Center for Applied Mathematics, Wuhan University, China · Department of Ultrasound, The Central Hospital of Wuhan, China +4

Ultrasound Imaging Robotic Ultrasound

Ask PDF

Jul 24, 2026

Action-Conditioned World Model for Goal Plane Guidance in Robotic Neck Ultrasound

Ultrasound examinations typically follow specific scanning protocols that require navigating through an ordered sequence of anatomical landmarks and target imaging planes. We present a local action-conditioned world model framework for goal plane probe guidance in robotic ultrasound. Autonomous ultrasound tasks often require large numbers of probe-motion trajectories for training, but collecting high-quality demonstrations is labor-intensive and explicit simulators are difficult to build because ultrasound appearance depends on contact, tissue deformation, and view-dependent acoustic artifacts. We address this problem with a two-stage model-based learning pipeline. First, a latent conditional diffusion world model predicts future ultrasound observations from recent context frames, probe motions and temporal offset. Second, a goal-conditioned temporal transformer predicts ordered probe motions and is fine-tuned using rewards from the frozen world model. Experiments on the self-collected dataset show that the world model preserves action-dependent anatomical structure on target-directed scans. In real-world closed loop experiments, the framework achieves success rates of 70.0% for carotid guidance and 65.0% for thyroid guidance. These results demonstrate the potential of learned ultrasound dynamics for training goal-directed robotic probe navigation within protocol-constrained organ specific tasks.

Siqi Fan, Ran Liu, Zixuan Yang +6

Department of Mechanical and Automation Engineering, Chinese University of Hong Kong, Hong Kong SAR · Centre for Artificial Intelligence and Robotics Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong SAR · Department of Vascular Ultrasound, Xuanwu Hospital, the Capital Medical University +3

World Model Learning Action-Conditioned World Models Robotics Robotic Ultrasound Goal-Conditioned RL

Ask PDF

Jun 4, 2026

Visuotactile and Explicitly Force-Controlled Robotic Ultrasound for Abdominal Volumetric Reconstruction

In this paper, we present a robotic ultrasound acquisition system that integrates stereo vision, touch-based feedback, and expert-informed strategies to perform autonomous and adaptive abdominal scans. The system records freehand motion and force data from expert radiologists, creating a framework to capture transducer motion, applied forces, and anatomical scanning strategies. This expert data is replayed to replicate characteristic scans with the robot, forming a foundation for further autonomous capabilities. Using stereo vision, the system generates three-dimensional topography maps of the patient's abdomen, which are refined through stiffness measurements at key points to delineate the rib cage boundary. These combined techniques enable the robot to execute two distinct scanning paths: an upward-angled sweep beneath the rib cage to visualize structures near the upper abdomen and a perpendicular sweep across soft tissue regions. A compliant, torque-controlled seven degree-of-freedom robotic manipulator is controlled to maintain consistent probe contact through closed-loop force control over the varied anatomical surfaces. Physical experiments demonstrate that the system achieves high-quality imaging comparable to expert scans while dynamically adapting to patient-specific topographies. Furthermore, the robotic system surpasses expert capabilities by enabling three-dimensional volume acquisition, which enhances diagnostic potential and provides volumetric data for advanced analyses. This work highlights the integration of expert knowledge into autonomous robotic systems and underscores the potential of combining perception-based autonomy with physical reasoning for enhanced diagnostic performance.

Adrian Piedra, R Brooke Jeffrey, Oussama Khatib

Stanford Robotics Laboratory, Computer Science Department, Stanford University, Stanford, CA 94305 USA · Department of Radiology, School of Medicine, Stanford University, Stanford, CA 94305 USA

Robotic Control 3D Medical Imaging Ultrasound Imaging Robotic Ultrasound

Ask PDF

Abstract

Robotic Ultrasound

Explore similar work

SAMe: A Semantic Anatomy Mapping Engine for Robotic Ultrasound

Action-Conditioned World Model for Goal Plane Guidance in Robotic Neck Ultrasound

Visuotactile and Explicitly Force-Controlled Robotic Ultrasound for Abdominal Volumetric Reconstruction