Recon2Servo: Robotic Ultrasound Visual Servoing via Learned Image-to-Motion Inference

Yameng Zhang; Pei Liu; Dianye Huang; Yizhao Qian; Zhongyu Chen; Xiangyu Chu; K. W. Samuel Au; Zhongliang Jiang

cs.ROOct 5, 2026

Recon2Servo: Robotic Ultrasound Visual Servoing via Learned Image-to-Motion Inference

Authors: Yameng Zhang, Pei Liu, Dianye Huang, Yizhao Qian, Zhongyu Chen, Xiangyu Chu, K. W. Samuel Au, Zhongliang Jiang

Organizations: Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR. · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong SAR. · School of Computation, Information and Technology, Technical University of Munich, Germany. · Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR. · School of Optometry, Hong Kong Polytechnic University, Hong Kong SAR.

Abstract

Ultrasound visual servoing is essential for autonomous robotic ultrasound, yet 6-DoF probe control from 2D B-mode images remains challenging due to limited and ambiguous out-of-plane motion cues. Existing methods typically rely on anatomical priors or handcrafted visual features, limiting their generalizability across imaging targets. Inspired by trackerless 3D ultrasound reconstruction, we propose Recon2Servo, a visual servoing framework that learns image-to-motion inference directly from B-mode images for 6-DoF probe control. A DINOv3 encoder with low-rank adaptation and a bidirectional relation module estimate the relative probe pose between current and target images to guide iterative closed-loop target-view alignment. The framework combines supervised relative-pose learning, reconstruction-guided closed-loop adaptation, and bounded residual pose correction to improve motion inference during servoing. Evaluations on a public dataset and an in-house dataset collected from 12 healthy volunteers using different ultrasound systems demonstrate its effectiveness in reconstructed-volume servoing. Additional real-robot demonstrations of target-view alignment and dynamic tracking on a human forearm are provided in the supplementary video: https://youtu.be/qwsOdI-GMYk.

Figures & tables

Fig. 1: Overview of the proposed ultrasound servoing framework. Tracked B-mode images are reconstructed into a 3-D volume, which serves as an interactive simulation environment for training and adapting the probe control policy. The resulting policy is deployed using real-time ultrasound image feedback to iteratively guide the probe toward a target view.

Fig. 2: Relative-pose estimation network. A shared LoRA-adapted DINOv3 encoder and bidirectional relation module map current and target images to translation and rotation predictions. Snowflakes and flames denote frozen and trainable components, respectively.

Fig. 3: Reconstruction-guided closed-loop adaptation. Tracked images and calibrated poses define the reconstruction environment. A frozen pose estimator drives fixed-gain rollouts, and the renderer returns images at the updated poses. The known current and target poses provide oracle labels for visited states, which are combined with recorded training pairs to update the pose estimator.

Fig. 4: Bounded residual pose correction. The adapted encoder, relation module, and pose heads are frozen. The two motion tokens and normalized mean estimate are concatenated and passed to a trainable residual MLP. Its output is bounded in physical twist coordinates and added to the mean.

Fig. 5: Robotic ultrasound acquisition setup for the in-house dataset. A SonoScape L741 linear transducer connected to a SonoScape E2 system is mounted on a Realman RM75 robotic arm for automated forearm scanning.

Fig. 6: Reconstructed ultrasound volumes and image regions used for servoing in (a) the public TUS-REC2024 dataset and (b) the in-house dataset. Left: reconstructed volumes with the imaging planes outlined in red. Right: corresponding B-mode views, with red boxes indicating the $30\times 30$ mm ROIs resized to $256\times 256$ pixels for model input.

Fig. 7: Dataset-specific training protocols. The public-dataset estimator is trained on recorded pairs, adapted using 80% recorded and 20% rollout pairs, and refined using rollout pairs alone. For the in-house dataset, the public-dataset Stage-2 estimator initializes adaptation using labeled in-house rollout pairs, followed by residual-head training on the same rollout data.

Dataset	Model	Overall $\uparrow$	2-4 mm $\uparrow$	4-7 mm $\uparrow$	7-10 mm $\uparrow$	10-15 mm $\uparrow$
Public	A: Initial estimator	512/720 (71.11%)	86.67%	71.67%	65.00%	61.11%
	B: A + DAgger	589/720 (81.81%)	92.22%	80.00%	78.89%	76.11%
	C: A + Residual	594/720 (82.50%)	90.56%	81.67%	82.22%	75.56%
	D: A + DAgger + Residual	640/720 (88.89%)	95.56%	88.89%	88.89%	82.22%
In-house	Public A (zero-shot)	189/576 (32.81%)	52.1%	38.2%	27.1%	13.9%
	Public B (zero-shot)	228/576 (39.58%)	59.03%	44.44%	34.03%	20.83%

TABLE I: Overall and distance-stratified reaching success.

Dataset	Model	Out of range $\downarrow$	Step limit $\downarrow$	Out of volume $\downarrow$
Public	A: Initial estimator	16 (2.22%)	192 (26.67%)	0 (0.00%)
	B: A + DAgger	14 (1.94%)	116 (16.11%)	1 (0.14%)
	C: A + Residual	65 (9.03%)	61 (8.47%)	0 (0.00%)
	D: A + DAgger + Residual	30 (4.17%)	50 (6.94%)	0 (0.00%)
In-house	Public A (zero-shot)	170 (29.51%)	199 (34.55%)	18 (3.13%)
	Public B (zero-shot)	154 (26.74%)	177 (30.73%)	17 (2.95%)

TABLE II: Failure counts and percentages by termination category.

Dataset	Model	Overall $\downarrow$	2-4 mm $\downarrow$	4-7 mm $\downarrow$	7-10 mm $\downarrow$	10-15 mm $\downarrow$
Public	A: Initial estimator	$6.2\pm 9.0$	$3.8\pm 7.1$	$6.9\pm 9.4$	$6.4\pm 8.3$	$8.7\pm 10.7$
	B: A + DAgger	$5.2\pm 6.9$	$2.4\pm 4.3$	$5.3\pm 6.7$	$5.3\pm 5.1$	$8.3\pm 9.6$
	C: A + Residual	$3.2\pm 3.0$	$1.4\pm 1.1$	$3.5\pm 4.1$	$3.7\pm 2.8$	$4.4\pm 2.3$
	D: A + DAgger + Residual	$\boldsymbol{2.8\pm 2.1}$	$\boldsymbol{1.3\pm 1.1}$	$\boldsymbol{2.6\pm 1.7}$	$\boldsymbol{3.4\pm 1.9}$	$\boldsymbol{4.2\pm 2.4}$
In-house	Public A (zero-shot)	13.7 $\pm$ 12.2	8.1 $\pm$ 8.5	15.4 $\pm$ 11.1	17.9 $\pm$ 13.9	22.1 $\pm$ 14.8
	Public B (zero-shot)	$13.0\pm 11.4$	$7.9\pm 8.0$	$14.1\pm 11.7$	$17.7\pm 11.9$	$17.5\pm 12.4$

TABLE III: Control steps to success.

Explore similar work

Seeing Globally, Refining Locally: Global Visual Guidance and Local Ultrasound Cues for Robust Freehand 3-D Ultrasound Reconstruction
Jul 14, 2026Yameng Zhang, Zhongyu Chen, Dianye Huang +3Ultrasound Single Trajectory
Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China
Contact-Aware Impedance Controller for Robot-Assisted Ultrasound Imaging
Sep 28, 2026MD Miraj Arefin, M Efe TiryakiRobotic Ultrasound Impedance Control
Center for Robotics and AI (ROMER), Middle East Technical University, Ankara, Turkey
From Self-Normal-Positioning to Omni-Directional Tracking: Real-Time Surface Modeling Enabled Probe Tilt Control for Robotic Ultrasound Imaging
Aug 11, 2026Xihan Ma, Haichong ZhangRobotic Ultrasound Echocardiography
Department of Robotics Engineering, Worcester Polytechnic Institute, 100 Institute Road, Worcester, MA 01609 USA

Jul 14, 2026, cs.CVJ/K move · Enter open · S save

Seeing Globally, Refining Locally: Global Visual Guidance and Local Ultrasound Cues for Robust Freehand 3-D Ultrasound Reconstruction

Yameng Zhang, Zhongyu Chen, Dianye Huang, Xiangyu Chu+2

Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China

Ultrasound Single Trajectory

Open paper AskPDF arXiv

Freehand 3-D ultrasound (US) imaging has attracted increasing attention owing to its intuitive volumetric visualization, ease of use, and low cost. However, accurate 3-D reconstruction critically depends on stable probe pose estimation, yet existing trackerless methods remain susceptible to accumulated pose errors, particularly over long scanning trajectories. To address this limitation, we propose a global-to-local pose estimation framework that exploits external camera observations for globally stable localization and B-mode US images for anatomy-aware local refinement. Specifically, the framework comprises a dual-camera branch that performs contextual feature aggregation across camera views and temporal observations to estimate a globally consistent probe trajectory, and a B-mode branch that performs anatomical feature aggregation from sequential US images to capture tissue-dependent local motion cues. A cross-modal fusion module subsequently integrates the contextual camera features and anatomical US features to predict pose residuals and refine the camera-derived estimates in the transformation space. Furthermore, a multi-scale pose loss constrains relative motion over multiple temporal horizons to suppress accumulated drift during extended scans. The proposed framework is validated on phantom and in vivo datasets. On two in-house datasets (FUSION-J and FUSION-L) collected using different machines, the proposed US + Dual-Cam model reduces average trajectory drift to 1.67 mm and 1.29 mm, representing improvement of 16.50% and 27.12%, respectively, over a strong dual-camera baseline, while substantially outperforming US-only pose estimation (>13 mm drift). In in vivo forearm arteries reconstruction, it achieves Hausdorff distances of 1.58 mm, demonstrating the effectiveness of the proposed method on real clinical scenarios.

Abstract

Explore similar work

Seeing Globally, Refining Locally: Global Visual Guidance and Local Ultrasound Cues for Robust Freehand 3-D Ultrasound Reconstruction

Contact-Aware Impedance Controller for Robot-Assisted Ultrasound Imaging

From Self-Normal-Positioning to Omni-Directional Tracking: Real-Time Surface Modeling Enabled Probe Tilt Control for Robotic Ultrasound Imaging