Towards Automated Initial Probe Placement in Transthoracic Teleultrasound Using Human Mesh and Skeleton Recovery

Yu Chung Lee; David G. Black; Ryan S. Yeung; Septimiu E. Salcudean

cs.CVMar 11, 2026

Towards Automated Initial Probe Placement in Transthoracic Teleultrasound Using Human Mesh and Skeleton Recovery

Authors: Yu Chung Lee, David G. Black, Ryan S. Yeung, Septimiu E. Salcudean

Organizations: Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, British Columbia, Canada

Abstract

Cardiac and lung ultrasound are technically demanding because operators must identify patient-specific intercostal acoustic windows and then navigate between standard views by adjusting probe position, rotation, and force across different imaging planes. These challenges are amplified in teleultrasound, where the examination proceeds without in-person expert assistance: once the probe is approximately positioned, the expert can navigate in ultrasound image space, but guiding the initial placement remotely remains difficult given the limited 3D perception of the patient. We present a framework for automating patient registration and anatomy-informed initial probe placement guidance (PIPG) using RGB images obtained from a calibrated camera and a point cloud accumulated from depth images. The novice first captures the patient using the camera on a mixed reality (MR) head-mounted display (HMD), and an edge server then infers a patient-specific body-surface and skeleton model. By leveraging the patient's spatial and temporal consistency across multiview and point cloud data, we achieved robust, training-free human registration, verified in a healthcare setting. Using bony landmarks from the predicted skeleton, we estimate the intercostal region and project the guidance back onto the reconstructed body surface. To validate the framework, we rendered the reconstructed body mesh and the virtual probe pose guidance in the MR headset across multiple transthoracic echocardiography scan planes in situ and measured the quantitative placement error. Pilot experiments with five healthy volunteers suggest that the proposed probe placement prediction and MR guidance yield consistent initial placement, with a mean surface error of 15 mm, positional error to palpated anatomical landmarks of 41.0 mm, and torso orientation errors within 9 deg, acceptable for the teleultrasound setup.

Figures & tables

Figure 1: Teleultrasound’s patient-side and remote console with bidirectional streaming and MR guidance overlays via the WebRTC protocol. The remote console end could monitor the examination from the headset’s point-of-view camera and ultrasound video.

Figure 2: Body-part point cloud segmentation via majority-vote label transfer from nearby MHR mesh vertices. Left column: left lateral decubitus; right column: supine. From top to bottom rows: (a) RGB image captured by MR headset, (b) recorded point cloud with body part segmentation, (c) segmented point cloud $\mathcal{P}_{\text{bparts}}$ with orphan vertices $\mathcal{V}_{\text{orphans}}$ , (d) back view of the combined points.

Figure 3: Overall framework of the proposed Patient Registration and Initial Probe Placement Guidance (PIPG) pipeline. Given a batch of $K$ RGB images and an associated point cloud, the system exploits the patient’s spatial and temporal consistency to estimate the patient-specific anatomy and register it to the system coordinate frame for real-time guidance generation. The first processing block illustrates the RGB image stream, from which MHR predictions are obtained using SAM 3D Body. These predictions are subsequently registered to the point cloud via Iterative Closest Point (ICP). The second processing block depicts the reconstruction of the SMPL-X model from the segmented point cloud, partitioned by anatomical regions as detailed in Section 2 . The recovered SMPL-X model is then employed to compute the initial probe placement guidance, while both are rendered in the MR headset.

Panel of Figure 4 — Figure 4: (a) First-person perspective from the MR headset during probe tracking and guidance. (b) The predicted probe pose is visualized in green, the guided pose in blue, and the ground-truth pose in red, corresponding to the upper and lower sternum and the point of maximal impulse (PMI). (c) The PMI is initially localized manually with the patient in the supine position and iteratively adjusted until the relevant anatomical structures can be approximately identified. Note that additional fine-tuning is typically required to achieve the final optimal imaging view.

Metrics	${e}_{depth}$	${e}_{anatomy}$	${e}_{utility}$
Supine position $e_{\text{pos}}$	$14.4\pm 10.6$	$42.5\pm 13.0$	$43.9\pm 15.1$
Left lateral decubitus $e_{\text{pos}}$	$15.8\pm 8.0$	$39.0\pm 15.3$	$36.1\pm 19.3$
Subject-level mean $e_{\text{pos}}$	$15.1\pm 3.1$	$41.1\pm 9.1$	$40.3\pm 11.4$
All positional errors $e_{\text{pos}}$	$15.0\pm 9.5$	$41.0\pm 14.1$	$40.7\pm 17.4$
All Tilt errors $e_{\text{tilt}}$	–	$8.79\pm 6.20$	–
All Spin errors $e_{\text{spin}}$	–	$5.65\pm 4.19$	–

Table 1: Error distribution reported as Mean

\pm

Standard Deviation. Units:

e_{\text{pos}}

in mm,

e_{\text{orient}}

in degee

Panel of Figure 5 — Figure 5: (a) Error distribution per subject, (b) Error distribution of all probe poses. Detailed statistics are summarized in Table 1 .

Metrics ${e}_{pos}$	Depth & surface	Anatomy
RGB only (No ICP & $\mathcal{P}_{\text{bparts}}$ )	36.0 $\pm$ 17.6	51.0 $\pm$ 22.0
SAM3DB + ICP (No $\mathcal{P}_{\text{bparts}}$ )	22.1 $\pm$ 9.1	43.8 $\pm$ 16.3
Full (SAM3D + ICP + $\mathcal{P}_{\text{bparts}}$ )	15.0 $\pm$ 9.5	41.0 $\pm$ 14.1

Table 2: Ablation study of the registration pipeline. Error distribution reported as Mean

\pm

Standard Deviation.

Explore similar work

Seeing Globally, Refining Locally: Global Visual Guidance and Local Ultrasound Cues for Robust Freehand 3-D Ultrasound Reconstruction
Jul 14, 2026Yameng Zhang, Zhongyu Chen, Dianye Huang +3Object Pose Estimation Ultrasound Imaging
Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China
Echo-POSED: Geometric Self-Distillation for Echocardiography Guidance
May 30, 2026Elias Stenhede, Edvart Grüner Bjerke, Joanna Sulkowska +4Self-Supervised Learning Echocardiography
Medical Technology & E-Health, Akershus University Hospital, Norway · Department of Technology Systems, University of Oslo, Norway · K.G. Jebsen Center for Cardiac Biomarkers, University of Oslo, Norway +3
MLRecon: Robust Markerless Freehand 3D Ultrasound Reconstruction via Coarse-to-Fine Pose Estimation
Mar 1, 2026Yi Zhang, Puxun Tu, Kun Wang +3Ultrasound Imaging Object Pose Estimation
Institute of Biomedical Manufacturing and Life Quality Engineering, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China · Department of Ultrasound in Medicine, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, China · Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China

Jul 14, 2026J/K move · Enter open · S save

Seeing Globally, Refining Locally: Global Visual Guidance and Local Ultrasound Cues for Robust Freehand 3-D Ultrasound Reconstruction

Yameng Zhang, Zhongyu Chen, Dianye Huang, Xiangyu Chu+2

Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China

Object Pose Estimation Ultrasound Imaging Multimodal Sensor Fusion

Open paper AskPDF arXiv

Freehand 3-D ultrasound (US) imaging has attracted increasing attention owing to its intuitive volumetric visualization, ease of use, and low cost. However, accurate 3-D reconstruction critically depends on stable probe pose estimation, yet existing trackerless methods remain susceptible to accumulated pose errors, particularly over long scanning trajectories. To address this limitation, we propose a global-to-local pose estimation framework that exploits external camera observations for globally stable localization and B-mode US images for anatomy-aware local refinement. Specifically, the framework comprises a dual-camera branch that performs contextual feature aggregation across camera views and temporal observations to estimate a globally consistent probe trajectory, and a B-mode branch that performs anatomical feature aggregation from sequential US images to capture tissue-dependent local motion cues. A cross-modal fusion module subsequently integrates the contextual camera features and anatomical US features to predict pose residuals and refine the camera-derived estimates in the transformation space. Furthermore, a multi-scale pose loss constrains relative motion over multiple temporal horizons to suppress accumulated drift during extended scans. The proposed framework is validated on phantom and in vivo datasets. On two in-house datasets (FUSION-J and FUSION-L) collected using different machines, the proposed US + Dual-Cam model reduces average trajectory drift to 1.67 mm and 1.29 mm, representing improvement of 16.50% and 27.12%, respectively, over a strong dual-camera baseline, while substantially outperforming US-only pose estimation (>13 mm drift). In in vivo forearm arteries reconstruction, it achieves Hausdorff distances of 1.58 mm, demonstrating the effectiveness of the proposed method on real clinical scenarios.

Abstract

Explore similar work

Seeing Globally, Refining Locally: Global Visual Guidance and Local Ultrasound Cues for Robust Freehand 3-D Ultrasound Reconstruction

Echo-POSED: Geometric Self-Distillation for Echocardiography Guidance

MLRecon: Robust Markerless Freehand 3D Ultrasound Reconstruction via Coarse-to-Fine Pose Estimation