Towards Automated Initial Probe Placement in Transthoracic Teleultrasound Using Human Mesh and Skeleton Recovery

Yu Chung Lee; David G. Black; Ryan S. Yeung; Septimiu E. Salcudean

cs.CVMar 11, 2026

Towards Automated Initial Probe Placement in Transthoracic Teleultrasound Using Human Mesh and Skeleton Recovery

Authors: Yu Chung Lee, David G. Black, Ryan S. Yeung, Septimiu E. Salcudean

Organizations: Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, British Columbia, Canada

Abstract

Cardiac and lung ultrasound are technically demanding because operators must identify patient-specific intercostal acoustic windows and then navigate between standard views by adjusting probe position, rotation, and force across different imaging planes. These challenges are amplified in teleultrasound, where the examination proceeds without in-person expert assistance: once the probe is approximately positioned, the expert can navigate in ultrasound image space, but guiding the initial placement remotely remains difficult given the limited 3D perception of the patient. We present a framework for automating patient registration and anatomy-informed initial probe placement guidance (PIPG) using RGB images obtained from a calibrated camera and a point cloud accumulated from depth images. The novice first captures the patient using the camera on a mixed reality (MR) head-mounted display (HMD), and an edge server then infers a patient-specific body-surface and skeleton model. By leveraging the patient's spatial and temporal consistency across multiview and point cloud data, we achieved robust, training-free human registration, verified in a healthcare setting. Using bony landmarks from the predicted skeleton, we estimate the intercostal region and project the guidance back onto the reconstructed body surface. To validate the framework, we rendered the reconstructed body mesh and the virtual probe pose guidance in the MR headset across multiple transthoracic echocardiography scan planes in situ and measured the quantitative placement error. Pilot experiments with five healthy volunteers suggest that the proposed probe placement prediction and MR guidance yield consistent initial placement, with a mean surface error of 15 mm, positional error to palpated anatomical landmarks of 41.0 mm, and torso orientation errors within 9 deg, acceptable for the teleultrasound setup.

Figures & tables

Figure 1: Teleultrasound’s patient-side and remote console with bidirectional streaming and MR guidance overlays via the WebRTC protocol. The remote console end could monitor the examination from the headset’s point-of-view camera and ultrasound video.

Figure 2: Body-part point cloud segmentation via majority-vote label transfer from nearby MHR mesh vertices. Left column: left lateral decubitus; right column: supine. From top to bottom rows: (a) RGB image captured by MR headset, (b) recorded point cloud with body part segmentation, (c) segmented point cloud $\mathcal{P}_{\text{bparts}}$ with orphan vertices $\mathcal{V}_{\text{orphans}}$ , (d) back view of the combined points.

Figure 3: Overall framework of the proposed Patient Registration and Initial Probe Placement Guidance (PIPG) pipeline. Given a batch of $K$ RGB images and an associated point cloud, the system exploits the patient’s spatial and temporal consistency to estimate the patient-specific anatomy and register it to the system coordinate frame for real-time guidance generation. The first processing block illustrates the RGB image stream, from which MHR predictions are obtained using SAM 3D Body. These predictions are subsequently registered to the point cloud via Iterative Closest Point (ICP). The second processing block depicts the reconstruction of the SMPL-X model from the segmented point cloud, partitioned by anatomical regions as detailed in Section 2 . The recovered SMPL-X model is then employed to compute the initial probe placement guidance, while both are rendered in the MR headset.

Panel of Figure 4 — Figure 4: (a) First-person perspective from the MR headset during probe tracking and guidance. (b) The predicted probe pose is visualized in green, the guided pose in blue, and the ground-truth pose in red, corresponding to the upper and lower sternum and the point of maximal impulse (PMI). (c) The PMI is initially localized manually with the patient in the supine position and iteratively adjusted until the relevant anatomical structures can be approximately identified. Note that additional fine-tuning is typically required to achieve the final optimal imaging view.

Metrics	${e}_{depth}$	${e}_{anatomy}$	${e}_{utility}$
Supine position $e_{\text{pos}}$	$14.4\pm 10.6$	$42.5\pm 13.0$	$43.9\pm 15.1$
Left lateral decubitus $e_{\text{pos}}$	$15.8\pm 8.0$	$39.0\pm 15.3$	$36.1\pm 19.3$
Subject-level mean $e_{\text{pos}}$	$15.1\pm 3.1$	$41.1\pm 9.1$	$40.3\pm 11.4$
All positional errors $e_{\text{pos}}$	$15.0\pm 9.5$	$41.0\pm 14.1$	$40.7\pm 17.4$
All Tilt errors $e_{\text{tilt}}$	–	$8.79\pm 6.20$	–
All Spin errors $e_{\text{spin}}$	–	$5.65\pm 4.19$	–

Table 1: Error distribution reported as Mean

\pm

Standard Deviation. Units:

e_{\text{pos}}

in mm,

e_{\text{orient}}

in degee

Panel of Figure 5 — Figure 5: (a) Error distribution per subject, (b) Error distribution of all probe poses. Detailed statistics are summarized in Table 1 .

Metrics ${e}_{pos}$	Depth & surface	Anatomy
RGB only (No ICP & $\mathcal{P}_{\text{bparts}}$ )	36.0 $\pm$ 17.6	51.0 $\pm$ 22.0
SAM3DB + ICP (No $\mathcal{P}_{\text{bparts}}$ )	22.1 $\pm$ 9.1	43.8 $\pm$ 16.3
Full (SAM3D + ICP + $\mathcal{P}_{\text{bparts}}$ )	15.0 $\pm$ 9.5	41.0 $\pm$ 14.1

Table 2: Ablation study of the registration pipeline. Error distribution reported as Mean

\pm

Standard Deviation.

Explore similar work

Jul 14, 2026

Seeing Globally, Refining Locally: Global Visual Guidance and Local Ultrasound Cues for Robust Freehand 3-D Ultrasound Reconstruction

Freehand 3-D ultrasound (US) imaging has attracted increasing attention owing to its intuitive volumetric visualization, ease of use, and low cost. However, accurate 3-D reconstruction critically depends on stable probe pose estimation, yet existing trackerless methods remain susceptible to accumulated pose errors, particularly over long scanning trajectories. To address this limitation, we propose a global-to-local pose estimation framework that exploits external camera observations for globally stable localization and B-mode US images for anatomy-aware local refinement. Specifically, the framework comprises a dual-camera branch that performs contextual feature aggregation across camera views and temporal observations to estimate a globally consistent probe trajectory, and a B-mode branch that performs anatomical feature aggregation from sequential US images to capture tissue-dependent local motion cues. A cross-modal fusion module subsequently integrates the contextual camera features and anatomical US features to predict pose residuals and refine the camera-derived estimates in the transformation space. Furthermore, a multi-scale pose loss constrains relative motion over multiple temporal horizons to suppress accumulated drift during extended scans. The proposed framework is validated on phantom and in vivo datasets. On two in-house datasets (FUSION-J and FUSION-L) collected using different machines, the proposed US + Dual-Cam model reduces average trajectory drift to 1.67 mm and 1.29 mm, representing improvement of 16.50% and 27.12%, respectively, over a strong dual-camera baseline, while substantially outperforming US-only pose estimation (>13 mm drift). In in vivo forearm arteries reconstruction, it achieves Hausdorff distances of 1.58 mm, demonstrating the effectiveness of the proposed method on real clinical scenarios.

Yameng Zhang, Zhongyu Chen, Dianye Huang +3

Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China

Object Pose Estimation Ultrasound Imaging Multimodal Sensor Fusion

Ask PDF

May 30, 2026

Echo-POSED: Geometric Self-Distillation for Echocardiography Guidance

We introduce Echo-POSED, a self-supervised framework for real-time transthoracic echocardiography (TTE) guidance that recommends probe adjustments directly from 2D ultrasound images, without the need for expert-labelled views or tracked probe trajectories. Instead, it trains on 2D views sliced from routinely acquired 3D echocardiography volumes, enforcing equivariance to probe motions while remaining invariant to cardiac phase, yielding a pose representation on

\mathrm{SO}(3)\times\mathrm{SO}(3)

. Across a held-out split and public external 3D--TTE datasets (including vendor shift), Echo-POSED maintains geometric consistency under virtual perturbations and enables intra- and inter-patient guidance simulations, achieving a combined mean angular error of 8.2 degrees between the guided and target views in intra-patient simulations with cardiac motion.

Elias Stenhede, Edvart Grüner Bjerke, Joanna Sulkowska +4

Medical Technology & E-Health, Akershus University Hospital, Norway · Department of Technology Systems, University of Oslo, Norway · K.G. Jebsen Center for Cardiac Biomarkers, University of Oslo, Norway +3

Self-Supervised Learning Echocardiography Self-Distillation

Ask PDF

Mar 1, 2026

MLRecon: Robust Markerless Freehand 3D Ultrasound Reconstruction via Coarse-to-Fine Pose Estimation

Freehand 3D ultrasound (US) reconstruction promises volumetric imaging with the flexibility of standard 2D probes, yet existing tracking paradigms face a restrictive trilemma: marker-based systems demand prohibitive costs, inside-out methods require intrusive sensor attachment, and sensorless approaches suffer from severe cumulative drift. To overcome these limitations, we present MLRecon, a robust markerless 3D US reconstruction framework delivering drift-resilient 6D probe pose tracking using a single commodity RGB-D camera. Leveraging the generalization power of vision foundation models, our pipeline enables continuous markerless tracking of the probe, augmented by a vision-guided divergence detector that autonomously monitors tracking integrity and triggers failure recovery to ensure uninterrupted scanning. Crucially, we further propose a dual-stage pose refinement network that explicitly disentangles high-frequency jitter from low-frequency bias, effectively denoising the trajectory while maintaining the kinematic fidelity of operator maneuvers. Experiments demonstrate that MLRecon significantly outperforms competing sensorless and sensor-aided methods, achieving average position errors as low as 0.88 mm on complex trajectories and yielding high-quality 3D reconstructions with sub-millimeter mean surface accuracy. This establishes a new benchmark for low-cost, accessible volumetric US imaging in resource-limited clinical settings.

Yi Zhang, Puxun Tu, Kun Wang +3

Institute of Biomedical Manufacturing and Life Quality Engineering, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China · Department of Ultrasound in Medicine, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, China · Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China

Ultrasound Imaging Object Pose Estimation

Ask PDF