Scaling Ultrasound Volumetric Reconstruction via Mobile Augmented Reality

Kian Wei Ng; Yujia Gao; Deborah Khoo; Ying Zhen Tan; Chengzheng Mao; Haojie Cheng; Andrew Makmur; Kee Yuan Ngiam

cs.CVFeb 17, 2026

Scaling Ultrasound Volumetric Reconstruction via Mobile Augmented Reality

Authors: Kian Wei Ng, Yujia Gao, Deborah Khoo, Ying Zhen Tan, Chengzheng Mao, Haojie Cheng, Andrew Makmur, Kee Yuan Ngiam,

Organizations: National University Health System, Singapore · College of Design and Engineering, National University of Singapore, Singapore

Abstract

Accurate volumetric characterization of lesions is essential for oncologic diagnosis, risk stratification, and treatment planning. While imaging modalities such as Computed Tomography provide high-quality 3D data, 2D ultrasound (2D-US) remains the preferred first-line modality for breast and thyroid imaging due to cost, portability, and safety factors. However, volume estimates derived from 2D-US suffer from high inter-user variability even among experienced clinicians. Existing 3D ultrasound (3D-US) solutions use specialized probes or external tracking hardware, but such configurations increase costs and diminish portability, constraining widespread clinical use. To address these limitations, we present Mobile Augmented Reality Volumetric Ultrasound (MARVUS), a resource-efficient system designed to increase accessibility to accurate and reproducible volumetric assessment. MARVUS is interoperable with conventional ultrasound (US) systems, using a foundation model to enhance cross-specialty generalization while minimizing hardware requirements relative to current 3D-US solutions. In a user study involving experienced clinicians performing measurements on breast phantoms, MARVUS yielded a substantial improvement in volume estimation accuracy (mean difference: 0.469 cm3) with reduced inter-user variability (mean difference: 0.417 cm3). Additionally, we prove that augmented reality (AR) visualizations enhance objective performance metrics and clinician-reported usability. Collectively, our findings suggests that MARVUS can enhance US-based cancer screening, diagnostic workflows, and treatment planning in a scalable, cost-conscious, and resource-efficient manner. Usage video demonstration available (https://youtu.be/m4llYcZpqmM).

Figures & tables

	Ultrasound	Tracking	Augmented	Segmentation
Ref.	Probe	Sensor	Reality Device	Algorithm
[ 24 ]	Standard (2D)	Robotic Arm	-	Specialized
[ 32 ]	Standard (2D)	Depth Camera	-	Specialized
[ 25 ]	Standard (2D)	Stereo Camera	Headset	Manual
[ 10 ]	Specialized (3D)	Stereo Camera	Headset	Specialized
Ours	Standard (2D)	Phone	Phone	Prompt-based

Table 1: System comparison for related 3D-US and Augmented Reality works

Figure 1: The MARVUS workflow. Calibration – US probe with strapped ArUco markers (a) is tracked via a mobile phone (b). Calibration is done via a simple phantom (c), with region-of-interest and keypoints detected from a raw screen capture (d). Reconstruction – A freehand sweep imaging a nodule is done with accompanying augmented reality visualizations (e). Nodule segmentation is obtained in a semi-automated manner (f) and compounded into 3D for volume estimation. Verification – The 3D mesh is verified in augmented reality (g) by comparing the mesh-plane intersection with a live ultrasound feed (green denotes intersection between the mesh and the live US image).

Figure 2: US Calibration Phantom . A single-piece 3D-printed phantom is used for single-frame spatial calibration using US-visible "ledges" (a) with intersections enabling calibration. The "ledges" $(L_{1},L_{2},L_{3})$ are brightest when viewed directly in-plane due to US physics. (b) Visible markers are used for extrinsics calibration (right).

Figure 3: Nodule Reconstruction Steps . (a) Segmentation of tracked US sequence produces a point cloud. (b) Voxelization resamples and fills sparsely sampled locations. (c) A meshing algorithm is applied to smoothen noise-induced highly irregular surfaces.

	Image Acquisition	Volume Estimation	Nodule Assessment
Control	Manual	Ellipsoid Formula	N/A
Recon	Sweep	Automated	Mesh
Recon+AR	Sweep (AR)	Automated	Mesh (AR)

Table 2: User Study: Volume Assessment Methods

Figure 4: Nodule Volume Estimation . Our AR 3D-US setup improves accuracy and reduces operator variability (left) , notably for complex non-spherical nodules (right) .

Tracker	Phantom Type	CR (mm)
Optotrak Certus (NDI) [ 12 ]	Double-N Wire	$0.938\pm 0.118$
Vicra (NDI) [ 30 ]	Arbitrary Wire	$0.896\pm 0.075$
Polaris Spectra (NDI) [ 8 ]	Triple-N Wire	$0.690\pm 0.310$
Mobile Phone (Ours)	3D-printed Ledge	$0.826\pm 0.447$

Table 3: US Calibration: Calibration Reproducibility (CR) against Related Works

Ultrasound Imaging

Ultrasound imaging uses sound waves to visualize anatomy and function, with AI improving image reconstruction, diagnosis, measurement, and scan guidance.

See all 150 papers

MLRecon: Robust Markerless Freehand 3D Ultrasound Reconstruction via Coarse-to-Fine Pose Estimation
Mar 1, 2026Yi Zhang, Puxun Tu, Kun Wang +3Ultrasound Imaging Object Pose Estimation
Institute of Biomedical Manufacturing and Life Quality Engineering, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China · Department of Ultrasound in Medicine, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, China · Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China
US-VLA: An Ultrasound Vision-Language-Action Model for Embodied Abdomina
Aug 17, 2026Cheng Zhang, Xingzheng Wu, Guihao Yan +4Robotic Ultrasound Vision-Language-Action Models
Innovation School of Artificial Intelligence, Hefei University of Technology Hefei, China
Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports
Jul 2, 2026Bingcong Yan, Chunlei Li, Jingliang Hu +3VLM Adaptation Ultrasound Imaging
1MedAI Technology (Wuxi) Co. Ltd. · 2Technical University of Munich

Mar 1, 2026J/K move · Enter open · S save

MLRecon: Robust Markerless Freehand 3D Ultrasound Reconstruction via Coarse-to-Fine Pose Estimation

Yi Zhang, Puxun Tu, Kun Wang, Yulin Yan+2

Institute of Biomedical Manufacturing and Life Quality Engineering, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China · Department of Ultrasound in Medicine, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, China · Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China

Ultrasound Imaging Object Pose Estimation

Open paper AskPDF arXiv

Freehand 3D ultrasound (US) reconstruction promises volumetric imaging with the flexibility of standard 2D probes, yet existing tracking paradigms face a restrictive trilemma: marker-based systems demand prohibitive costs, inside-out methods require intrusive sensor attachment, and sensorless approaches suffer from severe cumulative drift. To overcome these limitations, we present MLRecon, a robust markerless 3D US reconstruction framework delivering drift-resilient 6D probe pose tracking using a single commodity RGB-D camera. Leveraging the generalization power of vision foundation models, our pipeline enables continuous markerless tracking of the probe, augmented by a vision-guided divergence detector that autonomously monitors tracking integrity and triggers failure recovery to ensure uninterrupted scanning. Crucially, we further propose a dual-stage pose refinement network that explicitly disentangles high-frequency jitter from low-frequency bias, effectively denoising the trajectory while maintaining the kinematic fidelity of operator maneuvers. Experiments demonstrate that MLRecon significantly outperforms competing sensorless and sensor-aided methods, achieving average position errors as low as 0.88 mm on complex trajectories and yielding high-quality 3D reconstructions with sub-millimeter mean surface accuracy. This establishes a new benchmark for low-cost, accessible volumetric US imaging in resource-limited clinical settings.

Abstract

Ultrasound Imaging

Explore similar work

MLRecon: Robust Markerless Freehand 3D Ultrasound Reconstruction via Coarse-to-Fine Pose Estimation

US-VLA: An Ultrasound Vision-Language-Action Model for Embodied Abdomina

Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports