UltraWorld: Learning Interactive Ultrasound World Models from Untracked Clinical Videos with Acoustic Sampling Map

Keke Yang; Erqi Wang; Sainan Guan; Hongliang Ren

cs.CVOct 7, 2026

UltraWorld: Learning Interactive Ultrasound World Models from Untracked Clinical Videos with Acoustic Sampling Map

Authors: Keke Yang, Erqi Wang, Sainan Guan, Hongliang Ren

Organizations: The Chinese University of Hong Kong · The Eighth Affiliated Hospital, Sun Yat-sen University

Abstract

World models can enable autonomous ultrasound scanning by predicting the outcomes of probe motions from local observations. Learning this action--observation relationship typically relies on synchronized video--pose pairs, which are costly to collect at scale and largely unavailable in routine clinical recordings. Reliable action following further requires modeling ultrasound's cross-sectional sampling geometry. We present UltraWorld, a self-distillation recipe that transfers priors from clinical ultrasound videos into interactive world models without real action annotations. Starting from clinical videos, we adapt a video foundation model into an ultrasound generator conditioned on reference images and anatomical masks. Anatomical masks sampled along programmable trajectories through 3D anatomy provide spatial guidance for synthesizing action--video pairs. We then use these synthetic pairs to self-distill the generator into a world model that predicts future observations from local observations and actions, without requiring anatomical masks or other 3D assets at inference time. To further improve action following, we introduce the Acoustic Sampling Map (AsMap), which represents probe poses and imaging settings as pixel-wise 3D sampling positions, beam directions, and depths. Experiments demonstrate improved prediction fidelity and action following. Across nine simulated closed-loop local planning episodes, UltraWorld reduces the mean final distance to the goal and orientation error by 29% and 38%, respectively, compared with visual servoing. Project Page: https://ultraworld-project.github.io/.

Figures & tables

Figure 1: Learning ultrasound world models from untracked clinical videos. (a) Clinical videos provide appearance priors, while 3D anatomy provides motion supervision. Together, they yield synthetic action–video pairs for self-distillation. (b) Complex scanning operations change which anatomical cross-sections are imaged and how they are sampled. AsMap encodes the resulting sampling positions, beam directions, and depths to support reliable action control.

Figure 2: Overall framework of UltraWorld. (a) A shared video backbone supports mask-conditioned generation and action-conditioned world modeling through AsMap. (b) 3D anatomy and programmable trajectories provide synthetic action–video supervision, transferring the learned video prior to the world model through self-distillation and flow matching. (c) At inference, the world model predicts future ultrasound video from the current observation and prescribed acquisition trajectory.

Figure 3: Acoustic Sampling Map and geometry conditioning. (a) Probe pose and imaging parameters determine each pixel’s physical sampling position $\mathbf{q}$ , beam direction $\mathbf{b}$ , and sampling depth $\rho$ . Positions and directions are expressed in the first-frame probe coordinate system, forming a seven-channel AsMap. (b) A geometry encoder $\Phi$ aligns the AsMap sequence with the video tokens, providing pixel-aligned conditioning before the first transformer block.

Figure 4: Self-distillation for ultrasound world modeling. The transformation consists of three steps: (1) Prior weight transfer : initialize the world model from the learned video generator; (2) Condition replacement : replace anatomical-mask conditioning with AsMap conditioning while retaining the observation input; (3) Two-stage adaptation : train the geometry encoder with a frozen video backbone, then jointly fine-tune both. Both stages use synthesized action–video trajectories.

Figure 5: Mask-conditioned ultrasound video generation. Generated sequences follow the prescribed anatomy while preserving realistic ultrasound appearance.

Conditioning	Prediction fidelity			Action following
Conditioning	a PSNR $\uparrow$	LPIPS $\downarrow$	Latent $\ell_{2}\downarrow$	$\Delta$ -LPIPS $\downarrow$	Motion err. $\downarrow$
Fourier Enc. ( Fan et al., 2026 )	22.946	0.274	0.120	0.509	4.540
MLP Enc. ( Yue et al., 2025 )	23.031	0.262	0.121	0.486	4.471
PRoPE ( Li et al., 2025 )	22.634	0.275	0.127	0.504	4.457
AsMap (Ours)	24.994	0.236	0.095	0.466	4.155

Table 1: Comparison of different action-conditioning representations. AsMap consistently improves both prediction fidelity and action following over global action encodings and projective geometric conditioning. Motion error is measured in pixels.

Figure 6: Qualitative comparison across scanning operations. UltraWorld follows prescribed operations more consistently. White arrows highlight differences in action following; yellow insets highlight missing or unstable hypoechoic regions and anatomical features in baseline predictions.

Variant	Prediction fidelity			Action following
Variant	PSNR $\uparrow$	LPIPS $\downarrow$	Latent $\ell_{2}\downarrow$	$\Delta$ -LPIPS $\downarrow$	Motion err. $\downarrow$
Matched MLP conditioning	23.402	0.258	0.113	0.482	4.429
Wan2.1 initialization	24.640	0.235	0.099	0.472	4.414
Full model (Ours)	24.994	0.236	0.095	0.466	4.155

Table 2: Ablation studies on action encoding and initialization. “Matched” approximately matches the AsMap encoder parameter count and uses the same first-frame-relative coordinates. For each ablation, the remaining components follow the full model configuration.

Figure 8: Closed-loop robotic ultrasound planning with UltraWorld. A representative episode showing imagined rollouts, executed observations, and the target view. Only the first optimized action is executed before replanning from the new observation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Encoder	Architecture	Parameters
Fourier Enc.	Fourier features + MLP	1.11M
MLP Enc.	MLP	0.79M
MLP Enc. (Matched)	Matched-size MLP	53.49M
PRoPE	30-layer Linear	70.82M
AsMap (Ours)	Convolution + ResBlock	53.48M

Appendix

Table 4: Architectures of token-level action encoders.

Figure 9: Qualitative comparison under revisiting. UltraWorld and alternative conditioning methods predict future ultrasound frames from the same initial observation and prescribed trajectory. Comparisons with reference futures highlight differences in action following and the continuity of local tissue appearance.

Figure 10: Additional qualitative rollouts for robotic planning. Additional closed-loop examples showing UltraWorld rollouts and the corresponding simulator executions, together with the probe-pose error to the goal during planning.

Figure 11: Action-conditioned rollouts from real clinical ultrasound observations. Each method is initialized from the same real ultrasound image and follows a prescribed acquisition trajectory. The intermediate predictions visualize how the predicted anatomy evolves in response to the prescribed action.

Figure 12: Counterfactual predictions under alternative acquisition trajectories. Starting from the same ultrasound observation, we prescribe sweep, slide, rock, and depth-change trajectories and compare the resulting predictions across action conditioning methods.

Explore similar work

Action-Conditioned World Model for Goal Plane Probe Guidance in Robotic Ultrasound
Jul 24, 2026Siqi Fan, Mingcong Chen, Ran Liu +5Robotic Ultrasound Efficient World-Action Model
Department of Mechanical and Automation Engineering, Chinese University of Hong Kong, Hong Kong SAR · Centre for Artificial Intelligence and Robotics Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong SAR · Department of Biomedical Engineering, City University of Hong Kong, Hong Kong SAR +3
Ultrasound Operator Guidance Using World Modeling and Retrieval Based Action Planning
Oct 5, 2026Noortje I. P. Schueler, Hans van Gorp, Ruud J. G. van SlounGeometric Guidance
Department of Electrical Engineering, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands
DreamReg: Belief-Driven World Model for 2D-3D Ultrasound Registration
Jun 17, 2026Luoyao Kang, Yuelin Zhang, Jiwei Shan +3Image Registration Dream
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong SAR, China · Perelman School of Medicine, University of Pennsylvania, PA, USA · Shun Hing Institute of Advanced Engineering and Multi-scale Medical Robotics Center, Hong Kong SAR, China

Jul 24, 2026, cs.ROJ/K move · Enter open · S save

Action-Conditioned World Model for Goal Plane Probe Guidance in Robotic Ultrasound

Siqi Fan, Mingcong Chen, Ran Liu, Zixuan Yang+4

Department of Mechanical and Automation Engineering, Chinese University of Hong Kong, Hong Kong SAR · Centre for Artificial Intelligence and Robotics Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong SAR · Department of Biomedical Engineering, City University of Hong Kong, Hong Kong SAR+3

Robotic Ultrasound Efficient World-Action Model Scalable Robot Learning Inverse Dynamics Geometric Guidance World

Open paper AskPDF arXiv

We present an action-conditioned world model framework for goal plane probe guidance in robotic ultrasound, with a focus on neck ultrasound scanning. Autonomous ultrasound tasks often require large numbers of probe-motion trajectories for training, but collecting high-quality demonstrations is labor-intensive and explicit simulators are difficult to build because ultrasound appearance depends on contact, tissue deformation, and view-dependent acoustic artifacts. We address this problem with a two-stage model-based learning pipeline. First, a latent conditional diffusion world model predicts future ultrasound observations from recent context frames, probe motions and temporal offset. Second, a goal-conditioned temporal transformer predicts ordered probe motions and is fine-tuned using rewards from the frozen world model. Experiments on the self-collected dataset show that the world model preserves action-dependent anatomical structure on target-directed scans. In real-world closed loop experiments, the framework achieves success rates of 70.0% for carotid guidance and 65.0% for thyroid guidance. These results demonstrate the potential of learned ultrasound dynamics for training goal-directed robotic probe navigation.

Abstract

Explore similar work

Action-Conditioned World Model for Goal Plane Probe Guidance in Robotic Ultrasound

Ultrasound Operator Guidance Using World Modeling and Retrieval Based Action Planning

DreamReg: Belief-Driven World Model for 2D-3D Ultrasound Registration