Robotic ultrasound can standardize image acquisition and reduce operator workload and variability, but policy learning remains limited by the lack of realistic, controllable simulation environments. While controllable world models have advanced rapidly in natural-image domains, ultrasound world modeling remains challenging because each frame captures only a cross-section of unobserved 3D anatomy, causing small probe motions to induce large appearance changes. Existing approaches rely on robot-tracked poses and latent-space prediction, but do not directly generate the observations required for interactive control. We propose SonoWorld, a causal ultrasound world model that learns video dynamics from unlabeled clinical videos and grounds them to 6-DoF control using a small set of freehand clinical scans tracked with an EM sensor. SonoWorld is trained in three stages: pretraining a latent-action-conditioned video diffusion model on around 6.9 million unlabeled multi-organ ultrasound frames; grounding the learned video dynamics to metric probe poses with a small EM-tracked thyroid dataset; and applying distribution matching distillation to enable causal, long-horizon autoregressive generation with selectively caching eviction tokens at inference time. We evaluate controllability through pose-following fidelity, round-trip consistency, and axis-specific control under synthetic actions on held-out tracked scans, and further validate it with a user study. In model-based action discrimination, SonoWorld correctly ranks the target-directed probe motion over an equal-magnitude opposite motion in up to 74.8% of trials. To our knowledge, SonoWorld is the first causal, pose-controllable ultrasound world model learned primarily from freehand clinical videos, providing a data-driven interactive environment for robotic ultrasound data acquisition and policy evaluation.
Figures & tables
Figure 1: Starting from the same initial ultrasound frame, SonoWorld generates distinct sequences of future ultrasound views along different 6-DoF probe scan paths, which captures how probe motion determines the evolving image appearance and ultimately the reconstructed 3D thyroid anatomy.
Figure 2: Three stages of ultrasound world model development. (a) Pretraining. A bidirectional DiT is trained on large-scale unlabeled multi-organ ultrasound videos, conditioned on the diffusion timestep t and latent actions z via adaptive layer normalization. (b) Post-training. The model is then post-trained on high-quality thyroid ultrasound videos paired with EM tracker poses, replacing the latent action with the physical action. (c) Causal transform. The bidirectional model is finally distilled into a causal DiT via distribution matching distillation (DMD) Yin et al. (2024) .
Figure 3: EM-tracked pose preprocessing pipeline (a) Raw EM sensor poses are expressed in world frame and are affected by the lever arm between the sensor and probe tip. (b) A fixed sensor-to-tip calibration transforms the measurements to probe-tip poses, such that the resulting motion corresponds to that of the imaging plane. (c) Consecutive probe-tip poses are converted to relative motions expressed in the preceding probe frame, yielding 6-DoF actions used to condition SonoWorld.
Models
Tasks
PSNR ↑
SSIM ↑
LPIPS ↓
Bidirectional
Original pose
16.84 ± 1.17
0.402 ± 0.060
0.413 ± 0.056
Zero motion
16.87 ± 1.12
0.385 ± 0.066
0.392 ± 0.076
Speed × 2
16.89 ± 0.98
0.414 ± 0.053
0.411 ± 0.051
Round trip
16.41 ± 1.09
0.376 ± 0.059
0.431 ± 0.054
Causal
Original pose
16.09 ± 0.95
0.386 ± 0.054
0.439 ± 0.044
Zero motion
17.76 ± 1.13
0.409 ± 0.062
0.390 ± 0.065
Table 1: Quantitative controllability evaluation on EM-tracked scans. Ground-truth sequence is generated from original sequence according to the specific task. For zero motion it is the first image.
Figure 4: Visualizations on video generation of our ultrasound world model. We utilize the causal model to autoregressively generate future frames conditioned on the same initial ultrasound frame (t=0s) and different probe trajectories.
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
w/o latent actions
17.78 ± 2.59
0.613 ± 0.110
0.266 ± 0.087
w/ latent actions
21.38 ± 2.80
0.696 ± 0.108
0.199 ± 0.072
Table 2: Effect of latent actions on ultrasound video diffusion model pretraining.
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
Bidirectional Model
16.52 ± 0.93
0.389 ± 0.061
0.422 ± 0.051
Teacher Forcing ( Gao et al., 2024a )
16.44 ± 0.76
0.387 ± 0.044
0.515 ± 0.055
Diffusion Forcing ( Chen et al., 2024 )
16.22 ± 0.81
0.373 ± 0.039
0.525 ± 0.046
Rolling Forcing ( Liu et al., 2026 )
15.77 ± 0.90
0.372 ± 0.055
0.456 ± 0.047
Reward Forcing ( Lu et al., 2026 )
15.60 ± 0.81
0.371 ± 0.051
0.469 ± 0.041
SonoWorld (Ours)
16.37 ± 0.98
0.394 ± 0.052
0.436 ± 0.047
Table 3: Comparison of causal model training methods on 8 second rollouts. For each metric, the best result among causal methods is shown in boldface and the second best is underlined.
Cache Config.
PSNR ↑
SSIM ↑
LPIPS ↓
Full KV
16.23 ± 0.93
0.390 ± 0.049
0.437 ± 0.046
Local
16.26 ± 0.86
0.387 ± 0.049
0.440 ± 0.045
Sink+Local
16.31 ± 0.96
0.391 ± 0.051
0.436 ± 0.048
Eviction (strided)+Local
16.27 ± 0.94
0.391 ± 0.051
0.437 ± 0.047
SonoWorld (Ours)
16.37 ± 0.98
0.394 ± 0.052
0.436 ± 0.047
Table 4: Results on KV-cache configurations at inference time. We conducted causal rollouts of 8 second ultrasound videos conditioned on an initial frame and 6 DoF poses.
Image quality
Video consistency
Probe–anatomy alignment
(a) Probe scan path varies; conditioning frame held fixed
Path A
3.15± 0.19
3.83± 0.26
4.08± 0.35
Path B
3.35± 0.23
3.70± 0.36
3.73± 0.40
Path C
3.60± 0.25
3.80± 0.13
3.92± 0.18
(b) Conditioning frame shifted laterally; path held fixed
δx=−2 (left lobe)
3.58± 0.23
4.12± 0.12
4.35± 0.21
Table 5: Human evaluation of pose-conditioned rollouts. Five experienced ultrasound raters scored 55 generated clips on three 5-point scales (1 = worst, 3=acceptable, 5 = perfect): image quality, video consistency, and probe–anatomy alignment. Values are mean ± 95% CI over conditioning frames, after averaging raters within each clip.
Threshold
x
y
rx
ry
rz
Avg.
15
73.91
78.26
61.54
86.96
66.67
74.76
20
88.89
77.78
66.67
77.78
44.44
66.67
Table 6: Results on model-based action discrimination. Entries are success rates (%), the fraction of rollouts in which the action reconstructs the goal frame more accurately than its negation;
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Network architecture of the latent action model.
Hyperparameter
Value
Frame resolution / patch size p
256×256 / 16
Patch tokens per frame / token dim.
256 / 768
Width / attention heads
1024 / 16 (head dim. 64 )
Encoder / decoder blocks
24 / 24
Positional encoding
sinusoidal (spatial), RoPE (temporal)
Latent action dim. dz
32
Appendix
Table 7: Hyperparameters of the latent action model.
Augmentation
Prob.
Index map
Effect on the action
Identity
0.25
[s,s+45)
unchanged
Reverse
0.25
[s+44,s] descending
motion played backward
Stride
0.25
s+σ[0,…,44] , σ∈{2,3}
composed, faster motion
Stop-motion
0.25
monotonic non-decreasing with repeats
exact zero-motion steps
Appendix
Table 8: Online temporal augmentation used in post-training. Each case is an index map into the source segment; the action is recomputed from the indexed pose rather than edited.
Setting
Pretraining
Post-training
Training clips
153,892
262
Batch (per GPU × GPUs × accum.)
64×4×3
8×4×3
Effective batch
768 clips
96 clips
Training steps
7,236
18,000
Prefix
k=3 , 9 generated
k=1 , 11 generated
Optimizer
AdamW, β=(0.9,0.999) , weight decay 0.01
Appendix
Table 9: Optimization settings. Both stages train the full network with identical optimizer, schedule and precision; the effective batch follows the corpus size.
Models
Time
Peak Memory
PSNR ↑ (dB)
SSIM ↑
LPIPS ↓
Wan2.1-T2V-1.3B
0.71 clips/s
2.22 GB
36.50
0.900
0.033
Wan2.2-TI2V-5B-Diffusers
0.68 clips/s
3.71 GB
36.24
0.922
0.035
HunyuanVideo-1.5
0.35 clips/s
18.85 GB
36.24
0.921
0.062
LongCat-Video
0.68 clips/s
2.36 GB
36.62
0.916
0.033
Appendix
Table 10: Comparison of open-source video autoencoders for ultrasound video reconstruction.
Figure 6: Latent action model predictions across organs. Rows correspond to thyroid, cardiac, and abdominal scans. Given only the current frame and the 32-dimensional latent action, the decoder predicts the next frame; the rightmost two columns show the absolute difference of the predicted and of the ground-truth next frame from the current frame.
w/o latent actions
w/ latent actions
Organ
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Abdomen
16.74
0.618
0.300
23.16
0.740
0.180
Cardiac
17.45
0.660
0.259
22.02
0.763
0.156
Thyroid
14.99
0.378
0.429
18.20
0.496
0.290
Average
16.39
0.552
0.330
21.13
0.666
0.209
Appendix
Table 11: Effect of latent actions on video generation across organs. We evaluate 8-second video generation conditioned on 9 initial frames (3 latent frames) on abdominal, cardiac and thyroid scans.
Figure 7: Generation conditioned on latent actions. Rows from top to bottom are thyroid, cardiac, and abdominal scans. The model receives the first 9 ultrasound frames and the latent action sequence, then generates the remaining ultrasound frames. Generated frames follow the ground-truth anatomy as the probe moves across all three anatomies, showing that latent-action conditioning consistently captures useful transition dynamics across all three anatomical regions.
Pretraining
Tasks
PSNR ↑
SSIM ↑
LPIPS ↓
w/ latent actions
Original pose
16.09 ± 0.95
0.386 ± 0.054
0.439 ± 0.044
Zero motion
17.76 ± 1.13
0.409 ± 0.062
0.390 ± 0.065
Speed × 2
16.10 ± 1.05
0.388 ± 0.051
0.445 ± 0.048
Round trip
15.17 ± 0.83
0.349 ± 0.044
0.465 ± 0.041
Long (8 s)
16.37 ± 0.98
0.394 ± 0.052
0.436 ± 0.047
w/o latent actions
Original pose
15.69 ± 1.12
0.375 ± 0.058
0.447 ± 0.054
Appendix
Table 12: Effect of Latent-Action Pretraining on Pose-Conditioned Models. Causal inference results under different probe trajectories.
Figure 8: Eight-second rollouts from the causal model on three held-out scans. Each row is one scan: the model receives the initial frame and the measured 6-DoF pose sequence, then generates remaining frames autoregressively in four denoising steps. The right column shows the corresponding probe trajectory in the patient frame.
Figure 9: Generation under customized actions. Starting from the same initial frame ( t=0 s), SonoWorld autoregressively generates 3-second rollouts, each under a constant increment ( Δx , Δy , Δrx , Δry , or Δrz ) in a single degree of freedom. We set the constant increment to 1 mm in translation and 1° in rotation. The x , y , and z axes correspond to the inferior–superior, left–right, and posterior–anterior directions, respectively.
Figure 10: User study interface. Users are required to evaluate ultrasound video generated by actions according to three metrics and provide the evaluation score.
World models can enable autonomous ultrasound scanning by predicting the outcomes of probe motions from local observations. Learning this action--observation relationship typically relies on synchronized video--pose pairs, which are costly to collect at scale and largely unavailable in routine clinical recordings. Reliable action following further requires modeling ultrasound's cross-sectional sampling geometry. We present UltraWorld, a self-distillation recipe that transfers priors from clinical ultrasound videos into interactive world models without real action annotations. Starting from clinical videos, we adapt a video foundation model into an ultrasound generator conditioned on reference images and anatomical masks. Anatomical masks sampled along programmable trajectories through 3D anatomy provide spatial guidance for synthesizing action--video pairs. We then use these synthetic pairs to self-distill the generator into a world model that predicts future observations from local observations and actions, without requiring anatomical masks or other 3D assets at inference time. To further improve action following, we introduce the Acoustic Sampling Map (AsMap), which represents probe poses and imaging settings as pixel-wise 3D sampling positions, beam directions, and depths. Experiments demonstrate improved prediction fidelity and action following. Across nine simulated closed-loop local planning episodes, UltraWorld reduces the mean final distance to the goal and orientation error by 29% and 38%, respectively, compared with visual servoing. Project Page: https://ultraworld-project.github.io/.
Keke Yang, Erqi Wang, Sainan Guan +1
The Chinese University of Hong Kong · The Eighth Affiliated Hospital, Sun Yat-sen University
Ultrasound examinations typically follow specific scanning protocols that require navigating through an ordered sequence of anatomical landmarks and target imaging planes. We present a local action-conditioned world model framework for goal plane probe guidance in robotic ultrasound. Autonomous ultrasound tasks often require large numbers of probe-motion trajectories for training, but collecting high-quality demonstrations is labor-intensive and explicit simulators are difficult to build because ultrasound appearance depends on contact, tissue deformation, and view-dependent acoustic artifacts. We address this problem with a two-stage model-based learning pipeline. First, a latent conditional diffusion world model predicts future ultrasound observations from recent context frames, probe motions and temporal offset. Second, a goal-conditioned temporal transformer predicts ordered probe motions and is fine-tuned using rewards from the frozen world model. Experiments on the self-collected dataset show that the world model preserves action-dependent anatomical structure on target-directed scans. In real-world closed loop experiments, the framework achieves success rates of 70.0% for carotid guidance and 65.0% for thyroid guidance. These results demonstrate the potential of learned ultrasound dynamics for training goal-directed robotic probe navigation within protocol-constrained organ specific tasks.
Siqi Fan, Ran Liu, Zixuan Yang +6
Department of Mechanical and Automation Engineering, Chinese University of Hong Kong, Hong Kong SAR · Centre for Artificial Intelligence and Robotics Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong SAR · Department of Vascular Ultrasound, Xuanwu Hospital, the Capital Medical University +3
Ultrasound is widely used, but acquisition quality is heavily dependent on the operator's knowledge and expertise. With demand for examinations outpacing the supply of trained sonographers, operator-guidance systems aim to close this gap by instructing a less trained user how to move the probe toward a target view. In this paper, we propose a retrieval-induced latent transition model for ultrasound acquisition dynamics, formulating ultrasound operator guidance as multi-step planning and retrieval in a world model. Using a V-JEPA 2.1 backbone, observations are first encoded into a latent space where anatomically related views lie close together. We then retrieve similar views from a reference database containing encoded latent states and corresponding probe positions and orientations. Rather than learning a parametric transition function, we directly use physically executed transitions from the database to establish our nonparametric, retrieval-induced transition model that supports receding-horizon planning. At deployment, guidance is generated from the live ultrasound image feed alone, without any probe tracking hardware. Applied to carotid ultrasound, the proposed planner reaches the target view in 86% of retrospective closed-loop episodes, versus 52% and 43% for representative baselines, outperforming both on every target view, including the challenging longitudinal internal and external carotid artery views. A prospective feasibility study on unseen volunteers, run in real time on a CPU using distillation, reaches 83% target-view reachability. Because planning is driven by proximity to any encodable goal latent, the same world model can navigate back to any previously acquired, patient-specific frame, supporting reproducible longitudinal imaging for e.g. perioperative or follow-up monitoring.
Noortje I. P. Schueler, Hans van Gorp, Ruud J. G. van Sloun
Department of Electrical Engineering, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands