Ultrasound Operator Guidance Using World Modeling and Retrieval Based Action Planning

Noortje I. P. Schueler; Hans van Gorp; Ruud J. G. van Sloun

cs.LGOct 5, 2026

Ultrasound Operator Guidance Using World Modeling and Retrieval Based Action Planning

Authors: Noortje I. P. Schueler, Hans van Gorp, Ruud J. G. van Sloun

Organizations: Department of Electrical Engineering, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands

Abstract

Ultrasound is widely used, but acquisition quality is heavily dependent on the operator's knowledge and expertise. With demand for examinations outpacing the supply of trained sonographers, operator-guidance systems aim to close this gap by instructing a less trained user how to move the probe toward a target view. In this paper, we propose a retrieval-induced latent transition model for ultrasound acquisition dynamics, formulating ultrasound operator guidance as multi-step planning and retrieval in a world model. Using a V-JEPA 2.1 backbone, observations are first encoded into a latent space where anatomically related views lie close together. We then retrieve similar views from a reference database containing encoded latent states and corresponding probe positions and orientations. Rather than learning a parametric transition function, we directly use physically executed transitions from the database to establish our nonparametric, retrieval-induced transition model that supports receding-horizon planning. At deployment, guidance is generated from the live ultrasound image feed alone, without any probe tracking hardware. Applied to carotid ultrasound, the proposed planner reaches the target view in 86% of retrospective closed-loop episodes, versus 52% and 43% for representative baselines, outperforming both on every target view, including the challenging longitudinal internal and external carotid artery views. A prospective feasibility study on unseen volunteers, run in real time on a CPU using distillation, reaches 83% target-view reachability. Because planning is driven by proximity to any encodable goal latent, the same world model can navigate back to any previously acquired, patient-specific frame, supporting reproducible longitudinal imaging for e.g. perioperative or follow-up monitoring.

Figures & tables

Fig. 1: Navigation in carotid ultrasound is difficult for both human operators and AI guidance models alike, due to high intra-class variation and inter-class similarity between different ultrasound views. This figure illustrates both effects using representative images from different volunteers: intra-class variation, where the same anatomical view (transverse bulb, ICA, or ECA) appears visually different across volunteers, and inter-class similarity, where different anatomical views (e.g., transverse bulb (right image) and transverse CCA) can appear visually similar.

Fig. 2: Visual overview of the ultrasound world model and planning in its latent space. A) Training sequences are encoded into latent states using the learned encoder $f_{\theta}$ , based purely on the visual ultrasound frames; each training frame additionally has a ground-truth recorded pose, used to calculate actions between training frames. B) The live imaging sequence is encoded with the same encoder $f_{\theta}$ . Many candidate trajectories are then randomly sampled by alternating between retrieval hops and within-trajectory hops. Retrieval hops sample the top- $K$ nearest neighbors of the current latent state to create the anchor points $z^{(h)}$ . Within-trajectory hops then follow a training sequence forward or backward in time from anchor $z^{(h)}$ to $\bar{z}^{(h)}$ , yielding a recorded action $a^{(h)}$ . Here, we show one example of a sampled trajectory. From all generated trajectories, the lowest cost trajectory is chosen, and its first action is communicated to the user.

	Linear probe		Top-K
Method	Balanced accuracy	Macro F1	Purity@10
EchoJepa	0.75	0.75	0.65
ImageJepa	0.52	0.51	0.49
V-JEPA 2.1	0.81	0.81	0.69

TABLE I: Latent Space Analysis.

	Translation dynamics		Rotation dynamics
Method	Pearson	Spearman	Pearson	Spearman
EchoJEPA	0.20	0.16	0.30	0.34
ImageJEPA	0.19	0.15	0.31	0.33
V-JEPA 2.1	0.20	0.13	0.41	0.47

TABLE II: Representation dynamics results.

Fig. 3: 2D T-SNE embedding of the latent space of the validation set, colored by both view label and volunteer. Clear clusters appear over views, while the method generalizes neatly across volunteers.

Fig. 4: Success rates across test-set rollouts, separated by navigation initialization from a standard view and a random image.

Fig. 5: Fraction of consecutive nonzero global actions with non-positive cosine similarity, reported separately for translation and rotation, computed over successful episodes with more than two actions, up to the first arrival at the target view. Lower values indicate smoother trajectories with fewer direction reversals.

Fig. 6: Success rate per target view, highlighting easier target views and more challenging ones, with our method having a higher success rate for all views.

Fig. 7: Example executed trajectory of offline planning, starting at the ICA navigating to the cross-sectional bifurcation.

Horizon	1	2	3	4	5	6	7	8
Success rate $\uparrow$	0.76	0.86	0.88	0.88	0.91	0.92	0.92	0.93
Rot backtrack $\downarrow$	0.22	0.29	0.28	0.27	0.28	0.26	0.26	0.27
Trans backtrack $\downarrow$	0.37	0.41	0.38	0.35	0.35	0.35	0.37	0.37

TABLE III: Ablation: horizon sweep (validation set)

Ablation	Success rate $\uparrow$	Rot backtrack $\downarrow$	Trans backtrack $\downarrow$
no $J_{\mathrm{traj}}$	0.89	0.24	0.31
no $J_{\mathrm{mom}}$	0.84	0.22	0.29
no $J_{\mathrm{xstep}}$	0.90	0.34	0.39
Proposed	0.92	0.26	0.35

TABLE IV: Ablation cost function (validation set)

Stepsize	Success rate $\uparrow$	Rot backtrack $\downarrow$	Trans backtrack $\downarrow$
30	0.86	0.31	0.46
80	0.82	0.34	0.44
130	0.85	0.40	0.48
Proposed	0.92	0.26	0.35

TABLE V: Naive database retrieval (validation set)

Fig. 8: Acquisition results guided by both CardiacCopilot style method (top) and our proposed method (bottom). For views where no image is shown, the target was not reached. The view probabilities p indicate the view classification output for the corresponding target view.

Explore similar work

Jul 24, 2026cs.RO

Action-Conditioned World Model for Goal Plane Probe Guidance in Robotic Ultrasound

We present an action-conditioned world model framework for goal plane probe guidance in robotic ultrasound, with a focus on neck ultrasound scanning. Autonomous ultrasound tasks often require large numbers of probe-motion trajectories for training, but collecting high-quality demonstrations is labor-intensive and explicit simulators are difficult to build because ultrasound appearance depends on contact, tissue deformation, and view-dependent acoustic artifacts. We address this problem with a two-stage model-based learning pipeline. First, a latent conditional diffusion world model predicts future ultrasound observations from recent context frames, probe motions and temporal offset. Second, a goal-conditioned temporal transformer predicts ordered probe motions and is fine-tuned using rewards from the frozen world model. Experiments on the self-collected dataset show that the world model preserves action-dependent anatomical structure on target-directed scans. In real-world closed loop experiments, the framework achieves success rates of 70.0% for carotid guidance and 65.0% for thyroid guidance. These results demonstrate the potential of learned ultrasound dynamics for training goal-directed robotic probe navigation.

Siqi Fan, Mingcong Chen, Ran Liu +5

Department of Mechanical and Automation Engineering, Chinese University of Hong Kong, Hong Kong SAR · Centre for Artificial Intelligence and Robotics Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong SAR · Department of Biomedical Engineering, City University of Hong Kong, Hong Kong SAR +3

Robotic Ultrasound Efficient World-Action Model Scalable Robot Learning Inverse Dynamics Geometric Guidance World

Ask PDF

Oct 7, 2026cs.CV

UltraWorld: Learning Interactive Ultrasound World Models from Untracked Clinical Videos with Acoustic Sampling Map

World models can enable autonomous ultrasound scanning by predicting the outcomes of probe motions from local observations. Learning this action--observation relationship typically relies on synchronized video--pose pairs, which are costly to collect at scale and largely unavailable in routine clinical recordings. Reliable action following further requires modeling ultrasound's cross-sectional sampling geometry. We present UltraWorld, a self-distillation recipe that transfers priors from clinical ultrasound videos into interactive world models without real action annotations. Starting from clinical videos, we adapt a video foundation model into an ultrasound generator conditioned on reference images and anatomical masks. Anatomical masks sampled along programmable trajectories through 3D anatomy provide spatial guidance for synthesizing action--video pairs. We then use these synthetic pairs to self-distill the generator into a world model that predicts future observations from local observations and actions, without requiring anatomical masks or other 3D assets at inference time. To further improve action following, we introduce the Acoustic Sampling Map (AsMap), which represents probe poses and imaging settings as pixel-wise 3D sampling positions, beam directions, and depths. Experiments demonstrate improved prediction fidelity and action following. Across nine simulated closed-loop local planning episodes, UltraWorld reduces the mean final distance to the goal and orientation error by 29% and 38%, respectively, compared with visual servoing. Project Page: https://ultraworld-project.github.io/.

Keke Yang, Erqi Wang, Sainan Guan +1

The Chinese University of Hong Kong · The Eighth Affiliated Hospital, Sun Yat-sen University

Topic pending

Ask PDF

Feb 22, 2026cs.CV

US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound

Ultrasound (US) imaging poses unique challenges for representation learning due to its inherently noisy acquisition process. The low signal-to-noise ratio and stochastic speckle patterns hinder standard self-supervised learning methods relying on a pixel-level reconstruction objective. Joint-Embedding Predictive Architectures (JEPAs) address this drawback by predicting masked latent representations rather than raw pixels. However, standard approaches depend on hyperparameter-brittle and computationally expensive online teachers updated via exponential moving average. We propose US-JEPA, a self-supervised framework that adopts the Static-teacher Asymmetric Latent Training (SALT) objective. By using a frozen, domain-specific teacher to provide stable latent targets, US-JEPA decouples student-teacher optimization and pushes the student to expand upon the semantic priors of the teacher. In addition, we provide the first rigorous comparison of all publicly available state-of-the-art ultrasound foundation models on UltraBench, a public dataset benchmark spanning multiple organs and pathological conditions. Under linear probing for diverse classification tasks, US-JEPA achieves performance competitive with or superior to domain-specific and universal vision foundation model baselines. Our results demonstrate that masked latent prediction provides a stable and efficient path toward robust ultrasound representations.

Ashwath Radhachandran, Vedrana Ivezić, Shreeram Athreya +2

Biomedical Artificial Intelligence Research Lab, University of California, Los Angeles, Los Angeles, United States of America · Medical Informatics, University of California, Los Angeles, Los Angeles, United States of America · Department of Electrical and Computer Engineering, University of California, Los Angeles, Los Angeles, United States of America +3

Ultrasound Joint-Embedding Predictive Architectures S-Jepa

Ask PDF