In medical imaging, it is common to use learned perceptual image patch similarity (LPIPS) to compare images semantically in feature space. Although backbones pretrained on natural images are widely used for LPIPS computation, B-mode ultrasound images possess distinct speckle patterns and acoustic-specific image statistics that are fundamentally different from natural images and even from other images in radiology. Consequently, we propose that domain-specific models are needed to measure perceptual similarity in ultrasound data, a finding which is not necessarily the case for other imaging modalities. We compare LPIPS metrics across downstream tasks like classification, segmentation and reconstruction using natural image, medical generalist and ultrasound backbone models and show that selection of LPIPS backbone is a non-trivial design choice. In particular, the ultrasound backbone models were more correlated with downstream performance of supervised models than classical and natural image models, and optimization of the LPIPS loss with an ultrasound backbone achieved a strong balance between reconstruction quality and realism. Our code is available at https://github.com/talg2324/UltraPIPS and introduces the UltraPIPS library, a set of LPIPS metrics based on the open-source foundation models analyzed in this paper.
Figures & tables
Natural Image
Radiology
Ultrasound
AlexNet
RadImageNet
USFM
VGG-16
MedSAM
TUSA
ViT
BiomedCLIP
Ultrasound-CLIP
SwinViT
CLIP
Table 1: Candidate LPIPS backbone models grouped by pre-training domain.
Figure 1: Spearman correlations of supervised models on CAMUS and EchoNet datasets under ultrasound augmentation. In the view classification task (a) correlation is calculated between the log-softmax confidence score of EchoPrime’s view classifier and each distance metric, averaged across augmentations and frames. For segmentation (b), the correlation is calculated between UltraSam’s dice score and and each of the distance metrics, averaged across augmentations and frames
Null Hypothesis
Classification
Segmentation
ρUS≤ρClassical
7.8×10−3
7.8×10−3
ρUS≤ρNatural Images
0.016
0.039
ρUS≤ρRadiology
0.078
0.078
Table 2: Wilcoxon test of LPIPS computed with ultrasound models vs. comparison groups.
Figure 2: Implicit Neural Representation (INR) visual comparison on the follicles dataset. Models are grouped and color-coded by their pre-training domain: Classical (dark blue), Natural Images (light blue), Radiology (orange), and Ultrasound (green).
USOVA
MicroSegNet
L2
Natural Images
Radiology
Ultrasound
L2
Natural Images
Radiology
Ultrasound
↑ SSIM
0.76 ± 0.09
0.45 ± 0.17
0.47 ± 0.13
0.70 ± 0.13
0.65 ± 0.04
0.36 ± 0.13
0.39 ± 0.09
0.55 ± 0.10
↓ HFEN
0.34 ± 0.04
1.12 ± 0.45
1.06 ± 0.27
0.46 ± 0.15
0.37 ± 0.04
1.41 ± 0.42
1.36 ± 0.25
0.71 ± 0.29
↓ GLCM Contrast
2.84 ± 1.92
1.81 ± 1.43
1.79 ± 1.37
1.96 ± 1.89
5.42 ± 1.84
3.61 ± 2.07
3.87 ± 1.83
3.24 ± 2.86
↓ GLCM Dissimilarity
0.45 ± 0.26
0.24 ± 0.18
0.23 ± 0.16
0.29 ± 0.26
0.62 ± 0.21
0.28 ± 0.18
0.36 ± 0.19
0.33 ± 0.32
↓ GLCM Homogeneity
0.07 ± 0.04
0.04 ± 0.04
0.03 ± 0.02
0.04 ± 0.04
0.06 ± 0.03
0.04 ± 0.03
0.03 ± 0.02
0.03 ± 0.03
Table 3: Image reconstruction metrics of INRs trained with each group of distance metrics. For each reconstruction metric, the best score for each dataset among the groups of distance metrics is displayed in bold .
Figure S1: Ultrasound augmentations provided by EchoGains. The pre-trained EchoGains diffusion model is used to induce realistic controlled perturbation in B-mode images.
Figure S2: UltraSam segmentation of left ventricle and atrium. As increasingly stronger rotation is applied to the B-mode image (a), the model still recognizes the heart chambers but becomes more error prone (b). This corresponds to increased perturbation captured by distance metrics (c).
CAMUS
EchoNet
Overall
Model
Depth
Rotation
Sector Width
Translation
Depth
Rotation
Sector Width
Translation
L2
0.57 ± 0.25
0.36 ± 0.29
0.58 ± 0.27
0.26 ± 0.28
0.50 ± 0.27
0.40 ± 0.29
0.53 ± 0.27
0.37 ± 0.25
0.45 ± 0.28
SSIM
0.42 ± 0.26
0.20 ± 0.28
0.51 ± 0.28
0.18 ± 0.25
0.24 ± 0.27
0.09 ± 0.28
0.23 ± 0.27
0.02 ± 0.22
0.20 ± 0.27
AlexNet
0.58 ± 0.25
0.45 ± 0.26
0.62±0.25
0.27 ± 0.27
0.54 ± 0.28
0.51 ± 0.29
0.56 ± 0.27
0.39 ± 0.25
0.49 ± 0.28
VGG-16
0.54 ± 0.25
0.37 ± 0.27
0.61 ± 0.26
0.25 ± 0.27
0.55±0.27
0.52 ± 0.29
0.57 ± 0.27
0.39 ± 0.25
0.48 ± 0.28
ViT
0.52 ± 0.26
0.38 ± 0.26
0.62 ± 0.26
0.24 ± 0.27
0.53 ± 0.27
0.50 ± 0.28
0.55 ± 0.27
0.38 ± 0.25
0.47 ± 0.28
Table S1: Spearman correlation of EchoPrime view classifier confidence with distance metrics on augmented data. For each augmentation type, the augmentation is applied with increased strength. Correlations are measured between view classifier log-softmax output at the correct view class and each of the candidate distance metrics, averaged across frames. In each column, the highest correlation is displayed in bold .
CAMUS
EchoNet
Overall
Model
Depth
Rotation
Sector Width
Translation
Depth
Rotation
Sector Width
Translation
L2
0.04 ± 0.25
0.46 ± 0.29
0.45 ± 0.28
0.38 ± 0.25
0.10 ± 0.25
0.39 ± 0.30
0.31 ± 0.28
0.36 ± 0.25
0.30 ± 0.28
SSIM
0.01 ± 0.22
0.20 ± 0.27
0.39 ± 0.28
0.25 ± 0.24
0.06 ± 0.23
0.04 ± 0.30
0.14 ± 0.26
0.01 ± 0.22
0.12 ± 0.26
AlexNet
0.04 ± 0.25
0.67 ± 0.26
0.51±0.28
0.40±0.25
0.13±0.27
0.57±0.29
0.35±0.29
0.39±0.25
0.37±0.29
VGG-16
0.02 ± 0.23
0.50 ± 0.28
0.48 ± 0.28
0.35 ± 0.24
0.13±0.27
0.55 ± 0.29
0.35±0.28
0.38 ± 0.25
0.34 ± 0.28
ViT
0.02 ± 0.24
0.59 ± 0.27
0.48 ± 0.28
0.37 ± 0.25
0.12 ± 0.26
0.54 ± 0.30
0.34 ± 0.28
0.37 ± 0.25
0.35 ± 0.28
Table S2: Spearman correlation of UltraSam dice scores with distance metrics on augmented data. For each augmentation type, the augmentation is applied with increased strength. Correlations are measured between dice score and each of the candidate distance metrics, averaged across frames. In each column, the highest correlation is displayed in bold .
Ultrasound imaging uses sound waves to visualize anatomy and function, with AI improving image reconstruction, diagnosis, measurement, and scan guidance.
Benchmarking is an increasingly critical part of research in machine learning and the domains where it is applied, including healthcare. Yet, despite the steady development of new ultrasound foundation models in recent years, the development of well-designed benchmarks to evaluate them has lagged behind. This deficiency has led to fragmented and inconsistent evaluations of competing models, making it difficult to measure progress. To address this issue, we introduce UltraBench 2, a comprehensive benchmark with wide anatomical and task coverage, and a focus on standardization, reproducibility, and ease-of-use. Using this benchmark, we compare existing vision foundation models for ultrasound image analysis. Our analyses demonstrate that ultrasound-specific pretraining still leads on classification, but that state-of-the-art general-purpose models have drawn level on segmentation.
Ashwath Radhachandran, Adam Tupper, Christian Gagné +1
Bioengineering Department, University of California, Los Angeles · Institut Intelligence et Données (IID), Université Laval Mila – Quebec AI Institute · Radiological Sciences Department, UCLA David Geffen School of Medicine
Ultrasound is one of the most widely used medical imaging modalities, and recent large vision-language models(VLMs) have shown increasing capabilities in ultrasound image understanding. However, these models fail to provide pixel-level visual evidence aligned with their semantic predictions, and their fine-grained grounding capability in ultrasound remains largely unclear. We introduce UltraG-Bench, a large-scale multi-task benchmark for evaluating pixel-level evidence grounding in ultrasound. UltraG-Bench is built by annotating 40 public ultrasound segmentation datasets spanning 13 anatomical categories, and comprises three progressive tasks: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation, with 331125, 666779, and 138832 annotations, respectively. Comprehensive evaluation of 14 state-of-the-art models reveals a substantial gap between semantic understanding and fine-grained pixel-level localization. We further propose UltraG-Agent, which combines the semantic reasoning capabilities of a VLM with the ultrasound-specific segmentation capability of UltraSAM3. Experiments show that UltraG-Agent substantially improves both semantic prediction and pixel-level visual grounding. Our dataset and code are available at https://github.com/zhuqh19/UltraG-Bench.
Large vision-language models (LVLMs) have achieved strong performance across many medical imaging tasks, yet their application to ultrasound remains limited due to its inherent complexity and variability. In this work, we revisit what is truly needed to enable real-world ultrasound understanding. Instead of introducing complex architectures or elaborate training strategies, we show that data scale and clinically faithful data alignment are the key factors. We construct a large-scale dataset of 1.5M real-world ultrasound examinations, containing 17.7M images, multi-organ coverage, and paired uncurated clinical reports. Crucially, we organize the data at the examination level, aligning multiple images with their corresponding reports to reflect real clinical workflows. We then fine-tune a standard LVLM using low-rank adaptation (LoRA) on this dataset without task-specific modifications. Surprisingly, this simple recipe already leads to strong performance across diverse ultrasound understanding tasks, outperforming prior methods designed with more complex pipelines. Beyond these results, we present model and data scaling analyses that provide insights into the role of scale in ultrasound LVLMs.
Bingcong Yan, Chunlei Li, Jingliang Hu +3
1MedAI Technology (Wuxi) Co. Ltd. · 2Technical University of Munich