UltraPIPS: Improving model perception in B-mode ultrasound with foundation models

Tal Grutman; Tali Ilovitsh

cs.CVAug 26, 2026

UltraPIPS: Improving model perception in B-mode ultrasound with foundation models

Authors: Tal Grutman, Tali Ilovitsh

Organizations: School of Biomedical Engineering, Tel Aviv University, Tel Aviv, Israel

Abstract

In medical imaging, it is common to use learned perceptual image patch similarity (LPIPS) to compare images semantically in feature space. Although backbones pretrained on natural images are widely used for LPIPS computation, B-mode ultrasound images possess distinct speckle patterns and acoustic-specific image statistics that are fundamentally different from natural images and even from other images in radiology. Consequently, we propose that domain-specific models are needed to measure perceptual similarity in ultrasound data, a finding which is not necessarily the case for other imaging modalities. We compare LPIPS metrics across downstream tasks like classification, segmentation and reconstruction using natural image, medical generalist and ultrasound backbone models and show that selection of LPIPS backbone is a non-trivial design choice. In particular, the ultrasound backbone models were more correlated with downstream performance of supervised models than classical and natural image models, and optimization of the LPIPS loss with an ultrasound backbone achieved a strong balance between reconstruction quality and realism. Our code is available at https://github.com/talg2324/UltraPIPS and introduces the UltraPIPS library, a set of LPIPS metrics based on the open-source foundation models analyzed in this paper.

Figures & tables

Natural Image	Radiology	Ultrasound
AlexNet	RadImageNet	USFM
VGG-16	MedSAM	TUSA
ViT	BiomedCLIP	Ultrasound-CLIP
SwinViT
CLIP

Table 1: Candidate LPIPS backbone models grouped by pre-training domain.

Panel of Figure 1 — Figure 1: Spearman correlations of supervised models on CAMUS and EchoNet datasets under ultrasound augmentation. In the view classification task (a) correlation is calculated between the log-softmax confidence score of EchoPrime’s view classifier and each distance metric, averaged across augmentations and frames. For segmentation (b), the correlation is calculated between UltraSam’s dice score and and each of the distance metrics, averaged across augmentations and frames

Null Hypothesis	Classification	Segmentation
$\rho_{\text{US}}\leq\rho_{\text{Classical}}$	$7.8\times 10^{-3}$	$7.8\times 10^{-3}$
$\rho_{\text{US}}\leq\rho_{\text{Natural Images}}$	0.016	0.039
$\rho_{\text{US}}\leq\rho_{\text{Radiology}}$	0.078	0.078

Table 2: Wilcoxon test of LPIPS computed with ultrasound models vs. comparison groups.

Panel of Figure 2 — Figure 2: Implicit Neural Representation (INR) visual comparison on the follicles dataset. Models are grouped and color-coded by their pre-training domain: Classical (dark blue), Natural Images (light blue), Radiology (orange), and Ultrasound (green).

	USOVA				MicroSegNet
	$L_{2}$	Natural Images	Radiology	Ultrasound	$L_{2}$	Natural Images	Radiology	Ultrasound
$\uparrow$ SSIM	0.76 $\pm$ 0.09	0.45 $\pm$ 0.17	0.47 $\pm$ 0.13	0.70 $\pm$ 0.13	0.65 $\pm$ 0.04	0.36 $\pm$ 0.13	0.39 $\pm$ 0.09	0.55 $\pm$ 0.10
$\downarrow$ HFEN	0.34 $\pm$ 0.04	1.12 $\pm$ 0.45	1.06 $\pm$ 0.27	0.46 $\pm$ 0.15	0.37 $\pm$ 0.04	1.41 $\pm$ 0.42	1.36 $\pm$ 0.25	0.71 $\pm$ 0.29
$\downarrow$ GLCM Contrast	2.84 $\pm$ 1.92	1.81 $\pm$ 1.43	1.79 $\pm$ 1.37	1.96 $\pm$ 1.89	5.42 $\pm$ 1.84	3.61 $\pm$ 2.07	3.87 $\pm$ 1.83	3.24 $\pm$ 2.86
$\downarrow$ GLCM Dissimilarity	0.45 $\pm$ 0.26	0.24 $\pm$ 0.18	0.23 $\pm$ 0.16	0.29 $\pm$ 0.26	0.62 $\pm$ 0.21	0.28 $\pm$ 0.18	0.36 $\pm$ 0.19	0.33 $\pm$ 0.32
$\downarrow$ GLCM Homogeneity	0.07 $\pm$ 0.04	0.04 $\pm$ 0.04	0.03 $\pm$ 0.02	0.04 $\pm$ 0.04	0.06 $\pm$ 0.03	0.04 $\pm$ 0.03	0.03 $\pm$ 0.02	0.03 $\pm$ 0.03

Table 3: Image reconstruction metrics of INRs trained with each group of distance metrics. For each reconstruction metric, the best score for each dataset among the groups of distance metrics is displayed in bold .

Panel of Figure 6 — Figure S1: Ultrasound augmentations provided by EchoGains. The pre-trained EchoGains diffusion model is used to induce realistic controlled perturbation in B-mode images.

Panel of Figure 7 — Figure S2: UltraSam segmentation of left ventricle and atrium. As increasingly stronger rotation is applied to the B-mode image (a), the model still recognizes the heart chambers but becomes more error prone (b). This corresponds to increased perturbation captured by distance metrics (c).

	CAMUS				EchoNet				Overall
Model	Depth	Rotation	Sector Width	Translation	Depth	Rotation	Sector Width	Translation
$L_{2}$	0.57 $\pm$ 0.25	0.36 $\pm$ 0.29	0.58 $\pm$ 0.27	0.26 $\pm$ 0.28	0.50 $\pm$ 0.27	0.40 $\pm$ 0.29	0.53 $\pm$ 0.27	0.37 $\pm$ 0.25	0.45 $\pm$ 0.28
SSIM	0.42 $\pm$ 0.26	0.20 $\pm$ 0.28	0.51 $\pm$ 0.28	0.18 $\pm$ 0.25	0.24 $\pm$ 0.27	0.09 $\pm$ 0.28	0.23 $\pm$ 0.27	0.02 $\pm$ 0.22	0.20 $\pm$ 0.27
AlexNet	0.58 $\pm$ 0.25	0.45 $\pm$ 0.26	$\mathbf{0.62\pm 0.25}$	0.27 $\pm$ 0.27	0.54 $\pm$ 0.28	0.51 $\pm$ 0.29	0.56 $\pm$ 0.27	0.39 $\pm$ 0.25	0.49 $\pm$ 0.28
VGG-16	0.54 $\pm$ 0.25	0.37 $\pm$ 0.27	0.61 $\pm$ 0.26	0.25 $\pm$ 0.27	$\mathbf{0.55\pm 0.27}$	0.52 $\pm$ 0.29	0.57 $\pm$ 0.27	0.39 $\pm$ 0.25	0.48 $\pm$ 0.28
ViT	0.52 $\pm$ 0.26	0.38 $\pm$ 0.26	0.62 $\pm$ 0.26	0.24 $\pm$ 0.27	0.53 $\pm$ 0.27	0.50 $\pm$ 0.28	0.55 $\pm$ 0.27	0.38 $\pm$ 0.25	0.47 $\pm$ 0.28

Table S1: Spearman correlation of EchoPrime view classifier confidence with distance metrics on augmented data. For each augmentation type, the augmentation is applied with increased strength. Correlations are measured between view classifier log-softmax output at the correct view class and each of the candidate distance metrics, averaged across frames. In each column, the highest correlation is displayed in bold .

	CAMUS				EchoNet				Overall
Model	Depth	Rotation	Sector Width	Translation	Depth	Rotation	Sector Width	Translation
$L_{2}$	0.04 $\pm$ 0.25	0.46 $\pm$ 0.29	0.45 $\pm$ 0.28	0.38 $\pm$ 0.25	0.10 $\pm$ 0.25	0.39 $\pm$ 0.30	0.31 $\pm$ 0.28	0.36 $\pm$ 0.25	0.30 $\pm$ 0.28
SSIM	0.01 $\pm$ 0.22	0.20 $\pm$ 0.27	0.39 $\pm$ 0.28	0.25 $\pm$ 0.24	0.06 $\pm$ 0.23	0.04 $\pm$ 0.30	0.14 $\pm$ 0.26	0.01 $\pm$ 0.22	0.12 $\pm$ 0.26
AlexNet	0.04 $\pm$ 0.25	0.67 $\pm$ 0.26	$\mathbf{0.51\pm 0.28}$	$\mathbf{0.40\pm 0.25}$	$\mathbf{0.13\pm 0.27}$	$\mathbf{0.57\pm 0.29}$	$\mathbf{0.35\pm 0.29}$	$\mathbf{0.39\pm 0.25}$	$\mathbf{0.37\pm 0.29}$
VGG-16	0.02 $\pm$ 0.23	0.50 $\pm$ 0.28	0.48 $\pm$ 0.28	0.35 $\pm$ 0.24	$\mathbf{0.13\pm 0.27}$	0.55 $\pm$ 0.29	$\mathbf{0.35\pm 0.28}$	0.38 $\pm$ 0.25	0.34 $\pm$ 0.28
ViT	0.02 $\pm$ 0.24	0.59 $\pm$ 0.27	0.48 $\pm$ 0.28	0.37 $\pm$ 0.25	0.12 $\pm$ 0.26	0.54 $\pm$ 0.30	0.34 $\pm$ 0.28	0.37 $\pm$ 0.25	0.35 $\pm$ 0.28

Table S2: Spearman correlation of UltraSam dice scores with distance metrics on augmented data. For each augmentation type, the augmentation is applied with increased strength. Correlations are measured between dice score and each of the candidate distance metrics, averaged across frames. In each column, the highest correlation is displayed in bold .

Ultrasound Imaging

Ultrasound imaging uses sound waves to visualize anatomy and function, with AI improving image reconstruction, diagnosis, measurement, and scan guidance.

See all 150 papers

UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasound
Sep 23, 2026Ashwath Radhachandran, Adam Tupper, Christian Gagné +1Ultrasound Image Segmentation Medical Imaging Foundation Models
Bioengineering Department, University of California, Los Angeles · Institut Intelligence et Données (IID), Université Laval Mila – Quebec AI Institute · Radiological Sciences Department, UCLA David Geffen School of Medicine
UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound
Sep 25, 2026Quanhao Zhu, Bo Xu, Rui Lin +7Medical VLMs Medical Image Grounding
Dalian University of Technology · RMIT University
Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports
Jul 2, 2026Bingcong Yan, Chunlei Li, Jingliang Hu +3VLM Adaptation Ultrasound Imaging
1MedAI Technology (Wuxi) Co. Ltd. · 2Technical University of Munich

Sep 23, 2026J/K move · Enter open · S save

UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasound

Ashwath Radhachandran, Adam Tupper, Christian Gagné, William Speier

Bioengineering Department, University of California, Los Angeles · Institut Intelligence et Données (IID), Université Laval Mila – Quebec AI Institute · Radiological Sciences Department, UCLA David Geffen School of Medicine

Ultrasound Image Segmentation Medical Imaging Foundation Models Vision Foundation Models Medical Image Classification Ultrasound Imaging Medical Image Benchmarks

Open paper AskPDF arXiv

Benchmarking is an increasingly critical part of research in machine learning and the domains where it is applied, including healthcare. Yet, despite the steady development of new ultrasound foundation models in recent years, the development of well-designed benchmarks to evaluate them has lagged behind. This deficiency has led to fragmented and inconsistent evaluations of competing models, making it difficult to measure progress. To address this issue, we introduce UltraBench 2, a comprehensive benchmark with wide anatomical and task coverage, and a focus on standardization, reproducibility, and ease-of-use. Using this benchmark, we compare existing vision foundation models for ultrasound image analysis. Our analyses demonstrate that ultrasound-specific pretraining still leads on classification, but that state-of-the-art general-purpose models have drawn level on segmentation.

Abstract

Ultrasound Imaging

Explore similar work

UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasound

UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports