Pretext Matters: An Empirical Study of SSL Methods in Medical Imaging

Vedrana Ivezić; Mara Pleasure; Ashwath Radhachandran; Saarang Panchavati; Shreeram Athreya; Vivek Sant; Benjamin Emert; Gregory Fishbein

cs.CVMar 23, 2026

Pretext Matters: An Empirical Study of SSL Methods in Medical Imaging

Authors: Vedrana Ivezić, Mara Pleasure, Ashwath Radhachandran, Saarang Panchavati, Shreeram Athreya, Vivek Sant, Benjamin Emert, Gregory Fishbein,

Organizations: Biomedical AI Research Lab, University of California, Los Angeles, Los Angeles, CA, USA · Department of Radiology, University of California, Los Angeles, Los Angeles, CA, USA · Department of Bioengineering, University of California, Los Angeles, Los Angeles, CA, USA · Department of Electrical and Computer Engineering, University of California, Los Angeles, Los Angeles, CA, USA ·

Abstract

Though self-supervised learning (SSL) has demonstrated incredible ability to learn robust representations from unlabeled data, the choice of optimal SSL strategy can lead to vastly different performance outcomes in specialized domains. Joint embedding architectures (JEAs) and joint embedding predictive architectures (JEPAs) have shown robustness to noise and strong semantic feature learning compared to pixel reconstruction-based SSL methods, leading to widespread adoption in medical imaging. However, no prior work has systematically investigated which SSL objective is better aligned with the spatial organization of clinically relevant signal. In this work, we empirically investigate how the choice of SSL method impacts the learned representations in medical imaging. We select two representative imaging modalities characterized by unique noise profiles: ultrasound and histopathology. When informative signal is spatially localized, as in histopathology, JEAs are more effective due to their view-invariance objective. In contrast, when diagnostically relevant information is globally structured, such as the macroscopic anatomy present in liver ultrasounds, JEPAs are optimal. These differences are especially evident in the clinical relevance of the learned features, as independently validated by board-certified radiologists and pathologists. Together, our results provide a framework for matching SSL objectives to the structural and noise properties of medical imaging modalities.

Figures & tables

Figure 1: SSL pretext tasks on a lung image for (a) MAE, (b) I-JEPA, and (c) DINOv3.

Ultrasound		Histopathology
Dataset	Task	Dataset	Task
MMOTU	Ovarian Tumor	CPTAC-Lung	LUAD vs LSCC
AUL	Liver Mass Malignancy	PLCO-Lung	LUAD vs LSCC
TN5000	Thyroid Tumor	PLCO-Breast	Breast Cancer Subtype
Fatty Liver	Hepatic Steatosis (NAFLD)	Ovarian	Ovarian Cancer Subtype
POCUS	Lung Pathology	PANDA	Prostate Biopsy ISUP Grade

Table 1: Overview of downstream tasks. (Acronyms: NAFLD - Non Alcoholic Fatty Liver Disease; LUAD - Lung Adenocarcinoma; LSCC - Lung Squamous Cell Carcinoma; ISUP - International Society of Urological Pathology.) See 0.D.1 for more info.

Figure 2: Example visualizations including attention maps, cosine similarity maps (anchor points denoted by red X), and PCA maps for both ultrasound and pathology.

	MAE		DINOv3		I-JEPA
Dataset	AUROC (%)	F1 (%)	AUROC (%)	F1 (%)	AUROC (%)	F1 (%)
MMOTU	78.43 $\pm$ 0.12	38.04 $\pm$ 1.4	84.57 $\pm$ 0.07	50.31 $\pm$ 0.65	82.59 $\pm$ 0.06	42.03 $\pm$ 0.58
AUL	81.86 $\pm$ 0.21	62.52 $\pm$ 2.1	86.77 $\pm$ 0.23	66.33 $\pm$ 0.52	86.47 $\pm$ 0.21	64.71 $\pm$ 0.81
TN5000	69.88 $\pm$ 0.2	58.25 $\pm$ 2.2	72.36 $\pm$ 0.42	63.34 $\pm$ 1.5	75.60 $\pm$ 0.12	65.83 $\pm$ 1.6
Fatty Liver	65.89 $\pm$ 0.6	50.76 $\pm$ 5.7	86.66 $\pm$ 1.5	64.06 $\pm$ 1.7	98.70 $\pm$ 0.26	92.80 $\pm$ 1.2
POCUS	96.12 $\pm$ 0.19	91.01 $\pm$ 0.11	99.49 $\pm$ 0.07	94.00 $\pm$ 0.19	97.18 $\pm$ 0.06	90.48 $\pm$ 0.52

Table 2: Performance comparison across SSL methods on eight ultrasound classification tasks averaged across five seeds. All values are reported as percentages (%).

Figure 3: Attention heatmaps from each attention head of the final ViT layer across models pre-trained on ultrasounds. The dark oval area in the GBCU image is the gallbladder with the liver region to the left. The oval area of the Fatty Liver image is the kidney, the white line above is the diaphragm, and the shaded region to the right is the liver. The TN5000 image contains a thyroid nodule marked with calipers.

Figure 4: Cosine similarity maps computed in relation to anchor patches (red X).

	MAE		DINOv3		I-JEPA
Dataset	AUROC (%)	F1 (%)	AUROC (%)	F1 (%)	AUROC (%)	F1 (%)
CPTAC-Lung	77.1 $\pm$ 4.2	27.2 $\pm$ 33.3	91.4 $\pm$ 1.0	83.3 $\pm$ 1.4	54.8 $\pm$ 1.3	31.6 $\pm$ 29.4
PLCO-Lung	63.8 $\pm$ 0.9	30.6 $\pm$ 13.7	82.9 $\pm$ 1.3	66.8 $\pm$ 5.3	57.4 $\pm$ 8.4	21.8 $\pm$ 26.7
PLCO-Breast	49.2 $\pm$ 2.2	26.5 $\pm$ 6.2	65.2 $\pm$ 1.1	38.7 $\pm$ 8.5	52.3 $\pm$ 6.2	11.6 $\pm$ 8.6
Ovarian	73.5 $\pm$ 0.7	34.1 $\pm$ 7.3	92.0 $\pm$ 1.8	73.3 $\pm$ 2.4	55.2 $\pm$ 4.3	4.5 $\pm$ 2.0
PANDA	81.9 $\pm$ 0.2	44.2 $\pm$ 0.3	87.0 $\pm$ 0.3	53.9 $\pm$ 0.8	64.0 $\pm$ 0.6	22.7 $\pm$ 1.5

Table 3: Performance comparison across SSL methods on five histopathology classification tasks. All values are reported as percentages (%).

Figure 5: Attention heatmaps from each attention head of the final ViT layer across models pre-trained on histopathology data for three downstream datasets.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Hyperparameter	Value
MAE	ViT	ViT-small
	Batch size	256
	Learning rate	$1.5\times 10^{-4}$
	Warmup epochs	10
DINOv3	ViT	ViT-small
	Batch size	128

Appendix

Table 4: Comparison of Hyperparameters across three Models. Learning rate peak for DINOv3 histopathology model was

1\times 10^{-5}

otherwise all parameters were shared across ultrasound and histopathology.

Figure 6: Example attention maps from DINOv3 and MAE comparing [CLS] token and mean patch attentions for an ultrasound example. I-JEPA mean patch attention map is provided as a reference.

Figure 7: Example attention maps from DINOv3 and MAE comparing [CLS] token and mean patch attentions for a histopathology example. I-JEPA mean patch attention map is provided as a reference.

Figure 8: Attention maps from each SSL method for gallbladder ultrasounds, including both a benign and malignant example.

Figure 9: Attention maps from each SSL method for ultrasounds of normal liver and non-alcoholic fatty liver disease (NFLD).

Figure 10: Attention maps from each SSL method for thyroid ultrasounds. Both ultrasounds show benign nodules but of varying sizes. Nodules are denoted by the calipers.

Figure 11: Attention maps from each SSL method for ultrasounds of the carotid and the four chamber view of the heart.

Figure 12: Carotid artery ultrasound cosine similarity map for each model. The anchor point (red X) is placed on the boundary of the carotid artery to demonstrate I-JEPA’s ability to learn anatomical structures.

Figure 13: PCA Visualizations of two histopathology patches from three downstream examples.

Self-Supervised Learning

Self-supervised learning uses structure within unlabeled data to train representations and models for downstream tasks.

See all 258 papers

Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelines
May 18, 2026Chathura Wimalasiri, Kishor Nandakishor, Marimuthu PalaniswamiContrastive Learning Self-Supervised Learning
Department of Electrical and Electronic Engineering, University of Melbourne, Parkville, VIC 3052, Australia
US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound
Feb 22, 2026Ashwath Radhachandran, Vedrana Ivezić, Shreeram Athreya +2Joint-Embedding Predictive Architecture Ultrasound Imaging
Biomedical Artificial Intelligence Research Lab, University of California, Los Angeles, Los Angeles, United States of America · Medical Informatics, University of California, Los Angeles, Los Angeles, United States of America · Department of Electrical and Computer Engineering, University of California, Los Angeles, Los Angeles, United States of America +3
Which Pretext Task Transfers? Self-Supervised Pretraining Objectives for Lung Ultrasound
Sep 15, 2026Moein Heidari, Junbo Rao, Jai Choraria +3Contrastive Learning Masked Autoencoders
University of British Columbia, Vancouver, BC, Canada · Rutgers Cancer Institute of New Jersey, New Brunswick, NJ, USA

May 18, 2026J/K move · Enter open · S save

Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelines

Chathura Wimalasiri, Kishor Nandakishor, Marimuthu Palaniswami

Department of Electrical and Electronic Engineering, University of Melbourne, Parkville, VIC 3052, Australia

Contrastive Learning Self-Supervised Learning Medical Image Analysis Medical Image Representation Learning

Open paper AskPDF arXiv

Self-supervised learning (SSL) is increasingly used in medical image analysis to reduce dependence on costly expert annotations by learning transferable representations from unlabeled data. However, SSL performance depends not only on model architecture, but also on whether the pretext task preserves information required by the downstream clinical objective. This review presents a task-oriented synthesis of SSL methods for medical imaging, focusing on how pretext-task design interacts with imaging modality, label availability, and downstream performance. We analyze 75 studies published from 2017 to 2025 and organize them into four paradigms: contrastive learning, non-contrastive and predictive learning, generative and reconstruction-based learning, and hybrid learning. Rather than cataloging methods chronologically, we examine how these paradigms support classification, segmentation, detection, reconstruction, and regression. The evidence suggests that no SSL strategy is universally optimal. Contrastive objectives generally encourage global discriminative representations and are well aligned with classification, but may underrepresent subtle or localized pathology. Spatial prediction, masked modeling, and reconstruction-based objectives better preserve anatomical structure and are often more suitable for segmentation and dense prediction. Hybrid methods can provide balanced representations, although they increase training complexity. Across modalities, SSL is most beneficial in low-label and few-shot regimes, but its effectiveness depends on modality-aware augmentation, pathology-preserving corruption, and clinically meaningful evaluation. We conclude with practical design guidelines and identify open challenges, including pathology-aware pretext tasks, resource-efficient training for high-dimensional data, and standardized evaluation protocols.

Abstract

Self-Supervised Learning

Explore similar work

Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelines

US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound

Which Pretext Task Transfers? Self-Supervised Pretraining Objectives for Lung Ultrasound