Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift via Pseudo Label Discrepancy

Juan Iñaki Larrea; Lucas Mansilla; Enzo Ferrante

cs.CVAug 26, 2026

Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift via Pseudo Label Discrepancy

Authors: Juan Iñaki Larrea, Lucas Mansilla, Enzo Ferrante

Organizations: Applied Artificial Intelligence Lab (LIAA), Computer Sciences Institute (ICC), Universidad de Buenos Aires - CONICET, Argentina · Research Institute for Signals, Systems and Computational Intelligence, sinc(i), Universidad Nacional del Litoral - CONICET, Argentina

Abstract

Foundation models are increasingly deployed for medical image analysis. However, under the inter-institutional distribution shift typical of deployment, their performance varies widely and cannot be known without target-domain labels, which are rarely available. This leaves a practical question unresolved: given several candidate foundational models and labeled-data from a source domain, which one to deploy in an unlabeled target domain? We propose a label-free selection criterion built on SUDO, a framework for evaluating clinical AI systems without ground-truth annotations. SUDO partitions the unlabeled target data by predicted probability and, for each region, measures a pseudo-label discrepancy reflecting class contamination; aggregated across regions, this yields a score (AURCC) requiring neither target annotation nor fine-tuning. We show that AURCC can be used to rank a variety of vision-language models (BioMedCLIP, CXR-CLIP, CheXzero, MedCLIP, MedImageInsight, CLIP) on chest X-ray classification across three inter-hospital shift scenarios, under zero-shot and MLP-probe regimes. The AURCC ranking recovers the ground-truth ranking with Spearman rho up to 0.943 (p<0.05). Against the natural baseline of ranking by held-out source accuracy, AURCC is competitive when the labeled source is large and yields a more accurate ranking once it is small; the regime of interest in resource-constrained settings.

Figures & tables

Figure 1: Overview of the label-free model ranking framework. Given a labeled source dataset and an unlabeled target dataset, each candidate model

\phi_{1},\ldots,\phi_{M}

produces embeddings that drive the SUDO procedure, yielding a ranking of models by their AURCC without target labels.

Dataset	$N$	Positive rate
MIMIC-CXR	37,457	41.6%
NIH ChestX-ray14	4,770	30.0%
CheXpert	563	67.5%
PadChest	88,018	3.2%

Table 1: Dataset sizes and positive rates after filtering for a binary pneumonia label. Each source is split 80/20 into train and held-out portions.

		AURCC vs. AUC ${}_{\text{target}}$			AUC ${}_{\text{source}}$ vs. AUC ${}_{\text{target}}$
Source $\to$ Target	Regime	K=5	K=10	K=20
MIMIC $\to$ PadChest	ZS	0.886*	0.714	0.714	0.943*
MIMIC $\to$ PadChest	MP	0.943*	0.943*	0.943*	1.000*
NIH $\to$ PadChest	ZS	0.886*	0.886*	0.886*	1.000*
NIH $\to$ PadChest	MP	0.943*	0.943*	0.943*	0.829*
CheXpert $\to$ PadChest	ZS	0.943*	0.943*	0.943*	0.886*

Table 2: Spearman

\rho

between model rankings and ground-truth

\text{AUC}_{\text{target}}

, for

M=6

foundation models. AURCC = label-free SUDO score;

\text{AUC}_{\text{source}}

= held-out source baseline.

{}^{*}p<0.05

Figure 2: Label-free AURCC (x-axis, MLP-probe, $K=10$ ) versus true $\text{AUC}_{\text{target}}$ (y-axis) for the three source–target scenarios. Each point is one of the six foundation models. The shaded band indicates the 95% confidence interval of the regression line.

Source $\to$ Target	Method	100%	50%	25%	$N$ =100	$N$ =50
MIMIC $\to$ PadChest	AURCC	0.943	0.943	0.943	0.851	0.683
MIMIC $\to$ PadChest	AUC ${}_{\text{source}}$	1.000	0.997	0.989	0.916	0.605
NIH $\to$ PadChest	AURCC	0.994	0.977	0.963	0.911	0.877
NIH $\to$ PadChest	AUC ${}_{\text{source}}$	0.829	0.809	0.734	0.749	0.752
CheXpert $\to$ PadChest	AURCC	0.980	0.897	—	0.977	0.880
CheXpert $\to$ PadChest	AUC ${}_{\text{source}}$	0.943	0.749	—	0.897	0.706

Table 3: Mean Spearman

\rho

(over 20 repetitions) of AURCC and source-AUC rankings versus

\text{AUC}_{\text{target}}

, as a function of source held-out size. MP regime,

K=10

Figure 3: Sensitivity of AURCC and source-AUC rankings to held-out source size (MP,

K=10

). Each point is the mean Spearman

\rho

over 20 stratified subsampling repetitions.

Explore similar work

Sep 26, 2026cs.LG

STAMP: Predicting Out-of-Distribution Generalization without Target Data

Predicting whether a trained model will generalize under distribution shift remains difficult, especially when target-domain data are unavailable. We introduce STAMP (Semantic Temporal Augmented Model Prediction), a source-only, target-label-free criterion that estimates out-of-distribution (OOD) performance from paired source-domain images. STAMP computes the output-space correlation ratio

η^2=S_B/S_T

by contrasting semantically stable pairs with random pairs: higher

η^2

indicates that model outputs vary with semantic identity rather than nuisance variation. On 44 chest X-ray models spanning CNNs, ViTs, MetaFormers, foundation models, and SSL/VLM probes, temporal STAMP attains Spearman correlations of

0.844

0.855

with macro AUROC on VinDr-CXR, CheXpert, and MIMIC-CXR; a class-matched variant improves single-class RSNA from

0.311

0.663

. STAMP attains the best average source-only medical ranking and outperforms the target-domain ATC and AoTL estimators without any target data. On 27 ImageNet models, temperature-scaled STAMPTS attains

ρ{=}0.984

on ObjectNet and

ρ\geq0.905

on four additional distribution shifts, with partial correlations of

0.662

0.949

after controlling for ImageNet accuracy. Requiring approximately 12s per model on one GPU, STAMP is a practical pre-deployment model-selection and auditing tool.

Md Kawsher Mahbub, Milon Biswas

Department of Computer Science and Engineering NPI University of Bangladesh Manikganj, Bangladesh · Department of Computer & Information Sciences Towson University Towson, MD 21252

OOD Generalization

Ask PDF

Jul 11, 2026cs.CV

Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift

Foundation models are increasingly used as image feature extractors for mammography, but their robustness under external domain shift remains unclear. We benchmark 15 foundation-model backbones across breast density, BI-RADS severity, and cancer status using a unified frozen-backbone linear-probe protocol, training on 3 source datasets and evaluating on 12 task-compatible out-of-distribution (OOD) datasets after label harmonization. Mammography-specific vision-language models (Mammo-FM and MaMA) provide the strongest mean OOD performance, but robustness is not explained by mammography exposure alone. DINOv3 remains a competitive vision-only baseline, and mammography-adapted pretraining does not consistently improve generalization. Dataset-level analysis further shows that even leading models show heterogeneous performance across datasets. Feature-space inspection reveals that useful representations can preserve clinical signal while retaining dataset and acquisition structure. These findings highlight dataset-level OOD evaluation as a central criterion for assessing mammography representations. Our code is publicly available: https://github.com/biomedia-mira/mammo-ood.

Giang Nguyen, Raghav Mehta, Emma A. M. Stanley +4

College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam · Imperial College London, London, UK · Radiology Department, Vietnam National Cancer Hospital, Hanoi, Vietnam +2

Distribution Shift Robustness Medical Imaging Foundation Models OOD Generalization Medical Image Classification Mammography Medical Image Benchmarks

Ask PDF

Aug 12, 2026cs.CV

How Far from Clinical Deployment? Evaluating the Complete Unsupervised Domain Adaptation Pipeline in Medical Imaging

Deploying unsupervised domain adaptation (UDA) in clinical practice requires choosing which algorithm to use and which of its trained models to ship. However, the deployment (target) domain is unlabeled, so models cannot be evaluated directly on it, leaving it unclear which to select. We address this by evaluating the complete UDA pipeline, considering both adaptation and label-free selection together. Our study covers eleven clinically relevant cross-domain scenarios from nine medical imaging datasets, with ten UDA algorithms and 13 label-free selection methods (validators), evaluating over 80,000 trained models in total. By this, we find that a capable adapted model usually exists, but identifying it without target labels is difficult: the validator-selected models leave a large and structural target performance gap to the best available one, with no evaluated validator consistently reliable. Towards closing it, we explore two strategies, ensembling and a small target-labeling budget; both narrow this gap but do not close it entirely. Overall, deployable UDA depends on the complete pipeline; addressing the less explored selection step could bring much of current UDA closer to clinical use.

Yiheng Xiong, Luisa Gallée, Daniel Santak Wolf +2

1Section of Experimental Radiology, Ulm University Medical Center · 2Visual Computing Group, Ulm University

Unsupervised Domain Adaptation

Ask PDF