Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift via Pseudo Label Discrepancy

Juan Iñaki Larrea; Lucas Mansilla; Enzo Ferrante

cs.CVAug 26, 2026

Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift via Pseudo Label Discrepancy

Authors: Juan Iñaki Larrea, Lucas Mansilla, Enzo Ferrante

Organizations: Applied Artificial Intelligence Lab (LIAA), Computer Sciences Institute (ICC), Universidad de Buenos Aires - CONICET, Argentina · Research Institute for Signals, Systems and Computational Intelligence, sinc(i), Universidad Nacional del Litoral - CONICET, Argentina

Abstract

Foundation models are increasingly deployed for medical image analysis. However, under the inter-institutional distribution shift typical of deployment, their performance varies widely and cannot be known without target-domain labels, which are rarely available. This leaves a practical question unresolved: given several candidate foundational models and labeled-data from a source domain, which one to deploy in an unlabeled target domain? We propose a label-free selection criterion built on SUDO, a framework for evaluating clinical AI systems without ground-truth annotations. SUDO partitions the unlabeled target data by predicted probability and, for each region, measures a pseudo-label discrepancy reflecting class contamination; aggregated across regions, this yields a score (AURCC) requiring neither target annotation nor fine-tuning. We show that AURCC can be used to rank a variety of vision-language models (BioMedCLIP, CXR-CLIP, CheXzero, MedCLIP, MedImageInsight, CLIP) on chest X-ray classification across three inter-hospital shift scenarios, under zero-shot and MLP-probe regimes. The AURCC ranking recovers the ground-truth ranking with Spearman rho up to 0.943 (p<0.05). Against the natural baseline of ranking by held-out source accuracy, AURCC is competitive when the labeled source is large and yields a more accurate ranking once it is small; the regime of interest in resource-constrained settings.

Figures & tables

Figure 1: Overview of the label-free model ranking framework. Given a labeled source dataset and an unlabeled target dataset, each candidate model

\phi_{1},\ldots,\phi_{M}

produces embeddings that drive the SUDO procedure, yielding a ranking of models by their AURCC without target labels.

Dataset	$N$	Positive rate
MIMIC-CXR	37,457	41.6%
NIH ChestX-ray14	4,770	30.0%
CheXpert	563	67.5%
PadChest	88,018	3.2%

Table 1: Dataset sizes and positive rates after filtering for a binary pneumonia label. Each source is split 80/20 into train and held-out portions.

		AURCC vs. AUC ${}_{\text{target}}$			AUC ${}_{\text{source}}$ vs. AUC ${}_{\text{target}}$
Source $\to$ Target	Regime	K=5	K=10	K=20
MIMIC $\to$ PadChest	ZS	0.886*	0.714	0.714	0.943*
MIMIC $\to$ PadChest	MP	0.943*	0.943*	0.943*	1.000*
NIH $\to$ PadChest	ZS	0.886*	0.886*	0.886*	1.000*
NIH $\to$ PadChest	MP	0.943*	0.943*	0.943*	0.829*
CheXpert $\to$ PadChest	ZS	0.943*	0.943*	0.943*	0.886*

Table 2: Spearman

\rho

between model rankings and ground-truth

\text{AUC}_{\text{target}}

, for

M=6

foundation models. AURCC = label-free SUDO score;

\text{AUC}_{\text{source}}

= held-out source baseline.

{}^{*}p<0.05

Figure 2: Label-free AURCC (x-axis, MLP-probe, $K=10$ ) versus true $\text{AUC}_{\text{target}}$ (y-axis) for the three source–target scenarios. Each point is one of the six foundation models. The shaded band indicates the 95% confidence interval of the regression line.

Source $\to$ Target	Method	100%	50%	25%	$N$ =100	$N$ =50
MIMIC $\to$ PadChest	AURCC	0.943	0.943	0.943	0.851	0.683
MIMIC $\to$ PadChest	AUC ${}_{\text{source}}$	1.000	0.997	0.989	0.916	0.605
NIH $\to$ PadChest	AURCC	0.994	0.977	0.963	0.911	0.877
NIH $\to$ PadChest	AUC ${}_{\text{source}}$	0.829	0.809	0.734	0.749	0.752
CheXpert $\to$ PadChest	AURCC	0.980	0.897	—	0.977	0.880
CheXpert $\to$ PadChest	AUC ${}_{\text{source}}$	0.943	0.749	—	0.897	0.706

Table 3: Mean Spearman

\rho

(over 20 repetitions) of AURCC and source-AUC rankings versus

\text{AUC}_{\text{target}}

, as a function of source held-out size. MP regime,

K=10

Figure 3: Sensitivity of AURCC and source-AUC rankings to held-out source size (MP,

K=10

). Each point is the mean Spearman

\rho

over 20 stratified subsampling repetitions.

Explore similar work

STAMP: Predicting Out-of-Distribution Generalization without Target Data
Sep 26, 2026Md Kawsher Mahbub, Milon BiswasOOD Generalization
Department of Computer Science and Engineering NPI University of Bangladesh Manikganj, Bangladesh · Department of Computer & Information Sciences Towson University Towson, MD 21252
Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
Jul 11, 2026Giang Nguyen, Raghav Mehta, Emma A. M. Stanley +4Distribution Shift Robustness Medical Imaging Foundation Models
College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam · Imperial College London, London, UK · Radiology Department, Vietnam National Cancer Hospital, Hanoi, Vietnam +2
How Far from Clinical Deployment? Evaluating the Complete Unsupervised Domain Adaptation Pipeline in Medical Imaging
Aug 12, 2026Yiheng Xiong, Luisa Gallée, Daniel Santak Wolf +2Unsupervised Domain Adaptation
1Section of Experimental Radiology, Ulm University Medical Center · 2Visual Computing Group, Ulm University

Sep 26, 2026, cs.LGJ/K move · Enter open · S save

STAMP: Predicting Out-of-Distribution Generalization without Target Data

Md Kawsher Mahbub, Milon Biswas

Department of Computer Science and Engineering NPI University of Bangladesh Manikganj, Bangladesh · Department of Computer & Information Sciences Towson University Towson, MD 21252

OOD Generalization

Open paper AskPDF arXiv

Predicting whether a trained model will generalize under distribution shift remains difficult, especially when target-domain data are unavailable. We introduce STAMP (Semantic Temporal Augmented Model Prediction), a source-only, target-label-free criterion that estimates out-of-distribution (OOD) performance from paired source-domain images. STAMP computes the output-space correlation ratio

η^2=S_B/S_T

by contrasting semantically stable pairs with random pairs: higher

η^2

indicates that model outputs vary with semantic identity rather than nuisance variation. On 44 chest X-ray models spanning CNNs, ViTs, MetaFormers, foundation models, and SSL/VLM probes, temporal STAMP attains Spearman correlations of

0.844

0.855

with macro AUROC on VinDr-CXR, CheXpert, and MIMIC-CXR; a class-matched variant improves single-class RSNA from

0.311

0.663

. STAMP attains the best average source-only medical ranking and outperforms the target-domain ATC and AoTL estimators without any target data. On 27 ImageNet models, temperature-scaled STAMPTS attains

ρ{=}0.984

on ObjectNet and

ρ\geq0.905

on four additional distribution shifts, with partial correlations of

0.662

0.949

after controlling for ImageNet accuracy. Requiring approximately 12s per model on one GPU, STAMP is a practical pre-deployment model-selection and auditing tool.

Abstract

Explore similar work

STAMP: Predicting Out-of-Distribution Generalization without Target Data

Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift

How Far from Clinical Deployment? Evaluating the Complete Unsupervised Domain Adaptation Pipeline in Medical Imaging