PRISM: Pathology Reliability In Scarce-label Medicine. A Tile-level Benchmark of Pathology Foundation Models

Alicem Koyun

cs.LGOct 27, 2026

PRISM: Pathology Reliability In Scarce-label Medicine. A Tile-level Benchmark of Pathology Foundation Models

Authors: Alicem Koyun

Organizations: Westcliff University

Abstract

Pathology foundation models are typically benchmarked on classification performance at full label availability and on in-distribution test sets, while calibration, behavior at low label budgets, and robustness under cross-institutional shift are seldom evaluated jointly. We introduce PRISM, a tile-level benchmark that measures the reliability of pathology foundation models along four axes: model choice, label fraction, calibration, and out-of-distribution transfer. PRISM evaluates eight foundation models (CLIP, PLIP, CONCH, UNI, Virchow2, GigaPath, H-Optimus-0, Midnight) on six tile-level datasets at six label fractions (1% to 100%) with three seeds (864 linear probe runs), extended with a five-hospital Camelyon17 study of 20 directed transfer pairs. Four phenomena emerge. (i) Under full supervision, AUROC rank predicts post-temperature calibration rank consistently across datasets (Spearman rho = 0.52, I^2 = 0%), whereas at 1% labels the relationship becomes dataset-specific (rho = -0.24, I^2 = 67%). (ii) On harder datasets the linear probe collapses to a single predicted class below a dataset-specific label fraction. (iii) Whether additional source-domain labels help or harm target calibration tracks how well the probe transfers: below a transfer AUROC of roughly 0.85, more labels worsen target ECE in 86-88% of combinations. (iv) Temperature scaling fails to reduce ECE in 49 of 288 cells, miscalibration is predominantly under-confidence, and a source-fitted temperature is harmful under failing transfer. PRISM is released as a pip-installable package with reference results for all 288 cells. Code and data: https://github.com/alicemkyn/prism-benchmark

Figures & tables

Figure 1: PRISM overview. Eight foundation models are evaluated on six tile-level datasets at six label fractions with three seeds (864 linear probe runs), plus a five-hospital Camelyon17 grid of 20 directed transfer pairs (3,600 runs), measuring discrimination, calibration, out-of-distribution transfer and post-hoc recoverability. Two transfer designs separate label-definition shift from covariate shift. Right: the four findings, with the numbers reported in Section 4 (finding 3 counts FMs whose probe collapses in all three seeds). The Composite Reliability Index is a screening heuristic read alongside its three components.

Dataset	Task	Classes	Samples	License	Source
PCam	Binary (tumor)	2	327,680	CC0	Veeling et al. [ 39 ]
CRC	Multiclass	9	100,000	CC-BY 4.0	Kather et al. [ 18 ]
MHIST	Binary (HP/SSA)	2	3,152	RUA	Wei et al. [ 40 ]
BRACS	Multiclass	7	4,539	CC-BY-NC 4.0	Brancati et al. [ 5 ]
SPIDER-Breast	Multiclass	18	92,900	CC-BY-NC 4.0	Nechaev et al. [ 26 ]
LungHist700	Multiclass	7	691	CC-BY 4.0	Diosdado et al. [ 9 ]

Table 1: Tile-level datasets included in PRISM, ordered by sample count: six datasets spanning binary to 18-class classification, with counts varying by nearly three orders of magnitude. CRC is NCT-CRC-HE-100K, split 70/15/15. RUA: research-use agreement without redistribution rights.

		$n_{\text{train}}$ at label fraction
Dataset	$K$	1%	5%	10%	100%	$n_{\text{val}}$	val. share @1%
PCam	2	2,620	13,106	26,214	262,144	32,768	93%
BRACS	7	33	180	362	3,657	312	90%
CRC	9	696	3,495	6,996	70,000	15,000	96%
MHIST	2	18	92	184	1,849	326	95%
LungHist700	7	7	22	45	483	103	94%

Table 2: Absolute label budget. Training labels

n_{\text{train}}

at each label fraction and the size of the validation split used for temperature scaling (and CRI). The validation split is held at its full size at every fraction, so at 1% it accounts for 90–96% of all labels used; raw ECE uses no validation labels. Camelyon17 counts are per source hospital.

Figure 2: Discrimination rank against calibration rank. Each point is one (model, dataset) cell: its position on the horizontal axis is the model’s AUROC rank within that dataset, on the vertical axis its rank by post-temperature ECE, both with 1 best. Points on the diagonal are models whose two ranks agree and the shaded band marks agreement to within one place. Mean displacement is 3.0 of 8 positions at 1% labels against 1.8 under full supervision. The largest discords at 1% labels are CLIP on BRACS, LungHist700 and PCam (last on AUROC, first on calibration) and H-Optimus-0 on PCam (first on AUROC, seventh on calibration).

Figure 3: Reverse OOD calibration scaling. Raw ECE on the transfer target against source-domain label fraction, at all six fractions rather than the two endpoints. On MHIST

\to

PCam the four large vision-only SSL models (UNI, VIRCHOW2, GigaPath, H-Optimus-0; solid lines) rise from roughly 0.23 to between 0.44 and 0.49, monotonically in the label fraction and consistently across seeds; CONCH (dashed) rises as strongly, CLIP and MIDNIGHT more weakly, and PLIP stays flat. The three remaining pairs show the effect weakly or not at all, so the phenomenon is pair-dependent and we do not claim it holds for transfer in general.

Panel of Figure 4 — Figure 4: Reverse calibration scaling tracks transfer quality, not the kind of shift. (a) The same protocol, opposite directions: raw target ECE against source label fraction, averaged over pairs within each design. Top: the four label-definition pairs (transfer AUROC near 0.5); bottom: twenty Camelyon17 hospital pairs with one label definition. (b) Each point is one (model, pair) combination; the vertical axis is the slope of target ECE in $\log_{10}$ label fraction, so points in the shaded region are combinations where additional source labels worsen calibration. The strip above marks where each design sits on the AUROC axis; the two designs overlap through the transition zone.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Type	Size	$d$	Pre-training corpus
CLIP	Vision-Language	86M	768	WebImageText (general)
PLIP	Vision-Language	86M	768	Twitter pathology pairs
CONCH	Vision-Language	86M	512	Curated pathology pairs
UNI	Vision	307M	1024	100k WSIs (Mass-100K)
VIRCHOW2	Vision	632M	2560	3.1M WSIs
GigaPath	Vision	1.1B	1536	171k WSIs (Prov-Path)

Appendix

Table 3: The eight foundation models evaluated. Sizes range from 86M to 1.1B parameters.

Model(s)	Resize	Crop	Normalization
CLIP, PLIP, MIDNIGHT, UNI, VIRCHOW2	224	224	ImageNet
GigaPath	256	224	ImageNet
H-Optimus-0	224	224	model-specific
CONCH	transform supplied by the model loader

Appendix

Table 4: Preprocessing applied to each FM during feature extraction. Each model is embedded once under one pipeline, held fixed thereafter.

Benchmark	FMs	Label fraction	Calibration	OOD	Year
THUNDER [ 25 ]	23	few-shot only	✓	adversarial only	2026
HistoVL [ 1 ]	4 VLMs	✗	✓	✗	2025
Neidlinger et al. [ 27 ]	19	✗	✗	✗	2025
PathBench [ 24 ]	19	✗	✗	✗	2025
Lee et al. [ 21 ]	4	few-shot only	✗	near/mid/out-domain	2025
Nunes et al. [ 29 ]	–	✗	✓	✗	2025

Appendix

Table 5: Comparison of pathology FM benchmarks and adjacent calibration studies. Only PRISM evaluates the joint interaction of FM choice, label fraction, calibration, and cross-domain OOD shift on pathology FMs. † Image classification (not pathology); included as the closest methodological precedent.

Model	Resize	Normalization	Dim
CLIP, PLIP	224	CLIP statistics	768
CONCH	model loader	model loader	512
UNI	224	ImageNet	1024
VIRCHOW2	224	ImageNet	2560
GigaPath	224	ImageNet	1536
H-Optimus-0	256	model-specific	1536

Appendix

Table 6: Preprocessing resolved from each model for the Camelyon17 extraction.

Figure 5: Calibration under transfer depends mostly on the target hospital. Transfer AUROC and raw target ECE at full supervision, averaged over the eight foundation models, for each ordered pair of Camelyon17 hospitals. Marginal bars give column and row means. The diagonal is empty because source and target are always distinct. For raw ECE, column (target) means vary about twice as much as row (source) means; for AUROC the two spreads are similar.

Transfer AUROC	$n$	Rising	Rate	Mean $\Delta$ ECE
$[0.00,0.55)$	16	14	88%	$+0.081$
$[0.55,0.70)$	17	15	88%	$+0.089$
$[0.70,0.85)$	7	6	86%	$+0.110$
$[0.85,0.95)$	13	1	8%	$-0.074$
$[0.95,1.00]$	139	2	1%	$-0.172$

Appendix

Table 7: Rate of rising target ECE by transfer AUROC, over all 192 (model, pair) combinations from both transfer designs. The two middle bands contain combinations from both designs.

Pair	Source positive	Target positive
PCam $\to$ MHIST	tumour tissue, lymph node	SSA polyp
MHIST $\to$ PCam	SSA polyp	tumour tissue, lymph node
CRC $\to$ BRACS	TUM	IC, DCIS
BRACS $\to$ CRC	IC, DCIS	TUM

Appendix

Table 8: Binary label mapping for the four transfer pairs. The positive class is defined differently on each side, so these settings combine covariate shift with a change of label definition.

Figure 6: Breaking points. Macro-F1 versus label fraction for each (model, dataset) pair. PCam and CRC are close to saturation (macro-F1 at least 0.80 already at 1% labels); {BRACS, MHIST, LungHist700} expose label-efficiency differences across FM classes. CLIP and PLIP never break F1 = 0.5 on BRACS regardless of label budget.

Figure 7: Composite Reliability Index. CRI per (model, label fraction) cell, aggregating discrimination, calibration and OOD stability with $\mathrm{OOD\_Stability}$ as defined in Equation 1 . VIRCHOW2 leads in both regimes (0.56 at 1% labels, 0.64 at 100%). Two features of the index are worth reading off the figure rather than inferring from it. The spread between the best and worst model is 0.15 at 1% labels and 0.17 at full supervision, so the index separates models less sharply than AUROC alone does. And CLIP places second at 1% labels despite being last on AUROC on five of six datasets, because a ratio-based stability term rewards models that had little in-distribution performance to lose under transfer. We read CRI as a screening heuristic subject to that caveat, not as a ranking.

Figure 8: Single-class collapse of the linear probe. Share of the three seeds in which the probe assigns more than 99% of the test set to one class, per (model, dataset, label fraction). On MHIST every FM collapses at 1% labels and most remain collapsed at 5%; on PCam and CRC collapse never occurs. Stratified sampling does not remove the effect, so it is not purely an artefact of subset selection: at these fractions the probe has enough examples of each class and still does not move its decision threshold off the majority. Where a cell is dark, AUROC may remain informative while macro-F1 and ECE describe a classifier that makes one prediction.

Figure 9: Rank correlation between discrimination and calibration. Left: per-dataset Spearman

\rho

between AUROC rank and calibration rank at 1% and 100% labels, with bootstrap intervals over the eight models. Right: the same quantity pooled over all 48 (model, dataset) cells at each label fraction, with intervals from a cluster bootstrap that resamples whole datasets rather than models, since models within a dataset are not independent. The pooled estimate rises from

-0.24

at 1% labels to

0.52

at full supervision; the contrast between the two regimes is significant (

\Delta\rho=0.62

, 95% CI

[0.08,1.13]

when the bootstrap resamples seeds as well as datasets) even though the individual per-fraction estimates do not all exclude zero on their own.

Figure 10: Raw miscalibration at 1% labels. Raw ECE for every (model, dataset) cell. On SPIDER-Breast raw ECE reaches 0.42–0.47 for UNI, GigaPath and H-Optimus-0 against 0.21 for CLIP. The fitted temperatures in these cells are below 1 (Figure 15 ), so this is under-confidence of the regularized linear probe, not over-confidence, and temperature scaling largely removes it (Figure 11 ). Complementary to Figure 2 : AUROC rankings hide the magnitude of raw miscalibration.

Figure 11: Calibration recoverability via temperature scaling. Improvement in ECE (raw minus scaled) per (model, dataset) cell, plotted against label fraction. Recoverability ranges from 0.45 (H-Optimus-0 on SPIDER-Breast at 1% labels, raw ECE 0.465 to scaled 0.020) to zero or slightly negative (UNI on MHIST at 5% and 10% labels); scaling fails to reduce ECE in 49 of the 288 cells. Practitioners can use this map to anticipate when a single scalar

T

suffices and when post-hoc calibration is inert.

Figure 12: ECE after temperature scaling across label fractions. ECE per dataset as a function of label fraction, after applying the per-cell temperature

T

fitted on the validation split. From 5% labels upward, scaling keeps ECE below 0.1 for every FM on PCam, CRC and SPIDER-Breast and in all but one BRACS cell, but not on MHIST and LungHist700, where 62 of the 80 cells remain above 0.1. Scaling also does not repair the OOD calibration failure documented in Figure 3 .

Figure 13: OOD transfer: AUROC versus source label fraction. Companion to Figure 3 , showing OOD discrimination on the four transfer pairs. On MHIST

\to

PCam, several FMs hold AUROC near random (

0.4

–

0.5

) across the entire 1–100% range while their OOD ECE rises monotonically: confidence increasing without competence.

Figure 14: OOD AUROC drop heatmap. Per (model, transfer pair) drop in AUROC from in-distribution to OOD, at 1% and 100% labels. Drops above 0.4 (e.g., CONCH on CRC $\to$ BRACS at 100% labels) indicate that the model’s discrimination collapses near random under domain shift, even before calibration is considered.

Figure 15: Fitted temperature versus model size. Validation-fitted temperature

T

(on the probe’s own logit scale) against parameter count, at 1% and 100% labels;

T>1

indicates an over-confident probe and

T<1

an under-confident one. The correlation between

\log

size and

T

is negligible (

r=-0.08

at 1% labels,

r=+0.07

at 100%). Task type dominates:

T<1

in 189 of 192 multiclass cells, whereas binary cells straddle

T=1

. On LungHist700 at 1% labels

T

reaches the lower search bound (0.1) for four FMs, so scaled ECE there is bound-limited. This pattern mirrors the absence of a clean “bigger is better” rule in CRI rankings (Figure 7 ).

Figure 16: AUROC versus label fraction. Companion to Figure 6 , showing AUROC instead of macro-F1 across the same six datasets. Top-4 pathology SSL models (VIRCHOW2, UNI, GigaPath, H-Optimus-0) form a tight cluster from 1% labels upward, while CLIP, PLIP, and MIDNIGHT separate visibly on the harder triplet {BRACS, MHIST, LungHist700}. Shaded bands show

\pm 1

standard deviation across the three random seeds.

Figure 17: Bootstrap 95% confidence intervals. AUROC for the four strongest pathology SSL FMs (VIRCHOW2, UNI, GigaPath, H-Optimus-0) across all six datasets, with 1000-resample bootstrap CI bands computed over the three random seeds. With only three seeds these bands describe seed variability, not sampling uncertainty of the test set. Band width collapses at 100% labels because the deterministic LBFGS solver yields zero between-seed variance, as noted in Appendix D .

Figure 18: Reliability radar. Per-model summary at 100% labels across the four reliability axes: AUROC (in-distribution, mean over 6 datasets),

1-\mathrm{ECE}_{\mathrm{raw}}

1-\mathrm{Brier}

, and OOD_Stability (mean OOD AUROC divided by mean in-distribution AUROC across the four transfer pairs, clipped to

[0,1]

). The top-4 panel shows the four large pathology SSL FMs; the bottom-4 panel shows CLIP, PLIP, CONCH, and MIDNIGHT for visual separation.

Figure 19: Performance clustering. Hierarchical clustering of the $8\times 6$ AUROC matrix at 100% labels using $1-\rho$ correlation distance with average linkage on both axes. The dendrogram on the rows separates the four pathology SSL FMs (VIRCHOW2, UNI, GigaPath, H-Optimus-0) from CLIP and the vision-language pathology models (PLIP, CONCH), with MIDNIGHT joining the lighter-pretraining cluster despite its 1.1B parameter count. The grouping confirms that pre-training corpus rather than parameter count is the dominant predictor of in-distribution AUROC ranking on PRISM datasets.

Figure 20: Embedding alignment across foundation models. Mutual $k$ -nearest-neighbor alignment ( $k=50$ , Jaccard overlap of per-sample neighbor sets) between every pair of FMs, computed on each test split. Higher values indicate more similar embedding geometry. Alignment is uniformly low on PCam, BRACS, and CRC, where every FM produces a distinctive neighbor structure, and substantially higher on LungHist700 and SPIDER-Breast, where the four pathology SSL FMs (UNI, GigaPath, H-Optimus-0, VIRCHOW2) form a tightly aligned subgroup. MHIST sits in between, with moderate alignment that mirrors its position as the binary task with the most cross-model disagreement. The diagonal is omitted as self-alignment is identically 1.

Figure 21: Embedding alignment network on BRACS. Force-directed layout of the eight FMs with edge thickness proportional to mutual

k

-nearest-neighbor alignment (

k=50

). The strongest pairwise alignment is between GigaPath and H-Optimus-0, with VIRCHOW2 and UNI joining the same dense central cluster; CLIP and PLIP sit at the periphery, consistent with their non-pathology and Twitter-pathology pre-training corpora respectively. CONCH, despite being a vision-language model, integrates into the central pathology-SSL cluster, indicating that curated pathology image-caption pairs produce embedding geometry closer to the vision-only SSL family than to CLIP or PLIP. The complementary heatmaps for the remaining five datasets are reported in Figure 20 .

Model	PCam	BRACS	CRC	MHIST	LungHist700	SPIDER-Breast
CLIP	0.917 $\pm$ 0.000	0.814	0.998 $\pm$ 0.000	0.854	0.835	0.964 $\pm$ 0.000
PLIP	0.952	0.857	0.999	0.829	0.922	0.982 $\pm$ 0.000
CONCH	0.966 $\pm$ 0.000	0.905	0.999	0.851	0.971	0.988 $\pm$ 0.000
VIRCHOW2	0.986	0.904	1.000	0.916	0.972	0.995
UNI	0.979 $\pm$ 0.000	0.901	1.000	0.886	0.963	0.995
GigaPath	0.980	0.898	1.000	0.887	0.965	0.995 $\pm$ 0.000

Appendix

Table 9: AUROC at 100% label fraction (mean

\pm

std over 3 seeds). Best per dataset in bold.

Model	PCam	BRACS	CRC	MHIST	LungHist700	SPIDER-Breast
CLIP	0.881 $\pm$ 0.001	0.693 $\pm$ 0.035	0.978 $\pm$ 0.001	0.638 $\pm$ 0.047	0.594 $\pm$ 0.027	0.890 $\pm$ 0.006
PLIP	0.942 $\pm$ 0.001	0.728 $\pm$ 0.046	0.996 $\pm$ 0.000	0.704 $\pm$ 0.061	0.762 $\pm$ 0.033	0.945 $\pm$ 0.003
CONCH	0.955 $\pm$ 0.002	0.823 $\pm$ 0.021	0.998 $\pm$ 0.000	0.638 $\pm$ 0.048	0.873 $\pm$ 0.022	0.966 $\pm$ 0.001
VIRCHOW2	0.979 $\pm$ 0.000	0.792 $\pm$ 0.021	0.999 $\pm$ 0.000	0.629 $\pm$ 0.066	0.810 $\pm$ 0.024	0.988 $\pm$ 0.001
UNI	0.982 $\pm$ 0.001	0.786 $\pm$ 0.021	0.999 $\pm$ 0.000	0.693 $\pm$ 0.069	0.679 $\pm$ 0.018	0.987 $\pm$ 0.000
GigaPath	0.982 $\pm$ 0.001	0.748 $\pm$ 0.024	0.999 $\pm$ 0.000	0.670 $\pm$ 0.013	0.655 $\pm$ 0.037	0.987 $\pm$ 0.000

Appendix

Table 10: AUROC at 1% label fraction (mean

\pm

std over 3 seeds). Best per dataset in bold.

Model	PCam	BRACS	CRC	MHIST	LungHist700	SPIDER-Breast
CLIP	0.082 $\pm$ 0.002	0.048 $\pm$ 0.023	0.372 $\pm$ 0.004	0.128 $\pm$ 0.027	0.067 $\pm$ 0.073	0.209 $\pm$ 0.010
PLIP	0.072 $\pm$ 0.004	0.069 $\pm$ 0.042	0.287 $\pm$ 0.009	0.146 $\pm$ 0.025	0.307 $\pm$ 0.051	0.364 $\pm$ 0.008
CONCH	0.052 $\pm$ 0.003	0.162 $\pm$ 0.022	0.191 $\pm$ 0.003	0.119 $\pm$ 0.027	0.331 $\pm$ 0.007	0.316 $\pm$ 0.006
VIRCHOW2	0.062 $\pm$ 0.005	0.158 $\pm$ 0.022	0.182 $\pm$ 0.003	0.139 $\pm$ 0.045	0.294 $\pm$ 0.031	0.378 $\pm$ 0.011
UNI	0.073 $\pm$ 0.002	0.110 $\pm$ 0.033	0.221 $\pm$ 0.005	0.121 $\pm$ 0.027	0.213 $\pm$ 0.032	0.429 $\pm$ 0.006
GigaPath	0.071 $\pm$ 0.000	0.084 $\pm$ 0.055	0.241 $\pm$ 0.003	0.120 $\pm$ 0.019	0.186 $\pm$ 0.048	0.422 $\pm$ 0.005

Appendix

Table 11: ECE (raw, before temperature scaling) at 1% label fraction (mean

\pm

std over 3 seeds). Lower is better. Best per dataset in bold.

Model	PCam	BRACS	CRC	MHIST	LungHist700	SPIDER-Breast
CLIP	0.040 $\pm$ 0.000	0.018 $\pm$ 0.000	0.004 $\pm$ 0.000	0.107 $\pm$ 0.000	0.176 $\pm$ 0.000	0.029 $\pm$ 0.000
PLIP	0.056 $\pm$ 0.000	0.057 $\pm$ 0.000	0.002 $\pm$ 0.000	0.085 $\pm$ 0.000	0.107 $\pm$ 0.000	0.036 $\pm$ 0.000
CONCH	0.037 $\pm$ 0.000	0.025 $\pm$ 0.000	0.002 $\pm$ 0.000	0.085 $\pm$ 0.000	0.104 $\pm$ 0.000	0.025 $\pm$ 0.000
VIRCHOW2	0.035 $\pm$ 0.000	0.050 $\pm$ 0.000	0.002 $\pm$ 0.000	0.062 $\pm$ 0.000	0.102 $\pm$ 0.000	0.027 $\pm$ 0.000
UNI	0.041 $\pm$ 0.000	0.081 $\pm$ 0.000	0.001 $\pm$ 0.000	0.061 $\pm$ 0.000	0.122 $\pm$ 0.000	0.032 $\pm$ 0.000
GigaPath	0.039 $\pm$ 0.000	0.062 $\pm$ 0.000	0.002 $\pm$ 0.000	0.054 $\pm$ 0.000	0.078 $\pm$ 0.000	0.036 $\pm$ 0.000

Appendix

Table 12: ECE after post-hoc temperature scaling at 100% label fraction. Lower is better.

Model	PCam	BRACS	CRC	MHIST	LungHist700	SPIDER-Breast
CLIP	0.118 $\pm$ 0.000	0.095 $\pm$ 0.000	0.010 $\pm$ 0.000	0.168 $\pm$ 0.000	0.110 $\pm$ 0.000	0.024 $\pm$ 0.000
PLIP	0.092 $\pm$ 0.000	0.086 $\pm$ 0.000	0.004 $\pm$ 0.000	0.171	0.077	0.017 $\pm$ 0.000
CONCH	0.074 $\pm$ 0.000	0.072 $\pm$ 0.000	0.003 $\pm$ 0.000	0.161 $\pm$ 0.000	0.052	0.014 $\pm$ 0.000
VIRCHOW2	0.047 $\pm$ 0.000	0.072 $\pm$ 0.000	0.001 $\pm$ 0.000	0.130 $\pm$ 0.000	0.055	0.008 $\pm$ 0.000
UNI	0.053 $\pm$ 0.000	0.074 $\pm$ 0.000	0.001 $\pm$ 0.000	0.142 $\pm$ 0.000	0.062	0.008 $\pm$ 0.000
GigaPath	0.053 $\pm$ 0.000	0.073 $\pm$ 0.000	0.001 $\pm$ 0.000	0.141 $\pm$ 0.000	0.064	0.008 $\pm$ 0.000

Appendix

Table 13: Brier score at 100% label fraction. Lower is better.

Model	PCam	BRACS	CRC	MHIST	LungHist700	SPIDER-Breast
CLIP	1.04	0.74	0.56	0.75	0.36	0.61
PLIP	1.07	0.78	0.65	0.72	0.41	0.64
CONCH	1.04	0.97	0.78	0.88	0.48	0.71
VIRCHOW2	1.24	0.88	0.70	0.61	0.46	0.66
UNI	1.33	0.81	0.66	0.68	0.42	0.63
GigaPath	1.40	0.78	0.65	0.68	0.39	0.62

Appendix

Table 14: Fitted temperature

T

at 100% labels, on the probe’s own logit scale (binary values are the fitted two-column temperature divided by two).

T<1

indicates an under-confident probe,

T>1

an over-confident one; bold marks the value closest to 1 per dataset. Seeds agree exactly at 100% labels.

Model	PCam	BRACS	CRC	MHIST	LungHist700	SPIDER-Breast
CLIP	0.831	0.433	0.950	0.654	0.216	0.647
PLIP	0.872	0.498	0.976	0.642	0.597	0.738
CONCH	0.898	0.589	0.983	0.713	0.701	0.783
VIRCHOW2	0.937	0.596	0.995	0.815	0.760	0.873
UNI	0.935	0.596	0.995	0.773	0.753	0.875
GigaPath	0.934	0.605	0.994	0.785	0.725	0.866

Appendix

Table 15: Macro-F1 at 100% label fraction (seeds agree exactly). Best per dataset in bold.

Model	PCam $\to$ MHIST	MHIST $\to$ PCam	CRC $\to$ BRACS	BRACS $\to$ CRC
CLIP	0.484	0.628	0.661	0.527
PLIP	0.548	0.575	0.613	0.434
CONCH	0.633 $\pm$ 0.000	0.696	0.428	0.240
VIRCHOW2	0.585	0.685 $\pm$ 0.000	0.635	0.746
UNI	0.442	0.589	0.523	0.401
GigaPath	0.588	0.487	0.602	0.602

Appendix

Table 16: OOD AUROC on target dataset, trained at 100% source labels. Best per transfer pair in bold.

Model	PCam $\to$ MHIST	MHIST $\to$ PCam	CRC $\to$ BRACS	BRACS $\to$ CRC
CLIP	0.251	0.312	0.204	0.300
PLIP	0.261	0.207	0.179	0.261
CONCH	0.151	0.455	0.338	0.261
VIRCHOW2	0.123	0.453	0.268	0.138
UNI	0.285	0.439	0.272	0.188
GigaPath	0.288	0.462	0.232	0.278

Appendix

Table 17: OOD ECE (raw, 15 equal-width bins) on the target dataset at 100% source labels (seeds agree exactly). Lower is better.

Computational Pathology

Computational pathology uses algorithms to analyze digitized tissue images for diagnosis, prognosis, biomarker discovery, and clinical decision support.

See all 279 papers

The Good, the Bad, and the Brittle: Benchmarking Robustness and Generalisation of Histopathology Foundation Models
Jul 5, 2026Dhyey Yajnik, Amina Asif, Fayyaz MinhasDistribution Shift Robustness Neural Network Robustness
Predictive Systems in Biomedicine Lab, Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom
SCALE: Synthetic Calibration via Agreement Labeling in Embedding Space
Sep 30, 2026Wenjun Liu, Saeed HassanpourComputational Pathology Pathology Foundation Models
Department of Computer Science Dartmouth College Hanover, NH, USA · Departments of Biomedical Data Science, Computer Science, and Epidemiology Dartmouth College, Hanover, NH, USA
Robustifying pathology foundation models via fine-tuning
Jul 24, 2026Alexandre Filiot, Oskar Thaeter, Benoit Schmauch +1Computational Pathology Fine-Tuning
1Waiv · Institute of Pathology, Technical University of Munich · School of Computation, Information and Technology, Technical University of Munich

Jul 5, 2026J/K move · Enter open · S save

The Good, the Bad, and the Brittle: Benchmarking Robustness and Generalisation of Histopathology Foundation Models

Dhyey Yajnik, Amina Asif, Fayyaz Minhas

Predictive Systems in Biomedicine Lab, Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom

Distribution Shift Robustness Neural Network Robustness Pathology Foundation Models Histopathology Image Classification Cross-Dataset Generalization

Open paper AskPDF arXiv

How robust and generalisable are pathology foundation models and have their scaling limites been reached? We benchmarked twelve pathology foundation models (PFMs) and ResNet baselines using our Robustness Evaluation and Enhancement Toolbox (REET) across eleven clinically realistic perturbations and a dissimilarity-driven Non-Redundant K-fold validation (NR-Kfold) protocol. We introduce a Perturbation Performance Index (PPI) to summarise accuracy trends under controlled perturbation sweeps and analyse robustness scaling with parameter count. We show that PFMs consistently outperform CNNs in both robustness and domain generalisation, yet model scaling shows diminishing returns: mid-sized models such (UNI2/Virchow-2 etc.) achieve comparable or greater resilience than larger systems. NR-Kfold analysis further reveals systematic accuracy loss and increased variability when training-test similarity is broken, underscoring the need for explicit distribution-shift evaluation. These findings suggest that the next generation of pathology foundation models must prioritise data quality, multimodality information and domain alignment over parameter count to achieve genuine clinical reliability.

Abstract

Computational Pathology

Explore similar work

The Good, the Bad, and the Brittle: Benchmarking Robustness and Generalisation of Histopathology Foundation Models

SCALE: Synthetic Calibration via Agreement Labeling in Embedding Space

Robustifying pathology foundation models via fine-tuning