Enabling clinical use of foundation models for computational pathology
Authors: Audun L Henriksen, Ole-Johan Skrede, Lisa van der Schee, Enric Domingo, Karolina Cyll, Sepp de Raedt, Ilyá Kostolomov, Jennifer Hay, +26 more
Organizations: Institute for Cancer Genetics and Informatics, Oslo University Hospital, Oslo, Norway · Department of Pathology, University Medical Center Utrecht, Utrecht, The Netherlands · Department of Oncology, University of Oxford, Oxford, UK · CRUK Beatson Institute of Cancer Research, Garscube Estate, Glasgow, UK · Glasgow Tissue Research Facility, University of Glasgow, Queen Elizabeth University Hospital, Glasgow, UK · Area for Improvement and Digital Transformation, Norwegian Offshore Directorate, Stavanger, Norway · Pathology Department, Hospital Clínic, Barcelona, Spain · Institut d’Investigacions Biomèdiques August Pi I Sunyer (IDIBAPS), Barcelona, Spain · Department of Clinical Foundations, Universitat de Barcelona, Barcelona, Spain · School of Cancer Sciences, Wolfson Wohl Cancer Research Centre, University of Glasgow, Glasgow, UK · Institute of Clinical Medicine, University of Oslo, Oslo, Norway · Department of Gastrointestinal Surgery, Oslo University Hospital, Oslo, Norway · Gloucestershire Cellular Pathology Laboratory, Cheltenham General Hospital, Cheltenham, UK · Institute of Pathology, University Medical Center, Mainz, Germany · Department of Gastrointestinal Oncology, National Cancer Center Hospital East, Kashiwa, Japan · Translational Research Support Office, National Cancer Center Hospital East, Kashiwa, Japan · Department of Cancer Biology and Pathology, University College London, London, UK · Department of Gastroenterological Surgery, Osaka International Cancer Institute, Osaka, Japan · Centre for Human Genetics, University of Oxford, Oxford, UK · Oxford NIHR Comprehensive Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, John Radcliffe Hospital, Oxford, UK · Nuffield Division of Clinical Laboratory Sciences, University of Oxford, Oxford, UK · Department of Informatics, University of Oslo, Oslo, Norway · Centre for Research-based Innovation Visual Intelligence, UiT The Arctic University of Norway, Tromsø, Norway
Foundation models for computational pathology are expected to facilitate the development of high-performing, generalisable deep learning systems. However, in addition to biologically relevant features, current foundation models also capture pre-analytic and scanner-specific variation that bias the predictions made by downstream task-specific models trained on these features. Here we show that introducing novel robustness losses during downstream model training reduces sensitivity to technical variability. A purpose-designed comprehensive experimentation setup with 27,042 whole-slide images from 6,155 patients is used to train thousands of models from the features of eight well-known foundation models for computational pathology. In addition to a substantial improvement in robustness, our approach improves classification accuracy by focusing on biologically relevant features. It mitigates robustness limitations of foundation models for computational pathology without retraining the foundation models themselves, enabling development of models that are more suitable in real-world clinical use.
Figures & tables
Figure 1: Method overview
Figure 2: Foundation model robustness issue
Figure 3: Survival prediction
Figure 4: Spatial robustness
Figure 5: Robustness in each layer
Figure 6: Prediction of lymph node metastasis
Ahus
Aker
Gloucester
VICTOR
Training
QUASAR 2
TransSCOT
Patient count
58
388
265
427
1138
1112
2856
Age
Years
71
(64–77)
70
(60–77)
68
(62–75)
64
(57–71)
67
(59–75)
65
(59–71)
65
(58–70)
Sex
Female
34
(59%)
195
(50%)
144
(54%)
149
(35%)
522
(46%)
474
(43%)
1126
(39%)
Male
24
(41%)
193
(50%)
121
(46%)
278
(65%)
616
(54%)
638
(57%)
1730
(61%)
Table 1: Baseline characteristics for included cohorts in CRC outcome prediction. Data are given as median (interquartile range) or count (percentage) . Time to event statistics are based only on patients with the respective event. CSD: cancer-specific death.
Aker
Ahus
Gloucester
VICTOR
Mainz
DENEB
Training
Dutch T1
Patient count
69
243
444
282
135
61
1234
290
Age
Years
71
(59–77)
73
(63–79)
70
(64–77)
64
(58–71)
73
(66–79)
61
(52–72)
69
(61–77)
68
(63–74)
Sex
Female
36
(52%)
128
(53%)
182
(41%)
102
(36%)
60
(44%)
31
(51%)
539
(44%)
126
(43%)
Male
33
(48%)
115
(47%)
262
(59%)
180
(64%)
75
(56%)
30
(49%)
695
(56%)
164
(57%)
Table 2: Baseline characteristics for included cohorts in CRC T1 LNM prediction. Data are given as median (interquartile range) or count (percentage) . Time to event statistics are based only on patients with the respective event.
Figure 1: WSI prediction score comparison between tissue sections. Comparison of model score between different tissue sections from the same patient. Each dot represents one tissue section. For each patient, the x-axis is the prediction score on a tissue section scanned at ICGI, on 5 different scanners, and the y-axis is the score of a original tissue section. For each scatterplot, kernel density estimates are shown for both the original tissue sections and the tissue sections scanned at ICGI.
Model
No robustness loss
Embedding loss
Score loss
Both losses
c-index
H-optimus-0
0.661 ± 0.014
0.682 ± 0.013
(6%)
0.689 ± 0.005
(7%)
0.694 ± 0.006
(10%)
H-optimus-1
0.692 ± 0.011
0.689 ± 0.009
(-2%)
0.695 ± 0.004
(1%)
0.699 ± 0.005
(2%)
Hibou-L
0.654 ± 0.012
0.678 ± 0.007
(7%)
0.672 ± 0.005
(6%)
0.680 ± 0.004
(7%)
Phikon-v2
0.654 ± 0.017
0.668 ± 0.014
(4%)
0.666 ± 0.004
(4%)
0.678 ± 0.006
(7%)
Prov-GigaPath
0.662 ± 0.013
0.676 ± 0.008
(1%)
0.673 ± 0.004
(0%)
0.680 ± 0.007
(5%)
UNI
0.681 ± 0.010
0.680 ± 0.012
(0%)
0.670 ± 0.005
(0%)
0.683 ± 0.005
(0%)
Table 3: Performance comparison between different loss terms in survival prediction. Metric summary results are given as mean ± standard deviation ( relative improvement ). For all columns with robustness loss, the best weight ( λ ) is used, and improvement is relative to no robustness loss ( λ = 0).
Model
No robustness loss
Embedding loss
Score loss
Both losses
Balanced accuracy
H-optimus-0
0.611 ± 0.014
0.608 ± 0.014
(0%)
0.647 ± 0.014
(10%)
0.620 ± 0.019
(2%)
H-optimus-1
0.631 ± 0.009
0.638 ± 0.011
(5%)
0.628 ± 0.018
(2%)
0.612 ± 0.026
(-5%)
Hibou-L
0.661 ± 0.013
0.655 ± 0.013
(2%)
0.662 ± 0.013
(6%)
0.688 ± 0.016
(8%)
Phikon-v2
0.618 ± 0.011
0.617 ± 0.014
(6%)
0.625 ± 0.015
(8%)
0.620 ± 0.008
(0%)
Prov-GigaPath
0.635 ± 0.011
0.620 ± 0.009
(0%)
0.654 ± 0.011
(10%)
0.651 ± 0.015
(4%)
UNI
0.636 ± 0.011
0.640 ± 0.014
(8%)
0.653 ± 0.014
(14%)
0.605 ± 0.013
(-8%)
Table 4: Performance comparison between different loss terms in prediction of lymph node metastasis. Metric summary results are given as mean ± standard deviation ( relative improvement ). For all columns with robustness loss, the best weight ( λ ) is used, and improvement is relative to no robustness loss ( λ = 0).
Figure 2: Performance vs robustness weight for outcome prediction. Effect of varying λ in eq. 1 on inconsistency and c-index. Statistics are gathered from 20 models per value of λ per foundation model.
Figure 3: Performance vs robustness weight for LNM prediction. Effect of varying λ in eq. 1 on inconsistency and AUC. Statistics are gathered from 20 models per value of λ per foundation model.
Figure 4: Effect of different robustness loss terms on robustness and classification performance in survival prediction. Box plots show the distribution of metric values for each foundation model when downstream models are trained without robustness loss, with the embedding loss alone, with the score loss alone, or with both loss terms combined. P values indicate comparisons of each loss setting with conventional training without robustness loss.
Figure 5: Effect of different robustness loss terms on robustness and classification performance in prediction of lymph node metastasis. Box plots show the distribution of metric values for each foundation model when downstream models are trained without robustness loss, with the embedding loss alone, with the score loss alone, or with both loss terms combined. P values indicate comparisons of each loss setting with conventional training without robustness loss.
Figure 6: Scanner prediction with linear probing. A single-layer linear classifier was trained to predict which scanner was used to form the input WSI, using tile feature vectors extracted with foundation models. Input is a WSI feature vector that is the average of its tile feature vectors, and the task was to predict whether a WSI had been imaged with one of the following five scanners: Aperio AT2, Aperio GT 450 DX, NanoZoomer XR, KF-PRO-400, Pannoramic 1000. We trained nine classifiers for each of the eight foundation models on WSIs from QUASAR 2 and show the average accuracy over all scanners on WSIs from TransSCOT in this figure. We used PyTorch’s implementation of the LBFGS optimisation algorithm with a learning rate of 1.0, 1000 as the maximum number of iterations, a history size of 50, with strong wolfe as the solver, and CrossEntropyLoss as the loss function. Prior to feeding each feature vector into the linear classifier, we normalise it based on mean and standard deviation calculated on the QUASAR 2 dataset.
Pathology foundation models (FMs) produce powerful tile-level representations which remain sensitive to scanner and staining variability, undermining deployment across laboratories. We develop a novel fine-tuning recipe that improves the robustness of pathology FMs to acquisition factors. Applied to ten different FMs, our fine-tuning strategy consistently improves robustness for every model as well as downstream performance, with no observed trade-off. On average, it raises the PathoROB robustness index by 23% (from 0.72 to 0.87) and increases the overall cross-benchmark performance by 43% on Patho-Bench, HEST and THUNDER combined, with individual gains reaching up to 72% in robustness (Phikon-v2) and 76% in performance (Midnight-12k). We publicly release the fine-tuned versions of Phikon-v2 (Phaet) and Midnight-12k (Mascaret) at https://huggingface.co/wearewaiv/models.
Alexandre Filiot, Oskar Thaeter, Benoit Schmauch +1
1Waiv · Institute of Pathology, Technical University of Munich · School of Computation, Information and Technology, Technical University of Munich
How robust and generalisable are pathology foundation models and have their scaling limites been reached? We benchmarked twelve pathology foundation models (PFMs) and ResNet baselines using our Robustness Evaluation and Enhancement Toolbox (REET) across eleven clinically realistic perturbations and a dissimilarity-driven Non-Redundant K-fold validation (NR-Kfold) protocol. We introduce a Perturbation Performance Index (PPI) to summarise accuracy trends under controlled perturbation sweeps and analyse robustness scaling with parameter count. We show that PFMs consistently outperform CNNs in both robustness and domain generalisation, yet model scaling shows diminishing returns: mid-sized models such (UNI2/Virchow-2 etc.) achieve comparable or greater resilience than larger systems. NR-Kfold analysis further reveals systematic accuracy loss and increased variability when training-test similarity is broken, underscoring the need for explicit distribution-shift evaluation. These findings suggest that the next generation of pathology foundation models must prioritise data quality, multimodality information and domain alignment over parameter count to achieve genuine clinical reliability.
Dhyey Yajnik, Amina Asif, Fayyaz Minhas
Predictive Systems in Biomedicine Lab, Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom
Foundation models trained on large pathology image corpora now provide strong, transferable representations for computational pathology. Over the past few years a series of such models has been released, each trained on more slides than the last; on standard classification and segmentation benchmarks, the leading models are now separated by small margins. In clinical use, however, the foundation model is applied to images from hospitals, scanners, and staining protocols outside its training data. Encoders generally embed these acquisition factors alongside biological information, which may introduce downstream errors and hinder safe clinical adoption. A pathology foundation model should therefore be robust to acquisition shift without giving up representation quality, yet robustness is seldom the axis along which models are compared. In this report, we introduce HERO (Histology Encoder for Robust Representation in Oncology), a ViT-G/14 pathology foundation model trained with the DINO and iBOT objectives and refined with high-resolution Gram anchoring on a morphology-balanced corpus of 500 million tiles from approximately 575,000 clinical whole-slide images. Across the evaluated public benchmarks, HERO shows the strongest robustness to center, scanner, and stain variation among the compared state-of-the-art foundation models, performs comparably on tile-level classification, segmentation, and gene-expression prediction, ranks first on average across 39 evaluated slide-level clinical tasks, and, under an equal-weighted framework-level analysis, has the best average rank across the six benchmark frameworks.