When Do Histology Foundation Models Need Geometric Test-Time Augmentation?
Authors: Lars Chen, Valentino F. Foit, Dorry Segev, Michal Mankowski
Organizations: Department of Surgery, New York University Langone Health, USA · Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Department of Physics, New York University, USA
Test-time augmentation (TTA) aggregates model predictions across multiple transformed views of an input during inference. Because histology images lack a canonical orientation, D4 TTA---pooling across the 8 rotations and reflections of a square---is a standard default in computational pathology. However, the rise of self-supervised histology foundation models (FMs) with inherent rotational invariance calls its continued utility into question. We benchmark 9 histology FMs and 6 general-purpose models across four patch-level and two whole-slide image (WSI) datasets. We find that D4 TTA remains a reliable post hoc improvement, but benefits diminish from +1.98 percentage points (pp) for general-purpose backbones to +0.50 pp for histology FMs on TCGA-UT, and shrink further for the largest models. To eliminate computational overhead, our experiments demonstrate that entropy routing---applying TTA selectively to uncertain predictions---recovers most of the gain while augmenting only ∼13% of samples. Finally, we show that patch-level TTA improvements do not transfer to slide-level multiple instance learning (MIL). Code: https://github.com/lars-chen/histology-tta
Figures & tables
Figure 1: D4 test-time augmentation with selective routing improves histology foundation model classification. (Left) A lightweight classifier is trained using FM-extracted patch embeddings zp . At inference, patch prediction entropy H gates augmentation: low-entropy patches receive a single-view prediction y^p ; high-entropy patches are passed through the D4 symmetry group and predictions are averaged across views. (Top right) D4 TTA improves balanced accuracy across both histology FMs and general-purpose models. Dots show per-model gains, on TCGA-UT, with the mean bar indicating +0.5 pp and +1.98 pp respectively. Applying TTA selectively with entropy thresholds (curves, averaged across models) recovers most of this gain while augmenting only a fraction of patches. (Bottom right) Applied to all patches in an exemplar Camelyon17 slide ( Bandi et al., 2018 ) , full TTA corrects predictions from UNI embeddings. Green patches indicate corrected and blue line shows tumor boundary. Zoomed image shows the overall increase in tumor probability within boundary.
Model
Type
Params
Patches
Arch.
CTransPath †
Histo
28
15
Swin-T
Phikon †
Histo
86
40
ViT-B/16
Phikon-v2 †
Histo
300
456
ViT-L/16
UNI
Histo
300
100
ViT-L/16
UNI-v2
Histo
681
200
ViT-H/14
Virchow
Histo
632
2000
ViT-H/14
Table 1: Models evaluated in this study. Histology FMs are pretrained on histopathology data via self-supervised learning. General-purpose models are pretrained on ImageNet. The equivariant model (D4WRN) has architectural D4 symmetry guarantees. Number of parameters and training patches are reported in millions (M). † TCGA-UT slides may appear in self-supervised pretraining, however without labels (Section 4.1 ).
Figure 2: D4 TTA consistently improves histopathology classifiers, with stronger gains for weaker baselines. (a) Distribution of ΔTTA in balanced accuracy (pp) across models for each dataset, split by model family (histology FMs vs. general-purpose backbones). Points show individual models; boxplots summarize the distribution; D4WRN (equivariant reference) exhibits near-zero gain by construction. (b) Relationship between baseline balanced accuracy and ΔTTA for each dataset: weaker baselines tend to benefit more from TTA.
Figure 3: Prediction entropy explains D4 TTA outcomes. (a) Per model (TCGA-UT; other datasets as offset segments), predictions flipped by D4 TTA split into corrections and corruptions , sorted by ΔTTA . The thick segment is the net effect; the right margin gives the correction/corruption ratio. Models with small ΔTTA flip few predictions. (b) On TCGA-UT, binned by baseline normalized entropy: flips and net gains concentrate in high-entropy predictions, where corrections outpace corruptions. (c) Model-level association (MHIST excluded). Mean entropy is moderately associated with ΔTTA (Spearman ρ=0.54 , p=10−4 ); within datasets ρ=0.89 (TCGA-UT), 0.71 (NCT-CRC-HE-100K) and 0.25 (NCT-CRC-HE-NoNorm, where stain variation dominates; Section 4.8 ). This trend is descriptive; routing rests on the per-patch results in (b) and Table 4 , and is evaluated in Section 4.7 .
Dataset
Classes
Train
Test
Domain
TCGA-UT
31
190k
40.9k
Multi-organ
NCT-CRC-HE-100K
9
100k
7.2k
Colorectal
NCT-CRC-HE-NoNorm
9
100k
7.2k
Colorectal
MHIST
2
2,175
977
Colorectal polyp
Table 2: Datasets used for evaluation. Train and test sizes are tile counts.
ΔTTA (pp)
Dataset
Family
Base (%)
V4
D4
95% CI ( D4 )
Corr./Corrupt.
Agree. (%)
TCGA-UT
Histology FM
73.8
+0.37
+0.50
[+0.34,+0.66]
1.42
95.6
General
46.4
+1.48
+1.98
[+1.21,+2.76]
1.74
83.0
NCT-CRC-HE-100K
Histology FM
94.0
+0.21
+0.29
[−0.14,+0.71]
2.15
98.7
General
89.1
+0.81
+0.93
[+0.50,+1.37]
1.66
96.2
NCT-CRC-HE-NoNorm
Histology FM
86.9
+0.30
+0.30
[−0.21,+0.82]
1.38
97.2
Table 3: Summary of D4 TTA effects by dataset and model family (frozen linear probes, probability-mean aggregation). Base is single-view balanced accuracy; ΔTTA is the mean gain in percentage points from V4 and D4 TTA, with the 95% confidence interval over models for D4 . Corr./Corrupt. is the ratio of predictions TTA fixes to those it breaks, and Agree. the base-vs- D4 prediction agreement rate. General-purpose backbones start lower, agree less across views, and gain more; histology FMs gain little beyond flips.
Figure 4: Selective TTA accuracy–efficiency trade-off on TCGA-UT. Each point corresponds to an entropy threshold t for routing samples to full D4 TTA; lower t applies TTA to more samples (higher cost) and vice versa. Histology FMs recover nearly the full D4 benefit while saving a substantial fraction of forward passes. Thin lines indicate individual model curves, the thick line indicates their mean, and the shaded area indicates the confidence interval.
Predictor
All
Histo FM
General
Orbit position relative to the boundary
Straddle ratio ∣r∣
0.954±0.035
0.974
0.924
Geometric margin ∣m∣
0.947±0.037
0.969
0.914
Softmax entropy (proxy)
0.925±0.054
0.950
0.887
Orbit size
Orbit spread s
0.586±0.056
0.600
0.566
Table 4: AUROC for predicting whether D4 TTA flips a patch prediction. Computed per patch for the 15 frozen linear probes on all four patch datasets (mean ± SD over the 60 model–dataset combinations). Where the orbit sits relative to the decision boundary predicts flips; how large the orbit is does not.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
TCGA-UT
NCT-100K
NCT-NoNorm
MHIST
Type
Model
Base
V4
D4
Base
V4
D4
Base
V4
D4
Base
V4
D4
General-purpose
ResNet-18
37.5
+2.48
+3.23∗∗∗
86.9
+1.06
+1.11∗∗∗
69.5
+0.72
+0.66∗
71.8
+0.94
+1.84∗∗∗
ResNet-50
42.5
+1.92
+2.34∗∗∗
87.4
+1.10
+1.52∗∗∗
80.4
+0.85
+0.82∗∗∗
72.8
+1.27
+2.01∗∗
ConvNeXtV2-Tiny
48.3
+1.31
+1.92∗∗∗
88.8
+0.43
+0.49∗∗
70.4
+0.47
+1.05∗∗∗
76.2
+1.43
+2.48∗∗
ConvNeXtV2-Base
49.6
+1.01
+1.32∗∗∗
90.1
+0.83
+1.22∗∗∗
69.3
+0.20
−0.24∗∗
77.8
+0.43
+0.94
Appendix
Table 5: Per-model TTA results across all four datasets (frozen backbone / SGD linear probe, probability-mean aggregation, 5 seeds). Base : no-TTA balanced accuracy (%). V4 and D4 : change in percentage points from 4-view (flips) and 8-view (full dihedral) TTA respectively. Significance of ΔTTA ( D4 ) vs. zero (two-sided paired t -test, n=5 seeds, uncorrected): ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 . Best ΔTTA ( D4 ) per dataset in bold . Class counts: TCGA-UT 31, NCT-100K / NCT-NoNorm 9, MHIST 2. D4WRN is finetuned (seeds vary by dataset; see † ) and shown as an uncontrolled equivariant reference (see Section 4.1 ).
TCGA-UT (31 cls)
NCT-CRC-100K (9 cls)
NCT-CRC-NoNorm (9 cls)
MHIST (2 cls)
Family
Head
Base
ΔTTA
Base
ΔTTA
Base
ΔTTA
Base
ΔTTA
General-purpose models
kNN ( k =10)
27.6
+0.86
83.9
+0.99
80.3
+1.19
71.9
+0.95
Linear
46.4
+1.98
89.1
+0.93
74.3
+0.64
76.1
+1.56
MLP-1h
48.4
+2.33
89.8
+1.17
73.8
+1.06
76.8
+1.73
MLP-2h
48.3
+2.38
89.6
+1.13
73.5
+1.37
77.3
+1.67
Appendix
Table 6: Mean D4 TTA benefit ( ΔTTA in balanced accuracy, pp) by classifier head type and model family (frozen backbone, probability-mean aggregation). Values averaged over all models within each family and 5 seeds (kNN: 1 seed, deterministic). Base : mean no-TTA balanced accuracy (%). ΔTTA : pp gain from D4 TTA.
Model
Repository
Revision
Access
Reference
Histology foundation models
CTransPath
1aurent/swin_tiny_patch4_window7_224.CTransPath
59c740e
open
Wang et al. (2022)
Phikon
owkin/phikon
057cc02
open
Filiot et al. (2023)
Phikon-v2
owkin/phikon-v2
2ae989a
open
Filiot et al. (2024)
UNI
MahmoodLab/UNI
b55a5ec
agreement
Chen et al. (2024)
UNI-v2
MahmoodLab/UNI2-h
d517a8d
agreement
Chen et al. (2024)
Appendix
Table 7: Pretrained backbones. Revision : commit id, model name, or torchvision weights. Agreement : model requires accepting a license agreement.
Figure 5: Reliability on TCGA-UT, single-view versus D4 TTA. Points below the diagonal indicate overconfidence; grey bars show the share of predictions in each confidence bin. Curves and ECE are computed per model and seed and averaged within each family; bins holding fewer than 0.5% of a model’s predictions are omitted from the curve but retained in the ECE. Histology FMs are overconfident, and D4 TTA moves them toward the diagonal (ECE 0.058→0.049 ); general-purpose backbones are near-calibrated, and TTA pushes them into underconfidence ( 0.011→0.036 ). Histology FMs place 65% of their predictions in the top confidence bin, against 21% for general-purpose backbones.
Test-time augmentation (TTA)--aggregating predictions over multiple augmented copies of a test input--is widely assumed to improve classification accuracy, particularly in medical imaging where it is routinely deployed in production systems and competition solutions. We present a systematic empirical study challenging this assumption across three MedMNIST v2 benchmarks and four architectures spanning three orders of magnitude in parameter count (21K to 11M). Our principal finding is that TTA with standard augmentation pipelines consistently degrades accuracy relative to single-pass inference, with drops as severe as 31.6 percentage points for ResNet-18 on pathology images. This degradation affects all architectures, including convolutional models, and worsens with more augmented views. The sole exception is ResNet-18 on dermatology images, which gains a modest +1.6%. We identify the distribution shift between augmented and training-time inputs--amplified by batch normalization statistics mismatch--as the primary mechanism. Our ablation studies show that augmentation strategy matters critically: intensity-only augmentations preserve more performance than geometric transforms, and including the original unaugmented image partially mitigates but does not eliminate the accuracy drop. These findings serve as a cautionary note for practitioners: TTA should not be applied as a default post-hoc improvement but must be validated on the specific model-dataset combination.
Daniel Nobrega Medeiros
University of Colorado at Boulder, MSc in Artificial Intelligence.
In recent years, foundation models such as CLIP, DINO,and CONCH have demonstrated remarkable domain generalization and unsupervised feature extraction capabilities across diverse imaging tasks. However, systematic and independent evaluations of these models for pixel-level semantic segmentation in histopathology remain scarce. In this study, we propose a robust benchmarking approach to asses 10 foundational models on four histopathological datasets covering both morphological tissue-region and cellular/nuclear segmentation tasks. Our method leverages attention maps of foundation models as pixel-wise features, which are then classified using a machine learning algorithm, XGBoost, enabling fast, interpretable, and model-agnostic evaluation without finetuning. We show that the vision language foundation model, CONCH performed the best across datasets when compared to vision-only foundation models, with PathDino as close second. Further analysis shows that models trained on distinct histopathology cohorts capture complementary morphological representations, and concatenating their features yields superior segmentation performance. Concatenating features from CONCH, PathDino and CellViT outperformed individual models across all the datasets by 7.95% (averaged across the datasets), suggesting that ensembles of foundation models can better generalize to diverse histopathological segmentation tasks.
Lavish Ramchandani, Aashay Tinaikar, Dev Kumar Das +2
Pathology foundation models (FMs) produce powerful tile-level representations which remain sensitive to scanner and staining variability, undermining deployment across laboratories. We develop a novel fine-tuning recipe that improves the robustness of pathology FMs to acquisition factors. Applied to ten different FMs, our fine-tuning strategy consistently improves robustness for every model as well as downstream performance, with no observed trade-off. On average, it raises the PathoROB robustness index by 23% (from 0.72 to 0.87) and increases the overall cross-benchmark performance by 43% on Patho-Bench, HEST and THUNDER combined, with individual gains reaching up to 72% in robustness (Phikon-v2) and 76% in performance (Midnight-12k). We publicly release the fine-tuned versions of Phikon-v2 (Phaet) and Midnight-12k (Mascaret) at https://huggingface.co/wearewaiv/models.
Alexandre Filiot, Oskar Thaeter, Benoit Schmauch +1
1Waiv · Institute of Pathology, Technical University of Munich · School of Computation, Information and Technology, Technical University of Munich