When Do Histology Foundation Models Need Geometric Test-Time Augmentation?
Authors: Lars Chen, Valentino F. Foit, Dorry Segev, Michal Mankowski
Organizations: Department of Surgery, New York University Langone Health, USA · Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Department of Physics, New York University, USA
Test-time augmentation (TTA) aggregates model predictions across multiple transformed views of an input during inference. Because histology images lack a canonical orientation, D4 TTA---pooling across the 8 rotations and reflections of a square---is a standard default in computational pathology. However, the rise of self-supervised histology foundation models (FMs) with inherent rotational invariance calls its continued utility into question. We benchmark 9 histology FMs and 6 general-purpose models across four patch-level and two whole-slide image (WSI) datasets. We find that D4 TTA remains a reliable post hoc improvement, but benefits diminish from +1.98 percentage points (pp) for general-purpose backbones to +0.50 pp for histology FMs on TCGA-UT, and shrink further for the largest models. To eliminate computational overhead, our experiments demonstrate that entropy routing---applying TTA selectively to uncertain predictions---recovers most of the gain while augmenting only ∼13% of samples. Finally, we show that patch-level TTA improvements do not transfer to slide-level multiple instance learning (MIL). Code: https://github.com/lars-chen/histology-tta
Figures & tables
Figure 1: D4 test-time augmentation with selective routing improves histology foundation model classification. (Left) A lightweight classifier is trained using FM-extracted patch embeddings zp . At inference, patch prediction entropy H gates augmentation: low-entropy patches receive a single-view prediction y^p ; high-entropy patches are passed through the D4 symmetry group and predictions are averaged across views. (Top right) D4 TTA improves balanced accuracy across both histology FMs and general-purpose models. Dots show per-model gains, on TCGA-UT, with the mean bar indicating +0.5 pp and +1.98 pp respectively. Applying TTA selectively with entropy thresholds (curves, averaged across models) recovers most of this gain while augmenting only a fraction of patches. (Bottom right) Applied to all patches in an exemplar Camelyon17 slide ( Bandi et al., 2018 ) , full TTA corrects predictions from UNI embeddings. Green patches indicate corrected and blue line shows tumor boundary. Zoomed image shows the overall increase in tumor probability within boundary.
Model
Type
Params
Patches
Arch.
CTransPath †
Histo
28
15
Swin-T
Phikon †
Histo
86
40
ViT-B/16
Phikon-v2 †
Histo
300
456
ViT-L/16
UNI
Histo
300
100
ViT-L/16
UNI-v2
Histo
681
200
ViT-H/14
Virchow
Histo
632
2000
ViT-H/14
Table 1: Models evaluated in this study. Histology FMs are pretrained on histopathology data via self-supervised learning. General-purpose models are pretrained on ImageNet. The equivariant model (D4WRN) has architectural D4 symmetry guarantees. Number of parameters and training patches are reported in millions (M). † TCGA-UT slides may appear in self-supervised pretraining, however without labels (Section 4.1 ).
Figure 2: D4 TTA consistently improves histopathology classifiers, with stronger gains for weaker baselines. (a) Distribution of ΔTTA in balanced accuracy (pp) across models for each dataset, split by model family (histology FMs vs. general-purpose backbones). Points show individual models; boxplots summarize the distribution; D4WRN (equivariant reference) exhibits near-zero gain by construction. (b) Relationship between baseline balanced accuracy and ΔTTA for each dataset: weaker baselines tend to benefit more from TTA.
Figure 3: Prediction entropy explains D4 TTA outcomes. (a) Per model (TCGA-UT; other datasets as offset segments), predictions flipped by D4 TTA split into corrections and corruptions , sorted by ΔTTA . The thick segment is the net effect; the right margin gives the correction/corruption ratio. Models with small ΔTTA flip few predictions. (b) On TCGA-UT, binned by baseline normalized entropy: flips and net gains concentrate in high-entropy predictions, where corrections outpace corruptions. (c) Model-level association (MHIST excluded). Mean entropy is moderately associated with ΔTTA (Spearman ρ=0.54 , p=10−4 ); within datasets ρ=0.89 (TCGA-UT), 0.71 (NCT-CRC-HE-100K) and 0.25 (NCT-CRC-HE-NoNorm, where stain variation dominates; Section 4.8 ). This trend is descriptive; routing rests on the per-patch results in (b) and Table 4 , and is evaluated in Section 4.7 .
Dataset
Classes
Train
Test
Domain
TCGA-UT
31
190k
40.9k
Multi-organ
NCT-CRC-HE-100K
9
100k
7.2k
Colorectal
NCT-CRC-HE-NoNorm
9
100k
7.2k
Colorectal
MHIST
2
2,175
977
Colorectal polyp
Table 2: Datasets used for evaluation. Train and test sizes are tile counts.
ΔTTA (pp)
Dataset
Family
Base (%)
V4
D4
95% CI ( D4 )
Corr./Corrupt.
Agree. (%)
TCGA-UT
Histology FM
73.8
+0.37
+0.50
[+0.34,+0.66]
1.42
95.6
General
46.4
+1.48
+1.98
[+1.21,+2.76]
1.74
83.0
NCT-CRC-HE-100K
Histology FM
94.0
+0.21
+0.29
[−0.14,+0.71]
2.15
98.7
General
89.1
+0.81
+0.93
[+0.50,+1.37]
1.66
96.2
NCT-CRC-HE-NoNorm
Histology FM
86.9
+0.30
+0.30
[−0.21,+0.82]
1.38
97.2
Table 3: Summary of D4 TTA effects by dataset and model family (frozen linear probes, probability-mean aggregation). Base is single-view balanced accuracy; ΔTTA is the mean gain in percentage points from V4 and D4 TTA, with the 95% confidence interval over models for D4 . Corr./Corrupt. is the ratio of predictions TTA fixes to those it breaks, and Agree. the base-vs- D4 prediction agreement rate. General-purpose backbones start lower, agree less across views, and gain more; histology FMs gain little beyond flips.
Figure 4: Selective TTA accuracy–efficiency trade-off on TCGA-UT. Each point corresponds to an entropy threshold t for routing samples to full D4 TTA; lower t applies TTA to more samples (higher cost) and vice versa. Histology FMs recover nearly the full D4 benefit while saving a substantial fraction of forward passes. Thin lines indicate individual model curves, the thick line indicates their mean, and the shaded area indicates the confidence interval.
Predictor
All
Histo FM
General
Orbit position relative to the boundary
Straddle ratio ∣r∣
0.954±0.035
0.974
0.924
Geometric margin ∣m∣
0.947±0.037
0.969
0.914
Softmax entropy (proxy)
0.925±0.054
0.950
0.887
Orbit size
Orbit spread s
0.586±0.056
0.600
0.566
Table 4: AUROC for predicting whether D4 TTA flips a patch prediction. Computed per patch for the 15 frozen linear probes on all four patch datasets (mean ± SD over the 60 model–dataset combinations). Where the orbit sits relative to the decision boundary predicts flips; how large the orbit is does not.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
TCGA-UT
NCT-100K
NCT-NoNorm
MHIST
Type
Model
Base
V4
D4
Base
V4
D4
Base
V4
D4
Base
V4
D4
General-purpose
ResNet-18
37.5
+2.48
+3.23∗∗∗
86.9
+1.06
+1.11∗∗∗
69.5
+0.72
+0.66∗
71.8
+0.94
+1.84∗∗∗
ResNet-50
42.5
+1.92
+2.34∗∗∗
87.4
+1.10
+1.52∗∗∗
80.4
+0.85
+0.82∗∗∗
72.8
+1.27
+2.01∗∗
ConvNeXtV2-Tiny
48.3
+1.31
+1.92∗∗∗
88.8
+0.43
+0.49∗∗
70.4
+0.47
+1.05∗∗∗
76.2
+1.43
+2.48∗∗
ConvNeXtV2-Base
49.6
+1.01
+1.32∗∗∗
90.1
+0.83
+1.22∗∗∗
69.3
+0.20
−0.24∗∗
77.8
+0.43
+0.94
Appendix
Table 5: Per-model TTA results across all four datasets (frozen backbone / SGD linear probe, probability-mean aggregation, 5 seeds). Base : no-TTA balanced accuracy (%). V4 and D4 : change in percentage points from 4-view (flips) and 8-view (full dihedral) TTA respectively. Significance of ΔTTA ( D4 ) vs. zero (two-sided paired t -test, n=5 seeds, uncorrected): ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 . Best ΔTTA ( D4 ) per dataset in bold . Class counts: TCGA-UT 31, NCT-100K / NCT-NoNorm 9, MHIST 2. D4WRN is finetuned (seeds vary by dataset; see † ) and shown as an uncontrolled equivariant reference (see Section 4.1 ).
TCGA-UT (31 cls)
NCT-CRC-100K (9 cls)
NCT-CRC-NoNorm (9 cls)
MHIST (2 cls)
Family
Head
Base
ΔTTA
Base
ΔTTA
Base
ΔTTA
Base
ΔTTA
General-purpose models
kNN ( k =10)
27.6
+0.86
83.9
+0.99
80.3
+1.19
71.9
+0.95
Linear
46.4
+1.98
89.1
+0.93
74.3
+0.64
76.1
+1.56
MLP-1h
48.4
+2.33
89.8
+1.17
73.8
+1.06
76.8
+1.73
MLP-2h
48.3
+2.38
89.6
+1.13
73.5
+1.37
77.3
+1.67
Appendix
Table 6: Mean D4 TTA benefit ( ΔTTA in balanced accuracy, pp) by classifier head type and model family (frozen backbone, probability-mean aggregation). Values averaged over all models within each family and 5 seeds (kNN: 1 seed, deterministic). Base : mean no-TTA balanced accuracy (%). ΔTTA : pp gain from D4 TTA.
Model
Repository
Revision
Access
Reference
Histology foundation models
CTransPath
1aurent/swin_tiny_patch4_window7_224.CTransPath
59c740e
open
Wang et al. (2022)
Phikon
owkin/phikon
057cc02
open
Filiot et al. (2023)
Phikon-v2
owkin/phikon-v2
2ae989a
open
Filiot et al. (2024)
UNI
MahmoodLab/UNI
b55a5ec
agreement
Chen et al. (2024)
UNI-v2
MahmoodLab/UNI2-h
d517a8d
agreement
Chen et al. (2024)
Appendix
Table 7: Pretrained backbones. Revision : commit id, model name, or torchvision weights. Agreement : model requires accepting a license agreement.
Figure 5: Reliability on TCGA-UT, single-view versus D4 TTA. Points below the diagonal indicate overconfidence; grey bars show the share of predictions in each confidence bin. Curves and ECE are computed per model and seed and averaged within each family; bins holding fewer than 0.5% of a model’s predictions are omitted from the curve but retained in the ECE. Histology FMs are overconfident, and D4 TTA moves them toward the diagonal (ECE 0.058→0.049 ); general-purpose backbones are near-calibrated, and TTA pushes them into underconfidence ( 0.011→0.036 ). Histology FMs place 65% of their predictions in the top confidence bin, against 21% for general-purpose backbones.