I Can't Believe TTA Is Not Better: When Test-Time Augmentation Hurts Medical Image Classification

Daniel Nobrega Medeiros

cs.CVApr 6, 2026

I Can't Believe TTA Is Not Better: When Test-Time Augmentation Hurts Medical Image Classification

Authors: Daniel Nobrega Medeiros

Organizations: University of Colorado at Boulder, MSc in Artificial Intelligence.

Abstract

Test-time augmentation (TTA)--aggregating predictions over multiple augmented copies of a test input--is widely assumed to improve classification accuracy, particularly in medical imaging where it is routinely deployed in production systems and competition solutions. We present a systematic empirical study challenging this assumption across three MedMNIST v2 benchmarks and four architectures spanning three orders of magnitude in parameter count (21K to 11M). Our principal finding is that TTA with standard augmentation pipelines consistently degrades accuracy relative to single-pass inference, with drops as severe as 31.6 percentage points for ResNet-18 on pathology images. This degradation affects all architectures, including convolutional models, and worsens with more augmented views. The sole exception is ResNet-18 on dermatology images, which gains a modest +1.6%. We identify the distribution shift between augmented and training-time inputs--amplified by batch normalization statistics mismatch--as the primary mechanism. Our ablation studies show that augmentation strategy matters critically: intensity-only augmentations preserve more performance than geometric transforms, and including the original unaugmented image partially mitigates but does not eliminate the accuracy drop. These findings serve as a cautionary note for practitioners: TTA should not be applied as a default post-hoc improvement but must be validated on the specific model-dataset combination.

Figures & tables

Model	Parameters	Architecture
LogReg	$\sim$ 21K	Linear on flattened pixels
MLP	$\sim$ 670K	2-layer, 256 hidden, ReLU, dropout
SmallCNN	$\sim$ 95K	3-layer CNN, BatchNorm, MaxPool
ResNet-18	$\sim$ 11M	Adapted for $28\times 28$ (no initial pool)

Table 1: Model architectures.

Dataset	Model	Params	Base Acc	TTA@50	$\Delta$
PathMNIST	LogReg	21K	0.513	0.460	$-$ 5.3%
	MLP	670K	0.504	0.499	$-$ 0.6%
	SmallCNN	95K	0.881	0.596	$-$ 28.6%
	ResNet-18	11M	0.870	0.554	$-$ 31.6%
DermaMNIST	LogReg	16K	0.673	0.669	$-$ 0.5%
	MLP	670K	0.684	0.684	$-$ 0.1%

Table 2: Baseline accuracy vs. TTA accuracy at

N=50

views (mixed augmentation, mean aggregation).

\Delta

shows the change from baseline. Bold indicates the only improvement. TTA hurts in 11/12 cases.

Figure 1: TTA scaling curves across models and datasets. Dotted lines show single-pass baselines. For most combinations, accuracy drops sharply at $N=1$ (a single augmented view replaces the clean input) and fails to recover even at $N=100$ , indicating systematic distribution shift rather than variance.

Figure 2: TTA accuracy gain vs. model parameter count at $N=50$ . All gains are negative. The largest models (ResNet-18) and mid-sized convolutional models (SmallCNN) suffer the most, not the smallest.

Figure 3: ECE vs. number of TTA views. While TTA sometimes improves calibration for poorly-calibrated models (ResNet-18 on DermaMNIST, where ECE drops from 0.196 to 0.039), it frequently worsens calibration for already well-calibrated models (SmallCNN on BloodMNIST, ECE increases from 0.011 to 0.136).

Figure 4: Test accuracy by augmentation strategy at $N=25$ . Intensity-only augmentations preserve significantly more accuracy than geometric or mixed augmentations, particularly for high-performing convolutional models. On BloodMNIST, intensity-only TTA achieves 0.901 for ResNet-18 vs. 0.854 for mixed.

Figure 5: Aggregation method comparison at $N=25$ (mixed strategy). The choice of aggregation method has minimal impact compared to the choice of augmentation strategy. All three methods produce similar accuracy.

Figure 6: Compute-accuracy tradeoff. More inference compute (more TTA views) provides worse accuracy for most model-dataset combinations, inverting the expected relationship.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Effect of including the original image and BatchNorm adaptation on TTA performance. Dashed lines: single-pass baselines. Including the original image (orange) helps at low $N$ but converges to augmentation-only (blue) at high $N$ . BN adaptation (green) helps on BloodMNIST but not on PathMNIST.

Explore similar work

SegTTA: Training-Free Test-Time Augmentation for Zero-Shot Medical Imaging Segmentation
Apr 19, 2026Yihong Yao, Chunlei Li, Canxuan Gang +4Image Segmentation Medical Imaging Foundation Models
AI Geeks · Qingdao Municipal Hospital · University of Chinese Academy of Sciences
Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation
Jul 20, 2026Lingrui Li, Nan Pu, Dong Zhao +4Memory-Augmented VLMs VLM Adaptation
School of Computer Science, University of Nottingham, UK · School of Computer Science and Information Engineering, Hefei University of Technology, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore
Long-Tailed Medical Image Classification
Jul 26, 2026Nathanael Ren, Saagar AryaClass-Imbalanced Learning Long-Tailed Classification
Pratt School of Engineering Duke University Durham, NC 27708, USA

Apr 19, 2026J/K move · Enter open · S save

SegTTA: Training-Free Test-Time Augmentation for Zero-Shot Medical Imaging Segmentation

Yihong Yao, Chunlei Li, Canxuan Gang, Wenzhi Hu+3

AI Geeks · Qingdao Municipal Hospital · University of Chinese Academy of Sciences

Image Segmentation Medical Imaging Foundation Models Medical Image Segmentation Test-Time Augmentation Zero-Shot Medical Image Segmentation

Open paper AskPDF arXiv

Increasingly advanced data augmentation techniques have greatly aided clinical medical research, increasing data diversity and improving model generalization capabilities. Although most current basic models exhibit strong generalization abilities, image quality varies due to differences in equipment and operators. To address these challenges, we present SegTTA, a framework that improves medical image segmentation without model retraining by combining four augmentations (Gamma correction, Contrast enhancement, Gaussian blur, Gaussian noise) with weighted voting across multiple MedSAM2 checkpoints. Experiments demonstrate consistent improvements across three diverse datasets: healthy uterus segmentation, uterine myoma detection, and multi class hepatic structure segmentation. Ablation studies reveal that large organs benefit from intensity augmentations while small lesions require noise augmentations. The voting threshold controls the coverage precision trade off, enabling task specific optimization for different clinical requirements. Ultimately, on a multiclass hepatic vessel dataset, compared to MedSAM2 baselines, our method achieves an increase of 1.6 in mIoU and 1.9 in aIoU, along with a reduction of approximately 2.0 in HD95. Code will be available at https://github.com/AIGeeksGroup/SegTTA.

Abstract

Explore similar work

SegTTA: Training-Free Test-Time Augmentation for Zero-Shot Medical Imaging Segmentation

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

Long-Tailed Medical Image Classification