ZACH-ViT: Regime-Dependent Inductive Bias in Compact Vision Transformers for Medical Imaging

Athanasios Angelakis

cs.CVFeb 20, 2026

ZACH-ViT: Regime-Dependent Inductive Bias in Compact Vision Transformers for Medical Imaging

Authors: Athanasios Angelakis

Organizations: BioML Lab, RI CODE, UniBw, Munich, Germany · EDS, Amsterdam UMC, Amsterdam, Netherlands

Abstract

Vision Transformers rely on positional embeddings and class tokens encoding fixed spatial priors. While effective for natural images, these priors may be suboptimal when spatial layout is weakly informative, a frequent condition in medical imaging. We introduce ZACH-ViT (Zero-token Adaptive Compact Hierarchical Vision Transformer), a compact Vision Transformer that removes positional embeddings and the [CLS] token, achieving permutation-invariant patch processing via global average pooling. Zero-token denotes removal of the dedicated aggregation token and positional encodings. Patch tokens remain unchanged. Adaptive residual projections preserve training stability under strict parameter constraints. We evaluate ZACH-ViT across seven MedMNIST datasets under a strict few-shot protocol (50 samples/class, fixed hyperparameters, five seeds). Results reveal regime-dependent behavior: ZACH-ViT (0.25M parameters, trained from scratch) achieves strongest advantage on BloodMNIST and remains competitive on PathMNIST, while relative advantage decreases on datasets with stronger anatomical priors (OCTMNIST, OrganAMNIST), consistent with our hypothesis. Component and pooling ablations show positional support becomes mildly beneficial as spatial structure increases, whereas reintroducing a [CLS] token is consistently unfavorable. These findings support that architectural alignment with data structure can outweigh universal benchmark dominance. Despite minimal size and no pretraining, ZACH-ViT achieves competitive performance under data-scarce conditions, relevant for compact medical imaging and low-resource settings. Code: https://github.com/Bluesman79/ZACH-ViT

Figures & tables

Family	Model	Params (M)	Initialization	Weights (MB)
Scratch Models	ABMIL [ 6 ]	0.09	Random	1.082
	ZACH-ViT (Ours) [ 3 ]	0.25	Random	2.949
	TransMIL [ 4 ]	0.26	Random	3.051
	Minimal-ViT [ 1 ]	0.62	Random	7.458
	CNN-ABMIL [ 6 , 7 ]	1.12	Random	103.205
CNN (ImageNet)	MobileNetV2 [ 8 ]	2.39	ImageNet	27.959

Table 1: Complete list of evaluated models trained under the identical few-shot protocol. Initialization indicates whether ImageNet-pretrained weights were used. Parameter counts are exact. Disk footprint reports saved weight file size measured on BreastMNIST (seed=7, 50-shot, batch size 16, 23 epochs) to approximate deployment-oriented storage cost. Citations correspond to original architecture publications.

Figure 1: Regime spectrum analysis. Each point shows ZACH-ViT’s advantage over the mean performance of other scratch-trained baselines on a dataset. Datasets are ordered by an ordinal spatial-structure strength index (1 = very weak, 5 = strong). The trend suggests that ZACH-ViT’s relative advantage is larger when spatial layout is weakly informative and smaller when anatomical structure is fixed. The downward trend confirms the hypothesis that ZACH-ViT’s permutation-invariant bias is most beneficial when spatial layout is weakly informative.

Figure 2: Global parameter efficiency across all evaluated models. ZACH-ViT competes with substantially larger pretrained architectures despite scratch training.

Figure 3: Parameter efficiency across individual MedMNIST datasets. Each subplot reports model performance versus parameter count within a specific spatial-structure regime. The star indicates ZACH-ViT. Results illustrate regime-dependent efficiency: ZACH-ViT shows strong competitiveness in weak-structure datasets, while pretrained models generally achieve higher performance in strongly structured anatomical regimes.

Dataset	Spatial Strength	ZACH-ViT	ABMIL	CNN-ABMIL	Minimal-ViT	TransMIL
BloodMNIST	1 (weakest)	0.600 $\pm$ 0.071	0.211 $\pm$ 0.050	0.125 $\pm$ 0.067	0.515 $\pm$ 0.086	0.538 $\pm$ 0.065
PathMNIST	2	0.578 $\pm$ 0.041	0.291 $\pm$ 0.084	0.239 $\pm$ 0.028	0.443 $\pm$ 0.061	0.577 $\pm$ 0.048
BreastMNIST	3	0.577 $\pm$ 0.018	0.526 $\pm$ 0.058	0.532 $\pm$ 0.072	0.572 $\pm$ 0.014	0.599 $\pm$ 0.027
PneumoniaMNIST	3	0.707 $\pm$ 0.017	0.688 $\pm$ 0.026	0.711 $\pm$ 0.042	0.607 $\pm$ 0.027	0.674 $\pm$ 0.019
DermaMNIST	4	0.293 $\pm$ 0.029	0.216 $\pm$ 0.023	0.062 $\pm$ 0.058	0.241 $\pm$ 0.033	0.283 $\pm$ 0.024
OCTMNIST	5 (strongest)	0.239 $\pm$ 0.056	0.192 $\pm$ 0.059	0.179 $\pm$ 0.016	0.196 $\pm$ 0.069	0.258 $\pm$ 0.053

Table 2: Test MacroF1/[email protected] (mean

\pm

std) for scratch models across MedMNIST7. Bold indicates best performer per dataset.

Figure 4: Generalization gap (Train-Test) for scratch models. ZACH-ViT exhibits consistently small gaps across datasets.

Figure 5: Inference time versus performance. ZACH-ViT occupies an efficient region of the accuracy-latency trade-off.

Figure 6: Mean rank across all fifteen models over seven MedMNIST datasets. Lower rank indicates better average performance.

Figure 7: Critical Difference diagram (Nemenyi test, $\alpha=0.05$ ) for all models. Connected groups indicate non-significant differences.

Figure 8: Mean rank restricted to scratch-trained models.

Figure 9: Critical Difference diagram (Nemenyi test, $\alpha=0.05$ ) for scratch-trained models.

Variant	Patch	Heads	TU	MLP	Params (M)	Test MacroF1	Test Acc
Baseline	16	8	128-64	128-64	0.25	0.561 $\pm$ 0.051	0.609 $\pm$ 0.054
PS=8	8	8	128-64	128-64	0.17	0.609 $\pm$ 0.038	0.640 $\pm$ 0.035
PS=32	32	8	128-64	128-64	0.54	0.457 $\pm$ 0.034	0.450 $\pm$ 0.037
H=4	16	4	128-64	128-64	0.25	0.579 $\pm$ 0.039	0.609 $\pm$ 0.041
Deeper TU	16	8	128-128-64	128-64	0.33	0.497 $\pm$ 0.095	0.516 $\pm$ 0.102
Wider TU	16	8	256-128	128-64	0.75	0.638 $\pm$ 0.036	0.673 $\pm$ 0.038

Table 3: Hyperparameter sensitivity analysis on BloodMNIST. Test MacroF1 and accuracy are reported as mean

\pm

std across five seeds. Bold indicates statistically significant after correction for multiple comparisons over the baseline (two-tailed paired t-test,

p<0.01

, Bonferroni correction applied across all 11 variant-vs-baseline comparisons). The baseline configuration (PS=16, H=8, TU=[128,64]) represents the primary parameter-efficient setting at 0.25M parameters. All values correspond to the final experimental protocol (23 epochs, five seeds) and replace preliminary exploratory runs used during early development.

Figure 10: Hyperparameter sensitivity on BloodMNIST showing test MacroF1 distributions across five random seeds. Smaller patch sizes generally yield higher performance.

Figure 11: Parameter efficiency versus performance on BloodMNIST. Higher-performing configurations often require increased capacity, while the baseline provides a favorable efficiency-performance trade-off.

Dataset	Variant	Test MacroF1	Test Acc
BloodMNIST	Full ZACH-ViT	0.605 $\pm$ 0.029	0.630 $\pm$ 0.041
	+ Positional	0.568 $\pm$ 0.073	0.592 $\pm$ 0.083
	- Adaptive Residuals	0.612 $\pm$ 0.060	0.644 $\pm$ 0.075
	Random Shuffle	0.573 $\pm$ 0.055	0.594 $\pm$ 0.063
	[CLS] token	0.512 $\pm$ 0.088	0.556 $\pm$ 0.096
PathMNIST	Full ZACH-ViT	0.554 $\pm$ 0.026	0.633 $\pm$ 0.030

Table 4: Component ablation study across BloodMNIST, PathMNIST, and OCTMNIST. Values are reported as mean

\pm

std across five seeds. Bold indicates the best mean Test MacroF1 within each dataset; for Test Accuracy, bold indicates the best mean value within each dataset.

Figure 12: Component ablation study across BloodMNIST, PathMNIST, and OCTMNIST showing Test MacroF1 distributions across five seeds for each architectural variant. The effect of the ablated components is regime-dependent: positional embeddings become more favorable in PathMNIST and OCTMNIST, while the [CLS] token is consistently associated with lower performance.

Figure 13: Change in mean Test MacroF1 relative to the full ZACH-ViT for each ablation variant and dataset. Positive values indicate improvement over the full model and negative values indicate degradation. The figure highlights the mild benefit of positional embeddings in the more structured regimes and the consistently negative effect of the [CLS] token.

Dataset	Pooling Operator	Test MacroF1	Test Acc
BloodMNIST	Global Average Pooling (GAP)	0.605 $\pm$ 0.029	0.630 $\pm$ 0.041
	Attention Pooling	0.584 $\pm$ 0.039	0.603 $\pm$ 0.057
	Global Max Pooling	0.551 $\pm$ 0.080	0.566 $\pm$ 0.091
	[CLS] Token	0.471 $\pm$ 0.089	0.497 $\pm$ 0.105
PathMNIST	Global Average Pooling (GAP)	0.554 $\pm$ 0.039	0.637 $\pm$ 0.030
	Attention Pooling	0.557 $\pm$ 0.039	0.634 $\pm$ 0.042

Table 5: Pooling operator ablation across BloodMNIST, PathMNIST, and OCTMNIST. Values are reported as mean

\pm

std across five seeds. Bold indicates the best mean Test MacroF1 within each dataset; for Test Accuracy, bold indicates the best mean value within each dataset.

Figure 14: Pooling operator ablation across BloodMNIST, PathMNIST, and OCTMNIST showing Test MacroF1 distributions across five seeds for each aggregation strategy. GAP is the strongest default overall, attention pooling becomes mildly competitive in the more structured regimes, and [CLS] pooling is consistently associated with the lowest performance.

Figure 15: Change in mean Test MacroF1 relative to GAP for each pooling variant and dataset. Positive values indicate improvement over GAP and negative values indicate degradation. The figure highlights the small gains of attention pooling in the more structured regimes and the consistently negative effect of max and [CLS] pooling.

Vision Transformer

Vision Transformers (ViTs) use self-attention over image patches to learn visual representations for recognition, segmentation, and other tasks.

See all 352 papers

Extending ZACH-ViT to Robust Medical Imaging: Corruption and Adversarial Stress Testing in Low-Data Regimes
Apr 7, 2026Athanasios Angelakis, Marta Gomez-BarreroVision Transformer Medical Image Classification
BioML Lab, RI CODE, UniBw, Munich, Germany · AUMC, Amsterdam, Netherlands
hZACH-ViT: Curved Latent Geometry for Compact Vision Transformers in Low-Data Medical Imaging
May 30, 2026Athanasios AngelakisGeometric Representation Learning Efficient ViTs
BioML Lab, Research Institute CODE, UniBw, Munich, Germany · Department of Epidemiology and Data Science, Amsterdam UMC, Amsterdam, Netherlands
Spectral Vision Transformer for Efficient Tokenization with Limited Data
May 12, 2026Alexandra G. Roberts, Maneesh John, Jinwei Zhang +10Medical Imaging Image Tokenization

Apr 7, 2026J/K move · Enter open · S save

Extending ZACH-ViT to Robust Medical Imaging: Corruption and Adversarial Stress Testing in Low-Data Regimes

Athanasios Angelakis, Marta Gomez-Barrero

BioML Lab, RI CODE, UniBw, Munich, Germany · AUMC, Amsterdam, Netherlands

Vision Transformer Medical Image Classification Image Corruption Robustness Adversarial Robustness Medical Image Benchmarks

Open paper AskPDF arXiv

The recently introduced ZACH-ViT (Zero-token Adaptive Compact Hierarchical Vision Transformer) formalized a compact permutation-invariant Vision Transformer for medical imaging and argued that architectural alignment with spatial structure can matter more than universal benchmark dominance. Its design was motivated by the observation that positional embeddings and a dedicated class token encode fixed spatial assumptions that may be suboptimal when spatial organization is weakly informative, locally distributed, or variable across biomedical images. The foundational study established a regime-dependent clean performance profile across MedMNIST, but did not examine robustness in detail. In this work, we present the first robustness-focused extension of ZACH-ViT by evaluating its behavior under common image corruptions and adversarial perturbations in the same low-data setting. We compare ZACH-ViT with three scratch-trained compact baselines, ABMIL, Minimal-ViT, and TransMIL, on seven MedMNIST datasets using 50 samples per class, fixed hyperparameters, and five random seeds. Across the benchmark, ZACH-ViT achieves the best overall mean rank on clean data (1.57) and under common corruptions (1.57), indicating a favorable balance between baseline predictive performance and robustness to realistic image degradation. Under adversarial stress, all models deteriorate substantially; nevertheless, ZACH-ViT remains competitive, ranking first under FGSM (2.00) and second under PGD (2.29), where ABMIL performs best overall. These results extend the original ZACH-ViT narrative: the advantages of compact permutation-invariant transformers are not limited to clean evaluation, but can persist under realistic perturbation stress in low-data medical imaging, while adversarial robustness remains an open challenge for all evaluated models.

Abstract

Vision Transformer

Explore similar work

Extending ZACH-ViT to Robust Medical Imaging: Corruption and Adversarial Stress Testing in Low-Data Regimes

hZACH-ViT: Curved Latent Geometry for Compact Vision Transformers in Low-Data Medical Imaging

Spectral Vision Transformer for Efficient Tokenization with Limited Data