Extending ZACH-ViT to Robust Medical Imaging: Corruption and Adversarial Stress Testing in Low-Data Regimes

Athanasios Angelakis; Marta Gomez-Barrero

cs.CVApr 7, 2026

Extending ZACH-ViT to Robust Medical Imaging: Corruption and Adversarial Stress Testing in Low-Data Regimes

Authors: Athanasios Angelakis, Marta Gomez-Barrero

Organizations: BioML Lab, RI CODE, UniBw, Munich, Germany · AUMC, Amsterdam, Netherlands

Abstract

The recently introduced ZACH-ViT (Zero-token Adaptive Compact Hierarchical Vision Transformer) formalized a compact permutation-invariant Vision Transformer for medical imaging and argued that architectural alignment with spatial structure can matter more than universal benchmark dominance. Its design was motivated by the observation that positional embeddings and a dedicated class token encode fixed spatial assumptions that may be suboptimal when spatial organization is weakly informative, locally distributed, or variable across biomedical images. The foundational study established a regime-dependent clean performance profile across MedMNIST, but did not examine robustness in detail. In this work, we present the first robustness-focused extension of ZACH-ViT by evaluating its behavior under common image corruptions and adversarial perturbations in the same low-data setting. We compare ZACH-ViT with three scratch-trained compact baselines, ABMIL, Minimal-ViT, and TransMIL, on seven MedMNIST datasets using 50 samples per class, fixed hyperparameters, and five random seeds. Across the benchmark, ZACH-ViT achieves the best overall mean rank on clean data (1.57) and under common corruptions (1.57), indicating a favorable balance between baseline predictive performance and robustness to realistic image degradation. Under adversarial stress, all models deteriorate substantially; nevertheless, ZACH-ViT remains competitive, ranking first under FGSM (2.00) and second under PGD (2.29), where ABMIL performs best overall. These results extend the original ZACH-ViT narrative: the advantages of compact permutation-invariant transformers are not limited to clean evaluation, but can persist under realistic perturbation stress in low-data medical imaging, while adversarial robustness remains an open challenge for all evaluated models.

Figures & tables

Figure 1: Clean performance and corruption-averaged robustness across the seven MedMNIST datasets. The clean performance pattern reported in the foundational ZACH-ViT study is largely preserved, and the corruption analysis further shows that ZACH-ViT provides the most favorable overall balance between baseline predictive performance and robustness to realistic image degradation.

Figure 2: Adversarial stress testing under FGSM and PGD. All models degrade substantially relative to the clean setting. ZACH-ViT remains competitive and is the best-ranked transformer-based baseline on average under PGD, but no architecture is consistently best across all datasets.

Figure 3: Representative corruption-specific severity plots from the benchmark outputs. The selected panels illustrate that the favorable corruption profile of ZACH-ViT is visible across multiple corruption families, including Gaussian noise, Gaussian blur, brightness-contrast adjustment, JPEG compression, and cutout.

Figure 4: Dataset-agnostic summary of robustness behavior. Left: mean rank across clean, corruption, FGSM, and PGD regimes (lower is better). Right: average retention relative to each model’s own clean performance. ZACH-ViT is the most balanced model overall, while ABMIL shows the strongest proportional retention under adversarial stress.

	Clean				Common corruptions
Dataset	ABMIL	Minimal-ViT	TransMIL	ZACH-ViT	ABMIL	Minimal-ViT	TransMIL	ZACH-ViT
BloodMNIST	0.232 $\pm$ 0.053	0.502 $\pm$ 0.079	0.634 $\pm$ 0.042	0.597 $\pm$ 0.025	0.214 $\pm$ 0.033	0.465 $\pm$ 0.073	0.596 $\pm$ 0.040	0.559 $\pm$ 0.023
PathMNIST	0.257 $\pm$ 0.028	0.432 $\pm$ 0.066	0.523 $\pm$ 0.067	0.461 $\pm$ 0.054	0.253 $\pm$ 0.031	0.398 $\pm$ 0.049	0.501 $\pm$ 0.097	0.443 $\pm$ 0.054
BreastMNIST	0.706 $\pm$ 0.071	0.611 $\pm$ 0.015	0.633 $\pm$ 0.016	0.642 $\pm$ 0.010	0.699 $\pm$ 0.064	0.611 $\pm$ 0.014	0.630 $\pm$ 0.011	0.639 $\pm$ 0.006
PneumoniaMNIST	0.804 $\pm$ 0.019	0.681 $\pm$ 0.031	0.808 $\pm$ 0.014	0.813 $\pm$ 0.012	0.753 $\pm$ 0.019	0.681 $\pm$ 0.029	0.774 $\pm$ 0.012	0.775 $\pm$ 0.010
DermaMNIST	0.202 $\pm$ 0.028	0.211 $\pm$ 0.029	0.298 $\pm$ 0.044	0.301 $\pm$ 0.035	0.188 $\pm$ 0.023	0.188 $\pm$ 0.025	0.259 $\pm$ 0.042	0.272 $\pm$ 0.035

Table 1: Clean and corruption-averaged performance across the seven MedMNIST datasets. The reported metric is AUC for binary tasks and Macro-F1 for multiclass tasks. Values are mean

\pm

standard deviation over five seeds after averaging across corruption settings within each seed. We note that absolute differences between top-performing models are often small and should be interpreted with caution in the absence of formal hypothesis testing.

	FGSM				PGD
Dataset	ABMIL	Minimal-ViT	TransMIL	ZACH-ViT	ABMIL	Minimal-ViT	TransMIL	ZACH-ViT
BloodMNIST	0.110 $\pm$ 0.005	0.170 $\pm$ 0.015	0.134 $\pm$ 0.021	0.152 $\pm$ 0.018	0.102 $\pm$ 0.006	0.150 $\pm$ 0.015	0.108 $\pm$ 0.014	0.125 $\pm$ 0.020
PathMNIST	0.084 $\pm$ 0.010	0.162 $\pm$ 0.010	0.142 $\pm$ 0.026	0.160 $\pm$ 0.037	0.079 $\pm$ 0.008	0.139 $\pm$ 0.011	0.111 $\pm$ 0.021	0.127 $\pm$ 0.036
BreastMNIST	0.086 $\pm$ 0.064	0.094 $\pm$ 0.021	0.111 $\pm$ 0.056	0.103 $\pm$ 0.038	0.082 $\pm$ 0.061	0.071 $\pm$ 0.019	0.079 $\pm$ 0.050	0.071 $\pm$ 0.031
PneumoniaMNIST	0.316 $\pm$ 0.070	0.153 $\pm$ 0.055	0.301 $\pm$ 0.061	0.282 $\pm$ 0.015	0.309 $\pm$ 0.068	0.141 $\pm$ 0.051	0.242 $\pm$ 0.047	0.231 $\pm$ 0.011
DermaMNIST	0.074 $\pm$ 0.026	0.050 $\pm$ 0.021	0.089 $\pm$ 0.011	0.091 $\pm$ 0.015	0.073 $\pm$ 0.026	0.043 $\pm$ 0.018	0.070 $\pm$ 0.011	0.074 $\pm$ 0.015

Table 2: FGSM- and PGD-averaged adversarial performance across the seven MedMNIST datasets. Values are mean

\pm

standard deviation over five seeds after averaging across attack strengths within each seed. FGSM uses

\epsilon\in\{1/255,2/255,4/255,8/255\}

. PGD uses the same

\epsilon

values, 10 steps, and step size

\epsilon/4

. We note that absolute differences between top-performing models are often small and should be interpreted with caution in the absence of formal hypothesis testing.

Explore similar work

Feb 20, 2026

ZACH-ViT: Regime-Dependent Inductive Bias in Compact Vision Transformers for Medical Imaging

Vision Transformers rely on positional embeddings and class tokens encoding fixed spatial priors. While effective for natural images, these priors may be suboptimal when spatial layout is weakly informative, a frequent condition in medical imaging. We introduce ZACH-ViT (Zero-token Adaptive Compact Hierarchical Vision Transformer), a compact Vision Transformer that removes positional embeddings and the [CLS] token, achieving permutation-invariant patch processing via global average pooling. Zero-token denotes removal of the dedicated aggregation token and positional encodings. Patch tokens remain unchanged. Adaptive residual projections preserve training stability under strict parameter constraints. We evaluate ZACH-ViT across seven MedMNIST datasets under a strict few-shot protocol (50 samples/class, fixed hyperparameters, five seeds). Results reveal regime-dependent behavior: ZACH-ViT (0.25M parameters, trained from scratch) achieves strongest advantage on BloodMNIST and remains competitive on PathMNIST, while relative advantage decreases on datasets with stronger anatomical priors (OCTMNIST, OrganAMNIST), consistent with our hypothesis. Component and pooling ablations show positional support becomes mildly beneficial as spatial structure increases, whereas reintroducing a [CLS] token is consistently unfavorable. These findings support that architectural alignment with data structure can outweigh universal benchmark dominance. Despite minimal size and no pretraining, ZACH-ViT achieves competitive performance under data-scarce conditions, relevant for compact medical imaging and low-resource settings. Code: https://github.com/Bluesman79/ZACH-ViT

Athanasios Angelakis

BioML Lab, RI CODE, UniBw, Munich, Germany · EDS, Amsterdam UMC, Amsterdam, Netherlands

Vision Transformer Efficient ViTs Medical Imaging

Ask PDF

May 30, 2026

hZACH-ViT: Curved Latent Geometry for Compact Vision Transformers in Low-Data Medical Imaging

Compact Vision Transformers are attractive for medical imaging in low-data and resource-constrained settings, but most existing variants assume that Euclidean latent geometry is sufficient for organizing image representations. We introduce hZACH-ViT, a family of curved-geometry extensions of ZACH-ViT, a compact zero-token Vision Transformer that removes positional embeddings and the class token and relies on global average pooling over patch representations. To isolate the role of geometry, we preserve the verified ZACH-ViT backbone and modify only the final representation space and prototype-based classifier head, enabling a controlled comparison between Euclidean, hyperbolic, and spherical latent geometries. We evaluate Poincaré, Klein, and spherical hZACH-ViT heads on seven MedMNIST datasets under an identical few-shot protocol with 50 samples per class and five random seeds. The completed benchmark contains 770 training runs spanning seven datasets, three non-Euclidean geometries, seven curvature magnitudes, and a Euclidean baseline. Across all seven datasets, the best non-Euclidean hZACH-ViT configuration improves over Euclidean ZACH-ViT, with an average gain of +0.021 in the dataset-specific primary metric and the largest improvement on OCTMNIST (+0.055 MacroF1). Fixed low-curvature configurations retain positive gains on the majority of datasets, and low curvature values (c = 0.1 or 0.2) account for six of the seven dataset-level winners. Rather than identifying a universally optimal manifold, our results establish geometry and curvature as dataset-dependent model-selection variables, with fixed low-curvature analyses confirming that gains persist beyond exhaustive per-dataset tuning.

Athanasios Angelakis

BioML Lab, Research Institute CODE, UniBw, Munich, Germany · Department of Epidemiology and Data Science, Amsterdam UMC, Amsterdam, Netherlands

Geometric Representation Learning Efficient ViTs Medical Image Representation Learning

Ask PDF

May 18, 2026

MedFM-Robust: Benchmarking Robustness of Medical Foundation Models

Medical foundation models have achieved remarkable clinical performance, yet their robustness under real-world perturbations remains underexplored. We present a robustness benchmark comprising 40 perturbation types (12 base, 28 medical-specific) across eight imaging modalities, evaluating five VLMs (LLaVA-Med, MedGemma, MedGemma-1.5, Gemini-2.5-flash and GPT-4o-mini) on VQA, visual grounding, and captioning, alongside two segmentation models (MedSAM, SAM-Med2D) with five fine-tuning strategies. Our findings reveal: (1) Fine-tuning strategy dominates robustness, with LoRA exhibiting nearly double the degradation of full fine-tuning, while SAM-Med2D's Adapter offers favorable efficiency-robustness trade-off. (2) Medical-specific perturbations disproportionately damage segmentation, with 9 of 15 top corruptions being domain-specific. (3) LoRA-tuned visual grounding drops over 40 points, whereas zero-shot captioning remains stable (<7% drop). Zero-shot VQA shows model-dependent robustness--medical models drop under 20% while Gemini-2.5-flash drops 54%. General-purpose VLMs achieve higher VQA accuracy but fail on grounding; among medical VLMs, MedGemma demonstrates the best overall stability. These results provide deployment guidelines and underscore the necessity of domain-specific robustness evaluation for medical AI. Our code is available at: https://abnerai.github.io/MedFM-Robust.

Xiangxiang Cui, Tianjin Huang, Yifang Wang +2

Beijing Normal University, China · University of Exeter, United Kingdom · University College London, United Kingdom +2

Multimodal Robustness Neural Network Robustness Vision Foundation Model Adaptation Medical VLMs Medical Image Segmentation Medical Image Benchmarks

Ask PDF