SAP-ViT: A Self-Attributing Patch Vision Transformer for Medical Image Classification

Wei Zhang; Wolfgang Nejdl; Ming Tang

cs.AIOct 26, 2026

SAP-ViT: A Self-Attributing Patch Vision Transformer for Medical Image Classification

Authors: Wei Zhang, Wolfgang Nejdl, Ming Tang

Organizations: L3S Research Center, Leibniz University Hannover, 30167 Hannover, Germany

Abstract

Vision Transformer (ViT) classifiers commonly rely on a global classification (CLS) token for image-level prediction, which is poorly aligned with local interpretability requirements in medical imaging. We propose SAP-ViT, a Self-Attributing Patch Vision Transformer that replaces CLS-based classification with direct aggregation of last-layer patch embeddings. This design makes class predictions decomposable into patch-level contributions as part of the forward pass. To encourage concise and readable evidence, SAP-ViT applies sparsity regularization to the patch-weight distribution. Since self-attention mixes information across patch tokens, we further introduce an input-space back-mapping mechanism to estimate which input patches contribute to influential output patch embeddings. We evaluate SAP-ViT qualitatively and quantitatively on multiple medical image classification datasets spanning different imaging modalities. The results show that SAP-ViT maintains strong classification performance while providing consistent and interpretable patch-level explanations across datasets. Ablation results further show that both sparsity regularization and the alignment component of input-space back-mapping contribute to the final explanation quality. Overall, SAP-ViT provides a favorable trade-off between predictive accuracy and interpretability for ViT-based medical image classification.

Figures & tables

Fig. 1: Overview of SAP-ViT. The input image is divided into patches and processed by the ViT to obtain contextualized patch embeddings. SAP-ViT learns a weight for each output patch, while sparsity regularization encourages the model to focus on a compact set of informative patches. The weighted patch embeddings are additively aggregated and passed to a linear classification head to obtain the predicted class. Because self-attention mixes information across patches, an influential last-layer patch embedding $e_{N}^{k}$ may contain both local information from its corresponding input patch embedding $e_{0}^{k}$ and contextual information from other image regions, represented by the latent context embedding $\hat{e}^{k}$ . This motivates input-space back-mapping, which uses the pair $(e_{0}^{k},e_{N}^{k})$ to estimate the contextual representation $\hat{e}^{k}$ through a conditional VAE. The recovered context representation is compared with all input patch embeddings, and the resulting matching maps for the top- $K$ output patches are combined to produce an approximate input-space attribution map.

Dataset	Modality	# Classes	# Images
Brain tumor MRI	MRI	4	7023
COVID chest X-ray	Chest X-ray	4	21165
Pneumonia chest X-ray	Chest X-ray	2	5856
Breast ultrasound	Ultrasound	3	780

TABLE I: Dataset statistics.

Fig. 2: Qualitative attribution comparison on a Brain Tumor MRI example. The ground-truth label is meningioma. SAP-ViT produces a compact attribution map concentrated around the tumor region, while the baseline methods generate either less localized or more diffuse saliency patterns across surrounding anatomical structures and background regions.

Dataset	Entropy norm	Gini
Brain tumor MRI	$0.901$	$0.476$
COVID chest X-ray	$0.949$	$0.333$
Pneumonia chest X-ray	$0.953$	$0.322$
Breast ultrasound	$0.705$	$0.488$

TABLE IV: Model-independent patch-level radiomics-style concentration analysis. Entropy norm measures the spatial spread of patch-level texture responses, where higher values indicate more diffuse responses. Gini measures concentration, where higher values indicate stronger spatial concentration.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 3: Qualitative attribution comparison on a chest radiograph from the COVID chest X-ray dataset with the ground-truth label lung opacity. SAP-ViT produces a compact attribution map concentrated in an upper-lung region exhibiting visible opacity, whereas the baseline methods generate less localized or more diffuse saliency patterns across the lungs, surrounding anatomical structures, and background.

Fig. 4: Qualitative attribution comparison on an example from the pneumonia chest X-ray dataset with the ground-truth label pneumonia. SAP-ViT produces a compact attribution map concentrated in an upper-lung region exhibiting visible opacity, whereas the baseline methods generate less localized or more diffuse saliency patterns across the lungs, surrounding anatomical structures, and background.

Fig. 5: Qualitative attribution comparison on a breast ultrasound example. The ground-truth label is malignant. SAP-ViT produces a compact attribution map concentrated around the lesion, with its strongest hotspot overlapping the boundary of the supplied tumor mask. In contrast, the baseline methods generate either less localized or more diffuse saliency patterns across the surrounding tissue and background regions.

Study	Model	Accuracy (%)
Disci et al. [ 39 ]	Xception	98.73
Haque et al. [ 38 ]	NeuroNet19	99.30
Shahin [ 37 ]	Fine-tuned ResNet34	99.66
SAP-ViT (ours)	SAP-ViT	$99.30\pm 0.30$

Appendix

TABLE VII: Comparison with representative published classification results.

Model	Precision	Sensitivity	F1
Brain tumor MRI
SAP-ViT	$0.992\pm 0.003$	$0.992\pm 0.003$	$0.992\pm 0.003$
IA-ViT	$0.903\pm 0.020$	$0.900\pm 0.022$	$0.901\pm 0.021$
Vanilla ViT	$0.972\pm 0.008$	$0.971\pm 0.009$	$0.971\pm 0.009$
COVID chest X-ray
SAP-ViT	$0.965\pm 0.004$	$0.959\pm 0.004$	$0.962\pm 0.002$

Appendix

TABLE VIII: Additional classification metrics for models evaluated under the same experimental protocol.

Explainable Image Classification

Explainable image classification develops ways to show which pixels, regions, concepts, or model features support an image classifier’s predictions.

See all 97 papers

SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable
Aug 14, 2026Philip H. Lee, Parth PadalkarExplainable Image Classification Vision Transformer
Independent Researcher · Texas State University
Hierarchical Vision Transformer with Prototypes for Interpretable Medical Image Classification
Feb 13, 2025Luisa Gallée, Meinrad Beer, Michael GötzMedical Image Classification Explainable Medical Image Analysis
Experimental Radiology, Ulm University Medical Center, Germany · XAIRAD - Cooperation for Artificial Intelligence in Experimental Radiology, Germany · Department of Diagnostic and Interventional Radiology, Ulm University Medical Center, Germany +2
Decision-Aware Attention Propagation for Vision Transformer Explainability
Apr 20, 2026Sehyeong Jo, Gangjae Jang, Haesol ParkTransformer Interpretability Vision Transformer
Aarhus University · University of Colorado Boulder · KIST

Aug 14, 2026J/K move · Enter open · S save

SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable

Philip H. Lee, Parth Padalkar

Independent Researcher · Texas State University

Explainable Image Classification Vision Transformer Mechanistic Interpretability Sparse Autoencoders

Open paper AskPDF arXiv

Mechanistic interpretability has recently expanded to Vision Transformers (ViTs), with Sparse Autoencoders (SAEs) increasingly used as post-hoc tools to decompose internal representations into sparse and more interpretable features. However, because post-hoc SAEs are trained on frozen representations after the ViT has already been optimized, their latent features are not directly aligned with the downstream classification objective. We introduce SpIn-ViT, a framework that jointly trains a pretrained ViT and a modified SAE end-to-end, directly aligning sparse patch-level representations with image classification. SpIn-ViT learns semantically coherent neuron activations that localize meaningful image regions while maintaining competitive predictive performance. We evaluate SpIn-ViT across nine image-classification benchmarks using classification accuracy, quantitative interpretability metrics, AI-based and Human evaluations. Compared with the previous state-of-the-art post-hoc SAE method, SpIn-ViT achieves 8.84% higher average classification accuracy, an AI-based interpretability score nearly four times as high, and a human-evaluation score more than twice as high. We further extract interpretable rule-sets using the SAE neurons to create neurosymbolic models which achieve 5.97% higher average classification accuracy while requiring a 58.8% smaller rule-set than the neurosymbolic models created from the SOTA post-hoc SAE method.

Abstract

Explainable Image Classification

Explore similar work

SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable

Hierarchical Vision Transformer with Prototypes for Interpretable Medical Image Classification

Decision-Aware Attention Propagation for Vision Transformer Explainability