SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable

Philip H. Lee; Parth Padalkar

cs.CVAug 14, 2026

SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable

Authors: Philip H. Lee, Parth Padalkar

Organizations: Independent Researcher · Texas State University

Abstract

Mechanistic interpretability has recently expanded to Vision Transformers (ViTs), with Sparse Autoencoders (SAEs) increasingly used as post-hoc tools to decompose internal representations into sparse and more interpretable features. However, because post-hoc SAEs are trained on frozen representations after the ViT has already been optimized, their latent features are not directly aligned with the downstream classification objective. We introduce SpIn-ViT, a framework that jointly trains a pretrained ViT and a modified SAE end-to-end, directly aligning sparse patch-level representations with image classification. SpIn-ViT learns semantically coherent neuron activations that localize meaningful image regions while maintaining competitive predictive performance. We evaluate SpIn-ViT across nine image-classification benchmarks using classification accuracy, quantitative interpretability metrics, AI-based and Human evaluations. Compared with the previous state-of-the-art post-hoc SAE method, SpIn-ViT achieves 8.84% higher average classification accuracy, an AI-based interpretability score nearly four times as high, and a human-evaluation score more than twice as high. We further extract interpretable rule-sets using the SAE neurons to create neurosymbolic models which achieve 5.97% higher average classification accuracy while requiring a 58.8% smaller rule-set than the neurosymbolic models created from the SOTA post-hoc SAE method.

Figures & tables

Figure 1: Training SpIn-ViT. ViT patch embeddings are encoded into a sparse latent space, reconstructed through an SAE decoder, and aligned with CLS-token predictions using reconstruction, sparsity, and alignment losses.

Figure 2: Interpretability analysis performed after training. (a) Reference images for a given neuron (e.g., Neuron 50) are selected by ranking [CLS] token activations from the trained TopK SAE encoder. (b) Patch-token activations for that neuron are then used to localize the specific image regions responsible for the activation. (c) Neuron activations across images are binarized into tabular data, over which FOLD-SE-M creates neurosymbolic rules for explainability of the TopK SAE encoder.

Figure 3: Insertion and Deletion Plots of each model for average of all datasets

Dataset	PatchSAE	Vanilla ViT	SpIn-ViT (Separate)	SpIn-ViT (Frozen)	SpIn-ViT (Vanilla)	SpIn-ViT (Ours)
Flowers102	94.99 $\pm$ 0.42	98.54 $\pm$ 0.21	97.94 $\pm$ 0.38	98.71 $\pm$ 0.29	98.90 $\pm$ 0.19	99.88 $\pm$ 0.28
Caltech101	95.10 $\pm$ 0.31	96.80 $\pm$ 0.58	95.70 $\pm$ 0.24	93.47 $\pm$ 0.62	96.81 $\pm$ 0.23	97.02 $\pm$ 0.18
Stanford Cars	75.71 $\pm$ 1.53	82.37 $\pm$ 1.24	82.57 $\pm$ 1.39	66.46 $\pm$ 1.64	82.14 $\pm$ 1.21	84.61 $\pm$ 1.02
FGVC	55.43 $\pm$ 2.18	71.11 $\pm$ 1.67	68.32 $\pm$ 2.01	70.12 $\pm$ 1.75	67.63 $\pm$ 1.82	68.43 $\pm$ 1.79
EuroSAT	78.58 $\pm$ 1.08	97.41 $\pm$ 0.22	97.31 $\pm$ 0.21	62.42 $\pm$ 1.88	98.62 $\pm$ 0.18	98.58 $\pm$ 0.21
DTD	76.42 $\pm$ 1.32	80.90 $\pm$ 0.93	80.78 $\pm$ 0.98	53.44 $\pm$ 2.31	82.59 $\pm$ 1.11	84.42 $\pm$ 1.28

Table 1: Classification accuracy (%) across nine benchmark datasets (mean

\pm

std). Best results per dataset are highlighted in bold .

Dataset	PatchSAE	SpIn-ViT (Frozen)	SpIn-ViT (Separate)	SpIn-ViT (Vanilla)	SpIn-ViT (Ours)
Flowers102	$1.32\pm 0.18$	$1.60\pm 0.24$	$1.08\pm 0.12$	$3.00\pm 0.32$	$\mathbf{3.28\pm 0.35}$
Caltech101	$1.32\pm 0.22$	$1.56\pm 0.22$	$1.08\pm 0.11$	$2.96\pm 0.30$	$\mathbf{3.24\pm 0.33}$
Stanford Cars	$1.24\pm 0.15$	$1.40\pm 0.18$	$1.04\pm 0.10$	$2.76\pm 0.25$	$\mathbf{3.00\pm 0.28}$
FGVC	$1.16\pm 0.12$	$1.28\pm 0.15$	$1.04\pm 0.10$	$2.56\pm 0.20$	$\mathbf{2.72\pm 0.22}$
EuroSAT	$1.36\pm 0.25$	$1.64\pm 0.28$	$1.08\pm 0.12$	$3.04\pm 0.35$	$\mathbf{3.28\pm 0.36}$
DTD	$1.24\pm 0.16$	$1.44\pm 0.19$	$1.04\pm 0.10$	$2.76\pm 0.26$	$\mathbf{3.00\pm 0.27}$

Table 2: AI interpretability evaluations across nine image classification benchmarks (normalized to a 1–5 scale where 1.00 = 0.0 mIoU and 5.00 = 1.0 mIoU).

Method	Flowers102	FGVC	Average
PatchSAE	$1.78\pm 0.15$	$1.92\pm 0.17$	$1.85\pm 0.16$
SpIn-ViT (Frozen)	$2.45\pm 0.18$	$2.34\pm 0.16$	$2.40\pm 0.17$
SpIn-ViT (Separate)	$1.45\pm 0.11$	$1.32\pm 0.12$	$1.39\pm 0.11$
SpIn-ViT (Vanilla)	$3.43\pm 0.22$	$3.22\pm 0.24$	$3.33\pm 0.23$
SpIn-ViT (ours)	$\mathbf{4.20\pm 0.28}$	$\mathbf{3.99\pm 0.26}$	$\mathbf{4.10\pm 0.27}$

Table 3: Human-rated interpretability of neuron activations (1–5, higher is better) on Flowers102 and FGVC.

	PatchSAE		SpIn-ViT (Frozen)		SpIn-ViT (Separate)		SpIn-ViT (Vanilla)		SpIn-ViT (Ours)
Dataset	Acc ( $\uparrow$ )	Rules ( $\downarrow$ )	Acc ( $\uparrow$ )	Rules ( $\downarrow$ )	Acc ( $\uparrow$ )	Rules ( $\downarrow$ )	Acc ( $\uparrow$ )	Rules ( $\downarrow$ )	Acc ( $\uparrow$ )	Rules ( $\downarrow$ )
Flowers102	$90.91\pm 0.98$	$20\pm 3.4$	$94.24\pm 0.35$	$14\pm 1.2$	$92.21\pm 0.48$	$24\pm 2.8$	$95.04\pm 0.31$	$13\pm 2.5$	$\mathbf{96.48\pm 0.97}$	$\mathbf{10\pm 0.8}$
Caltech101	$89.62\pm 0.38$	$12\pm 1.8$	$93.56\pm 0.16$	$23\pm 4.5$	$90.97\pm 1.13$	$16\pm 2.1$	$94.38\pm 0.43$	$7\pm 2.7$	$\mathbf{95.16\pm 0.78}$	$\mathbf{5\pm 0.9}$
Stanford Cars	$84.36\pm 0.54$	$48\pm 5.8$	$86.32\pm 0.17$	$38\pm 3.2$	$84.82\pm 0.48$	$24\pm 4.6$	$87.91\pm 0.56$	$30\pm 2.9$	$\mathbf{88.78\pm 0.33}$	$\mathbf{23\pm 1.6}$
FGVC	$75.33\pm 0.55$	$106\pm 6.2$	$78.68\pm 1.12$	$67\pm 4.8$	$78.29\pm 0.67$	$60\pm 3.1$	$79.44\pm 0.52$	$65\pm 2.4$	$\mathbf{81.24\pm 0.96}$	$\mathbf{47\pm 1.1}$
EuroSAT	$94.16\pm 0.87$	$60\pm 4.9$	$95.94\pm 0.59$	$40\pm 3.7$	$95.18\pm 0.97$	$34\pm 2.3$	$96.59\pm 0.78$	$40\pm 1.4$	$\mathbf{97.89\pm 1.13}$	$\mathbf{29\pm 0.7}$

Table 4: Accuracy (%) and number of active rules (

\pm

standard deviation) across nine image classification benchmarks.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Insertion plots for all nine benchmark datasets. For each dataset, pixels are progressively inserted into a blank baseline image in order of attribution ranking (most salient first, per each model’s attribution map), and classification accuracy is plotted as a function of the percentage of pixels inserted. Higher curves and larger AUC indicate more faithful attributions.

Figure 5: Deletion plots for all nine benchmark datasets. For each dataset, pixels are progressively removed from the original image in order of attribution ranking (most salient first, per each model’s attribution map), and classification accuracy is plotted as a function of the percentage of pixels deleted. Steeper drops and lower AUC indicate more faithful attributions.

Figure 6: Examples of correspondence between sparse patch-level neuron activations and extracted symbolic rules. (a) Soft patch segmentations induced by individual neurons across example images. (b) FOLD-SE-M rules for the corresponding datasets, where each predicate maps to one of the visualized neurons.

Explore similar work

Inside the Visual Mind: Neuroscience-Motivated Concept Circuits for Interpreting and Steering Vision Transformers
Jun 4, 2026Tang Li, Yanlin Chen, Mengmeng Ma +1Transformer Interpretability Vision Transformer
Department of Computer & Information Science, University of Delaware, Newark, DE, USA · Department of Computer Science, University of Virginia, Charlottesville, VA, USA
Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision Transformers
Mar 16, 2026Siyu ZhangNeural Network Interpretability Transformer Interpretability
University of Texas at Austin, Austin TX 78712, USA
Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?
Sep 28, 2026Teodor Chiaburu, Franz Motzkus, Frank Haußer +1Linear Representation Hypothesis Sparse Autoencoders
Fraunhofer SIT · ATHENE National Research Center for Applied Cybersecurity · AUMOVIO +3

Jun 4, 2026J/K move · Enter open · S save

Inside the Visual Mind: Neuroscience-Motivated Concept Circuits for Interpreting and Steering Vision Transformers

Tang Li, Yanlin Chen, Mengmeng Ma, Xi Peng

Department of Computer & Information Science, University of Delaware, Newark, DE, USA · Department of Computer Science, University of Virginia, Charlottesville, VA, USA

Transformer Interpretability Vision Transformer Sparse Autoencoders Mechanistic Interpretability Concept-Based Explanations

Open paper AskPDF arXiv

Despite high accuracy, Vision Transformer (ViT) predictions can be driven by spurious cues, raising the need to understand their inner workings before safe deployment. Sparse autoencoders (SAEs) provide a promising lens for decomposing model representations into human-interpretable concepts, yet adapting SAE-based interpretation to ViTs remains challenging due to limited control over concept coverage and subjective, non-scalable feature interpretation. To fill the gaps, motivated by neuroscience-inspired principles, we propose ViSAE, a mechanistic interpretability toolbox for understanding ViT inner workings through concept circuits. ViSAE consists of three components: (1) A probing suite with 64K images and a 16K visually grounded concept vocabulary, improving concept coverage efficiency by 20x over ImageNet and interpretation accuracy by 28.7% over existing concept sets. (2) Top-down concept reading and Bottom-up circuit tracing algorithms that automatically recover ViT inner workings via concept circuits. (3) Applications for auditing and steering ViT behavior. Through concept editing, ViSAE improves the worst-group accuracy on WaterBirds by 48.2%, outperforming existing methods by 23.8%. Our data and code: https://github.com/deep-real/ViSAE.

Abstract

Explore similar work

Inside the Visual Mind: Neuroscience-Motivated Concept Circuits for Interpreting and Steering Vision Transformers

Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision Transformers

Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?