How do vision transformers (ViTs) represent and process the world? This paper addresses this long-standing question through the first systematic analysis of 6.6K features across all layers, extracted via sparse autoencoders, and by introducing the residual replacement model, which replaces ViT computations with interpretable features in the residual stream. Our analysis reveals not only a feature evolution from low-level patterns to high-level semantics, but also how ViTs encode curves and spatial positions through specialized feature types. The residual replacement model scalably produces a faithful yet parsimonious circuit for human-scale interpretability by significantly simplifying the original computations. As a result, this framework enables intuitive understanding of ViT mechanisms. Finally, we demonstrate the utility of our framework in debiasing spurious correlations.
Figures & tables
Figure 1: Overview. a. The residual replacement model replaces the original computations of a ViT with SAE features in the residual stream. b. We interpret features to understand what the ViT represents. c. We discover parsimonious circuits that explain the underlying mechanisms of the ViT. d. We apply the residual replacement model to debias spurious correlations.
Figure 2: Systematic Feature Analysis. a. Average interpretability scores of neurons and SAE features across all layers. Features are consistently more interpretable than neurons. b. Feature categorization by layer. Low-level features (e.g., color) emerge in early layers, while high-level features (e.g., objects) become more prominent in later layers. c. Example features from different categories.
Figure 3: Curve Detectors
Figure 4: Position Detectors
Faithfulness (%)
1 - Completeness (%)
Causality (%)
Unit
Strategy
ViT
DINOv2
CLIP
ViT
DINOv2
CLIP
ViT
DINOv2
CLIP
1
Neuron
Random Circuit
24.9
24.7
25.0
81.0
82.6
87.7
-
-
-
2
Naïve Circuit
23.0
25.8
29.2
95.2
94.9
97.1
-
-
-
3
+ Edge-based Discovery
26.3
26.7
30.2
92.4
93.7
97.1
-
-
-
4
+ Gradient Correction
61.4
42.8
32.4
94.2
94.4
95.4
-
-
-
5
Feature
Random Circuit
30.2
27.5
27.0
78.1
90.1
84.4
35.6
33.9
31.1
Table 1: Circuit Evaluation. We evaluate the faithfulness, completeness, and causality of the circuits using 1,500 randomly sampled images from the ImageNet validation set.
Figure 5: Granny Smith Circuit. Each node and edge is shaded according to its importance, with darker tones indicating higher importance. Node symbols represent the interpretation of each feature.
Figure 6: Trend of Feature Similarity
Figure 7: Curve Circuit (L3#801)
Figure 8: Position Circuit (L3#509)
Figure 9: freight car Circuit
Accuracy
mAUC
Original ViT
0.849
0.854
Intervened ViT
0.848
0.904
SpuFix (Upper bound)
0.848
0.917
Table 2: Debiasing Spurious Correlations
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Contour plots of the fitted scaling law for reconstruction loss with respect to the expansion rate R=f/d and the sparsity level k (ViT). Dotted line indicates the contour line corresponding to FVU=0.15 . The cross indicates the chosen hyperparameters for the SAE.
Figure 11: Contour plots of the fitted scaling law for reconstruction loss with respect to the expansion rate R=f/d and the sparsity level k (DINOv2). Dotted line indicates the contour line corresponding to FVU=0.15 . The cross indicates the chosen hyperparameters for the SAE.
Figure 12: Contour plots of the fitted scaling law for reconstruction loss with respect to the expansion rate R=f/d and the sparsity level k (CLIP). Dotted line indicates the contour line corresponding to FVU=0.15 . The cross indicates the chosen hyperparameters for the SAE.
Figure 13: FVU=0.15 contour line across layers. Dotted lines are extrapolated from the scaling law.
Figure 14: Activation frequency and mean activation value of the TopK SAE features (ViT).
Figure 15: Activation frequency and mean activation value of the TopK SAE features (DINOv2).
Figure 16: Activation frequency and mean activation value of the TopK SAE features (CLIP).
Figure 17: Visualization Example (1).
Figure 18: Visualization Example (2).
Figure 19: Category Proportions for ViT, DINOv2, and CLIP.
Figure 20: More ViT Feature Examples (Line and Shape).
Figure 21: More ViT Feature Examples (Color and Texture).
Figure 22: More ViT Feature Examples (Positional, Background, and Semantic).
Figure 23: More ViT Feature Examples (Object and Miscellaneous).
Figure 24: More DINOv2 Feature Examples (Line, Color, and Miscellaneous).
Figure 25: More DINOv2 Feature Examples (Shape and Texture).
Figure 26: More DINOv2 Feature Examples (Positional and Semantic).
Figure 27: More DINOv2 Feature Examples (Background and Object).
Figure 28: More CLIP Feature Examples (Line, Color, and Shape).
Figure 29: More CLIP Feature Examples (Texture and Miscellaneous).
Figure 30: More CLIP Feature Examples (Positional and Semantic).
Figure 31: More CLIP Feature Examples (Background and Object).
Faithfulness (%)
1 - Completeness (%)
Rule
ViT
DINOv2
CLIP
ViT
DINOv2
CLIP
top- k
94.1
85.1
82.3
99.6
99.8
99.7
top- p
95.0
85.4
82.9
99.7
99.7
99.7
threshold
93.6
84.6
82.2
99.6
99.8
99.7
Appendix
Table 3: Circuit Evaluation for Various Node Selection Rules. We evaluate the faithfulness and completeness of the circuits using 1,500 randomly sampled images from the ImageNet validation set.
Faithfulness (%)
1 - Completeness (%)
Causality (%)
Strategy
ViT
DINOv2
CLIP
ViT
DINOv2
CLIP
ViT
DINOv2
CLIP
Random
30.2
27.5
27.0
78.1
90.1
84.4
35.6
33.9
31.1
CLA
56.3
41.3
36.7
92.5
93.8
93.5
57.2
54.8
55.1
Ours
94.1
85.1
82.3
99.6
99.8
99.7
54.5
54.8
53.8
Appendix
Table 4: Comparison to Cross-Layer Attribution (CLA) Algorithm [ 8 ] . We evaluate the faithfulness, completeness, and causality of the circuits using 1,500 randomly sampled images from the ImageNet validation set.
Figure 32: Teapot circuit in ViT. The circuit reveals that the model initially attends to the grayish tone of the teapot (L1#12) and its specular highlights (L1#241) in the lower layers. As it progresses through the intermediate layers, the model gradually captures the overall shape of the teapot (see the heatmap of the input image in the sub-graph L3#208 → L4#258 → L5#1374 → L6#603). In the higher layers, information about the teapot’s handle (L8#2785) and body (L8#2560) is integrated, leading to the model’s prediction of a teapot.
Figure 33: Street sign circuit in ViT. As the input progresses through the early and intermediate layers, the model gradually develops an understanding of the text written on the street sign. For instance, at L0#140, the characters are perceived as a stripe pattern ; at L1#42, they are interpreted as a combination of diagonal lines ; and at L2#10, as a composition of curves . Starting from L3, the model begins to recognize the characters as alphabetical and Arabic numerals. This understanding of the text is subsequently integrated with the wide board feature (L8#3063), allowing the model to grasp the overall appearance of the signboard (L9#1321). Finally, this representation is combined with the signal light features (L9#2068, L10#1492) to yield the prediction of a traffic light.
Figure 34: Switch circuit in ViT. Up to the intermediate layers, the model captures information about the areas surrounding the switch—both laterally (L2#253, L3#439) and vertically (L2#13, L3#1275). This information is integrated to form a representation of the switch’s rounded shape (L6#2676, L7#40). Beginning from layer 5, the model starts recognizing the Roman numerals written on the switch (L5#855, L6#1429). The information combines with the switch’s round shape (L7#40, L8#63) ,leading the model to predict the presence of a switch.
Figure 35: Tabby circuit in ViT. Up to layer 7, the model captures features related to fur texture (L1#16, L2#516) and the cat’s whiskers (L2#658, L3#1102, L4#637, L5#1062). In layer 8, both the whiskers (L8#2319) and the cat’s eye (L8#1156) significantly contribute to the model’s ability to identify the cat’s face in layer 9 (L9#2381). Notably, the detection of the cat’s stripes (L9#377) and animal fur (L9#2651) in layer 9 plays an integral role in forming the final representation of a tabby cat (L10#404).
Figure 36: Typewriter keyboard circuit in DINOv2. In the early layers, the model interprets the spaces between keys either as lines ( e.g. , L0#98, L1#69) or as textures ( e.g. , L2#136, L3#904). In the middle and later layers, the model captures the round and repetitive shapes of the keys (L5#88, L6#1350), leading to the recognition of a keyboard (L8#455, L9#1213). By further detecting the characters engraved on the keys (L9#1091, L10#275), the model arrives at the prediction of a typewriter.
Figure 37: Scale circuit in DINOv2. Through the subgraph L1#131 → L2#296 → L3#806, the model captures the white, monotone color of the scale. Simultaneously, it interprets the text printed on the scale via the subgraph L1#132 → L2#244 → L3#40 → L4#1531 → L5#1114. Notably, in layer 6, the model appears to separately recognize the logo text (L6#1373) and the numerical values on the dial (L6#1177). In the later layers, the structural form of the scale (L8#744, L9#1106, L10#1332) is integrated with the textual and numerical information (L10#1135), ultimately leading the model to predict a scale.
Figure 38: Hare circuit in DINOv2. In the early layers, the model detects the hare’s eye as either a texture or a distinct shape (L1#1, L2#9, L3#1268), while simultaneously capturing the texture of the hare’s fur (L2#129, L3#1145). In the middle layers, the eye is interpreted more semantically as an animal eye (L4#316, L5#1225), and the model begins to represent the overall body of the hare (L4#633, L5#6). In the subsequent layers, the model exhibits increasingly fine-grained understanding of the hare’s face by attending to the eye (L6#609), the area below the eye (L6#1201), the region between the eye and nose (L6#209), and the snout (L9#696). In the final layer, the activation of the hare’s face (L10#1137) and the animal’s whiskers (L10#734) culminates in the model’s prediction of a hare.
Figure 39: Goose circuit in DINOv2. In the early layers, the model captures the goose’s coloration by separating it into white (L0#60), gray (L0#178), and edge regions (L2#224). In the middle layers, the model detects the left (L3#1268) and right (L3#1171) sides of the goose’s eye as distinct shapes, which are subsequently integrated in later layers to form a semantic understanding of an animal eye (L4#316, L5#1225, L6#1032). In layers 7 and 8, the model attends to the goose’s neck (L7#1471), cheek (L7#338), body (L8#1017), and beak (L8#983), ultimately leading to the prediction of a goose.
Figure 40: Cellular telephone circuit in CLIP. In the early layers, the model focuses on the blue, uniform color of the cell phone (L3#1379). In the middle layers, it identifies the keypad of the device (L6#688), and subsequently recognizes it as a button-equipped device (L7#518). This understanding is further refined as the model begins to interpret it as a time-displaying device (L8#520) and a communication device (L9#1163, L10#1382), ultimately leading to the prediction of a cell phone.
Figure 41: Tennis ball circuit in CLIP. In the early layers, the model captures the background of the tennis ball ( e.g. , L0#214, L1#17) as well as the texture of the ball ( e.g. , L2#41, L3#838). In the intermediate layers, it detects the edges on both sides of the tennis ball (L4#703) and the lower edge (L4#1406), leading to a representation of a round object (L5#540). This understanding is further refined into multiple round objects (L6#0), and in the subsequent layers, the model recognizes the object as a ball (L7#139, L8#274). By identifying the characteristic stripes on the ball (see the maximum activating patches in L9#957), the model ultimately predicts a tennis ball.
Figure 42: Wall clock circuit in CLIP. In the early layers, the model focuses on the background of the wall clock (L0#101, L0#157, L1#236). In the middle layers, it begins to recognize the numbers on the clock (L6#178), followed by the detection of the tick marks in the subsequent layer (L7#1248). The model then forms a representation of a round-shaped clock (L8#876), eventually activating general clock features (L9#1246, L10#568), which leads to the final prediction of a wall clock.
Figure 43: Oscilloscope circuit in CLIP. In the early and middle layers, the model recognizes the background of the oscilloscope ( e.g. , L0#210, L1#236, L2#111). In the later layers, it identifies the oscilloscope’s buttons (L7#518), display (L8#1472), and body (L8#456). Subsequently, the red line connected to the oscilloscope (L9#1396) is interpreted as a wire (L10#733). This understanding, combined with the oscilloscope-specific features (L10#361), leads the model to predict an oscilloscope.
Understanding the internal activations of Vision Transformers (ViTs) is critical for building interpretable and trustworthy models. While Sparse Autoencoders (SAEs) have been used to extract human-interpretable features, they operate on individual layers and fail to capture the cross-layer computational structure of Transformers, as well as the relative significance of each layer in forming the last-layer representation. Alternatively, we introduce the adoption of Cross-Layer Transcoders (CLTs) as reliable, sparse, and depth-aware proxy models for MLP blocks in ViTs. CLTs use an encoder-decoder scheme to reconstruct each post-MLP activation from learned sparse embeddings of preceding layers, yielding a linear decomposition that transforms the final representation of ViTs from an opaque embedding into an additive, layer-resolved construction that enables faithful attribution and process-level interpretability. We train CLTs on CLIP ViT-B/32 and ViT-B/16 across CIFAR-100, COCO, and ImageNet-100. We show that CLTs achieve high reconstruction fidelity with post-MLP activations while preserving and even improving, in some cases, CLIP zero-shot classification accuracy. In terms of interpretability, we show that the cross-layer contribution scores provide faithful attribution, revealing that the final representation is concentrated in a smaller set of dominant layer-wise terms whose removal degrades performance and whose retention largely preserves it. These results showcase the significance of adopting CLTs as an alternative interpretable proxy of ViTs in the vision domain.
Gerasimos Chatzoudis, Konstantinos D. Polyzos, Zhuowei Li +4
Rutgers University · University of California San Diego
Despite high accuracy, Vision Transformer (ViT) predictions can be driven by spurious cues, raising the need to understand their inner workings before safe deployment. Sparse autoencoders (SAEs) provide a promising lens for decomposing model representations into human-interpretable concepts, yet adapting SAE-based interpretation to ViTs remains challenging due to limited control over concept coverage and subjective, non-scalable feature interpretation. To fill the gaps, motivated by neuroscience-inspired principles, we propose ViSAE, a mechanistic interpretability toolbox for understanding ViT inner workings through concept circuits. ViSAE consists of three components: (1) A probing suite with 64K images and a 16K visually grounded concept vocabulary, improving concept coverage efficiency by 20x over ImageNet and interpretation accuracy by 28.7% over existing concept sets. (2) Top-down concept reading and Bottom-up circuit tracing algorithms that automatically recover ViT inner workings via concept circuits. (3) Applications for auditing and steering ViT behavior. Through concept editing, ViSAE improves the worst-group accuracy on WaterBirds by 48.2%, outperforming existing methods by 23.8%. Our data and code: https://github.com/deep-real/ViSAE.
Tang Li, Yanlin Chen, Mengmeng Ma +1
Department of Computer & Information Science, University of Delaware, Newark, DE, USA · Department of Computer Science, University of Virginia, Charlottesville, VA, USA
Mechanistic interpretability has recently expanded to Vision Transformers (ViTs), with Sparse Autoencoders (SAEs) increasingly used as post-hoc tools to decompose internal representations into sparse and more interpretable features. However, because post-hoc SAEs are trained on frozen representations after the ViT has already been optimized, their latent features are not directly aligned with the downstream classification objective. We introduce SpIn-ViT, a framework that jointly trains a pretrained ViT and a modified SAE end-to-end, directly aligning sparse patch-level representations with image classification. SpIn-ViT learns semantically coherent neuron activations that localize meaningful image regions while maintaining competitive predictive performance. We evaluate SpIn-ViT across nine image-classification benchmarks using classification accuracy, quantitative interpretability metrics, AI-based and Human evaluations. Compared with the previous state-of-the-art post-hoc SAE method, SpIn-ViT achieves 8.84% higher average classification accuracy, an AI-based interpretability score nearly four times as high, and a human-evaluation score more than twice as high. We further extract interpretable rule-sets using the SAE neurons to create neurosymbolic models which achieve 5.97% higher average classification accuracy while requiring a 58.8% smaller rule-set than the neurosymbolic models created from the SOTA post-hoc SAE method.