cs.CVSep 22, 2025
SaveInterpreting vision transformers via residual replacement model
Organizations: Yonsei University
Abstract
How do vision transformers (ViTs) represent and process the world? This paper addresses this long-standing question through the first systematic analysis of 6.6K features across all layers, extracted via sparse autoencoders, and by introducing the residual replacement model, which replaces ViT computations with interpretable features in the residual stream. Our analysis reveals not only a feature evolution from low-level patterns to high-level semantics, but also how ViTs encode curves and spatial positions through specialized feature types. The residual replacement model scalably produces a faithful yet parsimonious circuit for human-scale interpretability by significantly simplifying the original computations. As a result, this framework enables intuitive understanding of ViT mechanisms. Finally, we demonstrate the utility of our framework in debiasing spurious correlations.
Figures & tables
Figure 1: Overview. a. The residual replacement model replaces the original computations of a ViT with SAE features in the residual stream. b. We interpret features to understand what the ViT represents. c. We discover parsimonious circuits that explain the underlying mechanisms of the ViT. d. We apply the residual replacement model to debias spurious correlations.
Figure 2: Systematic Feature Analysis. a. Average interpretability scores of neurons and SAE features across all layers. Features are consistently more interpretable than neurons. b. Feature categorization by layer. Low-level features (e.g., color) emerge in early layers, while high-level features (e.g., objects) become more prominent in later layers. c. Example features from different categories.
Figure 3: Curve Detectors
Figure 4: Position Detectors
| Faithfulness (%) | 1 - Completeness (%) | Causality (%) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Unit | Strategy | ViT | DINOv2 | CLIP | ViT | DINOv2 | CLIP | ViT | DINOv2 | CLIP | |
| 1 | Neuron | Random Circuit | 24.9 | 24.7 | 25.0 | 81.0 | 82.6 | 87.7 | - | - | - |
| 2 | Naïve Circuit | 23.0 | 25.8 | 29.2 | 95.2 | 94.9 | 97.1 | - | - | - | |
| 3 | + Edge-based Discovery | 26.3 | 26.7 | 30.2 | 92.4 | 93.7 | 97.1 | - | - | - | |
| 4 | + Gradient Correction | 61.4 | 42.8 | 32.4 | 94.2 | 94.4 | 95.4 | - | - | - | |
| 5 | Feature | Random Circuit | 30.2 | 27.5 | 27.0 | 78.1 | 90.1 | 84.4 | 35.6 | 33.9 | 31.1 |
Table 1: Circuit Evaluation. We evaluate the faithfulness, completeness, and causality of the circuits using 1,500 randomly sampled images from the ImageNet validation set.
Figure 5: Granny Smith Circuit. Each node and edge is shaded according to its importance, with darker tones indicating higher importance. Node symbols represent the interpretation of each feature.
Figure 6: Trend of Feature Similarity
Figure 7: Curve Circuit (L3#801)
Figure 8: Position Circuit (L3#509)
Figure 9: freight car Circuit
| Accuracy | mAUC | |
|---|---|---|
| Original ViT | 0.849 | 0.854 |
| Intervened ViT | 0.848 | 0.904 |
| SpuFix (Upper bound) | 0.848 | 0.917 |
Table 2: Debiasing Spurious Correlations
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Contour plots of the fitted scaling law for reconstruction loss with respect to the expansion rate and the sparsity level (ViT). Dotted line indicates the contour line corresponding to . The cross indicates the chosen hyperparameters for the SAE.
Figure 11: Contour plots of the fitted scaling law for reconstruction loss with respect to the expansion rate and the sparsity level (DINOv2). Dotted line indicates the contour line corresponding to . The cross indicates the chosen hyperparameters for the SAE.
Figure 12: Contour plots of the fitted scaling law for reconstruction loss with respect to the expansion rate and the sparsity level (CLIP). Dotted line indicates the contour line corresponding to . The cross indicates the chosen hyperparameters for the SAE.
Figure 13: contour line across layers. Dotted lines are extrapolated from the scaling law.
Figure 14: Activation frequency and mean activation value of the TopK SAE features (ViT).
Figure 15: Activation frequency and mean activation value of the TopK SAE features (DINOv2).
Figure 16: Activation frequency and mean activation value of the TopK SAE features (CLIP).
Figure 17: Visualization Example (1).
Figure 18: Visualization Example (2).
Figure 19: Category Proportions for ViT, DINOv2, and CLIP.
Figure 20: More ViT Feature Examples (Line and Shape).
Figure 21: More ViT Feature Examples (Color and Texture).
Figure 22: More ViT Feature Examples (Positional, Background, and Semantic).
Figure 23: More ViT Feature Examples (Object and Miscellaneous).
Figure 24: More DINOv2 Feature Examples (Line, Color, and Miscellaneous).
Figure 25: More DINOv2 Feature Examples (Shape and Texture).
Figure 26: More DINOv2 Feature Examples (Positional and Semantic).
Figure 27: More DINOv2 Feature Examples (Background and Object).
Figure 28: More CLIP Feature Examples (Line, Color, and Shape).
Figure 29: More CLIP Feature Examples (Texture and Miscellaneous).
Figure 30: More CLIP Feature Examples (Positional and Semantic).
Figure 31: More CLIP Feature Examples (Background and Object).
| Faithfulness (%) | 1 - Completeness (%) | |||||
|---|---|---|---|---|---|---|
| Rule | ViT | DINOv2 | CLIP | ViT | DINOv2 | CLIP |
| top- | 94.1 | 85.1 | 82.3 | 99.6 | 99.8 | 99.7 |
| top- | 95.0 | 85.4 | 82.9 | 99.7 | 99.7 | 99.7 |
| threshold | 93.6 | 84.6 | 82.2 | 99.6 | 99.8 | 99.7 |
Table 3: Circuit Evaluation for Various Node Selection Rules. We evaluate the faithfulness and completeness of the circuits using 1,500 randomly sampled images from the ImageNet validation set.
| Faithfulness (%) | 1 - Completeness (%) | Causality (%) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Strategy | ViT | DINOv2 | CLIP | ViT | DINOv2 | CLIP | ViT | DINOv2 | CLIP |
| Random | 30.2 | 27.5 | 27.0 | 78.1 | 90.1 | 84.4 | 35.6 | 33.9 | 31.1 |
| CLA | 56.3 | 41.3 | 36.7 | 92.5 | 93.8 | 93.5 | 57.2 | 54.8 | 55.1 |
| Ours | 94.1 | 85.1 | 82.3 | 99.6 | 99.8 | 99.7 | 54.5 | 54.8 | 53.8 |
Table 4: Comparison to Cross-Layer Attribution (CLA) Algorithm [ 8 ] . We evaluate the faithfulness, completeness, and causality of the circuits using 1,500 randomly sampled images from the ImageNet validation set.
Figure 32: Teapot circuit in ViT. The circuit reveals that the model initially attends to the grayish tone of the teapot (L1#12) and its specular highlights (L1#241) in the lower layers. As it progresses through the intermediate layers, the model gradually captures the overall shape of the teapot (see the heatmap of the input image in the sub-graph L3#208 L4#258 L5#1374 L6#603). In the higher layers, information about the teapot’s handle (L8#2785) and body (L8#2560) is integrated, leading to the model’s prediction of a teapot.
Figure 33: Street sign circuit in ViT. As the input progresses through the early and intermediate layers, the model gradually develops an understanding of the text written on the street sign. For instance, at L0#140, the characters are perceived as a stripe pattern ; at L1#42, they are interpreted as a combination of diagonal lines ; and at L2#10, as a composition of curves . Starting from L3, the model begins to recognize the characters as alphabetical and Arabic numerals. This understanding of the text is subsequently integrated with the wide board feature (L8#3063), allowing the model to grasp the overall appearance of the signboard (L9#1321). Finally, this representation is combined with the signal light features (L9#2068, L10#1492) to yield the prediction of a traffic light.
Figure 34: Switch circuit in ViT. Up to the intermediate layers, the model captures information about the areas surrounding the switch—both laterally (L2#253, L3#439) and vertically (L2#13, L3#1275). This information is integrated to form a representation of the switch’s rounded shape (L6#2676, L7#40). Beginning from layer 5, the model starts recognizing the Roman numerals written on the switch (L5#855, L6#1429). The information combines with the switch’s round shape (L7#40, L8#63) ,leading the model to predict the presence of a switch.
Figure 35: Tabby circuit in ViT. Up to layer 7, the model captures features related to fur texture (L1#16, L2#516) and the cat’s whiskers (L2#658, L3#1102, L4#637, L5#1062). In layer 8, both the whiskers (L8#2319) and the cat’s eye (L8#1156) significantly contribute to the model’s ability to identify the cat’s face in layer 9 (L9#2381). Notably, the detection of the cat’s stripes (L9#377) and animal fur (L9#2651) in layer 9 plays an integral role in forming the final representation of a tabby cat (L10#404).
Figure 36: Typewriter keyboard circuit in DINOv2. In the early layers, the model interprets the spaces between keys either as lines ( e.g. , L0#98, L1#69) or as textures ( e.g. , L2#136, L3#904). In the middle and later layers, the model captures the round and repetitive shapes of the keys (L5#88, L6#1350), leading to the recognition of a keyboard (L8#455, L9#1213). By further detecting the characters engraved on the keys (L9#1091, L10#275), the model arrives at the prediction of a typewriter.
Figure 37: Scale circuit in DINOv2. Through the subgraph L1#131 L2#296 L3#806, the model captures the white, monotone color of the scale. Simultaneously, it interprets the text printed on the scale via the subgraph L1#132 L2#244 L3#40 L4#1531 L5#1114. Notably, in layer 6, the model appears to separately recognize the logo text (L6#1373) and the numerical values on the dial (L6#1177). In the later layers, the structural form of the scale (L8#744, L9#1106, L10#1332) is integrated with the textual and numerical information (L10#1135), ultimately leading the model to predict a scale.
Figure 38: Hare circuit in DINOv2. In the early layers, the model detects the hare’s eye as either a texture or a distinct shape (L1#1, L2#9, L3#1268), while simultaneously capturing the texture of the hare’s fur (L2#129, L3#1145). In the middle layers, the eye is interpreted more semantically as an animal eye (L4#316, L5#1225), and the model begins to represent the overall body of the hare (L4#633, L5#6). In the subsequent layers, the model exhibits increasingly fine-grained understanding of the hare’s face by attending to the eye (L6#609), the area below the eye (L6#1201), the region between the eye and nose (L6#209), and the snout (L9#696). In the final layer, the activation of the hare’s face (L10#1137) and the animal’s whiskers (L10#734) culminates in the model’s prediction of a hare.
Figure 39: Goose circuit in DINOv2. In the early layers, the model captures the goose’s coloration by separating it into white (L0#60), gray (L0#178), and edge regions (L2#224). In the middle layers, the model detects the left (L3#1268) and right (L3#1171) sides of the goose’s eye as distinct shapes, which are subsequently integrated in later layers to form a semantic understanding of an animal eye (L4#316, L5#1225, L6#1032). In layers 7 and 8, the model attends to the goose’s neck (L7#1471), cheek (L7#338), body (L8#1017), and beak (L8#983), ultimately leading to the prediction of a goose.
Figure 40: Cellular telephone circuit in CLIP. In the early layers, the model focuses on the blue, uniform color of the cell phone (L3#1379). In the middle layers, it identifies the keypad of the device (L6#688), and subsequently recognizes it as a button-equipped device (L7#518). This understanding is further refined as the model begins to interpret it as a time-displaying device (L8#520) and a communication device (L9#1163, L10#1382), ultimately leading to the prediction of a cell phone.
Figure 41: Tennis ball circuit in CLIP. In the early layers, the model captures the background of the tennis ball ( e.g. , L0#214, L1#17) as well as the texture of the ball ( e.g. , L2#41, L3#838). In the intermediate layers, it detects the edges on both sides of the tennis ball (L4#703) and the lower edge (L4#1406), leading to a representation of a round object (L5#540). This understanding is further refined into multiple round objects (L6#0), and in the subsequent layers, the model recognizes the object as a ball (L7#139, L8#274). By identifying the characteristic stripes on the ball (see the maximum activating patches in L9#957), the model ultimately predicts a tennis ball.
Figure 42: Wall clock circuit in CLIP. In the early layers, the model focuses on the background of the wall clock (L0#101, L0#157, L1#236). In the middle layers, it begins to recognize the numbers on the clock (L6#178), followed by the detection of the tick marks in the subsequent layer (L7#1248). The model then forms a representation of a round-shaped clock (L8#876), eventually activating general clock features (L9#1246, L10#568), which leads to the final prediction of a wall clock.
Figure 43: Oscilloscope circuit in CLIP. In the early and middle layers, the model recognizes the background of the oscilloscope ( e.g. , L0#210, L1#236, L2#111). In the later layers, it identifies the oscilloscope’s buttons (L7#518), display (L8#1472), and body (L8#456). Subsequently, the red line connected to the oscilloscope (L9#1396) is interpreted as a wire (L10#733). This understanding, combined with the oscilloscope-specific features (L10#361), leads the model to predict an oscilloscope.
Figure 44: Curve Circuit (L3#87) for DINOv2.
Figure 45: Curve Circuit (L3#141) for CLIP.
Figure 46: Position Circuit (L4#1203) for DINOv2.
Figure 47: Position Circuit (L4#70) for CLIP.
More in this topic
Transformer Interpretability
Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.
See all 336 papers