We present a novel view on feature evolution in Vision Transformers (ViTs) by visualizing the training process over two dimensions -- network depth (layer) and training time (epochs). We employ Sparse Autoencoders (SAEs) to extract candidate sparse features from CLS-token representations and compare their activation profiles across epoch--layer pairs. This allows us to study feature-level dynamics that are not directly visible from representation-level similarity measures. Furthermore, we demonstrate how this framework of feature evolution allows us to describe feature migration, the change in the layer where a feature is most detectable during training. Our experiments show that migration is concentrated early in training, occurs more often toward earlier layers than toward deeper layers, and declines as feature organization stabilizes. We further find that deeper layers stabilize earlier and more strongly than shallow layers. The results show that our approach can be employed as a tool for understanding how ViTs learn and evolve.
Figures & tables
Figure 1. Examples of distinct feature-evolution types observed during ViT training, shown in an epoch-by-layer view. ⋆ represents the reference feature.
Figure 2. Overview of the methodology: (A) extraction of image representations from every ViT checkpoint–layer pair; (B) SAE training at each pair; and (C) computation of feature-similarity scores for one example.
Figure 3. Feature-types based on ImageNet classes. Human-understandable SAE features are observed under 3 main categories: class-specific, superclass-specific and inter-class (see maximally activating images ( Zeiler and Fergus, 2014 ) for each feature in Appendix).
Figure 4. Percentage of features migrating towards deeper layers.
Figure 5. Percentage of features migrating towards earlier layers.
Figure 6. Mean signed active drift. For each reference epoch–layer pair, feature-level quantities are averaged across features. Average signed displacement of the layer center, in layer-index units; negative values indicate movement toward earlier layers.
Figure 7. Percentage of stable features.
Figure 8. Mean active lifetime of features, given in number of epochs.
Figure 9. Layer center-of-mass (COM) ± standard deviation trajectories averaged over all the features of selected reference checkpoints (0,100,200 and 300). The high-frequency epochs are depicted as a solid line around the reference checkpoints (marked as x) and the low-frequency epochs as dots. On the figure below, the average layer width ± standard deviation is depicted. The averaging is carried out only on active epochs per feature and therefore color fading signifies the lack of active features in that area.
Figure 10. Percentage of features that are widely spread across layers.
Figure 11. Feature migration analysis by layer center-of-mass (COM). Most of the features migrate towards earlier layers.
Reference epoch
class
superclass
other
total (active)
1
70
0
755
825
100
259
22
678
959
200
401
35
872
1308
300
461
42
903
1406
Table 1. Feature-type counts by reference epochs presented in Figure 11 .
Figure 12. The difference between two datasets in label entropy. Low label entropy indicates label-selective features, whereas high label entropy indicates features that activate across many labels.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13. Number of features found by SAE.
Figure 14. Percentage of features migrating towards deeper layers on Mixed 10.
Figure 15. Percentage of stable features on Mixed 10.
Figure 16. Mean active lifetime of features, given in number of epochs on Mixed 10.
Figure 17. Percentage of features that are widely spread across layers on Mixed 10.
Figure 18. Mean signed active drift. For each reference epoch–layer pair, feature-level quantities are averaged across features. Average signed displacement of the layer center, in layer-index units; negative values indicate movement toward earlier layers on Mixed 10.
Figure 19. Layer center-of-mass (COM) ± standard deviation trajectories averaged over all the features of selected reference checkpoints (0,100,200 and 300). The high-frequency epochs are depicted as a solid line around the reference checkpoints (marked as x) and the low-frequency epochs as dots. On the figure below, the average layer width ± standard deviation is depicted. The averaging is carried out only on active epochs per feature and therefore color fading signifies the lack of active features in that area.
Figure 20. This figure depicts an SAE feature 881 (marked with ⋆ ) evolution discovered from checkpoint epoch 1, layer 8 (top figure). Figures below show the best-matching target feature at each selected checkpoint (marked with ⋆ ) and the target feature’s top-5 activating images (example image thumbnails are from the ImageNet-1k dataset).
Figure 21. SAE test loss on a log-scale across time and depth on ImageNet-1k trained ViT checkpoints.
Figure 22. Linear CKA between the ImageNet-1k representation at epoch 1, layer 8 and every sampled epoch–layer pair.
Figure 23. This figure depicts a diverse selection of SAE feature evolutions decoded from checkpoint epoch 1, layer 8 (marked with ⋆ ) and the feature’s top-5 activating images (example image thumbnails are from the ImageNet-1k dataset).
Figure 24. Feature-type examples illustrated by maximally activating images ( Zeiler and Fergus, 2014 ) . Human-understandable SAE features are observed under 3 main categories: class-specific, superclass-specific and inter-class (Example image thumbnails are from the ImageNet-1k dataset).
Quantity
Value
Role
kon
3
Active checkpoints for emergence
koff
5
Inactive checkpoints for disappearance
σmax
4
Maximum layer width
fse
0.05
Fraction of valid active epochs at drift endpoints
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
Vision Transformers trained only on image classification routinely transfer to tasks that demand spatial understanding, yet they receive no spatial supervision during pretraining. We ask where and how robustly such structure is encoded. Probing a frozen ViT-B/16 layerwise for two complementary properties, local patch boundaries (BSDS500) and per-patch depth (NYU Depth V2), reveals a clear hierarchy: boundary structure becomes linearly decodable at layers 5-6 (AP = 0.833), while depth, which requires integrating global cues, peaks two to three layers later at layer 8 (MAE = 0.0875). Both signals collapse at the final classification layer, and random-weight controls confirm the encodings are learned rather than architectural. Causal interventions add specificity: ablating the single direction a linear depth probe reads degrades depth decoding by up to 165%, while ablating any other direction changes it by less than 1%. Targeted activation patching along that direction shows the depth signal is partially re-derived at each layer rather than passively carried in the residual stream, with mid-layer interventions persisting most strongly downstream. The result is that a classification-trained ViT develops an actively maintained spatial hierarchy that mirrors the early-to-late progression observed in the primate visual cortex.
How do vision transformers (ViTs) represent and process the world? This paper addresses this long-standing question through the first systematic analysis of 6.6K features across all layers, extracted via sparse autoencoders, and by introducing the residual replacement model, which replaces ViT computations with interpretable features in the residual stream. Our analysis reveals not only a feature evolution from low-level patterns to high-level semantics, but also how ViTs encode curves and spatial positions through specialized feature types. The residual replacement model scalably produces a faithful yet parsimonious circuit for human-scale interpretability by significantly simplifying the original computations. As a result, this framework enables intuitive understanding of ViT mechanisms. Finally, we demonstrate the utility of our framework in debiasing spurious correlations.
Understanding the internal activations of Vision Transformers (ViTs) is critical for building interpretable and trustworthy models. While Sparse Autoencoders (SAEs) have been used to extract human-interpretable features, they operate on individual layers and fail to capture the cross-layer computational structure of Transformers, as well as the relative significance of each layer in forming the last-layer representation. Alternatively, we introduce the adoption of Cross-Layer Transcoders (CLTs) as reliable, sparse, and depth-aware proxy models for MLP blocks in ViTs. CLTs use an encoder-decoder scheme to reconstruct each post-MLP activation from learned sparse embeddings of preceding layers, yielding a linear decomposition that transforms the final representation of ViTs from an opaque embedding into an additive, layer-resolved construction that enables faithful attribution and process-level interpretability. We train CLTs on CLIP ViT-B/32 and ViT-B/16 across CIFAR-100, COCO, and ImageNet-100. We show that CLTs achieve high reconstruction fidelity with post-MLP activations while preserving and even improving, in some cases, CLIP zero-shot classification accuracy. In terms of interpretability, we show that the cross-layer contribution scores provide faithful attribution, revealing that the final representation is concentrated in a smaller set of dominant layer-wise terms whose removal degrades performance and whose retention largely preserves it. These results showcase the significance of adopting CLTs as an alternative interpretable proxy of ViTs in the vision domain.
Gerasimos Chatzoudis, Konstantinos D. Polyzos, Zhuowei Li +4
Rutgers University · University of California San Diego