cs.CVOct 2, 2026

Decoding the Functional Roles of Register and High-Norm Patch Tokens in Vision Transformers

Authors: Neel Varma, Andrew Rufail, Dipika Khullar, Vasu Sharma

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Self-supervised Vision Transformers (ViTs), such as DINOv2, learn rich visual representations, but the functions of their internal tokens remain poorly understood. Recent architectures introduce dedicated register tokens to reduce high-norm out- lier patch tokens that emerge in background re- gions, yet the semantic and functional roles of both token types have not been fully established. In this paper, we analyze these roles by training sparse autoencoders (SAEs) on register-token and outlier-token activations in DINOv2. Using an automated interpretability pipeline, UMAP clus- tering, and CLIP-space cross-checks, we find that register-token features are more strongly associ- ated with high-level semantic concepts. Outlier- token features, by contrast, are more often associ- ated with lower-level structural, background, and texture-dominant patterns. Causal ablations fur- ther reveal a substantial functional asymmetry: disrupting top-activating register-derived features produces a 48.17% drop in representation cosine similarity, whereas disrupting outlier-derived fea- tures produces only a 0.31% drop. Together, our results provide evidence for token specialization in self-supervised ViTs.

Figures & tables

Figure 1: UMAP visualizations of the learned feature spaces. Register features (left) show semantic clustering, while outlier features (right) cluster around structural and texture-dominant information.
Figure 2: Visual impact of ablating the top-5 activating register features. The left column displays the resulting attention maps, while the right column shows PCA-based semantic maps where similar colors indicate similar semantic meanings. The interventions lead to significant disruption in object focus and semantic coherence.
Figure 3: Overview of the register–outlier analysis pipeline. We isolate register tokens and high-norm outlier patch tokens, train token-specific sparse autoencoders, interpret learned features using top-activating patches, VLM captions, UMAP structure, and CLIP-space checks, and measure functional importance through latent ablations.
 Cosine Similarity Drop
Ablation TargetTop-5Bottom-5
Register Tokens48.17%47.05%
Outlier Tokens0.31%0.28%
Full Normal Model (control)0.65%0.30%
Full Register Model (control)7.86%7.39%
Table 1: Reduction in cosine similarity after ablating the top-5 activating features in each condition. These causal ablations were performed on the model the SAE was trained on. The full models represent controls for their respective specialized tokens, both showing that the cosine similarity drops are significant.
Figure 4: UMAP of outlier activations after projection into CLIP space. The feature organization remains consistent with the texture-sink interpretation.
Figure 5: SAE training metrics and losses. Overall, all SAEs do not suffer from dead neurons. The full model SAEs have better reconstruction cosine similarities compared to outlier and register ones.
Figure 6: SAE latent activation-frequency distributions across token regimes. Most latents activate infrequently, indicating sparse feature usage across regimes.

Explore similar work

CardsList

  1. Zero-Ablation Overstates Register Content Dependence in DINO Vision Transformers

    Apr 15, 2026Felipe Parodi, Jordan Matelsky, Melanie SegadoVision TransformerNeural Network Interpretability

    University of Pennsylvania · Johns Hopkins Applied Physics Laboratory

  2. Vision Transformers Need More Than Registers

    Feb 25, 2026Cheng Shi, Yizhou Yu, Sibei YangVision TransformerVisual Representation Learning

    School of Computing and Data Science, The University of Hong Kong · Sun Yat-sen University

  3. Vision Transformers Need Better Token Interaction

    May 22, 2026Linxiang SuVisual Representation LearningVision Transformer

    University of Szeged