Zero-ablation -- replacing token activations with zero vectors -- is widely used to probe token function in vision transformers. Register zeroing in DINOv2+registers and DINOv3 produces large drops (up to −36.6,pp classification, −30.9,pp segmentation), suggesting registers are functionally indispensable. However, three replacement controls -- mean-substitution, noise-substitution, and cross-image register-shuffling -- preserve performance across classification, correspondence, and segmentation, remaining within ∼1,pp of the unmodified baseline. Per-patch cosine similarity shows these replacements genuinely perturb internal representations, while zeroing causes disproportionately large perturbations, consistent with why it alone degrades tasks. We conclude that zero-ablation overstates dependence on exact register content. In the frozen-feature evaluations we test, performance depends on plausible register-like activations rather than on exact image-specific values. Registers nevertheless buffer dense features from [CLS] dependence and are associated with compressed patch geometry. These findings, including the replacement-control results, replicate at ViT-B scale.
Figures & tables
Figure 1: Approach overview. We compare three ViT-S/B models with different register configurations. Hook-based ablations zero [CLS] or register hidden states at every block output, and we evaluate on global (classification, retrieval) and dense (correspondence, segmentation) tasks. Three replacement controls test whether observed deficits reflect genuine content dependence or distributional shift from out-of-distribution zero vectors.
Figure 2: Zero-ablation effects and patch geometry. (a) Task × Ablation heatmap ( Δ pp from Full). v2 = DINOv2, v2+R = DINOv2+reg, v3 = DINOv3. Zeroing registers produces large drops, but plausible replacement controls preserve all tasks ( Tab. 1 ). (b) Effective rank (median ± std): registers compress patch geometry; DINOv3 exhibits the most compression. (c) Normalized eigenspectrum (log scale, Full condition; eigenvalue index ascending): in DINOv3, variance is concentrated in the largest few eigenvalues. All models ViT-S.
Figure 3: PCA projection of patch features under ablation (ViT-S, layer 11, 3-component RGB). Rows : models. Columns : input image, Full (no ablation), Zero CLS, Zero Registers. Zero CLS barely alters spatial structure with registers present; Zero Registers drastically reorganizes the feature space.
Figure 4: Attention flow across all 12 layers (ViT-S, 200 images). Rows: source token type (CLS, registers, patches). Columns: model. Stacked areas show what fraction of attention each source directs to CLS (gray), registers (magenta), and patches (olive). Register attention builds gradually from mid-layers, yet classification dependence on registers emerges abruptly at layers 10–11 ( Fig. 8 b).
Figure 5: Attention patterns under ablation (1,000 images). (a) JS divergence vs. layer for ViT-S (dark) and ViT-B (light): register zeroing (solid) causes cascading divergence while mean-substitution (dashed) preserves attention patterns, supporting the distributional-shift interpretation. (b) CLS attention redistribution at last layer when registers are zeroed (ViT-S).
CLS (%)
Corr. (%)
Seg. (%)
Patch cos.
Condition
v2+R
v3
v2+R
v3
v2+R
v3
v2+R
v3
Full
67.3
62.0
69.1
78.9
71.3
78.5
1.000
1.000
Zero registers
48.4
25.4
64.3
57.8
61.7
47.6
0.605
0.584
Mean-sub
67.0
62.1
68.8
78.8
71.6
78.6
0.959
0.999
Noise-sub
67.0
62.0
68.7
78.7
71.5
78.6
0.955
0.998
Shuffle
67.8
62.0
68.5
78.6
71.2
78.6
0.952
0.998
Table 1: Register ablation controls (ViT-S, layer 11). Only zeroing degrades any task; all statistically plausible replacements preserve performance despite genuinely perturbing internal states (Patch cos. column shows per-patch cosine similarity to full-condition features).
Table 4: ViT-B replacement controls (layer 11). CLS: probe top-1 (%). Corr: correspondence (%). Seg: segmentation mIoU (%). All three controls preserve performance within ∼1 pp of Full, matching the ViT-S pattern ( Tab. 1 ).
Figure 6: Qualitative correspondence under ablation (DINOv3-B, layer 11, tolerance 1). Green lines: correct NN matches. Red: incorrect. Top : Full condition preserves correct correspondences. Middle : Zero CLS causes minimal disruption (register buffering). Bottom : Zero Registers collapses correspondence accuracy.
Figure 7: Scale comparison: ViT-S vs. ViT-B. (a) Classification and (b) segmentation under ablation. Solid: ViT-S; dashed: ViT-B. Ablation delta patterns are consistent across scales ( Tab. 2 ).
Figure 8: Patch compression and register dependence across layers. (a) Effective rank (Full condition) decreases across layers; DINOv3 is already compressed by layer 6. (b) CLS accuracy under Full (solid) vs. register-zeroed (dashed): dependence is layer-specific, emerging only at layers 10–11. ViT-S models.
Self-supervised Vision Transformers (ViTs), such as DINOv2, learn rich visual representations, but the functions of their internal tokens remain poorly understood. Recent architectures introduce dedicated register tokens to reduce high-norm out- lier patch tokens that emerge in background re- gions, yet the semantic and functional roles of both token types have not been fully established. In this paper, we analyze these roles by training sparse autoencoders (SAEs) on register-token and outlier-token activations in DINOv2. Using an automated interpretability pipeline, UMAP clus- tering, and CLIP-space cross-checks, we find that register-token features are more strongly associ- ated with high-level semantic concepts. Outlier- token features, by contrast, are more often associ- ated with lower-level structural, background, and texture-dominant patterns. Causal ablations fur- ther reveal a substantial functional asymmetry: disrupting top-activating register-derived features produces a 48.17% drop in representation cosine similarity, whereas disrupting outlier-derived fea- tures produces only a 0.31% drop. Together, our results provide evidence for token specialization in self-supervised ViTs.
Pretrained vision transformers learn structured image representations spanning spatial patch tokens and non-spatial tokens. Recent analyses suggest that non-spatial register tokens encode global scene properties such as illumination, blur, and style. Yet patch-based image decoders discard these outputs, leaving the decoder to recover image appearance from patches alone. We investigate whether retaining register features improves decoding when the complete spatial patch representation is already available. We study this question using representation autoencoders (RAEs) with frozen register-equipped encoders. Our decoder processes four register tokens together with all patch tokens through joint self-attention. Appearance probes, paired reconstruction-error analysis, and fixed-decoder register substitutions examine which global image properties are accessible from registers and how their values affect reconstructed pixels. For generation, we train a multi-head attention pooling (MHAP) predictor to recover register features from sampled patches, keeping the pretrained diffusion model and its samples unchanged. On ImageNet-1K, our ViT-B decoders achieve mean reconstruction FIDs of 0.472 with DINOv2 and 0.442 with DINOv3, reductions of 9.5% and 5.2% relative to their respective patch-only baselines. The corresponding PSNRs are 19.73 and 20.54 dB. For generation, our DINOv2 models achieve a mean FID of 1.641, 9.8% below the patch-only baseline on identical sampled patches.
David Jacob Drexlin, Julius Hense, Jonas Dippel
Machine Learning Group, Technische Universität Berlin · BIFOLD – Berlin Institute for the Foundations of Learning and Data Franklinstraße 28/29, 10587 Berlin, Germany · Aignostics, Germany
Most token reduction methods for Vision Transformers seek favorable tradeoffs between accuracy and efficiency by pruning, merging, or pooling patch tokens. REDI (Relevance for DINOv3 Token Reduction) studies this question through a controlled supervised reference: how should a fixed token budget be allocated across patches for image classification? REDI quantizes final block DINOv3 patch representations into a visual vocabulary and derives class conditioned corpus scores using supervised TF-IDF over visual words. For each validation image, the ground truth class selects a row of the TF-IDF table, and four transformed views produce a TF-IDF map aligned to a reference center crop. A separate dense pass on the same crop provides an attention map. After independent min max normalization, their elementwise product defines the REDI score. A fixed keep, merge, and compress operator then uses score rank to assign patch roles and score magnitude to weight merging and compression. With precomputed REDI scores, a frozen DINOv3 ViT-B/16 backbone, and the same linear classifier used for dense evaluation, the operator reduces the sequence length from 201 to 107 tokens, a 46.8% sequence reduction. The REDI variant based on incoming attention mass achieves 84.706% Top-1 accuracy on ImageNet-1K, compared with 83.514% for the dense baseline, 82.634% for incoming attention mass alone, and 81.796% for supervised TF-IDF alone. The same corpus term also improves reduced classification for three alternative attention formulations relative to their attention only counterparts. Together, these controlled comparisons indicate that class specific corpus statistics and image specific attention provide complementary signals for patch ranking in this setting.
Chanjong Im, Sebastian Diem, Thomas Mandl
University of Magdeburg · University of Hildesheim