Zero-Ablation Overstates Register Content Dependence in DINO Vision Transformers

Felipe Parodi; Jordan Matelsky; Melanie Segado

Figure 1: Approach overview. We compare three ViT-S/B models with different register configurations. Hook-based ablations zero [CLS] or register hidden states at every block output, and we evaluate on global (classification, retrieval) and dense (correspondence, segmentation) tasks. Three replacement controls test whether observed deficits reflect genuine content dependence or distributional shift from out-of-distribution zero vectors.

Figure 2: Zero-ablation effects and patch geometry. (a) Task

\times

Ablation heatmap (

\Delta

pp from Full). v2 = DINOv2, v2+R = DINOv2+reg, v3 = DINOv3. Zeroing registers produces large drops, but plausible replacement controls preserve all tasks ( Tab. 1 ). (b) Effective rank (median

\pm

std): registers compress patch geometry; DINOv3 exhibits the most compression. (c) Normalized eigenspectrum (log scale, Full condition; eigenvalue index ascending): in DINOv3, variance is concentrated in the largest few eigenvalues. All models ViT-S.

Figure 3: PCA projection of patch features under ablation (ViT-S, layer 11, 3-component RGB). Rows : models. Columns : input image, Full (no ablation), Zero CLS, Zero Registers. Zero CLS barely alters spatial structure with registers present; Zero Registers drastically reorganizes the feature space.

Figure 4: Attention flow across all 12 layers (ViT-S, 200 images). Rows: source token type (CLS, registers, patches). Columns: model. Stacked areas show what fraction of attention each source directs to CLS (gray), registers (magenta), and patches (olive). Register attention builds gradually from mid-layers, yet classification dependence on registers emerges abruptly at layers 10–11 ( Fig. 8 b).

Figure 5: Attention patterns under ablation (1,000 images). (a) JS divergence vs. layer for ViT-S (dark) and ViT-B (light): register zeroing (solid) causes cascading divergence while mean-substitution (dashed) preserves attention patterns, supporting the distributional-shift interpretation. (b) CLS attention redistribution at last layer when registers are zeroed (ViT-S).

	CLS (%)		Corr. (%)		Seg. (%)		Patch cos.
Condition	v2+R	v3	v2+R	v3	v2+R	v3	v2+R	v3
Full	$67.3$	$62.0$	$69.1$	$78.9$	$71.3$	$78.5$	$1.000$	$1.000$
Zero registers	$48.4$	$25.4$	$64.3$	$57.8$	$61.7$	$47.6$	$0.605$	$0.584$
Mean-sub	$67.0$	$62.1$	$68.8$	$78.8$	$71.6$	$78.6$	$0.959$	$0.999$
Noise-sub	$67.0$	$62.0$	$68.7$	$78.7$	$71.5$	$78.6$	$0.955$	$0.998$
Shuffle	$67.8$	$62.0$	$68.5$	$78.6$	$71.2$	$78.6$	$0.952$	$0.998$

Table 1: Register ablation controls (ViT-S, layer 11). Only zeroing degrades any task; all statistically plausible replacements preserve performance despite genuinely perturbing internal states (Patch cos. column shows per-patch cosine similarity to full-condition features).

		CLS	kNN	Corr	Seg
DINOv2	Full	73.2	99.7	72.0	70.8
DINOv2	Zero CLS	0 0.1 -73.1	0 0.1 -99.6	56.1 -15.9	33.7 -37.1
DINOv2 +reg	Full	67.3	99.6	69.1	71.3
	Zero CLS	0 0.1 -67.2	0 0.1 -99.5	68.3 -0.8	70.7 -0.7
	Zero Reg	48.4 -18.9	99.0 -0.6	64.3 -4.8	61.7 -9.6
DINOv3	Full	62.0	99.8	78.9	78.5

Table 2: Task

\times

ablation matrix (ViT-S, layer 11). CLS: probe top-1 (%). kNN: Recall@1 (%). Corr: correspondence (%). Seg: segmentation mIoU (%).

\Delta

: change from Full. Bootstrap 95% CIs on key deltas in text.

Model	Ablation	[email protected]	$\Delta$
DINOv2	Full	38.5 [38.0, 39.1]	—
DINOv2	Zero CLS	15.6 [15.3, 16.0]	$-22.9$
DINOv2 +reg	Full	35.7 [35.3, 36.2]	—
	Zero CLS	35.2 [34.7, 35.7]	$-0.5$
	Zero Reg	26.2 [25.7, 26.7]	$-9.5$
DINOv3	Full	31.9 [31.4, 32.3]	—

Table 3: SPair-71k [email protected] (%, ViT-S, layer 11, 12,234 pairs).

\Delta

: change from Full. Bootstrap 95% CIs in brackets.

	CLS (%)		Corr. (%)		Seg. (%)
Condition	v2-B+R	v3-B	v2-B+R	v3-B	v2-B+R	v3-B
Full	$74.5$	$73.3$	$71.2$	$77.1$	$72.3$	$83.4$
Zero registers	$55.2$	$36.8$	$63.3$	$61.3$	$64.1$	$59.6$
Mean-sub	$74.5$	$73.4$	$70.4$	$76.6$	$72.7$	$83.3$
Noise-sub	$74.3$	$73.4$	$70.4$	$76.5$	$72.6$	$83.4$
Shuffle	$74.7$	$73.4$	$70.5$	$76.8$	$72.3$	$83.4$

Table 4: ViT-B replacement controls (layer 11). CLS: probe top-1 (%). Corr: correspondence (%). Seg: segmentation mIoU (%). All three controls preserve performance within

{\sim}1

pp of Full, matching the ViT-S pattern ( Tab. 1 ).

Figure 6: Qualitative correspondence under ablation (DINOv3-B, layer 11, tolerance 1). Green lines: correct NN matches. Red: incorrect. Top : Full condition preserves correct correspondences. Middle : Zero CLS causes minimal disruption (register buffering). Bottom : Zero Registers collapses correspondence accuracy.

Figure 7: Scale comparison: ViT-S vs. ViT-B. (a) Classification and (b) segmentation under ablation. Solid: ViT-S; dashed: ViT-B. Ablation delta patterns are consistent across scales ( Tab. 2 ).

Figure 8: Patch compression and register dependence across layers. (a) Effective rank (Full condition) decreases across layers; DINOv3 is already compressed by layer 6. (b) CLS accuracy under Full (solid) vs. register-zeroed (dashed): dependence is layer-specific, emerging only at layers 10–11. ViT-S models.

		CLS	Corr	Seg	SPair
DINOv2-B	Full	78.7	72.9	72.0	41.2
DINOv2-B	Zero CLS	0.1 -78.6	58.9 -14.0	46.1 -25.9	21.3 -19.9
DINOv2-B +reg	Full	74.5	71.2	72.3	41.1
	Zero CLS	0.1 -74.4	70.4 -0.8	72.3 0.0	41.2 +0.1
	Zero Reg	55.2 -19.3	63.3 -7.9	64.1 -8.2	28.8 -12.3
DINOv3-B	Full	73.3	77.1	83.4	37.9

Table 5: ViT-B task

\times

ablation matrix (layer 11). CLS: probe top-1 (%). Corr: correspondence (%). Seg: segmentation mIoU (%). SPair: [email protected] (%).

\Delta

: change from Full.

Model	Ablation	mIoU
DINOv2	Full	$70.8\pm 0.10$
DINOv2	Zero CLS	$33.7\pm 0.07$
DINOv2+reg	Full	$71.3\pm 0.07$
	Zero CLS	$70.7\pm 0.11$
	Zero Reg	$61.7\pm 0.14$
DINOv3	Full	$78.5\pm 0.00$

Table 6: Segmentation probe mIoU (%, ViT-S, VOC2012 val, layer 11, mean

\pm

std over 3 seeds).

Model	Ablation	GT acc	Cycle
DINOv2	Full	72.0 [71.4, 72.7]	0.468
DINOv2	Zero CLS	56.1 [55.2, 57.1]	0.411
DINOv2+reg	Full	69.1 [68.4, 69.9]	0.454
	Zero CLS	68.3 [67.6, 69.2]	0.450
	Zero Reg	64.3 [63.4, 65.2]	0.433
DINOv3	Full	78.9 [78.2, 79.8]	0.477

Table 7: Patch correspondence (ViT-S, layer 11, 2,000 pairs). GT acc: ground-truth accuracy (%). Cycle: cycle consistency (A

\to

B

\to

A). Bootstrapped 95% CI in brackets.

Model	Ablation	Tol = 0	Tol = 1	Tol = 2
DINOv2	Full	38.5	72.0	80.3
DINOv2	Zero CLS	28.0	56.1	65.2
DINOv2 +reg	Full	37.5	69.1	77.1
	Zero CLS	36.8	68.3	76.4
	Zero Reg	34.0	64.3	72.7
DINOv3	Full	45.2	78.9	85.3

Table 8: Correspondence accuracy (%) at different tolerance thresholds (ViT-S, layer 11). Tol = 0: exact match; 1:

\pm

1 patch (default); 2:

\pm

2 patches.

Model	Condition	CLS (%)	$\Delta$	Reg-zero $\Delta$
DINOv2	Rand. patch	$72.4\pm 0.10$	$-0.8$	—
DINOv2+reg	Rand. patch	$66.3\pm 0.03$	$-1.0$	$-18.9$
DINOv3	Rand. patch	$61.5\pm 0.05$	$-0.5$	$-36.6$
DINOv2-B	Rand. patch	$78.4\pm 0.07$	$-0.3$	—
DINOv2-B+reg	Rand. patch	$74.5\pm 0.04$	$\phantom{-}0.0$	$-19.3$
DINOv3-B	Rand. patch	$71.0\pm 0.06$	$-2.3$	$-36.5$

Table 9: Random-patch negative control: CLS top-1 (%) when zeroing 4 random patch tokens (ViT-S and ViT-B, layer 11, 5 seeds).

\Delta

: change from Full baseline ( Tabs. 2 and 5 ). Reg-zero

\Delta

reproduced for comparison.

Comparison	Observed $\Delta$	$p$
DINOv3 vs. DINOv2+reg $\Delta_{\text{ZeroReg}}$ (CLS)	$17.7$ pp	$<0.001$
CLS-zeroing buffering (Seg, with vs. without regs)	$10.7$ pp	$<0.001$
ViT-S vs. ViT-B register dependence (DINOv3 $\Delta_{\text{ZeroReg}}$ )	$0.1$ pp	$0.80$ (ns)

Table 10: Paired permutation test p-values for headline comparisons (10,000 permutations, two-sided).

Model	Layer 6	Layer 11
DINOv2	$32.3\pm 6.3$	$13.5\pm 4.5$
DINOv2+reg	$41.7\pm 7.4$	$\phantom{0}8.7\pm 3.3$
DINOv3	$\phantom{0}6.4\pm 7.0$	$\phantom{0}4.0\pm 1.2$

Table 11: Effective rank (median

\pm

std) at layers 6 and 11 (ViT-S, 500 ImageNet images). Layer 11 values match the Full rows of Tab. 12 .

Model	Ablation	Erank	Entropy
DINOv2	Full	13.5	2.61
DINOv2	Zero CLS	18.1	2.90
DINOv2+reg	Full	0 8.7	2.16
	Zero CLS	0 9.8	2.29
	Zero Reg	11.3	2.43
DINOv3	Full	0 4.0	—

Table 12: Median Gram statistics (ViT-S, layer 11, 2,000 images). DINOv3 entropy omitted: its

14{\times}14

patch grid yields a

196{\times}196

Gram matrix vs.

256{\times}256

for DINOv2 models, making eigenspectrum entropy incomparable.

Figure 9: Task performance across transformer layers. (a) CLS classification (linear probe, 50 epochs, 1 seed). Classification emerges at layers 10–11. (b) Patch correspondence (tolerance = 1). Correspondence peaks at layers 6–8 then declines, except DINOv3 which maintains

78.9\%

at layer 11. ViT-S models.

Figure 10: CLS attention (ViT-S, last layer, 200 images). (a) CLS attention fraction per token type. DINOv2+reg:

17.9\%

to registers; DINOv3:

29.1\%

. (b) Per-register breakdown. This routing structure is maintained under all plausible register replacements.

Zero-Ablation Overstates Register Content Dependence in DINO Vision Transformers

Abstract

Figures & tables

Explore similar work

Decoding the Functional Roles of Register and High-Norm Patch Tokens in Vision Transformers

Decoding Register Tokens Preserves Global Scene Details

REDI: Corpus Aware Patch Ranking for DINOv3 Token Reduction