Vision Transformers (ViTs), when pre-trained on large-scale data, provide general-purpose representations for diverse downstream tasks. However, artifacts in ViTs are widely observed across different supervision paradigms and downstream tasks. Through systematic analysis of artifacts in ViTs, we find that their fundamental mechanisms have yet to be sufficiently elucidated. In this paper, through systematic analysis, we conclude that these artifacts originate from a lazy aggregation behavior: ViT uses semantically irrelevant background patches as shortcuts to represent global semantics, driven by global attention and Coarse-grained semantic supervision. Our solution selectively integrates patch features into the CLS token, reducing the influence of background-dominated shortcuts and consistently improving performance across 12 benchmarks under label-, text-, and self-supervision. We hope this work offers a new perspective on ViT behavior.
Figures & tables
Figure 2: Patch - score distribution and masking probe on ImageNet - 1k. (a) Normalized distributions of patch scores for foreground vs. background. (b) Removing top- k high - score patches (up to 70% ) does not hurt accuracy and even can improve it.
Figure 3: Training dynamics on ImageNet-1k. Left: top-1 accuracy; Right: Point-in-Box score. As training proceeds, ViT’s classification accuracy steadily improves, yet its Point-in-Box score remains nearly flat (around 0.42→0.44 ) and consistently lower than ResNet’s across the entire training process.
Figure 4: Effect of Coarse-grained semantic supervision. Increasing the patch size reduces background tokens by 10% . Effect: Point-in-Box rises from 0.44 to 0.52 , and high-score patches shift toward foreground. Trade-off: classification accuracy decreases, indicating that coarse-grained semantic supervision contributes to artifacts, while naive patch coarsening compromises recognition.
Figure 5: Where does the CLS token in LaSt-ViT “look at"? For each image, patches whose vote count exceeds 50%, 30%, or 20% of the largest vote count within the image are visualized in red from left to right, respectively. After the application of LaSt-ViT , highly voted patches consistently correspond to foreground regions, showing that the CLS token primarily aggregates foreground tokens rather than background ones.
Figure 6: Evaluation of the LaSt-ViT in feature norm. Specifically, the elimination of artifacts also removes the high-norm phenomena [ 5 ] , highlighting our deeper perspective on addressing artifacts.
Model
Backbone
COCO-Obj.
ADE20K
City.
VOC20
Context59
COCO-Stf.
CLIP [ 27 ]
ViT-B/16
8.8
3.1
6.5
49.0
11.2
7.2
CLIP ( + LazyStrike )
ViT-B/16
13.3 ( +4.5 )
8.3 ( +5.2 )
12.1 ( +5.6 )
75.0 ( +26.0 )
15.2 ( +4.0 )
11.8 ( +4.6 )
MetaCLIP [ 39 ]
ViT-B/16
4.8
2.9
5.8
39.6
9.3
6.2
MetaCLIP ( + LazyStrike )
ViT-B/16
14.1 ( +9.3 )
7.9 ( +5.0 )
11.1 ( +5.3 )
72.8 ( +33.2 )
15.5 ( +6.2 )
12.0 ( +5.8 )
EVACLIP [ 32 ]
ViT-B/16
15.0
6.7
12.2
56.5
14.1
9.7
EVACLIP ( + LazyStrike )
ViT-B/16
26.2 ( +11.2 )
14.8 ( +8.1 )
24.5 ( +12.3 )
79.6 ( +23.1 )
24.7 ( +10.6 )
18.3 ( +8.6 )
Table 4: Evaluation results (mIoU, %) on six semantic segmentation benchmarks . Our results are marked in gray . LazyStrike consistently improves semantic segmentation results under text supervision across different type of CLIP [ 27 ] and model sizes, demonstrating that, after understanding the essence of the problem, a simple approach can uniformly address issues across different models.
Method
Backbone
COCO Detection
LVIS Segmentation
AP50 box
AP50 basebox
AP50 novelbox
AP mask
AP freqmask
AP commmask
AP novelmask
ConvNet based
F-VLM [ 16 ]
RN50
1 39.6
1 /
1 28.0
1 24.2
1 26.9
1 24.0
1 18.6
F-VLM [ 16 ]
RN50x64
1 /
1 /
1 /
1 34.9
1 /
1 /
1 32.8
ViT based
F-ViT [ 37 ]
ViT-B/16
1 34.9
1 41.0
1 17.5
1 15.4
1 20.6
1 12.3
1 11.5
Table 5: Evaluation results on open-vocabulary benchmark . Our results are marked in gray . LazyStrike consistently enhances performance on open-vocabulary dense tasks, by demonstrating that frozen ViT can achieve comparable performance with ConvNet [ 11 ] .
Figure 7: Visualization of PCA components. We compute the PCA of the patch features and visualize the first 3 components for the foreground object. With LazyStrike , ViT under label-supervision also distinguish foreground from background and separate object parts, enhancing feature representation.
Training Vision Transformers (ViTs) presents significant challenges, one of which is the emergence of artifacts in attention maps, hindering their interpretability. Darcet et al. (2024) investigated this phenomenon and attributed it to the need of ViTs to store global information beyond the [CLS] token. They proposed a novel solution involving the addition of empty input tokens, named registers, which successfully eliminate artifacts and improve the clarity of attention maps. In this work, we reproduce the findings of Darcet et al. (2024) and evaluate the generalizability of their claims across multiple models, including DINO, DINOv2, OpenCLIP, and DeiT3. While we confirm the validity of several of their key claims, our results reveal that some claims do not extend universally to other models. Additionally, we explore the impact of model size, extending their findings to smaller models. Finally, we untie terminology inconsistencies found in the original paper and explain their impact when generalizing to a wider range of models.
Spiros Baxevanakis, Platon Karageorgis, Ioannis Dravilas +1
Vision Transformers (ViTs) can learn strong image-level representations while their patch representations become less effective for dense prediction during prolonged training. We revisit this dense degradation phenomenon and argue that it is not fully explained by high-norm artifacts alone. Instead, we characterize semantic diffusion: an optimization shortcut in which global semantic information spreads through patch tokens beyond what is locally justified. Our analysis shows that dense representation quality is not captured by locality alone: shallow features can remain better aligned with foreground regions yet underperform deeper features, and [CLS] features remain complementary for dense prediction. These observations suggest that the goal should not be to remove global context, but to make token interactions more selective. We therefore study sparse attention as a minimal intervention, replacing softmax attention with entmax-1.5 while preserving global token connectivity. On DINOv1 ViT-S/16 trained for 200 epochs on ImageNet-1K, this change preserves ImageNet linear probing accuracy and substantially improves semantic segmentation performance: VOC mIoU increases from 42.80 to 48.78, ADE20K from 19.85 to 21.97, and Cityscapes from 36.79 to 37.87. These results suggest that selective token mixing is a simple and effective bias for improving dense ViT representations.
Self-supervised Vision Transformers (ViTs), such as DINOv2, learn rich visual representations, but the functions of their internal tokens remain poorly understood. Recent architectures introduce dedicated register tokens to reduce high-norm out- lier patch tokens that emerge in background re- gions, yet the semantic and functional roles of both token types have not been fully established. In this paper, we analyze these roles by training sparse autoencoders (SAEs) on register-token and outlier-token activations in DINOv2. Using an automated interpretability pipeline, UMAP clus- tering, and CLIP-space cross-checks, we find that register-token features are more strongly associ- ated with high-level semantic concepts. Outlier- token features, by contrast, are more often associ- ated with lower-level structural, background, and texture-dominant patterns. Causal ablations fur- ther reveal a substantial functional asymmetry: disrupting top-activating register-derived features produces a 48.17% drop in representation cosine similarity, whereas disrupting outlier-derived fea- tures produces only a 0.31% drop. Together, our results provide evidence for token specialization in self-supervised ViTs.