cs.CVFeb 25, 2026

Vision Transformers Need More Than Registers

Authors: Cheng Shi, Yizhou Yu, Sibei Yang

Organizations: School of Computing and Data Science, The University of Hong Kong · Sun Yat-sen University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision Transformers (ViTs), when pre-trained on large-scale data, provide general-purpose representations for diverse downstream tasks. However, artifacts in ViTs are widely observed across different supervision paradigms and downstream tasks. Through systematic analysis of artifacts in ViTs, we find that their fundamental mechanisms have yet to be sufficiently elucidated. In this paper, through systematic analysis, we conclude that these artifacts originate from a lazy aggregation behavior: ViT uses semantically irrelevant background patches as shortcuts to represent global semantics, driven by global attention and Coarse-grained semantic supervision. Our solution selectively integrates patch features into the CLS token, reducing the influence of background-dominated shortcuts and consistently improving performance across 12 benchmarks under label-, text-, and self-supervision. We hope this work offers a new perspective on ViT behavior.

Figures & tables

Figure 2: Patch - score distribution and masking probe on ImageNet - 1k. (a) Normalized distributions of patch scores for foreground vs. background. (b) Removing top- kk high - score patches (up to 70%70\% ) does not hurt accuracy and even can improve it.
Figure 3: Training dynamics on ImageNet-1k. Left: top-1 accuracy; Right: Point-in-Box score. As training proceeds, ViT’s classification accuracy steadily improves, yet its Point-in-Box score remains nearly flat (around 0.42→0.440.42\rightarrow 0.44 ) and consistently lower than ResNet’s across the entire training process.
Figure 4: Effect of Coarse-grained semantic supervision. Increasing the patch size reduces background tokens by 10%10\% . Effect: Point-in-Box rises from 0.440.44 to 0.520.52 , and high-score patches shift toward foreground. Trade-off: classification accuracy decreases, indicating that coarse-grained semantic supervision contributes to artifacts, while naive patch coarsening compromises recognition.
Figure 5: Where does the CLS token in LaSt-ViT “look at"? For each image, patches whose vote count exceeds 50%, 30%, or 20% of the largest vote count within the image are visualized in red from left to right, respectively. After the application of LaSt-ViT , highly voted patches consistently correspond to foreground regions, showing that the CLS token primarily aggregates foreground tokens rather than background ones.
Figure 6: Evaluation of the LaSt-ViT in feature norm. Specifically, the elimination of artifacts also removes the high-norm phenomena [ 5 ] , highlighting our deeper perspective on addressing artifacts.
ModelBackboneCOCO-Obj.ADE20KCity.VOC20Context59COCO-Stf.
CLIP [ 27 ]ViT-B/168.83.16.549.011.27.2
CLIP ( + LazyStrike )ViT-B/1613.3 ( +4.5 )8.3 ( +5.2 )12.1 ( +5.6 )75.0 ( +26.0 )15.2 ( +4.0 )11.8 ( +4.6 )
MetaCLIP [ 39 ]ViT-B/164.82.95.839.69.36.2
MetaCLIP ( + LazyStrike )ViT-B/1614.1 ( +9.3 )7.9 ( +5.0 )11.1 ( +5.3 )72.8 ( +33.2 )15.5 ( +6.2 )12.0 ( +5.8 )
EVACLIP [ 32 ]ViT-B/1615.06.712.256.514.19.7
EVACLIP ( + LazyStrike )ViT-B/1626.2 ( +11.2 )14.8 ( +8.1 )24.5 ( +12.3 )79.6 ( +23.1 )24.7 ( +10.6 )18.3 ( +8.6 )
Table 4: Evaluation results (mIoU, %) on six semantic segmentation benchmarks . Our results are marked in gray . LazyStrike consistently improves semantic segmentation results under text supervision across different type of CLIP [ 27 ] and model sizes, demonstrating that, after understanding the essence of the problem, a simple approach can uniformly address issues across different models.
MethodBackboneCOCO Detection LVIS Segmentation
 AP50 box{}^{\text{box}}AP50 basebox{}^{\text{box}}_{\text{base}}AP50 novelbox{}^{\text{box}}_{\text{novel}} AP mask{}^{\text{mask}}AP freqmask{}^{\text{mask}}_{\text{freq}}AP commmask{}^{\text{mask}}_{\text{comm}}AP novelmask{}^{\text{mask}}_{\text{novel}}
ConvNet based         
F-VLM [ 16 ]RN501 39.61 /1 28.0 1 24.21 26.91 24.01 18.6
F-VLM [ 16 ]RN50x641 /1 /1 / 1 34.91 /1 /1 32.8
ViT based         
F-ViT [ 37 ]ViT-B/161 34.91 41.01 17.5 1 15.41 20.61 12.31 11.5
Table 5: Evaluation results on open-vocabulary benchmark . Our results are marked in gray . LazyStrike consistently enhances performance on open-vocabulary dense tasks, by demonstrating that frozen ViT can achieve comparable performance with ConvNet [ 11 ] .
Figure 7: Visualization of PCA components. We compute the PCA of the patch features and visualize the first 3 components for the foreground object. With LazyStrike , ViT under label-supervision also distinguish foreground from background and separate object parts, enhancing feature representation.

Explore similar work

CardsList

  1. Do All Vision Transformers Need Registers? A Cross-Architectural Reassessment

    Mar 26, 2026Spiros Baxevanakis, Platon Karageorgis, Ioannis Dravilas +1Vision TransformerTransformer Interpretability

    Informatics Institute, University of Amsterdam

  2. Vision Transformers Need Better Token Interaction

    May 22, 2026Linxiang SuVisual Representation LearningVision Transformer

    University of Szeged

  3. Decoding the Functional Roles of Register and High-Norm Patch Tokens in Vision Transformers

    Oct 2, 2026Neel Varma, Andrew Rufail, Dipika Khullar +1Vision TransformerTransformer Interpretability