The Locality Trade-off in Masked Patch Self-Distillation

Xi Weng; Zhaoyu Liu; Lianyu Hu; Jin Song Dong

cs.CVOct 23, 2026

The Locality Trade-off in Masked Patch Self-Distillation

Authors: Xi Weng, Zhaoyu Liu, Lianyu Hu, Jin Song Dong

Organizations: School of Computing, National University of Singapore · Nanyang Technological University

Abstract

Masked patch self-distillation is a key component of modern vision foundation models, including DINOv2 and DINOv3, yet its implications for local visual information remain unclear. We show that its semantic benefits can coexist with a scale-dependent locality cost. Using two annotation-free diagnostics, we distinguish recoverable local appearance from spatial correspondence. In smaller models, adding iBOT substantially weakens patch correspondence while largely preserving appearance decodability. In larger models, correspondence improves, but pixel and texture evidence becomes less linearly accessible in deep features. Thus, scaling changes the form of the locality cost rather than uniformly eliminating it. Our analysis of Gram anchoring further shows that preserving patch relations does not guarantee simultaneous improvements in both properties. These findings motivate evaluating and designing visual representations around the complementary demands of semantic abstraction, appearance preservation, and spatial correspondence. Our code is available at https://github.com/winci-ai/jea.

Figures & tables

Figure 1: Anchor-patch similarity heatmaps for DINO and DINO +iBOT on ADE20K examples, with semantic annotations shown for reference. Yellow boxes mark the anchor patches. For ViT-B, adding iBOT produces broader responses over the building and mountain, reducing distinctions associated with windows and rugged cliff structures. This effect is alleviated in ViT-L, where the $+$ iBOT model also exhibits more spatially selective responses to these structures. The semantic masks do not resolve many of these within-region differences, illustrating why segmentation performance alone cannot characterize fine-grained local representations. Additional examples appear in Figure 10 .

Figure 2: Downstream comparison of DINO and DINO +iBOT across ViT-S/16, ViT-B/16, ViT-L/16, and ViT-H/16, pretrained on ImageNet-1K for 300 epochs under matched settings within each scale. Adding iBOT generally benefits tasks dominated by semantic understanding, whereas its effects on tasks requiring fine-grained local information vary with model scale. In particular, the degradation in DAVIS tracking observed for smaller backbones reverses to modest gains for larger backbones. Task scores other than optical-flow EPE are percentages; normalized patch rank is multiplied by 100. Arrows indicate the preferred direction for task metrics:

\uparrow

denotes higher is better and

\downarrow

denotes lower is better. Patch effective rank measures spectral diversity. Numerical results and training trajectories are provided in Table 2 and Figure 11 , respectively.

Figure 3: DAVIS video object tracking visualization. Here we compare tracking masks produced by DINO and DINO +iBOT on videos containing visually similar same-category instances and occlusions. More visualization results can be seen in Figure 12 .

Figure 4: Quantitative results of Local appearance decodability across different layers.

\Delta

denotes the difference between the DINO +iBOT and its corresponding DINO model under the same pretraining setup. To validate that the diagnostic indeed captures local visual evidence, we also include raw-pixel prediction models, such as SimMIM and MAE, which are also pretrained using ViT-B/16 on ImageNet. We also report the evolution of each descriptor metric during training in Figure 13 , and show the decoding visualization in Figure 14 .

Figure 5: Quantitative results of local-to-global correspondence across different layers.

\Delta

denotes the difference between the DINO +iBOT model and its corresponding DINO model under the same pretraining setup. We also report the evolution of each metric during training in Figure 16 .

Figure 6: Local-to-global correspondence visualizations for DINO and DINO +iBOT , showing RANSAC box estimates, similarity heatmaps, and patch matches. With ViT-B/16, adding iBOT weakens correspondence: reliable matches are largely confined to larger foreground objects, while smaller objects are incorrectly matched or poorly localized. This failure pattern is not observed in the ViT-L/16 examples, where the $+$ iBOT model maintains correspondence for smaller objects. These illustrate the scale-dependent effect reported in Figure 5 . Additional examples appear in Figure 17 .

Figure 7: Downstream trajectories during epochs 250–300. Rows show ViT-B/16 and ViT-L/16; the dashed line marks Gram activation at epoch 260. Panels use independent linear vertical scales. Non-finite optical-flow results and EPE values above 100 are shown as gaps. Optical flow measures training-set fitting.

Figure 8: Local Appearance Decodability across layers at epoch 300. DINO-Gram improves deep-layer decodability more consistently for ViT-L/16 than for ViT-B/16. Values are

R^{2}\times 100

; nonlinear asinh axes emphasize deep-layer differences.

Figure 9: Local-to-Global Correspondence at epoch 300 with crop area ratio 0.1. Both Gram variants improve deep-layer correspondence for ViT-B/16, especially DINO-Gram. Neither consistently improves the four metrics for ViT-L/16. All values are percentages.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Objective	Batch size	Memory (GB)	Time
ViT-S/16	DINO	1024	28	1 d 7 h
	DINO +iBOT	1024	100	2 d 5 h
ViT-B/16	DINO	1024	47	3 d 1 h
	DINO +iBOT	1024	125	3 d 8 h
ViT-L/16	DINO	1024	100	5 d 15 h
	DINO +iBOT	1024	129	6 d 5 h

Appendix

Table 1: Compute resources for 300-epoch pretraining on ImageNet-1K. All runs use four NVIDIA H200 GPUs with bf16 mixed precision and FSDP. Batch size denotes the global batch size. Memory is the approximate peak usage per GPU, and time is the approximate wall-clock duration of a complete run.

Figure 10: Additional anchor-patch similarity heatmaps for DINO and DINO +iBOT using ViT-B/16 and ViT-L/16.

Arch.	Metric	Better	DINO	DINO +iBOT	$\Delta$
ViT-S/16	kNN top-1	$\uparrow$	70.88	71.27	+0.39
	VOC2012 Seg. mIoU	$\uparrow$	53.13	56.02	+2.89
	VOC2007 Det. mAP	$\uparrow$	42.88	43.96	+1.08
	RParis Hard mAP	$\uparrow$	29.75	30.78	+1.03
	ROxford Hard mAP	$\uparrow$	13.34	12.03	-1.31
	Copy strong mAP	$\uparrow$	84.14	83.69	-0.45

Appendix

Table 2: Numerical results for the four backbone scales in Figure 2 . All rows use epoch 300 except Sintel EPE, which uses the latest common retained checkpoint: epochs 180/260/300/270 for ViT-S/B/L/H. Sintel scores are evaluated on training videos rather than a held-out split. Performance metrics other than EPE are percentages. Normalized patch rank is the mean patch effective rank divided by the number of patches and multiplied by 100; it describes spectral dimensionality rather than task accuracy.

\Delta

denotes DINO +iBOT minus DINO, computed before rounding. Green and red indicate improved and degraded task performance, respectively; patch-rank changes are shown without a performance interpretation.

Figure 11: Training trajectories of DINO and DINO +iBOT across four backbone scales. The effect of iBOT varies with the task, backbone scale, and training stage. Improvements in classification and dense semantic prediction coexist with mixed effects on retrieval and correspondence. Non-finite optical-flow results and EPE values exceeding 100 are excluded.

Figure 12: Additional DAVIS video object tracking visualization for DINO and DINO +iBOT using ViT-B/16 and ViT-L/16.

Figure 13: Training trajectories of local appearance decodability for each descriptor group.

Figure 14: Visualization of local appearance decodability. Here we compare the RGB mean, HOG, DCT energy, and raw PCA results produced by ViT-B/16 and ViT-L/16 models of DINO and DINO +iBOT .

Figure 15: Local-to-global correspondence across crop area ratios. Each ratio specifies the approximate fraction of the source image used to construct a local query.

\Delta

denotes DINO +iBOT minus DINO under matched pretraining settings and identical sampled crops. Positive values favor DINO +iBOT for all reported metrics.

Figure 16: Training trajectories of local-to-global correspondence for DINO and DINO +iBOT across backbone scales. Rows show the backbone and evaluated layer: blocks 3/6/9/12 for ViT-S/B, 6/12/18/24 for ViT-L, and 8/16/24/32 for ViT-H. The sustained deterioration observed in smaller

+

iBOT models is alleviated in larger backbones.

Figure 17: Additional visualization of local-global correspondence for ViT-B/16 and ViT-L/16 models of DINO and DINO +iBOT .

Asset	Use in this paper	Credit / source	License or terms of use
ImageNet-1K	Self-supervised pretraining, $k$ -NN evaluation, and diagnostic image splits	Deng et al. [13] ; official access terms	Access provided for non-commercial research and educational use under ImageNet terms. ImageNet does not own the underlying image copyrights, which remain with the original owners.
PASCAL VOC 2007 / 2012	VOC2007 object detection and VOC2012 semantic segmentation	Everingham et al. [17] , Everingham et al. [16] ; VOC challenge pages	The official database-rights statement notes that VOC contains Flickr images, whose use must respect the corresponding Flickr terms of use.
MS COCO	Copy-detection distractors and whitening images	Lin et al. [26] ; COCO terms of use	COCO annotations are released under CC BY 4.0; COCO images are subject to the Flickr terms and to the image-level license metadata provided by COCO.
$\mathcal{R}$ Oxford5k / $\mathcal{R}$ Paris6k	Image retrieval evaluation	Radenović et al. [30] ; Revisited Oxford / Paris benchmark	Public benchmark data and annotations used for academic evaluation. The project page does not list a standard open-source license for the images; we do not redistribute the raw images.
ADE20K	Similarity heatmaps and reference semantic masks	Zhou et al. [42] ; official terms of use	Images are provided for non-commercial research and education; copyright remains with their owners. The official terms specify BSD-3 terms for the annotations and software.
INRIA Copydays	Copy detection and local-to-global correspondence diagnostics	Douze et al. [15] ; Copydays project page	INRIA holds copyright on the dataset images and distributes them “as is” without express or implied warranties. Used here only as a research benchmark; raw images are not redistributed.

Appendix

Table 3: Datasets and benchmarks used in the paper.

Asset	Use in this paper	Credit / source	License or terms of use
DINOv2	Pretraining framework and ViT implementation	Oquab et al. [28] ; DINOv2 repository	Apache License 2.0 for the DINOv2 code and standard model weights, per the repository license.
MMSegmentation	Linear semantic segmentation evaluation	MMSegmentation repository	Apache License 2.0.
MMDetection	Frozen-backbone object detection evaluation	MMDetection repository	Apache License 2.0.
MMFlow	Frozen-backbone optical-flow evaluation and PWCNet implementation	MMFlow repository ; PWCNet paper [ 34 ]	Apache License 2.0 for the MMFlow toolbox.
MAE	Raw-pixel masked-modeling baseline for local-appearance decodability	He et al. [21] ; MAE repository	CC BY-NC 4.0, per the repository license.
SimMIM	Raw-pixel masked-modeling baseline for local-appearance decodability	Xie et al. [40] ; SimMIM repository	MIT License, per the repository license.

Appendix

Table 4: Existing codebases, model releases, and software assets used in the paper.

Self-Supervised Visual Representation Learning

Self-supervised visual representation learning trains image, video, and 3D encoders on unlabeled data to produce transferable visual features.

See all 235 papers

CODI: Improving Self-Supervised Vision Transformers with Cross Distillation
Oct 22, 2026Siran Dai, Qianqian Xu, Peisong Wen +3Vision Transformer Self-Supervised Visual Representation Learning
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, CAS +3
Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning
Sep 23, 2026Basavaraj Sunagad, Artur Jesslen, Adam KortylewskiMulti-View Learning Self-Distillation
CISPA Helmholtz Center for Information Security · University of Freiburg
Scalable Patch-Level Self-Supervised Learning
Oct 7, 2026Maximilian Seitzer, Gabriele Trivigno, Antonín Vobecký +5Self-Supervised Learning Self-Supervised Visual Representation Learning
Meta FAIR

Oct 22, 2026J/K move · Enter open · S save

CODI: Improving Self-Supervised Vision Transformers with Cross Distillation

Siran Dai, Qianqian Xu, Peisong Wen, Yang Liu+2

Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, CAS+3

Vision Transformer Self-Supervised Visual Representation Learning Self-Supervised Knowledge Distillation Dense Prediction Knowledge Distillation

Open paper AskPDF arXiv

Self-supervised Vision Transformers learn strong image-level representations, but their patch-level representations can be unreliable for dense prediction. We study this mismatch and find that the class token concentrates global semantics on a small number of representative patches, while dense prediction errors often occur on low-attention regions. We provide a local gradient analysis showing that, along the [CLS] attention readout, image-level self-distillation gradients are weighted by [CLS]-to-patch attention mass. To provide masked patches with global semantics without changing the architecture, we propose Cross Distillation (CODI), which mixes a small amount of the teacher class-token distribution into each teacher patch-token target. CODI gives every masked patch an attention-independent semantic correction while preserving the standard self-distillation architecture. Experiments on dense and image-level benchmarks show that CODI improves both dense and image-level performance under aligned pretraining settings. Code is available at https://github.com/EldercatSAM/CODI.

Abstract

Self-Supervised Visual Representation Learning

Explore similar work

CODI: Improving Self-Supervised Vision Transformers with Cross Distillation

Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning

Scalable Patch-Level Self-Supervised Learning