Masked patch self-distillation is a key component of modern vision foundation models, including DINOv2 and DINOv3, yet its implications for local visual information remain unclear. We show that its semantic benefits can coexist with a scale-dependent locality cost. Using two annotation-free diagnostics, we distinguish recoverable local appearance from spatial correspondence. In smaller models, adding iBOT substantially weakens patch correspondence while largely preserving appearance decodability. In larger models, correspondence improves, but pixel and texture evidence becomes less linearly accessible in deep features. Thus, scaling changes the form of the locality cost rather than uniformly eliminating it. Our analysis of Gram anchoring further shows that preserving patch relations does not guarantee simultaneous improvements in both properties. These findings motivate evaluating and designing visual representations around the complementary demands of semantic abstraction, appearance preservation, and spatial correspondence. Our code is available at https://github.com/winci-ai/jea.
Figures & tables
Figure 1: Anchor-patch similarity heatmaps for DINO and DINO +iBOT on ADE20K examples, with semantic annotations shown for reference. Yellow boxes mark the anchor patches. For ViT-B, adding iBOT produces broader responses over the building and mountain, reducing distinctions associated with windows and rugged cliff structures. This effect is alleviated in ViT-L, where the + iBOT model also exhibits more spatially selective responses to these structures. The semantic masks do not resolve many of these within-region differences, illustrating why segmentation performance alone cannot characterize fine-grained local representations. Additional examples appear in Figure 10 .
Figure 2: Downstream comparison of DINO and DINO +iBOT across ViT-S/16, ViT-B/16, ViT-L/16, and ViT-H/16, pretrained on ImageNet-1K for 300 epochs under matched settings within each scale. Adding iBOT generally benefits tasks dominated by semantic understanding, whereas its effects on tasks requiring fine-grained local information vary with model scale. In particular, the degradation in DAVIS tracking observed for smaller backbones reverses to modest gains for larger backbones. Task scores other than optical-flow EPE are percentages; normalized patch rank is multiplied by 100. Arrows indicate the preferred direction for task metrics: ↑ denotes higher is better and ↓ denotes lower is better. Patch effective rank measures spectral diversity. Numerical results and training trajectories are provided in Table 2 and Figure 11 , respectively.
Figure 3: DAVIS video object tracking visualization. Here we compare tracking masks produced by DINO and DINO +iBOT on videos containing visually similar same-category instances and occlusions. More visualization results can be seen in Figure 12 .
Figure 4: Quantitative results of Local appearance decodability across different layers. Δ denotes the difference between the DINO +iBOT and its corresponding DINO model under the same pretraining setup. To validate that the diagnostic indeed captures local visual evidence, we also include raw-pixel prediction models, such as SimMIM and MAE, which are also pretrained using ViT-B/16 on ImageNet. We also report the evolution of each descriptor metric during training in Figure 13 , and show the decoding visualization in Figure 14 .
Figure 5: Quantitative results of local-to-global correspondence across different layers. Δ denotes the difference between the DINO +iBOT model and its corresponding DINO model under the same pretraining setup. We also report the evolution of each metric during training in Figure 16 .
Figure 6: Local-to-global correspondence visualizations for DINO and DINO +iBOT , showing RANSAC box estimates, similarity heatmaps, and patch matches. With ViT-B/16, adding iBOT weakens correspondence: reliable matches are largely confined to larger foreground objects, while smaller objects are incorrectly matched or poorly localized. This failure pattern is not observed in the ViT-L/16 examples, where the + iBOT model maintains correspondence for smaller objects. These illustrate the scale-dependent effect reported in Figure 5 . Additional examples appear in Figure 17 .
Figure 7: Downstream trajectories during epochs 250–300. Rows show ViT-B/16 and ViT-L/16; the dashed line marks Gram activation at epoch 260. Panels use independent linear vertical scales. Non-finite optical-flow results and EPE values above 100 are shown as gaps. Optical flow measures training-set fitting.
Figure 8: Local Appearance Decodability across layers at epoch 300. DINO-Gram improves deep-layer decodability more consistently for ViT-L/16 than for ViT-B/16. Values are R2×100 ; nonlinear asinh axes emphasize deep-layer differences.
Figure 9: Local-to-Global Correspondence at epoch 300 with crop area ratio 0.1. Both Gram variants improve deep-layer correspondence for ViT-B/16, especially DINO-Gram. Neither consistently improves the four metrics for ViT-L/16. All values are percentages.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
Objective
Batch size
Memory (GB)
Time
ViT-S/16
DINO
1024
28
1 d 7 h
DINO +iBOT
1024
100
2 d 5 h
ViT-B/16
DINO
1024
47
3 d 1 h
DINO +iBOT
1024
125
3 d 8 h
ViT-L/16
DINO
1024
100
5 d 15 h
DINO +iBOT
1024
129
6 d 5 h
Appendix
Table 1: Compute resources for 300-epoch pretraining on ImageNet-1K. All runs use four NVIDIA H200 GPUs with bf16 mixed precision and FSDP. Batch size denotes the global batch size. Memory is the approximate peak usage per GPU, and time is the approximate wall-clock duration of a complete run.
Figure 10: Additional anchor-patch similarity heatmaps for DINO and DINO +iBOT using ViT-B/16 and ViT-L/16.
Arch.
Metric
Better
DINO
DINO +iBOT
Δ
ViT-S/16
kNN top-1
↑
70.88
71.27
+0.39
VOC2012 Seg. mIoU
↑
53.13
56.02
+2.89
VOC2007 Det. mAP
↑
42.88
43.96
+1.08
RParis Hard mAP
↑
29.75
30.78
+1.03
ROxford Hard mAP
↑
13.34
12.03
-1.31
Copy strong mAP
↑
84.14
83.69
-0.45
Appendix
Table 2: Numerical results for the four backbone scales in Figure 2 . All rows use epoch 300 except Sintel EPE, which uses the latest common retained checkpoint: epochs 180/260/300/270 for ViT-S/B/L/H. Sintel scores are evaluated on training videos rather than a held-out split. Performance metrics other than EPE are percentages. Normalized patch rank is the mean patch effective rank divided by the number of patches and multiplied by 100; it describes spectral dimensionality rather than task accuracy. Δ denotes DINO +iBOT minus DINO, computed before rounding. Green and red indicate improved and degraded task performance, respectively; patch-rank changes are shown without a performance interpretation.
Figure 11: Training trajectories of DINO and DINO +iBOT across four backbone scales. The effect of iBOT varies with the task, backbone scale, and training stage. Improvements in classification and dense semantic prediction coexist with mixed effects on retrieval and correspondence. Non-finite optical-flow results and EPE values exceeding 100 are excluded.
Figure 12: Additional DAVIS video object tracking visualization for DINO and DINO +iBOT using ViT-B/16 and ViT-L/16.
Figure 13: Training trajectories of local appearance decodability for each descriptor group.
Figure 14: Visualization of local appearance decodability. Here we compare the RGB mean, HOG, DCT energy, and raw PCA results produced by ViT-B/16 and ViT-L/16 models of DINO and DINO +iBOT .
Figure 15: Local-to-global correspondence across crop area ratios. Each ratio specifies the approximate fraction of the source image used to construct a local query. Δ denotes DINO +iBOT minus DINO under matched pretraining settings and identical sampled crops. Positive values favor DINO +iBOT for all reported metrics.
Figure 16: Training trajectories of local-to-global correspondence for DINO and DINO +iBOT across backbone scales. Rows show the backbone and evaluated layer: blocks 3/6/9/12 for ViT-S/B, 6/12/18/24 for ViT-L, and 8/16/24/32 for ViT-H. The sustained deterioration observed in smaller + iBOT models is alleviated in larger backbones.
Figure 17: Additional visualization of local-global correspondence for ViT-B/16 and ViT-L/16 models of DINO and DINO +iBOT .
Asset
Use in this paper
Credit / source
License or terms of use
ImageNet-1K
Self-supervised pretraining, k -NN evaluation, and diagnostic image splits
Deng et al. [13] ; official access terms
Access provided for non-commercial research and educational use under ImageNet terms. ImageNet does not own the underlying image copyrights, which remain with the original owners.
PASCAL VOC 2007 / 2012
VOC2007 object detection and VOC2012 semantic segmentation
Everingham et al. [17] , Everingham et al. [16] ; VOC challenge pages
The official database-rights statement notes that VOC contains Flickr images, whose use must respect the corresponding Flickr terms of use.
MS COCO
Copy-detection distractors and whitening images
Lin et al. [26] ; COCO terms of use
COCO annotations are released under CC BY 4.0; COCO images are subject to the Flickr terms and to the image-level license metadata provided by COCO.
R Oxford5k / R Paris6k
Image retrieval evaluation
Radenović et al. [30] ; Revisited Oxford / Paris benchmark
Public benchmark data and annotations used for academic evaluation. The project page does not list a standard open-source license for the images; we do not redistribute the raw images.
ADE20K
Similarity heatmaps and reference semantic masks
Zhou et al. [42] ; official terms of use
Images are provided for non-commercial research and education; copyright remains with their owners. The official terms specify BSD-3 terms for the annotations and software.
INRIA Copydays
Copy detection and local-to-global correspondence diagnostics
Douze et al. [15] ; Copydays project page
INRIA holds copyright on the dataset images and distributes them “as is” without express or implied warranties. Used here only as a research benchmark; raw images are not redistributed.
Appendix
Table 3: Datasets and benchmarks used in the paper.
Asset
Use in this paper
Credit / source
License or terms of use
DINOv2
Pretraining framework and ViT implementation
Oquab et al. [28] ; DINOv2 repository
Apache License 2.0 for the DINOv2 code and standard model weights, per the repository license.
MMSegmentation
Linear semantic segmentation evaluation
MMSegmentation repository
Apache License 2.0.
MMDetection
Frozen-backbone object detection evaluation
MMDetection repository
Apache License 2.0.
MMFlow
Frozen-backbone optical-flow evaluation and PWCNet implementation
MMFlow repository ; PWCNet paper [ 34 ]
Apache License 2.0 for the MMFlow toolbox.
MAE
Raw-pixel masked-modeling baseline for local-appearance decodability
He et al. [21] ; MAE repository
CC BY-NC 4.0, per the repository license.
SimMIM
Raw-pixel masked-modeling baseline for local-appearance decodability
Xie et al. [40] ; SimMIM repository
MIT License, per the repository license.
Appendix
Table 4: Existing codebases, model releases, and software assets used in the paper.
Self-supervised Vision Transformers learn strong image-level representations, but their patch-level representations can be unreliable for dense prediction. We study this mismatch and find that the class token concentrates global semantics on a small number of representative patches, while dense prediction errors often occur on low-attention regions. We provide a local gradient analysis showing that, along the [CLS] attention readout, image-level self-distillation gradients are weighted by [CLS]-to-patch attention mass. To provide masked patches with global semantics without changing the architecture, we propose Cross Distillation (CODI), which mixes a small amount of the teacher class-token distribution into each teacher patch-token target. CODI gives every masked patch an attention-independent semantic correction while preserving the standard self-distillation architecture. Experiments on dense and image-level benchmarks show that CODI improves both dense and image-level performance under aligned pretraining settings. Code is available at https://github.com/EldercatSAM/CODI.
Siran Dai, Qianqian Xu, Peisong Wen +3
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, CAS +3
Self-supervised vision transformers trained with DINO-style objectives exhibit striking emergent semantic representation quality across visual tasks, yet the mechanisms underlying this behavior remain unclear. We present a systematic empirical dissection of the DINO family and show that semantic representations arise primarily from enforcing consistency between geometrically distinct global views of the same image instance. This instance-specific global alignment acts as the semantic anchor of DINO-style learning. Across controlled retraining experiments evaluated on semantic correspondence and a diverse suite of 2D and 3D downstream tasks, we find that patch-level masking objectives enhance semantics only when trained jointly with this global alignment, indicating that the iBOT objective refines and densifies existing semantic structure rather than creating it independently. In contrast, local-to-global view alignment does not substantially improve semantic qualities at fixed compute beyond a purely global alignment. Beyond training design, we revisit how semantic representation quality should be evaluated: while classification accuracy is the standard validation score, semantic correspondence provides a complementary axis that more reliably predicts downstream task performance. Together, these findings provide a functional decomposition of DINO-style learning and represent an important step toward understanding how semantic representations emerge in self-supervised vision models.
Basavaraj Sunagad, Artur Jesslen, Adam Kortylewski
CISPA Helmholtz Center for Information Security · University of Freiburg
Self-supervised learning (SSL) at scale produces powerful visual representations. However, most scalable SSL methods rely on ad hoc combinations of multiple objectives and stabilization mechanisms. Taking a step back, we ask if we can design a high-performing, yet principled SSL algorithm. Starting from the multi-view assumption, stipulating that task-relevant content is captured by the information common to different views, we construct an information-theoretic objective decomposing into interpretable terms. This derivation yields JEM, a student-teacher method that learns by aligning corresponding patch representations across views, explicitly regularized by information and structure preservation losses. JEM trains stably from 300M to 7B parameters, and, to our knowledge, is the first latent-space patch-level method demonstrated at 7B scale. Across all scales, JEM reaches strong performance on both global and dense probing tasks, on segmentation benchmarks consistently surpassing the DINOv2 algorithm, an influential foundation for today's strongest visual SSL methods. Notably, at 7B parameters, it exceeds the performance of DINOv3 on panoptic segmentation, despite being trained on 12× less data without refinement stages. These results demonstrate that we can indeed design an SSL algorithm that learns strong representations, is principled and stable.
Maximilian Seitzer, Gabriele Trivigno, Antonín Vobecký +5