CODI: Improving Self-Supervised Vision Transformers with Cross Distillation

Siran Dai; Qianqian Xu; Peisong Wen; Yang Liu; Xiaochun Cao; Qingming Huang

cs.CVOct 22, 2026

CODI: Improving Self-Supervised Vision Transformers with Cross Distillation

Authors: Siran Dai, Qianqian Xu, Peisong Wen, Yang Liu, Xiaochun Cao, Qingming Huang

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · Beijing Academy of Artificial Intelligence ·

Abstract

Self-supervised Vision Transformers learn strong image-level representations, but their patch-level representations can be unreliable for dense prediction. We study this mismatch and find that the class token concentrates global semantics on a small number of representative patches, while dense prediction errors often occur on low-attention regions. We provide a local gradient analysis showing that, along the [CLS] attention readout, image-level self-distillation gradients are weighted by [CLS]-to-patch attention mass. To provide masked patches with global semantics without changing the architecture, we propose Cross Distillation (CODI), which mixes a small amount of the teacher class-token distribution into each teacher patch-token target. CODI gives every masked patch an attention-independent semantic correction while preserving the standard self-distillation architecture. Experiments on dense and image-level benchmarks show that CODI improves both dense and image-level performance under aligned pretraining settings. Code is available at https://github.com/EldercatSAM/CODI.

Figures & tables

Figure 1: Comparison of CODI with baselines on Pascal VOC segmentation.

Figure 2: Left: During DINOv2 pretraining, downstream dense performance and [CLS] -to-patch attention mass follow similar trends. Right: The [CLS] token often focuses on representative patches, while prediction errors are more likely to occur on non-representative patches.

Panel of Figure 3 — Figure 3: The model tends to make prediction errors on patches with low attention scores.

Figure 4: [CLS] attention mass vs. DINO gradient norm.

Figure 5: Overview of CODI. In the self-distillation framework, CODI constructs the masked-patch target by mixing the teacher patch-token distribution with a small teacher class-token component.

Methods	Arch.	Dataset	Epochs	Semantic Segmentation						ImageNet-1K Classification
				COCO		ADE20K		VOC		$k$ -NN		Linear
				mIoU	Acc.	mIoU	Acc.	mIoU	Acc.	Top-1	Top-5	Top-1	Top-5
Distilled from Larger Models
DINOv2 [ 27 ]	ViT-S/14	LVD-142M	–	57.4	77.7	43.5	78.9	79.4	95.3	79.0	91.5	80.4	95.1
DINOv2 [ 27 ]	ViT-B/14	LVD-142M	–	59.1	78.4	46.3	79.8	81.3	95.7	82.0	92.3	83.9	96.6
DINOv2 [ 27 ]	ViT-L/14	LVD-142M	–	58.2	77.5	46.5	79.1	80.4	95.4	83.8	94.3	85.9	97.4

Table 1: Quantitative results of linear semantic segmentation and classification. Gray rows denote oracles trained with substantially larger data/model;

\dagger

denotes our reproduced result. The best and second-best results are highlighted with bold and underlined , respectively.

Methods	Arch.	Epochs	NYU-Depth V2			KITTI			NYU $\rightarrow$ SUNRGBD			Rank
			$\delta_{1}\uparrow$	Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	Rel $\downarrow$	RMSE $\downarrow$	$\downarrow$
Distilled from Larger Models
DINOv2 [ 27 ]	ViT-S/14	–	84.8	0.134	0.437	91.0	0.096	3.119	77.4	0.269	0.473	–
DINOv2 [ 27 ]	ViT-B/14	–	87.8	0.120	0.408	92.0	0.092	2.931	80.7	0.255	0.443	–
DINOv2 [ 27 ]	ViT-L/14	–	89.5	0.111	0.390	92.3	0.090	2.891	81.4	0.248	0.439	–
Small Models

Table 2: Depth estimation performance on NYU-Depth V2, KITTI, and SUNRGBD zero-shot transfer. Rank is computed within each model size.

Method	Pretraining	Fine-tuning	ImageNet	VOC
	iterations	iterations	$k$ -NN (%)	mIoU (%)
DINOv2	1M	–	76.9	68.3
DINOv2 (best checkpoint)	625K	–	75.9	69.4
DINOv2 + Gram anchoring	1M	100K	76.7	70.2
DINOv2 + CODI fine-tuning	1M	100K	76.8	71.4
CODI	1M	–	77.1	73.1

Table 5: Comparison of ViT-S/14 pretraining and fine-tuning settings on ImageNet-1K. Gram anchoring is applied to a DINOv2 checkpoint.

Table 10

Figure 7: Performance gain by [CLS] attention rank.

Methods	Intra- $d$ $\downarrow$	Inter- $d$ $\uparrow$	Gap $\uparrow$
DINOv2	0.6821	0.9168	0.2347
CODI	0.5864	0.8430	0.2566

Table 8: Representation distance analysis.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Top- $k$	Last frames	Neighborhood	Mask shape	Temperature
5	7	12	square	0.2
5	7	12	circle	0.2
5	7	5	square	0.2
5	7	24	square	0.2
5	7	$\infty$	none	0.2
5	7	12	square	0.01

Appendix

Table 9: VOS hyperparameter candidates used in the DAVIS holdout search.

Method	Unsupervised mIoU (%)	Linear-probe mIoU (%)
PiCIE+H [ 9 ]	14.4	14.8
STEGO (DINO ViT-S/8)	24.5	38.3
DINOv2	24.0	39.4
CODI	25.6	40.6

Appendix

Table 10: COCO-Stuff-27 validation mIoU under the STEGO protocol.

Method	Architecture	$\mathrm{AP}^{b}$	$\mathrm{AP}^{b}_{50}$	$\mathrm{AP}^{b}_{75}$	$\mathrm{AP}^{m}$	$\mathrm{AP}^{m}_{50}$	$\mathrm{AP}^{m}_{75}$
Supervised	Swin Transformer	48.1	67.1	52.5	41.7	64.4	45.0
MoBY [ 42 ]	Swin Transformer	48.1	67.1	52.1	41.5	64.0	44.7
Supervised	ViT-S/16	46.2	65.9	49.6	40.1	62.9	42.8
DINO	ViT-S/16	48.4	67.6	52.4	42.0	64.8	45.1
iBOT	ViT-S/16	49.4	68.7	53.3	42.6	65.6	45.8
DINOv2	ViT-S/14	49.9	69.0	53.8	43.0	66.0	46.4

Appendix

Table 11: Object detection and instance segmentation performance on COCO.

Method	ImageNet $k$ -NN (%)	VOC mIoU (%)
DINOv2	63.2	51.6
CODI	63.5	54.8

Appendix

Table 12: ViT-T/14 results after 200 epochs of ImageNet-1K pretraining.

Method	Architecture	Dataset	Epochs	ImageNet	VOC
				$k$ -NN (%)	mIoU (%)
DINOv3 (distilled)	ViT-L/16	LVD-1689M	–	85.3	85.1
SigLIP2 (distilled)	ViT-L/16	WebLI-10B	–	84.1	74.0
CODI	ViT-L/14	IN1K	200	81.2	80.1

Appendix

Table 13: Reference comparison with distilled DINOv3 and SigLIP2 checkpoints. Pretraining data and architectures differ across models.

COCO pretraining: 800 epochs
Method	$\alpha$	ImageNet $k$ -NN	VOC mIoU
DINOv2	–	41.1	49.2
CODI	0.05	41.8	50.8
CODI	0.075	41.8	50.9
CODI	0.1	41.6	51.3
CODI	0.2	40.9	48.4

Appendix

Table 14: Sensitivity to

\alpha

across pretraining datasets. ImageNet-22K excludes the ImageNet-1K subset. All downstream results are percentages.

Figure 9: Additional semantic segmentation visualizations from frozen-backbone linear probes. For each block, rows from top to bottom show the input image, iBOT, DINOv2, and CODI, while columns show different examples. Colored regions denote predicted semantic labels. CODI produces more complete and less fragmented object regions in these cases.

Panel of Figure 10 — Figure 10: Additional depth estimation visualizations from frozen-backbone linear depth probes. Rows from top to bottom show the input image, iBOT, DINOv2, and CODI, while columns show different indoor examples. Colors visualize relative predicted depth. CODI maintains coherent scene layouts and depth discontinuities while improving dense representation quality.

Vision Transformer

Vision Transformers (ViTs) use self-attention over image patches to learn visual representations for recognition, segmentation, and other tasks.

See all 362 papers

Distilling Vision Transformers for Distortion-Robust Representation Learning
Apr 24, 2026Konstantinos Alexis, Giorgos Giannopoulos, Dimitrios GunopulosRepresentation Learning Visual Representation Learning
Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Greece · National Observatory of Athens, Greece
Vision Transformers Need Better Token Interaction
May 22, 2026Linxiang SuVisual Representation Learning Vision Transformer
University of Szeged
The Locality Trade-off in Masked Patch Self-Distillation
Oct 23, 2026Xi Weng, Zhaoyu Liu, Lianyu Hu +1Self-Supervised Visual Representation Learning Vision Foundation Models
School of Computing, National University of Singapore · Nanyang Technological University

Apr 24, 2026J/K move · Enter open · S save

Distilling Vision Transformers for Distortion-Robust Representation Learning

Konstantinos Alexis, Giorgos Giannopoulos, Dimitrios Gunopulos

Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Greece · National Observatory of Athens, Greece

Representation Learning Visual Representation Learning Vision Transformer Knowledge Distillation

Open paper AskPDF arXiv

Self-supervised learning has achieved remarkable success in learning visual representations from clean data, yet remains challenging when clean observations are sparse or not available at all. In this paper, we demonstrate that pretrained vision models can be leveraged to learn distortion-robust representations, which can then be effectively applied to downstream tasks operating on distorted observations. In particular, we propose an asymmetric knowledge distillation framework in which both teacher and student are initialized from the same pretrained Vision Transformer but receive different views of each image: the teacher processes clean images, while the student sees their distorted versions. We introduce multi-level distillation that aligns global embeddings, patch-level features, and attention maps and show that the student is able to approximate clean-image representations despite never directly accessing clean data. We evaluate our approach on image classification tasks across several datasets and under various distortions, consistently outperforming existing alternatives for the same amount of human supervision.

Abstract

Vision Transformer

Explore similar work

Distilling Vision Transformers for Distortion-Robust Representation Learning

Vision Transformers Need Better Token Interaction

The Locality Trade-off in Masked Patch Self-Distillation