Knowledge distillation (KD) between architectural families, e.g., from a Transformer teacher to a CNN student, is difficult because the two networks learn different representations. Logits share a label space, but heterogeneous teachers differ in logit offset, scale, and sharpness, and their non-target classes are less semantically organized. Conventional logit distillation matches these values sample by sample, asking the student to imitate how another architecture expresses its predictions. We argue that a heterogeneous teacher should pass on the relational structure of its predictions, i.e., how it arranges samples relative to one another, and propose Heterogeneous Asymmetric Contrastive Distillation (HACD) to transfer this structure. HACD maps teacher and student logits to centered, temperature-softened probability directions on a shared unit hypersphere, which removes architecture-dependent offsets and lets the temperature control how sharply peaked outputs enter the representation while keeping informative non-target classes. Each student output must then identify its own teacher target within the batch. Because every teacher target also acts as a negative for the others, HACD computes the targets on a moderately augmented view and contrasts both a moderate and a strong student view against them. HACD adds no learnable parameters and needs one teacher pass per sample. It achieves the best or second-best accuracy on 11 of 12 heterogeneous pairs on CIFAR-100, without ground-truth labels, and on 14 of 15 pairs on ImageNet-1K, improving over the reported vanilla KD results by 8.82 and 1.33 percentage points on average. Checkpoints are available at https://github.com/BTAEYANG/HACD-Heterogeneous-Asymmetric-Contrastive-Distillation; the code will be released upon publication.
Figures & tables
Figure 1: CKA similarity analysis of representations learned by homogeneous and heterogeneous architectures. Each entry shows the CKA similarity between the representations of a layer of the first network (horizontal axis) and a layer of the second network (vertical axis), with darker colors indicating higher similarity. The CNN-to-CNN pair exhibits more consistent representation patterns, whereas heterogeneous pairs involving CNNs, Transformers, and MLPs show substantial discrepancies.
Figure 2: Output statistics of CNN, Transformer, and MLP teachers on the CIFAR-100 training dataset. Logits are standardized per sample, z~=(z−zˉ)/σz , to remove architecture-dependent scales. (a) Normalized entropy of softmax(z~) . (b) Top-1/top-2 margin of z~ . (c) Semantic consistency: fraction of the top- k non-target classes that share the superclass of the ground-truth class (dashed line: chance level, 4/99 ). In (a) and (b), half violins, boxes, and diamonds show the distribution, quartiles, and mean over samples. The Transformer and MLP teachers produce sharper outputs with less semantically organized non-target classes.
Figure 3: Overview of HACD. Each training image is augmented into a moderate view and a strong view. The frozen teacher processes only the moderate view, whereas the student processes both views. Teacher and student logits are mapped to centered, temperature-softened probability directions on a shared unit hypersphere (Eq. ( 2 )). The same teacher representations are the targets of bidirectional instance-level contrast for both student views (Eq. ( 5 )), and the two view losses are combined with weight α (Eq. ( 8 )). Only the student is used at inference.
Architecture
From Scratch
Feature-based
Logit-based
Gain
Teacher
Student
T.
S.
FitNet [ 28 ]
CC [ 26 ]
RKD [ 25 ]
CRD [ 32 ]
PAT [ 20 ]
RSD [ 46 ]
KD [ 13 ]
DKD [ 47 ]
DIST [ 15 ]
OFA [ 11 ]
TeKAP [ 14 ]
LDRLD [ 42 ]
HCD [ 43 ]
HACD
Δ
CNN-based students
Swin-T
ResNet-18
89.26
74.01
78.87
74.19
74.11
77.63
81.22
83.92
78.74
80.26
77.75
80.54
81.38 ∗
82.17
82.78
85.00
+6.26
ViT-S
ResNet-18
92.04
74.01
77.71
74.26
73.72
76.60
80.11
81.50
77.26
78.10
76.49
80.15
79.06 ∗
80.36
81.33
84.76
+7.50
Mixer-B/16
ResNet-18
87.29
74.01
77.15
74.26
73.75
76.42
80.07
81.85
77.79
78.67
76.36
79.39
80.05 ∗
80.69
81.53
83.95
+6.16
Swin-T
MobileNetV2
89.26
73.68
74.28
71.19
69.00
79.80
78.78
83.68
74.68
71.07
72.89
80.98
80.23 ∗
81.64
82.19
85.38
+10.70
Table 1: Top-1 classification accuracy (%) of cross-architecture knowledge distillation on CIFAR-100. All 12 teacher-student pairs involve heterogeneous architectures. The best and second-best results are highlighted in bold and underline , respectively. T. and S. denote the accuracies of the teacher and student models before distillation, respectively. Average Gain represents the mean accuracy improvement over the corresponding student trained from scratch, calculated across all 12 teacher-student pairs. Δ represents the improvement of HACD over vanilla KD. Average Gain and Δ are absolute differences in percentage points (pp). ∗ indicates results reproduced by LDRLD [ 42 ] . “–” indicates results not reported for the corresponding teacher-student pair. HCD [ 43 ] is excluded from the Average Gain comparison because results are available for only six pairs.
Architecture
From Scratch
Feature-based
Logit-based
Gain
Teacher
Student
T.
S.
FitNet [ 28 ]
CC [ 26 ]
RKD [ 25 ]
CRD [ 32 ]
PAT [ 20 ]
RSD [ 46 ]
KD [ 13 ]
DKD [ 47 ]
DIST [ 15 ]
OFA [ 11 ]
TeKAP [ 14 ]
HCD [ 43 ]
HACD
Δ
CNN-based students
DeiT-T
ResNet-18
72.19
69.75
70.44
69.77
69.47
69.25
–
71.70
70.22
69.39
70.64
71.34
–
–
71.50
+1.28
Swin-T
ResNet-18
81.38
69.75
71.18
70.07
68.89
69.09
71.54
72.13
71.14
71.10
70.91
71.85
71.25 †
71.91
72.76
+1.62
Mixer-B/16
ResNet-18
76.62
69.75
70.78
70.05
69.46
68.40
–
71.41
70.89
69.89
70.66
71.38
70.95 †
71.66
71.81
+0.92
DeiT-T
MobileNetV2
72.17
68.87
70.95
70.69
69.72
69.60
–
72.18
70.87
70.14
71.08
71.39
70.23 †
71.38
72.88
+2.01
Table 2: Top-1 classification accuracy (%) of cross-architecture knowledge distillation on ImageNet-1K. Notation follows Table 1 . Average Gain is computed over the 15 pairs and reported only for methods with complete results. ∗ indicates results obtained by combining the corresponding method with FitNets, following OFA. † indicates results taken from HCD [ 43 ] .
Objective
Student
Teacher
Swin-T → ResNet-18
ConvNeXt-T → DeiT-T
(a)
KD
–
–
78.74
72.99
(b)
KD ∗
M
M
79.68
81.21
(c)
Contrastive
M
M
83.35
81.84
(d)
Contrastive
S
M
83.08
74.27
(e)
Contrastive
M, S
S
84.41
80.96
(f)
Contrastive
M, S
M
85.00
82.50
Table 3: Ablation of the components of HACD on CIFAR-100 (top-1, %). M/S: moderate/strong view; Student/Teacher: views processed by each network. Row (a): KD result of Table 1 . KD ∗ : KD loss ( T=4 ) under the training setting of HACD. Row (f) is HACD.
Figure 4: Overhead of symmetric teacher views over HACD in training FLOPs, time per iteration, and peak GPU memory.
τ
λ
Pair
0.1
0.2
0.5
0
0.5
1.0
Swin-T → ResNet-18
84.23
85.00
84.39
85.00
83.93
83.23
ConvNeXt-T → DeiT-T
80.94
82.50
81.37
82.50
83.18
81.59
Table 4: Sensitivity to τ and λ on CIFAR-100 (top-1, %). Defaults are shaded.
Figure 5: Sensitivity to the view weight α on CIFAR-100. The dashed line marks the default α=0.7 .
Figure 6: Extra learnable parameters (solid, left axis) and peak GPU memory per training step (dashed, right axis) on ImageNet-1K.
Figure 7: Representation uT of three ImageNet-1K teachers as T decreases from ∞ to 1 (solid: T≥4 ; dashed: T<4 ; stars: T=4 ). (a) Energy in the ten highest-ranked classes (horizontal) versus energy in ranks 2–10 (vertical). (b) Class separation (horizontal) versus instance discrimination (vertical).