Classifier outputs provide a fundamental and univer- sally available knowledge interface for knowledge distil- lation (KD). However, standard logit objectives entangle transferable semantic relations with architecture-dependent parameterization artifacts, causing a classifier geometry mismatch that makes direct knowledge transfer unreliable. To address this, we propose Angular Conditional Distilla- tion (ACD). First, the Angular Coordinate Interface (ACI) reconstructs the student classifier using angular coordi- nates, suppressing magnitude and bias effects to provide a geometry-compatible space. Then, Scale-Invariant Condi- tional Distillation (SICD) standardizes outputs and exclu- sively transfers conditional non-target relations, filtering out untransferable absolute confidence. Together, ACI de- fines where to transfer, while SICD defines what to transfer. Extensive experiments show that ACD achieves state-of-the- art performance over existing KD methods. Moreover, as a universal drop-in replacement, it consistently improves existing distillation pipelines while requiring no feature adapters or additional inference-time cost. Code is avail- able at https://github.com/laniusnan/ACD.
Figures & tables
Figure 1: Classifier Geometry Mismatch. (a) Empirical evidence: fixing teacher/student features but restoring parameterization causes substantial deformation in non-target distributions (x-axis) and gradient deviations (y-axis). (b) Local gradient directional disagreement under different readouts (VGG13 → MobileNetV1). Darker overlays indicate higher disagreement 1−cos(gp,gpang) .
Figure 2: Output mismatch beyond scalar confidence. (a) Empirical cumulative distributions of the weighted KL divergence before and after per-image oracle calibration, with a symmetric-log horizontal axis. (b) Density of image-class pairs, comparing centered teacher scores with calibrated, centered student scores, both divided by T .
Figure 3: Sensitivity to score parameterization. Panels show (a) non-target conditional total variation at T=4 , (b) the fraction of images whose highest-scoring non-target class changes, and (c) KD feature-gradient direction deviation ( 1−cos ), all relative to the original model.
Figure 4: Relation distribution. (a) Teacher and student have different target probabilities, but identical non-target conditional distributions. (b) Varying qy while fixing these conditional distributions leaves the weighted conditional KL at zero. Full KL equals the target-mass term and remains positive unless qy=py .
Figure 5: Overview of ACI and SICD. ACI parameterizes student logits using cosine similarities with a shared scale, removing explicit feature magnitude, class-specific weight norms, and biases. The teacher retains its original classifier. SICD standardizes teacher and student logits and matches their non-target conditional distributions, weighted by the teacher’s non-target probability mass. A detached gradient-strength matching factor calibrates the conditional objective against standardized full KL. The student is trained jointly with ground-truth cross-entropy supervision.
Method
Teacher
ResNet56
ResNet110
ResNet32x4
WRN-40-2
WRN-40-2
VGG13
72.40
74.31
79.42
75.59
75.59
74.64
Student
ResNet20
ResNet32
ResNet8x4
WRN-16-2
WRN-40-1
VGG8
65.79
67.92
72.48
71.12
69.54
68.19
Feature
FitNet [29]
69.64
72.35
75.26
74.46
73.28
70.09
CRD
67.28
71.97
74.16
74.22
72.44
72.74
ReviewKD
69.54
71.12
75.71
73.92
74.50
72.29
Table 1: Comparison of Top-1 accuracy on CIFAR-100. We report the individual performance of the teacher and student models under homogeneous architecture settings.
Method
Teacher
ResNet32x4
WRN-40-2
VGG13
ResNet50
ResNet32x4
79.42
75.59
74.64
79.33
79.42
Student
ShuffleNetV1
ShuffleNetV1
MobileNetV1
MobileNetV1
ShuffleNetV2
69.43
69.43
60.09
60.09
73.5
Feature
FitNet [29]
74.09
73.49
64.98
64.06
75.42
CRD
75.50
76.15
66.55
66.28
74.66
ReviewKD
77.39
77.68
65.26
62.52
77.99
Table 2: Comparison of Top-1 accuracy on CIFAR-100. We report the individual performance of the teacher and student models under heterogeneous architecture settings.
Teacher
Student
KD [ 6 ]
DKD [ 2 ]
AT [ 28 ]
ReviewKD [ 27 ]
RKD [ 15 ]
SKD [ 7 ]
CD [ 9 ]
ACD (Ours)
R34 (73.30)
R18 (69.76)
69.11±0.21
70.88±0.16
70.54±0.15
70.51±0.18
69.94±0.20
71.13±0.12
71.01±0.10
71.64±0.12
R50 (76.14)
MV1 (66.51)
67.81±0.24
70.58±0.18
70.90±0.16
67.77±0.23
71.22±0.13
71.53±0.11
71.23±0.15
71.84±0.14
Table 3: Comparison of Top-1 accuracy on ImageNet with both homogeneous (e.g., R34 → R18) and heterogeneous (e.g., R50 → MV1) teacher-student architectures.
Method
AP
AP 50
AP 75
AP
AP 50
AP 75
AP
AP 50
AP 75
Teacher
ResNet101
ResNet101
ResNet50
42.04
62.48
45.88
42.04
62.48
45.88
40.22
61.02
43.81
Student
ResNet18
ResNet50
MobileNetV2
33.26
53.61
35.26
37.93
58.84
41.05
29.47
48.87
30.90
Feature
FitNet [ 18 ]
34.13
54.16
36.71
38.76
59.62
41.80
30.20
49.80
31.69
ReviewKD [ 27 ]
36.75
56.72
34.00
40.36
60.97
44.08
33.71
53.15
36.13
Table 4: Object detection results on MS-COCO using Faster R-CNN-FPN.
Configuration
V13 → V8
V13 → MNV1
KD (Aff.)
74.39±0.43
67.78±0.40
CI
75.10±0.19
66.60±0.16
CI + ETF (ACI)
75.80±0.15
68.10±0.14
ACI + DS
78.55±0.05
70.98±0.02
ACI + SICD + GM
80.52±0.02
73.30±0.01
Table 5: Component ablation. Accuracy (%); best results are bold. Aff.: affine head; CI: cosine interface; ACI: CI with equiangular tight frame (ETF) initialization; DS: dual standardization with full KL; GM: gradient matching. V13/V8: VGG13/VGG8; MNV1: MobileNetV1.
Method
T → S
KD
ACI
ACI + SICD
Δ
AT [ 28 ]
R32 × 4 → SV1
77.30
77.60
78.97
+1.67
ReviewKD [ 27 ]
V13 → MNV1
68.48
67.48
70.43
+1.95
Table 6: Compatibility with feature-based distillation on CIFAR-100. Top-1 accuracy (%). Δ : gain over KD in percentage points. Best results within each row are bold. T/S: teacher/student; R32 × 4: ResNet32 × 4; SV1: ShuffleNetV1; V13: VGG13; MNV1: MobileNetV1.
Figure 6: Non-target relation discrepancies. (a) Empirical cumulative distributions of conditional TV. (b) Medians (circles), means (diamonds), and image interquartile ranges (segments).