Classifier outputs provide a fundamental and univer- sally available knowledge interface for knowledge distil- lation (KD). However, standard logit objectives entangle transferable semantic relations with architecture-dependent parameterization artifacts, causing a classifier geometry mismatch that makes direct knowledge transfer unreliable. To address this, we propose Angular Conditional Distilla- tion (ACD). First, the Angular Coordinate Interface (ACI) reconstructs the student classifier using angular coordi- nates, suppressing magnitude and bias effects to provide a geometry-compatible space. Then, Scale-Invariant Condi- tional Distillation (SICD) standardizes outputs and exclu- sively transfers conditional non-target relations, filtering out untransferable absolute confidence. Together, ACI de- fines where to transfer, while SICD defines what to transfer. Extensive experiments show that ACD achieves state-of-the- art performance over existing KD methods. Moreover, as a universal drop-in replacement, it consistently improves existing distillation pipelines while requiring no feature adapters or additional inference-time cost. Code is avail- able at https://github.com/laniusnan/ACD.
Figures & tables
Figure 1: Classifier Geometry Mismatch. (a) Empirical evidence: fixing teacher/student features but restoring parameterization causes substantial deformation in non-target distributions (x-axis) and gradient deviations (y-axis). (b) Local gradient directional disagreement under different readouts (VGG13 → MobileNetV1). Darker overlays indicate higher disagreement 1−cos(gp,gpang) .
Figure 2: Output mismatch beyond scalar confidence. (a) Empirical cumulative distributions of the weighted KL divergence before and after per-image oracle calibration, with a symmetric-log horizontal axis. (b) Density of image-class pairs, comparing centered teacher scores with calibrated, centered student scores, both divided by T .
Figure 3: Sensitivity to score parameterization. Panels show (a) non-target conditional total variation at T=4 , (b) the fraction of images whose highest-scoring non-target class changes, and (c) KD feature-gradient direction deviation ( 1−cos ), all relative to the original model.
Figure 4: Relation distribution. (a) Teacher and student have different target probabilities, but identical non-target conditional distributions. (b) Varying qy while fixing these conditional distributions leaves the weighted conditional KL at zero. Full KL equals the target-mass term and remains positive unless qy=py .
Figure 5: Overview of ACI and SICD. ACI parameterizes student logits using cosine similarities with a shared scale, removing explicit feature magnitude, class-specific weight norms, and biases. The teacher retains its original classifier. SICD standardizes teacher and student logits and matches their non-target conditional distributions, weighted by the teacher’s non-target probability mass. A detached gradient-strength matching factor calibrates the conditional objective against standardized full KL. The student is trained jointly with ground-truth cross-entropy supervision.
Method
Teacher
ResNet56
ResNet110
ResNet32x4
WRN-40-2
WRN-40-2
VGG13
72.40
74.31
79.42
75.59
75.59
74.64
Student
ResNet20
ResNet32
ResNet8x4
WRN-16-2
WRN-40-1
VGG8
65.79
67.92
72.48
71.12
69.54
68.19
Feature
FitNet [29]
69.64
72.35
75.26
74.46
73.28
70.09
CRD
67.28
71.97
74.16
74.22
72.44
72.74
ReviewKD
69.54
71.12
75.71
73.92
74.50
72.29
Table 1: Comparison of Top-1 accuracy on CIFAR-100. We report the individual performance of the teacher and student models under homogeneous architecture settings.
Method
Teacher
ResNet32x4
WRN-40-2
VGG13
ResNet50
ResNet32x4
79.42
75.59
74.64
79.33
79.42
Student
ShuffleNetV1
ShuffleNetV1
MobileNetV1
MobileNetV1
ShuffleNetV2
69.43
69.43
60.09
60.09
73.5
Feature
FitNet [29]
74.09
73.49
64.98
64.06
75.42
CRD
75.50
76.15
66.55
66.28
74.66
ReviewKD
77.39
77.68
65.26
62.52
77.99
Table 2: Comparison of Top-1 accuracy on CIFAR-100. We report the individual performance of the teacher and student models under heterogeneous architecture settings.
Teacher
Student
KD [ 6 ]
DKD [ 2 ]
AT [ 28 ]
ReviewKD [ 27 ]
RKD [ 15 ]
SKD [ 7 ]
CD [ 9 ]
ACD (Ours)
R34 (73.30)
R18 (69.76)
69.11±0.21
70.88±0.16
70.54±0.15
70.51±0.18
69.94±0.20
71.13±0.12
71.01±0.10
71.64±0.12
R50 (76.14)
MV1 (66.51)
67.81±0.24
70.58±0.18
70.90±0.16
67.77±0.23
71.22±0.13
71.53±0.11
71.23±0.15
71.84±0.14
Table 3: Comparison of Top-1 accuracy on ImageNet with both homogeneous (e.g., R34 → R18) and heterogeneous (e.g., R50 → MV1) teacher-student architectures.
Method
AP
AP 50
AP 75
AP
AP 50
AP 75
AP
AP 50
AP 75
Teacher
ResNet101
ResNet101
ResNet50
42.04
62.48
45.88
42.04
62.48
45.88
40.22
61.02
43.81
Student
ResNet18
ResNet50
MobileNetV2
33.26
53.61
35.26
37.93
58.84
41.05
29.47
48.87
30.90
Feature
FitNet [ 18 ]
34.13
54.16
36.71
38.76
59.62
41.80
30.20
49.80
31.69
ReviewKD [ 27 ]
36.75
56.72
34.00
40.36
60.97
44.08
33.71
53.15
36.13
Table 4: Object detection results on MS-COCO using Faster R-CNN-FPN.
Configuration
V13 → V8
V13 → MNV1
KD (Aff.)
74.39±0.43
67.78±0.40
CI
75.10±0.19
66.60±0.16
CI + ETF (ACI)
75.80±0.15
68.10±0.14
ACI + DS
78.55±0.05
70.98±0.02
ACI + SICD + GM
80.52±0.02
73.30±0.01
Table 5: Component ablation. Accuracy (%); best results are bold. Aff.: affine head; CI: cosine interface; ACI: CI with equiangular tight frame (ETF) initialization; DS: dual standardization with full KL; GM: gradient matching. V13/V8: VGG13/VGG8; MNV1: MobileNetV1.
Method
T → S
KD
ACI
ACI + SICD
Δ
AT [ 28 ]
R32 × 4 → SV1
77.30
77.60
78.97
+1.67
ReviewKD [ 27 ]
V13 → MNV1
68.48
67.48
70.43
+1.95
Table 6: Compatibility with feature-based distillation on CIFAR-100. Top-1 accuracy (%). Δ : gain over KD in percentage points. Best results within each row are bold. T/S: teacher/student; R32 × 4: ResNet32 × 4; SV1: ShuffleNetV1; V13: VGG13; MNV1: MobileNetV1.
Figure 6: Non-target relation discrepancies. (a) Empirical cumulative distributions of conditional TV. (b) Medians (circles), means (diamonds), and image interquartile ranges (segments).
Knowledge distillation transfers knowledge by encouraging a student to match a teacher's predicted class probabilities. These probabilities express not only confidence in the correct class, but also relations among incorrect alternatives. Yet closer imitation does not necessarily yield a better student. A student may already distinguish the correct class more sharply than its teacher, so further imitation can require giving back discrimination it has acquired. Our main result is an exact separation between full imitation and conditional learning. When the correct class's score advantage over each alternative must be preserved, full teacher-to-student KL minimization is blocked exactly when the student assigns no more probability than the teacher to every incorrect class. Crucially, the teacher's relative probabilities among incorrect classes remain fully learnable. We characterize the exact price of this transfer: a minimum increase in correct-class log-odds that compensates for the largest conditional-probability mismatch. Label fitting and conditional matching can therefore be completed even as full teacher KL diverges. This separation motivates task-preserving knowledge distillation (TPKD), which keeps the label gradient intact and minimally corrects the conditional gradient so that its output update preserves the label step's gains against every incorrect alternative. The corrected conditional direction retains more than half of the original first-order conditional descent at the same step size, with a tight bound. For a fixed positive conditional target and sufficiently small constant output steps, label and conditional errors vanish together. Experiments trace this learning from exact head updates to ordinary network training. TPKD reaches 88.05% accuracy on CIFAR-100 and 93.81% on CLINC150, improving over standard distillation by 0.47 and 0.35 percentage points across three seeds.
Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher models. However, a substantial size gap between teacher and student models often impedes effective knowledge transfer. Most existing approaches adopt a static, one-way teacher-to-student distillation paradigm, which overlooks the dynamic nature of student learning and fails to provide targeted guidance on hard samples. In this paper, we propose adaptive reciprocal knowledge distillation (AR-KD), a novel method that improves knowledge transfer by simplifying the teacher's output distribution. Specifically, AR-KD performs reciprocal adaptation on the teacher by matching its class correlation matrix to the student's relational representation, which reshapes the teacher's prediction structure to better suit the student's capacity. This relational alignment mitigates the collapse of inter-class dark knowledge caused by overconfident teachers, enabling the student to learn from richer and more compatible supervisory signals. We evaluate AR-KD on CIFAR-100 and ImageNet-1k classification datasets, where it outperforms state-of-the-art knowledge distillation baselines. Specifically, AR-KD improves student performance across homogeneous and heterogeneous setups: up to 7.13% accuracy gain for students, 1.42% to 4.15% higher than vanilla KD on average, and further improvements when integrated with other advanced methods. Our code is available at https://anonymous.4open.science/r/ARKD/.
Dawen Jiang, Zhishu Shen, Zeyu Liu +1
Wuhan University of Technology Wuhan, Hubei, China · Tongji University Shanghai, China
Cross-modal knowledge distillation can incur negative transfer when source-modality supervision conflicts with the target modality's discriminative characteristics. We propose TALD, a two-stage framework for Transferability-Aware Logit Disentanglement in class-aligned logit space. Guided by task-dependent self-benefiting and cross-modal exclusivity criteria, a Residual Disentangler decomposes logits from a teacher and an independently trained target-modality reference student into modality-general and modality-specific components. A Sample-Adaptive Modulator regulates decomposition constraints and sample-specific fusion weights. The learned modality-general components are then adaptively fused to supervise the final student. Across five benchmarks spanning vision, audio, and text, TALD achieves the largest mean accuracy gain and highest positive-transfer frequency among the compared methods. Across ten transfer directions with five runs each, TALD improves over training without distillation in 84% of evaluations, compared with at most 70% for the baselines, while offering a favorable accuracy-efficiency trade-off. We plan to release the code.
Hui Li, Pengfei Yang, Xinyi Feng +7
1Xidian University · 2Beihang University · 3Xi’an Polytechnic University