Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

Qianfeng Yuan; Wenbing Tao

cs.LGSep 30, 2026

Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

Authors: Qianfeng Yuan, Wenbing Tao

Organizations: Huazhong University of Science and Technology

Abstract

Knowledge distillation transfers knowledge by encouraging a student to match a teacher's predicted class probabilities. These probabilities express not only confidence in the correct class, but also relations among incorrect alternatives. Yet closer imitation does not necessarily yield a better student. A student may already distinguish the correct class more sharply than its teacher, so further imitation can require giving back discrimination it has acquired. Our main result is an exact separation between full imitation and conditional learning. When the correct class's score advantage over each alternative must be preserved, full teacher-to-student KL minimization is blocked exactly when the student assigns no more probability than the teacher to every incorrect class. Crucially, the teacher's relative probabilities among incorrect classes remain fully learnable. We characterize the exact price of this transfer: a minimum increase in correct-class log-odds that compensates for the largest conditional-probability mismatch. Label fitting and conditional matching can therefore be completed even as full teacher KL diverges. This separation motivates task-preserving knowledge distillation (TPKD), which keeps the label gradient intact and minimally corrects the conditional gradient so that its output update preserves the label step's gains against every incorrect alternative. The corrected conditional direction retains more than half of the original first-order conditional descent at the same step size, with a tight bound. For a fixed positive conditional target and sufficiently small constant output steps, label and conditional errors vanish together. Experiments trace this learning from exact head updates to ordinary network training. TPKD reaches 88.05% accuracy on CIFAR-100 and 93.81% on CLINC150, improving over standard distillation by 0.47 and 0.35 percentage points across three seeds.

Figures & tables

Figure 1: Full imitation can worsen while conditional knowledge is learned. The safe path attains the minimum confidence

a_{\min}\approx 0.946

. Label CE and conditional KL fall while full KL rises; losses are normalized by their initial values.

Panel of Figure 2 — Figure 2: TPKD keeps the label signal intact. Only the teacher-conditional direction is corrected. The combined direction is passed to ordinary backpropagation through Eq. ( 8 ).

Panel of Figure 3 — Figure 3: Protection preserves a genuine conditional-learning direction. The corrected direction combines confidence compensation with a pure conditional component (a), which remains closely aligned with the original signal (b). The four-class example is constructed in Appendix B.5 .

Student state	Blocked (%)	Remaining conditional KL	Minimum log-odds cost
CE20	50.00	1.696	7.520
CE40	55.29	1.766	7.632
CE60	58.90	1.803	7.569

Table 1: Full-imitation blocking leaves conditional knowledge unlearned. Conditional error and transfer cost are means within the blocked set (nats).

State	Gain ( $10^{-3}$ )	KL change ( $10^{-3}$ )	Min. retained (%)	Min. cosine	$\Phi$ change ( $10^{-3}$ )
CE20	1.195	-1.359	52.13	0.7220	-3.302
CE60	1.213	-1.283	52.13	0.7221	-1.831

Table 2: Exact head steps learn conditional structure while preserving label-step progress.

State	Conditional test	Joint descent test	Positive pairs	Mean gain ( $10^{-3}$ )
CE20	16/16	16/16	16/16	8.252
CE60	16/16	16/16	16/16	9.886

Table 3: The conditional advantage survives ordinary backpropagation.

Method	CIFAR-100	CLINC150
	60 epochs	4 epochs
CE	$87.37\pm 0.07$	$93.24\pm 0.28$
KD	$87.58\pm 0.07$	$93.46\pm 0.34$
KL-Dist	$82.26\pm 0.05$	$93.38\pm 0.19$
DKD	$84.53\pm 0.09$	$93.59\pm 0.20$
DP-U	$83.14\pm 0.04$	$93.38\pm 0.19$

Table 4: Full-training accuracy (%). Mean

\pm

sample standard deviation over three seeds.

Condition	Direction	CIFAR-100	CLINC150
CE	$h$	$87.37\pm 0.07$	$93.24\pm 0.28$
Without CE gradient	$d/2$	$85.33\pm 0.05$	$91.60\pm 0.07$
Safe full-KL	$h+\Pi_{\mathcal{K}_{y}}(p-t)/2$	$87.21\pm 0.04$	$93.33\pm 0.29$
Without projection	$h+u/2$	$87.83\pm 0.03$	$93.83\pm 0.23$
TPKD	$h+d/2$	$88.05\pm 0.12$	$93.81\pm 0.27$

Table 5: Component ablations. Accuracy (%), mean

\pm

sample standard deviation over three seeds.

Explore similar work

Jun 10, 2026cs.CV

Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions

Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs. Their interaction, however, remains poorly understood, particularly when mixup is applied only during student training. In this setting, the teacher is queried on inputs drawn from a vicinal distribution it never saw during training, a controlled mismatch whose effect on knowledge transfer has not been characterised. We show that this mismatch causes the teacher's supervisory signal to be dominated by distributional confusion rather than inter-class structure. Despite it, the student does not merely imitate the teacher: it independently acquires greater linearity in the vicinal region, a structural property that the teacher lacks, and goes beyond dark-knowledge transfer. KD with mixup consistently improves student accuracy and reduces overconfidence by an order of magnitude relative to the baseline, across CIFAR and ImageNet with varying-capacity teachers. Crucially, calibration propagates from teacher to student independently of accuracy transfer, and temperature scaling governs a measurable accuracy-calibration trade-off that becomes more pronounced under vicinal training. These results reframe mixup distillation not as a degraded version of standard KD, but as a richer transfer channel that simultaneously shapes discriminative performance, uncertainty estimation, and representational geometry.

José Medina, Paul Honeine, Abdelaziz Bensrhair +1

ITS Lab, Institute of Computer Science, University of Tartu, Estonia. · LITIS, Universit´e de Rouen, France. · LITIS, INSA de Rouen, France.

Knowledge Distillation Few-Step Distillation Probe-Logit Distillation Discriminative

Ask PDF

May 19, 2026cs.LG

Consistently Informative Soft-Label Temperature for Knowledge Distillation

Knowledge distillation (KD) transfers knowledge from a high-capacity teacher to a compact student by matching their predictive distributions, with temperature scaling serving as a central mechanism for smoothing teacher predictions and exposing informative "dark knowledge" beyond the hard label. However, the standard fixed-temperature design is inherently sample-agnostic. Since samples differ in logit scale and learning difficulty, a single global temperature produces teacher soft labels with highly inconsistent entropy: some predictions remain overly sharp and provide limited inter-class information, whereas others become over-smoothed and lose class-discriminative information. Moreover, sharing the same temperature between teacher and student further imposes rigid logit-scale alignment despite their capacity mismatch. To address these limitations, we propose CIST (Consistently Informative Soft-label Temperature), which assigns separate sample-wise adaptive temperatures to the teacher and student. This design produces consistently informative teacher soft labels while relaxing rigid teacher--student logit-scale matching. It also reweights the distillation objective according to teacher confidence and student learning difficulty. Theoretically, we show that teacher-label entropy is largely governed by the ratio between the maximum teacher logit and the temperature, providing a principled basis for adaptive smoothing. Empirically, CIST mitigates the inconsistency induced by fixed temperature, and experiments on both vision and language distillation tasks show consistent improvements over standard KD and strong baselines with negligible computational overhead.

Hoang-Chau Luong, Nghia Van Vo, Kaiqi Zhao +1

Rochester Institute of Technology, Rochester, NY, USA · Oakland University, Rochester, MI, USA

Knowledge Distillation

Ask PDF

Sep 30, 2026cs.LG

K2P: Label-Free Knowledge to Prompt Distillation

Knowledge distillation can transfer reasoning from stronger teachers to frozen students through reusable prompts, but avoiding weight updates does not eliminate supervision. Without ground-truth answers, teacher solutions are unverified, and agreement with the teacher can reward shared mistakes. We introduce Knowledge-to-Prompt (K2P) for label-free knowledge distillation to prompts. K2P synthesizes reusable instructions from teacher solutions, refines them using paired teacher and student responses, and guides search and selection with answer agreement. It retains candidates that adaptive search may undervalue and selects on reserved questions. Deployment uses only the frozen student and selected prompt. Our theory separates generation and selection gaps and gives conditions under which agreement-guided construction yields accuracy guarantees despite imperfect teacher references. Across reasoning tasks and students, K2P outperforms label-free alternatives overall and remains competitive with supervised prompt optimization. Ablations and archive diagnostics assess the contributions of teacher solutions and refinement, while revealing the limits of agreement-guided selection.

Yingchuan Zhang, Haoran Lu, Wenxuan Zhong +1

Department of Statistics University of Georgia Athens, GA 30602

Knowledge Distillation Teacher Prompt Engineering Reasoning Skills Label-Free

Ask PDF