Knowledge distillation trains a small student model to reproduce the outputs of a large teacher model, and its progress is typically monitored through the teacher--student discrepancy. The quantity of ultimate interest, however, is the student's error with respect to the true task. We study the relation between these two objectives in a minimal three-party model, a true teacher (generative model), a teacher, and a student, all soft committee machines, in which the true teacher contains a shared latent factor that the teacher cannot represent, with mismatch strength controlled by a single scalar \dmiss. Within an order-parameter description of online distillation, and exploiting closed-form (arcsine-type) expressions for all errors under error-function activations, we prove that the learning dynamics and the distillation error \Ets are exactly invariant to \dmiss, whereas the true error \Etzs and the gap Δ=\Etzs−\Ets are strictly increasing in \dmiss, with a rate that is amplified linearly by the complexity M0 of the true teacher. Numerical phase diagrams over the plane spanned by true-teacher complexity and student capacity confirm the predicted deformation: the contours of \Ets do not move while the landscape of \Etzs rises systematically, and a teacher-miss regime, where mimicry succeeds but the task fails, expands with \dmiss. The results give a quantitative warning against evaluating distillation solely through teacher-mimicry metrics and identify the gap Δ as a minimal diagnostic for distinguishing teacher-miss from capacity-limited failure.
Figures & tables
Figure 1: Separation of the error landscapes under the mismatch strength δmiss . The background heat map shows the true error Et0s (color scale shared across panels); the solid contours show the distillation error Ets . Increasing δmiss leaves the Ets contours unchanged (Theorem 6 (i)) while the Et0s landscape rises systematically (Theorem 6 (ii)). Horizontal axis: student capacity Ktot ; vertical axis: true-teacher complexity M0 .
Figure 2: Phase diagram of the gap Δ=Et0s−Ets . Each panel corresponds to a different δmiss , with a color scale shared across panels for direct comparison. At δmiss=0 the mirror construction gives Δ=0 identically (Remark 8 ); as δmiss increases, the region with Δ>0 expands from the large- M0 side. Horizontal axis: Ktot ; vertical axis: M0 .
Figure 3: Auxiliary regime maps based on threshold classification with (τts,τt0s)=(0.15,0.18) , dividing the plane into good/teacher-miss/capacity-limited (and other). Because the classification depends on the thresholds, the main conclusions are drawn from the continuous quantities ( Ets,Et0s,Δ ) of Figs. 1 – 2 .
Figure 4: (Auxiliary) sensitivity to nsamp . Regime classification (colors) and contours of Δ (dashed) at δmiss=0.15 , compared for nsamp∈{500,2000,8000} (independent seeds). The global structure of the boundaries is preserved for all nsamp .
This position paper argues that knowledge distillation must account for what it loses: student models should be judged not only by retained task scores, but by whether they preserve the teacher capabilities that make those scores reliable. This matters because distillation is increasingly used to turn large teacher models into deployable students, yet headline metrics can obscure losses in the capabilities that make teacher behavior reliable. Conceptually, we show that current evaluation often assumes retained task scores imply retained teacher capabilities. Reframing distillation as a lossy projection exposes this flaw: students may match selected teacher observables without preserving the capabilities that make them reliable. We then synthesize existing evidence into a taxonomy of off-metric distillation losses, showing that such losses are concrete, recurring, and measurable, yet often unaccounted for when studies report what students retain rather than what they lose. To make the position actionable, we propose scenario-specific preservation targets and a Distillation Loss Statement that reports what was preserved, what was lost, and why the remaining losses are acceptable. The goal is not lossless distillation, but accountable distillation.
Wenshuo Wang
School of Future Technology, South China University of Technology, China
Knowledge distillation transfers knowledge by encouraging a student to match a teacher's predicted class probabilities. These probabilities express not only confidence in the correct class, but also relations among incorrect alternatives. Yet closer imitation does not necessarily yield a better student. A student may already distinguish the correct class more sharply than its teacher, so further imitation can require giving back discrimination it has acquired. Our main result is an exact separation between full imitation and conditional learning. When the correct class's score advantage over each alternative must be preserved, full teacher-to-student KL minimization is blocked exactly when the student assigns no more probability than the teacher to every incorrect class. Crucially, the teacher's relative probabilities among incorrect classes remain fully learnable. We characterize the exact price of this transfer: a minimum increase in correct-class log-odds that compensates for the largest conditional-probability mismatch. Label fitting and conditional matching can therefore be completed even as full teacher KL diverges. This separation motivates task-preserving knowledge distillation (TPKD), which keeps the label gradient intact and minimally corrects the conditional gradient so that its output update preserves the label step's gains against every incorrect alternative. The corrected conditional direction retains more than half of the original first-order conditional descent at the same step size, with a tight bound. For a fixed positive conditional target and sufficiently small constant output steps, label and conditional errors vanish together. Experiments trace this learning from exact head updates to ordinary network training. TPKD reaches 88.05% accuracy on CIFAR-100 and 93.81% on CLINC150, improving over standard distillation by 0.47 and 0.35 percentage points across three seeds.
Knowledge distillation is widely used to improve generalization in practice, yet its theoretical understanding remains elusive. In the standard distillation setting, a teacher model provides soft predictions to guide the training of a student model. We model teacher and student training as coupled stochastic processes and introduce a distillation divergence, defined as the Kullback-Leibler divergence between these two stochastic kernels. Within this framework, we derive two generalization bounds for the student model relative to the teacher's generalization gap: an upper bound under a sub-Gaussian assumption via algorithmic stability, and a lower bound under a central condition with sharper dependence on the distillation divergence. We further develop a loss-sharpness-aware bound with an explicit tightness regime, showing that the teacher's local flatness can strictly tighten the bound. Additionally, in a linear Gaussian case study, the distillation divergence admits an interpretable decomposition into bias, variance, and rank-bottleneck costs, yielding practical guidance for distillation design.
Bingying Li, Haiyun He
Internet of Things Thrust, Information Hub The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China