Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance

Kazuyuki Hara; Hideitsu Hino

cs.LGAug 29, 2026

Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance

Authors: Kazuyuki Hara, Hideitsu Hino

Organizations: Nihon University · The Institute of Statistical Mathematics

Abstract

Knowledge distillation trains a small student model to reproduce the outputs of a large teacher model, and its progress is typically monitored through the teacher--student discrepancy. The quantity of ultimate interest, however, is the student's error with respect to the true task. We study the relation between these two objectives in a minimal three-party model, a true teacher (generative model), a teacher, and a student, all soft committee machines, in which the true teacher contains a shared latent factor that the teacher cannot represent, with mismatch strength controlled by a single scalar $\dmiss$ . Within an order-parameter description of online distillation, and exploiting closed-form (arcsine-type) expressions for all errors under error-function activations, we prove that the learning dynamics and the distillation error $\Ets$ are exactly invariant to $\dmiss$ , whereas the true error $\Etzs$ and the gap $Δ=\Etzs-\Ets$ are strictly increasing in $\dmiss$ , with a rate that is amplified linearly by the complexity $M_0$ of the true teacher. Numerical phase diagrams over the plane spanned by true-teacher complexity and student capacity confirm the predicted deformation: the contours of $\Ets$ do not move while the landscape of $\Etzs$ rises systematically, and a teacher-miss regime, where mimicry succeeds but the task fails, expands with $\dmiss$ . The results give a quantitative warning against evaluating distillation solely through teacher-mimicry metrics and identify the gap $Δ$ as a minimal diagnostic for distinguishing teacher-miss from capacity-limited failure.

Figures & tables

Panel of Figure 1 — Figure 1: Separation of the error landscapes under the mismatch strength $\delta_{\mathrm{miss}}$ . The background heat map shows the true error $E_{t_{0}s}$ (color scale shared across panels); the solid contours show the distillation error $E_{ts}$ . Increasing $\delta_{\mathrm{miss}}$ leaves the $E_{ts}$ contours unchanged (Theorem 6 (i)) while the $E_{t_{0}s}$ landscape rises systematically (Theorem 6 (ii)). Horizontal axis: student capacity $K_{\mathrm{tot}}$ ; vertical axis: true-teacher complexity $M_{0}$ .

Figure 2: Phase diagram of the gap $\Delta=E_{t_{0}s}-E_{ts}$ . Each panel corresponds to a different $\delta_{\mathrm{miss}}$ , with a color scale shared across panels for direct comparison. At $\delta_{\mathrm{miss}}=0$ the mirror construction gives $\Delta=0$ identically (Remark 8 ); as $\delta_{\mathrm{miss}}$ increases, the region with $\Delta>0$ expands from the large- $M_{0}$ side. Horizontal axis: $K_{\mathrm{tot}}$ ; vertical axis: $M_{0}$ .

Figure 3: Auxiliary regime maps based on threshold classification with

(\tau_{ts},\tau_{t_{0}s})=(0.15,0.18)

, dividing the plane into good/teacher-miss/capacity-limited (and other). Because the classification depends on the thresholds, the main conclusions are drawn from the continuous quantities (

E_{ts},E_{t_{0}s},\Delta

) of Figs. 1 – 2 .

Figure 4: (Auxiliary) sensitivity to

n_{\mathrm{samp}}

. Regime classification (colors) and contours of

\Delta

(dashed) at

\delta_{\mathrm{miss}}=0.15

, compared for

n_{\mathrm{samp}}\in\{500,2000,8000\}

(independent seeds). The global structure of the boundaries is preserved for all

n_{\mathrm{samp}}

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 819 papers

Knowledge Distillation Must Account for What It Loses
Apr 28, 2026Wenshuo WangTeacher-Student Learning Knowledge Distillation
School of Future Technology, South China University of Technology, China
Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation
Sep 30, 2026Qianfeng Yuan, Wenbing TaoKnowledge Distillation
Huazhong University of Science and Technology
On the Generalization of Knowledge Distillation: An Information-Theoretic View
May 13, 2026Bingying Li, Haiyun HeInformation-Theoretic Generalization Bounds Statistical Learning Theory
Internet of Things Thrust, Information Hub The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China

Apr 28, 2026J/K move · Enter open · S save

Knowledge Distillation Must Account for What It Loses

Wenshuo Wang

School of Future Technology, South China University of Technology, China

Teacher-Student Learning Knowledge Distillation

Open paper AskPDF arXiv

This position paper argues that knowledge distillation must account for what it loses: student models should be judged not only by retained task scores, but by whether they preserve the teacher capabilities that make those scores reliable. This matters because distillation is increasingly used to turn large teacher models into deployable students, yet headline metrics can obscure losses in the capabilities that make teacher behavior reliable. Conceptually, we show that current evaluation often assumes retained task scores imply retained teacher capabilities. Reframing distillation as a lossy projection exposes this flaw: students may match selected teacher observables without preserving the capabilities that make them reliable. We then synthesize existing evidence into a taxonomy of off-metric distillation losses, showing that such losses are concrete, recurring, and measurable, yet often unaccounted for when studies report what students retain rather than what they lose. To make the position actionable, we propose scenario-specific preservation targets and a Distillation Loss Statement that reports what was preserved, what was lost, and why the remaining losses are acceptable. The goal is not lossless distillation, but accountable distillation.

Abstract

Knowledge Distillation

Explore similar work

Knowledge Distillation Must Account for What It Loses

Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

On the Generalization of Knowledge Distillation: An Information-Theoretic View