Many hypotheses compete to explain the successes of knowledge distillation. To help address this, we propose and analyze a mathematical model of distillation, which suggests that distillation's performance comes not from obtaining better models but from easier to optimize landscapes. For generalized linear models trained with stochastic gradient descent, we prove that distillation fits performant student models asymptotically more quickly than non-distilled models. In rank-1 matrix approximation, we characterize conditions on the target matrix under which gradient descent with distillation converges strictly faster than training on the supervised objective. The theory helps delineate the ways distillation provides benefits (i.e., in optimization speed, not in generalization), and experiments on real datasets corroborate the theoretical predictions.
Figures & tables
Figure 1: Asymptotic relative efficiency, given by the ratio of asymptotic covariance traces (supervised learning over knowledge distillation), as a function of label variance. The teacher dimension is fixed at p=1000 , and the student dimension is d . Each point represents an average over 100 sampled datasets and true parameters.
Figure 2: (Multinomial logistic regression, Fashion-MNIST) Normalized parameter error of SGD iterates under knowledge distillation and standard supervision for student dimensions d∈{100,300,500} . The initial step size η0 , optimal for standard supervision, is used for both methods. Results are averaged over 10 trials.
Figure 3: (Language model pretraining) Ratio of training steps required by knowledge distillation versus standard supervision across subsets of increasing entropy, evaluated at 0.5, 0.75, and 1.0 fractions of training steps. At each point, we record the maximum of the two validation losses, and measure how many steps each student needs to reach this loss. This is repeated over five trials.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Step size η0 , logistic regression on synthetic data.
d=3
d=5
d=7
d=9
Standard supervision η0
10−1.5
0.1
0.1
0.1
Distillation η0
1.0
1.0
1.0
1.0
Appendix
Table 1: Initial step size η0 for logistic regression on synthetic data.
Figure 4: (Logistic regression, synthetic data) Distance from optimal point γ^ for stochastic gradient descent iterates under knowledge distillation and standard supervision. All results are averaged over 100 trials. On the left, both trajectories use η0 that is optimal for standard supervision. On the right, the step η0 is chosen optimally for both methods.
Figure 5: (Rank-1 approximation) Parameter error of gradient descent iterates under knowledge distillation and supervised learning, averaged over 100 trials. In the plot on the left, the eigenvalue spectrum is fixed ( λr=0.7 , λn=0.65 ) and teacher ranks are varied. In the plot on the right, the teacher rank is fixed ( r=8 ) and the “tail“ spectrum λr through λn is varied.
Step size η0 , multinomial logistic regression.
d=100
d=300
d=500
Standard supervision η0
10−0.75
10−0.5
10−0.5
Distillation η0
10−0.75
10−0.5
10−0.25
Appendix
Table 2: Initial steps size η0 for multinomial logistic regression on the Fashion-MNIST dataset.
Figure 6: (Multinomial logistic regression, Fashion-MNIST) Normalized parameter error, averaged over 10 trials, of SGD iterates under knowledge distillation and standard supervision for student dimensions d∈{100,300,500} . The initial step size η0 is chosen optimally for both methods.
Knowledge distillation is widely used to improve generalization in practice, yet its theoretical understanding remains elusive. In the standard distillation setting, a teacher model provides soft predictions to guide the training of a student model. We model teacher and student training as coupled stochastic processes and introduce a distillation divergence, defined as the Kullback-Leibler divergence between these two stochastic kernels. Within this framework, we derive two generalization bounds for the student model relative to the teacher's generalization gap: an upper bound under a sub-Gaussian assumption via algorithmic stability, and a lower bound under a central condition with sharper dependence on the distillation divergence. We further develop a loss-sharpness-aware bound with an explicit tightness regime, showing that the teacher's local flatness can strictly tighten the bound. Additionally, in a linear Gaussian case study, the distillation divergence admits an interpretable decomposition into bias, variance, and rank-bottleneck costs, yielding practical guidance for distillation design.
Bingying Li, Haiyun He
Internet of Things Thrust, Information Hub The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China
Knowledge Distillation (KD) has emerged as a vital paradigm for transferring the capabilities of high-capacity models to efficient ``student'' counterparts, addressing critical challenges in computational cost, deployment constraints, and privacy-sensitive settings. Although KD is widely used in practice, it is often viewed primarily as an engineering technique, with a unified statistical perspective remaining less developed. This review bridges that gap by presenting a unified Bayesian formulation of KD that formulates teacher predictions as prior information. This provides a principled interpretation of how teacher information is incorporated into student learning and establishes a rigorous connection to uncertainty quantification. We demonstrate how this foundational lens reconciles classical distillation with modern extensions in generative and foundation-model systems, showing that contemporary developments remain rooted in these same statistical principles. By synthesizing theory with emerging methodologies and diverse applications, this review provides a conceptual roadmap and identifies critical open problems for the future of the field.
Luyang Fang, Haoran Lu, Jiazhang Cai +4
Department of Statistics, University of Georgia, Athens, Georgia 30602, USA · Department of Biostatistics, Boston University, Boston, Massachusetts 02118, USA
On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher. We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone. When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs. One forward pass over a few hundred rollouts provides them. We call this training-free on-policy distillation. What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision. On Olmo-3-7B, a few GPU-hours of forward computation per specialist yield a generalist that keeps most of each specialist's gain on each of seven mathematics and code benchmarks.
Bangji Yang, Jiajun Fan, Zhaoyu Zhu +2
University of Illinois at Urbana-Champaign · Tsinghua University