Many hypotheses compete to explain the successes of knowledge distillation. To help address this, we propose and analyze a mathematical model of distillation, which suggests that distillation's performance comes not from obtaining better models but from easier to optimize landscapes. For generalized linear models trained with stochastic gradient descent, we prove that distillation fits performant student models asymptotically more quickly than non-distilled models. In rank-1 matrix approximation, we characterize conditions on the target matrix under which gradient descent with distillation converges strictly faster than training on the supervised objective. The theory helps delineate the ways distillation provides benefits (i.e., in optimization speed, not in generalization), and experiments on real datasets corroborate the theoretical predictions.
Figures & tables
Figure 1: Asymptotic relative efficiency, given by the ratio of asymptotic covariance traces (supervised learning over knowledge distillation), as a function of label variance. The teacher dimension is fixed at p=1000 , and the student dimension is d . Each point represents an average over 100 sampled datasets and true parameters.
Figure 2: (Multinomial logistic regression, Fashion-MNIST) Normalized parameter error of SGD iterates under knowledge distillation and standard supervision for student dimensions d∈{100,300,500} . The initial step size η0 , optimal for standard supervision, is used for both methods. Results are averaged over 10 trials.
Figure 3: (Language model pretraining) Ratio of training steps required by knowledge distillation versus standard supervision across subsets of increasing entropy, evaluated at 0.5, 0.75, and 1.0 fractions of training steps. At each point, we record the maximum of the two validation losses, and measure how many steps each student needs to reach this loss. This is repeated over five trials.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Step size η0 , logistic regression on synthetic data.
d=3
d=5
d=7
d=9
Standard supervision η0
10−1.5
0.1
0.1
0.1
Distillation η0
1.0
1.0
1.0
1.0
Appendix
Table 1: Initial step size η0 for logistic regression on synthetic data.
Figure 4: (Logistic regression, synthetic data) Distance from optimal point γ^ for stochastic gradient descent iterates under knowledge distillation and standard supervision. All results are averaged over 100 trials. On the left, both trajectories use η0 that is optimal for standard supervision. On the right, the step η0 is chosen optimally for both methods.
Figure 5: (Rank-1 approximation) Parameter error of gradient descent iterates under knowledge distillation and supervised learning, averaged over 100 trials. In the plot on the left, the eigenvalue spectrum is fixed ( λr=0.7 , λn=0.65 ) and teacher ranks are varied. In the plot on the right, the teacher rank is fixed ( r=8 ) and the “tail“ spectrum λr through λn is varied.
Step size η0 , multinomial logistic regression.
d=100
d=300
d=500
Standard supervision η0
10−0.75
10−0.5
10−0.5
Distillation η0
10−0.75
10−0.5
10−0.25
Appendix
Table 2: Initial steps size η0 for multinomial logistic regression on the Fashion-MNIST dataset.
Figure 6: (Multinomial logistic regression, Fashion-MNIST) Normalized parameter error, averaged over 10 trials, of SGD iterates under knowledge distillation and standard supervision for student dimensions d∈{100,300,500} . The initial step size η0 is chosen optimally for both methods.
Department of Statistics, University of Georgia, Athens, Georgia 30602, USA · Department of Biostatistics, Boston University, Boston, Massachusetts 02118, USA