cs.LGNov 12, 2026

Two mathematical models of knowledge distillation

Authors: Audrey Xie, Ludwig Schmidt, John Duchi

Organizations: Stanford University

Abstract

Many hypotheses compete to explain the successes of knowledge distillation. To help address this, we propose and analyze a mathematical model of distillation, which suggests that distillation's performance comes not from obtaining better models but from easier to optimize landscapes. For generalized linear models trained with stochastic gradient descent, we prove that distillation fits performant student models asymptotically more quickly than non-distilled models. In rank-1 matrix approximation, we characterize conditions on the target matrix under which gradient descent with distillation converges strictly faster than training on the supervised objective. The theory helps delineate the ways distillation provides benefits (i.e., in optimization speed, not in generalization), and experiments on real datasets corroborate the theoretical predictions.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. On the Generalization of Knowledge Distillation: An Information-Theoretic View

    May 13, 2026Bingying Li, Haiyun HeInformation-Theoretic Generalization BoundsStatistical Learning Theory

  2. A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions

    Sep 27, 2026Luyang Fang, Haoran Lu, Jiazhang Cai +4Language Model DistillationKnowledge Distillation

  3. Training-Free On-Policy Distillation

    Oct 15, 2026Bangji Yang, Jiajun Fan, Zhaoyu Zhu +2On-Policy DistillationKnowledge Distillation