The Distributional View of Knowledge Distillation

Gordei Verbii; Juho Lee

stat.MLAug 15, 2026

The Distributional View of Knowledge Distillation

Authors: Gordei Verbii, Juho Lee

Organizations: Independent Researcher · KAIST

Abstract

Token-level knowledge distillation (KD) matches two conditional distributions per position, yet the standard objectives compare them pointwise: a Kullback-Leibler gradient is blind to which wrong token receives probability mass. We develop a distributional view in which the teacher is represented not by a single softened output but by a family of multi-temperature views - marginals of the annealing path of its logits - and the student is trained against a geometry-aware aggregate of these views under an embedding-based ground cost. We formalize the resulting design space (mixtures, log-linear pooling, entropic Wasserstein barycenters, and a debiased Sinkhorn-divergence flagship in hub and path forms), prove an exact collapse result showing log-linear pooling of tempered views is equivalent to a single temperature, and give a multi-marginal Schrodinger-bridge reading that yields falsifiable predictions. On instruction-tuned Pythia pairs, experiments yield three empirical laws: (i) dispersion law - the benefit of multi-temperature aggregation grows monotonically with the effective temperature dispersion of the views, not with their number; (ii) dispersed views unlock the aggregation operator - the barycenter separates from the arithmetic mixture exactly when transport-based aggregation starts to beat averaging; and (iii) two-regime picture governed by the ceiling gap $Γ=\mathrm{PPL}_{\mathrm{SFT}}-\mathrm{PPL}_{T}$ : when the fine-tuned teacher barely beats a supervised student the gentle transport objective is the best KD loss but no KD beats supervised fine-tuning, whereas at a real ceiling the ranking inverts - and the sign of the fidelity-generalization correlation flips. We argue that "which distillation loss is the best" is not a fixed property of the loss but a function of $Γ$ .

Figures & tables

Reference models
	small $160\text{M}\!\to\!31\text{M}$ , $\Gamma{=}{+}0.10$				big $410\text{M}\!\to\!70\text{M}$ , $\Gamma{=}{+}1.49$
method	PPL $\downarrow$	agree@1 $\uparrow$	predKL $\downarrow$	ECE $\downarrow$	PPL $\downarrow$	agree@1 $\uparrow$	predKL $\downarrow$	ECE $\downarrow$
Teacher (3-ep. FT)	10.664	–	–	.123	9.602	–	–	.195
SFT (no KD)	10.762 $\pm$ .017	.528	1.158	.060	11.089 $\pm$ .357	.468	1.587	.068
Pointwise divergences: the KL family
VanillaKD	11.923 $\pm$ .024	.550	0.984	.087	10.588 $\pm$ .049	.488	1.429	.153

Table 1: Main results (mean

\pm

s.d. over 5 seeds; held-out general_qa ). Left block: thin ceiling (

\Gamma=+0.10

); right block: real ceiling (

\Gamma=+1.49

). Bold = best KD method within a block and column; underline = beats SFT on PPL (

p<.05

, seed-paired). SFT retains the best ECE in both blocks; the bold ECE is the best among KD methods. The flagship row uses the endpoint views

\{0.5,1.5\}

in hub mode; both choices are varied in Table 2 .

View set (target-side aggregation)			Flagship objective and implementation
setting	$D_{\mathrm{eff}}$	PPL $\downarrow$	setting	PPL $\downarrow$
Dispersion sweep ( $\tau=1\pm\Delta$ ; $\Delta{=}0$ is one view)			Views $\times$ student-view mode
$\Delta{=}0$	0	13.08 $\pm$ .06	interior-hub	11.28
$\Delta{=}0.1$	0.10	12.54 $\pm$ .04	endpoints-hub	11.12
$\Delta{=}0.3$	0.30	11.90 $\pm$ .03	interior-path	10.88
$\Delta{=}0.5$	0.50	11.47 $\pm$ .01	endpoints-path	10.89

Table 2: Ablations (small pair,

\Gamma=+0.10

; held-out PPL). The view-set column uses SinkhornBary as the sensitive geometric probe and

D_{\mathrm{eff}}

is the effective dispersion ( 11 ); the flagship column varies BaryOT . Bold marks the best setting within each ablation. Seed counts differ by block – 3 for the dispersion sweep, 2 for the

K{=}3

probes, the flagship variants and the debiasing pair, 5 for the reference rows and for endpoints-hub – which is why these numbers carry small offsets from the 5-seed Table 1 . Uniform

K{=}3

sits

\approx 0.8

PPL above the

D_{\mathrm{eff}}

curve traced by the dispersion sweep (Fig. 3 a), whereas down-weighting the midpoint lands on it.

Figure 1: One point per run, grouped by experimental setting: what varies along the $x$ -axis is the teacher, not the method. All settings share the dataset, the training loop and the eight methods of Table 1 ; they differ in how strong the fine-tuned teacher ends up relative to an SFT student (the ceiling gap $\Gamma$ ) and, within the saturated settings, in the flagship’s view configuration. Left to right: an under-trained teacher (one fine-tuning epoch on 1k examples, fp16 evaluation) ends up $5.5$ PPL worse than its own student, so every faithful KD transfers weakness; longer teacher fine-tuning (three epochs, same data) closes most of the deficit but the teacher still trails; the saturated-task setting (three epochs, 3k examples, fp32 evaluation) brings the two within a tenth of a perplexity point ( $\Gamma=0.07$ - $0.10$ ) – the pretrained capacity gap is $+384$ PPL, but 3k-shot fine-tuning saturates the task – shown as two independent replications plus a third run whose flagship uses the endpoint views $\{0.5,1.5\}$ ; the capacity-ceiling setting swaps in the $410\text{M}\to 70\text{M}$ pair, which preserves $\Gamma=+1.49$ after identical fine-tuning. Circles: flagship gain (SFT $-$ BaryOT ); stars: best-KD gain; open triangle: the path-matching flagship variant of Eq. ( 9 ). Vertical differences at the same $\Gamma$ therefore isolate method effects (endpoint views $-0.15$ ; path mode a further $-0.23$ ); horizontal movement isolates the teacher effect. Wherever $\Gamma\lesssim 0.1$ no method beats SFT, and across the saturated settings the flagship pays a $0.35$ - $0.51$ tax; at $\Gamma=+1.49$ the best-KD star enters the winning half-plane – by JSKD ( $+1.20$ ), not the transport flagship ( $-0.46$ ). Hypothesis 1 brackets the crossover between the last two settings.

Figure 2: Real-ceiling run ( $410\text{M}\to 70\text{M}$ ): paired differences with bootstrap 95% CIs. Right panel: JSKD ( $-1.20$ , $p=.0023$ ) and VanillaKD ( $-0.50$ , $p=.043$ ) beat SFT – the first KD wins of the study – while the averaging and transport families fall behind. Left panel: the transport barycenter still beats the arithmetic mixture by $-0.99$ ( $p<10^{-4}$ ): the operator effect survives the regime flip.

Figure 3: The dispersion law (small pair, SinkhornBary target in (a), BaryOT in (b)). (a) Held-out PPL falls monotonically as the two views move apart symmetrically ( $\tau=1\pm\Delta$ ; $\Delta{=}0$ is the single-view control), and the two $K{=}3$ probes obey the same curve through their effective dispersion, not their count: the midpoint-down-weighted probe lands on it, uniform $K{=}3$ sits $\approx 0.8$ PPL above. The open marker is the measured sweep linearly interpolated to the uniform probe’s $D_{\mathrm{eff}}=1/3$ ; the quoted gap is that interpolation, not a separate run. (b) In the flagship, hub mode is view-sensitive while path mode is flat and uniformly better – the view-set effect in hub mode was a softening artifact. Error bars are seed s.d.; $n{=}3$ for the sweep, $n{=}2$ for the probes and flagship variants, $n{=}5$ for endpoints-hub. Both panels share the SFT floor and neither $y$ axis starts at zero.

Figure 4: Calibration. (a) Next-token top-1 reliability (small pair): every curve runs under the diagonal, so every model is over-confident, and SFT is the closest to it. (b) 15-bin ECE for all eight students in both regimes, against each regime’s teacher (filled = thin ceiling, hollow = real ceiling). The averaging family is the only one that fails to beat the teacher; BaryOT beats it in both regimes; and the ordering is essentially unchanged by the regime flip, unlike the PPL ranking of Table 1 .

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 783 papers

May 19, 2026

Consistently Informative Soft-Label Temperature for Knowledge Distillation

Knowledge distillation (KD) transfers knowledge from a high-capacity teacher to a compact student by matching their predictive distributions, with temperature scaling serving as a central mechanism for smoothing teacher predictions and exposing informative "dark knowledge" beyond the hard label. However, the standard fixed-temperature design is inherently sample-agnostic. Since samples differ in logit scale and learning difficulty, a single global temperature produces teacher soft labels with highly inconsistent entropy: some predictions remain overly sharp and provide limited inter-class information, whereas others become over-smoothed and lose class-discriminative information. Moreover, sharing the same temperature between teacher and student further imposes rigid logit-scale alignment despite their capacity mismatch. To address these limitations, we propose CIST (Consistently Informative Soft-label Temperature), which assigns separate sample-wise adaptive temperatures to the teacher and student. This design produces consistently informative teacher soft labels while relaxing rigid teacher--student logit-scale matching. It also reweights the distillation objective according to teacher confidence and student learning difficulty. Theoretically, we show that teacher-label entropy is largely governed by the ratio between the maximum teacher logit and the temperature, providing a principled basis for adaptive smoothing. Empirically, CIST mitigates the inconsistency induced by fixed temperature, and experiments on both vision and language distillation tasks show consistent improvements over standard KD and strong baselines with negligible computational overhead.

Hoang-Chau Luong, Nghia Van Vo, Kaiqi Zhao +1

Rochester Institute of Technology, Rochester, NY, USA · Oakland University, Rochester, MI, USA

Teacher-Student Learning Knowledge Distillation

Ask PDF

Mar 2, 2026

A Unified Revisit of Temperature in Classification-Based Knowledge Distillation

A central idea of knowledge distillation is to expose relational structure embedded in the teacher's weights for the student to learn, which is often facilitated using a temperature parameter. Despite its widespread use, there remains limited understanding on how to select an appropriate temperature value, or how this value depends on other training elements such as optimizer, teacher pretraining/finetuning, etc. In practice, temperature is commonly chosen via grid search or by adopting values from prior work, which can be time-consuming or may lead to suboptimal student performance when training setups differ. In this work, we posit that temperature is closely linked to these training components and present a unified study that systematically examines such interactions. From analyzing these cross-connections, we identify and present common situations that have a pronounced impact on temperature selection, providing valuable guidance for practitioners employing knowledge distillation in their work.

Logan Frank, Jim Davis

Ohio State University

Knowledge Distillation

Ask PDF

Aug 29, 2026

Temperature-Adaptive Transformed Teacher Matching

Temperature scaling is a core component of knowledge distillation, yet its role and effect are still not fully understood. Transformed Teacher Matching (TTM) clarifies the role of temperature scaling by applying it only to the teacher distribution and interpreting the resulting objective as standard distillation with an implicit Rényi entropy regularization on the student. However, TTM still relies on a fixed temperature and does not specify how the teacher-side temperature should be adapted for individual samples. In this paper, we introduce a sample-wise inverse-temperature update for TTM by locally minimizing the Kullback-Leibler divergence between the temperature-scaled teacher distribution and the student's prediction. We derive closed-form first and second derivatives with respect to the inverse temperature, and show that they can be expressed using variance and covariance statistics of centered teacher and student logits under the transformed teacher weighting. This yields an efficient curvature-aware update that requires one softmax evaluation and a constant number of class-wise weighted sums. Experiments on standard image classification distillation benchmarks show that our temperature adaptation generally improves TTM and WTTM, while remaining competitive with or outperforming prior temperature-adaptive distillation baselines.

Hiroaki Aizawa, Yoshikazu Hayashi

Graduate School of Advanced Science and Engineering Hiroshima University, Japan · Intelligent Production Technology Research & Development Center for Aerospace Gifu University, Japan

Knowledge Distillation Adaptive Knowledge Distillation

Ask PDF

Abstract

Knowledge Distillation

Explore similar work

Consistently Informative Soft-Label Temperature for Knowledge Distillation

A Unified Revisit of Temperature in Classification-Based Knowledge Distillation

Temperature-Adaptive Transformed Teacher Matching