Knowledge distillation compresses large teachers into smaller students, but performance saturates at a loss floor that persists across training methods and objectives. We argue this floor is geometric: neural networks represent far more features than dimensions through superposition, and a student of width dS can encode at most dS⋅g(α) features, where g(α)=1/((1−α)ln1−α1) is a sparsity-dependent capacity function. Features beyond this budget are permanently lost, yielding an importance-weighted loss floor. We validate on a toy model (48 configurations, median accuracy >93%) and on Pythia-410M, where sparse autoencoders measure F≈28,700 features at α≈0.992 (critical width dS∗≈1,065). Distillation into five student widths confirms the predicted monotonic floor ordering. The observed floor decomposes into a geometric component and a width-independent architectural baseline (R2=0.993). Linear probing shows coarse concepts survive even 88% feature loss, revealing the floor arises from aggregate loss of fine-grained features in the importance distribution's long tail. Our results connect representation geometry to distillation limits and provide a practical tool for predicting distillation performance from SAE measurements alone.
Figures & tables
Figure 1: Capacity function and critical width. Left: g(α) grows exponentially with sparsity; colored dots mark our toy model sparsity levels. Right: Critical width dS∗=F/g(α) shrinks as sparsity increases, since sparser features need fewer dimensions.
Figure 2: Loss floor vs. student width across 48 configurations (rows: n ; columns: α ). Solid = actual (mean ± std, 20 seeds); dashed = formula (Eq. 2 ); dotted = dS∗ . The formula captures both magnitude and shape.
Figure 3: Predicted vs. actual floor (log-log, 140 points). Left: Refined formula with g(α) (Pearson r=0.93 , MAPE =93.9% ). Right: Naive formula assuming one feature/dim ( R2=−0.04 ). Color = sparsity.
Figure 4: SAE training convergence. Layer 8 (blue) encodes a denser feature set; deeper layers 12 (orange) and 16 (green) show sparser, more selective representations with more feature death.
Layer
Alive F
Avg L0
α
g(α)
dS∗
8
31,006
234.9
0.9924
27.04
1,147
12
28,665
218.3
0.9924
26.92
1,065
16
29,169
249.0
0.9915
24.60
1,186
Table 1: SAE measurements on Pythia-410M. All layers have dS∗>1024 , confirming the teacher itself is in superposition.
Figure 5: (a) Feature importance follows a power law: the top ∼20 features dominate, with a cliff near rank ∼3,000 where thousands reach ∼10−7 . This heavy tail is why compression works. (b) Predicted floor vs. width at layers 8, 12, 16. All layers agree ( dS∗∈[1065,1186] ), converging near zero at dS=1024 .
dS
FS
Dropped
L^∗ (Eq. 2 )
Normalized
128
3,446
25,219
0.0795
1.000
256
6,892
21,773
0.0400
0.503
512
13,784
14,881
0.0111
0.140
768
20,676
7,989
0.0016
0.020
1024
27,568
1,097
0.0001
0.001
Table 2: Predicted floors (layer 12). Even dropping ∼25,000 features at dS=128 , the floor is small thanks to the power-law importance distribution.
dS
Params
Raw KL
Per-token KL
Normalized
128
18M
2,703
1.320
1.000
256
45M
2,064
1.008
0.764
512
127M
1,501
0.733
0.555
768
247M
1,335
0.652
0.494
1024
405M
1,200
0.586
0.444
Table 3: Distillation loss floors. Raw KL summed across 512 positions ×T2=4 . Per-token KL = raw /2048 . Normalized relative to dS=128 .
Figure 6: Distillation results. (a) Eval loss for all widths; narrower students plateau higher. (b) Floor decreases from 1.320 to 0.586 nats. Dotted line marks dS∗≈1065 .
Fit
Parameters
R2
Linear ( y=Cx )
C=8.97
−1.982
Affine ( y=Cx+B )
C=8.97 , B=0.623
0.993
Table 4: Calibration fits. The affine model succeeds because the floor has two components: geometric ( C⋅L^∗ , width-dependent) and architectural baseline ( B , width-independent).
Figure 7: Two-component decomposition. (a) Linear fit ( R2=−1.982 ) fails. (b) Affine fit ( R2=0.993 ): observed=8.97×predicted+0.623 . Baseline B=0.623 : architectural floor; slope C=8.97 : amplification through transformer layers.
dS
Observed
CL^∗
B
% Geom.
128
1.320
0.713
0.586
55.6%
256
1.008
0.359
0.586
41.9%
512
0.733
0.100
0.586
20.1%
768
0.652
0.014
0.586
10.1%
1024
0.586
0.001
0.586
0.1%
Table 5: Floor decomposition into geometric ( CL^∗ ) and baseline ( B ) components.
Concept
Teacher
dS=1024
dS=768
dS=128
Is question?
74.0
77.1
75.9
75.0
Is French?
99.4
99.2
99.5
98.9
Contains code?
84.0
83.8
83.6
86.9
About sports?
96.5
97.2
97.2
94.1
Legal text?
97.8
97.2
97.8
97.4
Medical text?
97.6
97.4
97.5
97.1
Table 6: Linear probe accuracy (%) at layer 12. All concepts stay far above chance (50%).
Figure 8: Linear probe results. (a) Heatmap: all concepts survive compression. (b) ±3 pp shifts reflect reallocation. (c) All above chance (50%).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
α
1−α
g(α)
Features/dim
0.00
1.000
1.0
1.0 ×
0.50
0.500
2.9
2.9 ×
0.80
0.200
3.1
3.1 ×
0.90
0.100
4.3
4.3 ×
0.95
0.050
6.7
6.7 ×
0.99
0.010
21.7
21.7 ×
Appendix
Table 7: Reference values for g(α) .
Figure 9: Effect of α on floor at dT=5 for n∈{10,20,40} . Higher sparsity (more features/dim) yields lower floors. Solid = actual; dashed = predicted.
Figure 10: Prediction error by sparsity. Refined (colored): median >90% accuracy; naive (gray): degrades at high α .
Figure 11: Accuracy heatmap (refined). Rows: α ; cols: dT ; panels: n . Green = >99% .
Figure 12: Zipf importance. Left: Linear scale. Right: Log-log confirms power law.
Figure 13: Normalized floor vs. dS/dS∗ . All configurations collapse: floor drops sharply at dS=dS∗ (dashed). This universal scaling confirms the phase transition.
Figure 14: Training curves at different widths for six configurations. Dashed = predicted floors. Rapid convergence confirms geometric origin.
dS
Layers
Heads
FFN dim
Params
Ratio
128
24
8
512
∼18 M
23 ×
256
24
16
1024
∼45 M
9 ×
512
24
16
2048
∼127 M
3.2 ×
768
24
16
3072
∼247 M
1.6 ×
1024
24
16
4096
∼405 M
1.0 ×
Appendix
Table 8: Student architectures. All share teacher’s depth (24), vocabulary (50,304), and positional encoding.
Figure 16: (a) Training loss for all widths. (b) Two seeds at dS=128 : floors differ by Δ=6.4 (0.24%), confirming the floor is deterministic.
Figure 17: Normalized predicted (SAE, dashed gray) vs. observed (distillation, solid red) floors. Both decrease monotonically; the widening gap reflects the constant baseline B dominating at larger widths (see Table 5 ).
Figure 18: Distillation summary. Top left: eval curves with floor estimates. Top right: per-token KL floor vs. width. Bottom left: normalized observed vs. predicted floors. Bottom right: seed variance at dS=128 .
Knowledge distillation is usually framed as a choice of what to match in the teacher - its logits, hidden features, or sample relations - which presupposes that the teacher's representation has absolute coordinates to match. It does not: a pretrained representation is identifiable only up to an orthogonal-and-isotropic-scaling equivalence class, so a student should learn the teacher's equivalence class, not its features. The organizing fact is that capability is the teacher's output function, a class invariant that factors through the quotient by the class action, so an objective recovers capability exactly when it is defined there. This makes absolute feature matching ill-posed, and admissible supervision a matter of targeting class invariants (Gram structure, CKA, principal subspaces) or aligning coordinates first, unifying feature matching, relational distillation, alignment, and grafting in one geometric account. We validate our framework on Qwen2.5 and Llama-3.1. A restoration study recovers a corrupted model's representation (CKA ~ 0.99) but not its capability, and an ablation isolates the cause: output-function (logit) matching drives capability, while matching hidden representations aligns geometry without restoring function. Recovery is confined to the corpus-covered region, and a graft study confirms that boundary overlap predicts transplant success but is necessary, not sufficient.
Sang Il Han
Korea University of Technology and Education · Biomonster Co.
This position paper argues that knowledge distillation must account for what it loses: student models should be judged not only by retained task scores, but by whether they preserve the teacher capabilities that make those scores reliable. This matters because distillation is increasingly used to turn large teacher models into deployable students, yet headline metrics can obscure losses in the capabilities that make teacher behavior reliable. Conceptually, we show that current evaluation often assumes retained task scores imply retained teacher capabilities. Reframing distillation as a lossy projection exposes this flaw: students may match selected teacher observables without preserving the capabilities that make them reliable. We then synthesize existing evidence into a taxonomy of off-metric distillation losses, showing that such losses are concrete, recurring, and measurable, yet often unaccounted for when studies report what students retain rather than what they lose. To make the position actionable, we propose scenario-specific preservation targets and a Distillation Loss Statement that reports what was preserved, what was lost, and why the remaining losses are acceptable. The goal is not lossless distillation, but accountable distillation.
Wenshuo Wang
School of Future Technology, South China University of Technology, China
Knowledge distillation compresses large language models into compact students that preserve aggregate metrics, yet the mechanistic consequences of depth-reduction distillation remain largely uncharacterized. This work applies linear CKA, logit lens analysis, attention pattern decomposition, sparse autoencoder feature statistics, and activation patching to compare GPT-2 and its six-layer student DistilGPT-2 across two canonical circuits: the Indirect Object Identification (IOI) circuit and the Greater-Than (GT) circuit. The results reveal a compression-collapse dichotomy: the IOI circuit suffers complete mechanistic collapse (DistilGPT-2: 35.2% accuracy vs. 94.8%, zero name-mover preservation), while the GT circuit survives with 87.7% task performance recovery through active remapping of the teacher's layer-9 comparison head into student layer~4 (mean Pearson r = 0.945). These findings demonstrate that circuit compressibility governs mechanistic survival under distillation, and that aggregate metrics are insufficient diagnostics for circuit integrity.
Nabeelah Faumi, Deshan Sumanathilaka
Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka