Geometric Limits of Knowledge Distillation: A Minimum-Width Theorem via Superposition Theory

Nilesh Sarkar; Dawar Jyoti Deka

cs.LGApr 5, 2026

Geometric Limits of Knowledge Distillation: A Minimum-Width Theorem via Superposition Theory

Authors: Nilesh Sarkar, Dawar Jyoti Deka

Organizations: Erd˝os AI Labs

Abstract

Knowledge distillation compresses large teachers into smaller students, but performance saturates at a loss floor that persists across training methods and objectives. We argue this floor is geometric: neural networks represent far more features than dimensions through superposition, and a student of width $d_S$ can encode at most $d_S \cdot g(α)$ features, where $g(α) = 1/((1-α)\ln\frac{1}{1-α})$ is a sparsity-dependent capacity function. Features beyond this budget are permanently lost, yielding an importance-weighted loss floor. We validate on a toy model (48 configurations, median accuracy >93%) and on Pythia-410M, where sparse autoencoders measure $F \approx 28{,}700$ features at $α\approx 0.992$ (critical width $d_S^* \approx 1{,}065$ ). Distillation into five student widths confirms the predicted monotonic floor ordering. The observed floor decomposes into a geometric component and a width-independent architectural baseline ( $R^2 = 0.993$ ). Linear probing shows coarse concepts survive even 88% feature loss, revealing the floor arises from aggregate loss of fine-grained features in the importance distribution's long tail. Our results connect representation geometry to distillation limits and provide a practical tool for predicting distillation performance from SAE measurements alone.

Figures & tables

Figure 1: Capacity function and critical width. Left: $g(\alpha)$ grows exponentially with sparsity; colored dots mark our toy model sparsity levels. Right: Critical width $d_{\mathrm{S}}^{*}=F/g(\alpha)$ shrinks as sparsity increases, since sparser features need fewer dimensions.

Figure 2: Loss floor vs. student width across 48 configurations (rows: $n$ ; columns: $\alpha$ ). Solid = actual (mean $\pm$ std, 20 seeds); dashed = formula (Eq. 2 ); dotted = $d_{\mathrm{S}}^{*}$ . The formula captures both magnitude and shape.

Figure 3: Predicted vs. actual floor (log-log, 140 points). Left: Refined formula with $g(\alpha)$ (Pearson $r=0.93$ , MAPE $=93.9\%$ ). Right: Naive formula assuming one feature/dim ( $R^{2}=-0.04$ ). Color = sparsity.

Panel of Figure 4 — Figure 4: SAE training convergence. Layer 8 (blue) encodes a denser feature set; deeper layers 12 (orange) and 16 (green) show sparser, more selective representations with more feature death.

Layer	Alive $F$	Avg $L_{0}$	$\alpha$	$g(\alpha)$	$d_{\mathrm{S}}^{*}$
8	31,006	234.9	0.9924	27.04	1,147
12	28,665	218.3	0.9924	26.92	1,065
16	29,169	249.0	0.9915	24.60	1,186

Table 1: SAE measurements on Pythia-410M. All layers have

d_{\mathrm{S}}^{*}>1024

, confirming the teacher itself is in superposition.

Panel of Figure 5 — Figure 5: (a) Feature importance follows a power law: the top ${\sim}20$ features dominate, with a cliff near rank ${\sim}3{,}000$ where thousands reach ${\sim}10^{-7}$ . This heavy tail is why compression works. (b) Predicted floor vs. width at layers 8, 12, 16. All layers agree ( $d_{\mathrm{S}}^{*}\in[1065,1186]$ ), converging near zero at $d_{\mathrm{S}}=1024$ .

$d_{\mathrm{S}}$	$F_{\mathrm{S}}$	Dropped	$\hat{L}^{*}$ (Eq. 2 )	Normalized
128	3,446	25,219	0.0795	1.000
256	6,892	21,773	0.0400	0.503
512	13,784	14,881	0.0111	0.140
768	20,676	7,989	0.0016	0.020
1024	27,568	1,097	0.0001	0.001

Table 2: Predicted floors (layer 12). Even dropping

{\sim}25{,}000

features at

d_{\mathrm{S}}=128

, the floor is small thanks to the power-law importance distribution.

$d_{\mathrm{S}}$	Params	Raw KL	Per-token KL	Normalized
128	18M	2,703	1.320	1.000
256	45M	2,064	1.008	0.764
512	127M	1,501	0.733	0.555
768	247M	1,335	0.652	0.494
1024	405M	1,200	0.586	0.444

Table 3: Distillation loss floors. Raw KL summed across 512 positions

\times\,T^{2}=4

. Per-token KL

=

raw

/2048

. Normalized relative to

d_{\mathrm{S}}=128

Panel of Figure 6 — Figure 6: Distillation results. (a) Eval loss for all widths; narrower students plateau higher. (b) Floor decreases from 1.320 to 0.586 nats. Dotted line marks $d_{\mathrm{S}}^{*}\approx 1065$ .

Fit	Parameters	$R^{2}$
Linear ( $y=Cx$ )	$C=8.97$	$-1.982$
Affine ( $y=Cx+B$ )	$C=8.97$ , $B=0.623$	$0.993$

Table 4: Calibration fits. The affine model succeeds because the floor has two components: geometric (

C\cdot\hat{L}^{*}

, width-dependent) and architectural baseline (

B

, width-independent).

Figure 7: Two-component decomposition. (a) Linear fit ( $R^{2}=-1.982$ ) fails. (b) Affine fit ( $R^{2}=0.993$ ): $\text{observed}=8.97\times\text{predicted}+0.623$ . Baseline $B=0.623$ : architectural floor; slope $C=8.97$ : amplification through transformer layers.

$d_{\mathrm{S}}$	Observed	$C\hat{L}^{*}$	$B$	% Geom.
128	1.320	0.713	0.586	55.6%
256	1.008	0.359	0.586	41.9%
512	0.733	0.100	0.586	20.1%
768	0.652	0.014	0.586	10.1%
1024	0.586	0.001	0.586	0.1%

Table 5: Floor decomposition into geometric (

C\hat{L}^{*}

) and baseline (

B

) components.

Concept	Teacher	$d_{\mathrm{S}}{=}1024$	$d_{\mathrm{S}}{=}768$	$d_{\mathrm{S}}{=}128$
Is question?	74.0	77.1	75.9	75.0
Is French?	99.4	99.2	99.5	98.9
Contains code?	84.0	83.8	83.6	86.9
About sports?	96.5	97.2	97.2	94.1
Legal text?	97.8	97.2	97.8	97.4
Medical text?	97.6	97.4	97.5	97.1

Table 6: Linear probe accuracy (%) at layer 12. All concepts stay far above chance (50%).

Panel of Figure 8 — Figure 8: Linear probe results. (a) Heatmap: all concepts survive compression. (b) $\pm 3$ pp shifts reflect reallocation. (c) All above chance (50%).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

$\alpha$	$1-\alpha$	$g(\alpha)$	Features/dim
0.00	1.000	1.0	1.0 $\times$
0.50	0.500	2.9	2.9 $\times$
0.80	0.200	3.1	3.1 $\times$
0.90	0.100	4.3	4.3 $\times$
0.95	0.050	6.7	6.7 $\times$
0.99	0.010	21.7	21.7 $\times$

Appendix

Table 7: Reference values for

g(\alpha)

Figure 9: Effect of $\alpha$ on floor at $d_{\mathrm{T}}=5$ for $n\in\{10,20,40\}$ . Higher sparsity (more features/dim) yields lower floors. Solid = actual; dashed = predicted.

Figure 10: Prediction error by sparsity. Refined (colored): median $>90\%$ accuracy; naive (gray): degrades at high $\alpha$ .

Figure 11: Accuracy heatmap (refined). Rows: $\alpha$ ; cols: $d_{\mathrm{T}}$ ; panels: $n$ . Green = $>99\%$ .

Figure 12: Zipf importance. Left: Linear scale. Right: Log-log confirms power law.

Figure 13: Normalized floor vs. $d_{\mathrm{S}}/d_{\mathrm{S}}^{*}$ . All configurations collapse: floor drops sharply at $d_{\mathrm{S}}=d_{\mathrm{S}}^{*}$ (dashed). This universal scaling confirms the phase transition.

Figure 14: Training curves at different widths for six configurations. Dashed = predicted floors. Rapid convergence confirms geometric origin.

$d_{\mathrm{S}}$	Layers	Heads	FFN dim	Params	Ratio
128	24	8	512	${\sim}18$ M	23 $\times$
256	24	16	1024	${\sim}45$ M	9 $\times$
512	24	16	2048	${\sim}127$ M	3.2 $\times$
768	24	16	3072	${\sim}247$ M	1.6 $\times$
1024	24	16	4096	${\sim}405$ M	1.0 $\times$

Appendix

Table 8: Student architectures. All share teacher’s depth (24), vocabulary (50,304), and positional encoding.

Panel of Figure 15 — Figure 15: Per-layer SAE curves: layers 8 (top), 12 (mid), 16 (bottom). Layer 8 has lower recon loss, higher $L_{0}$ , near-zero feature death.

Panel of Figure 16 — Figure 16: (a) Training loss for all widths. (b) Two seeds at $d_{\mathrm{S}}=128$ : floors differ by $\Delta=6.4$ (0.24%), confirming the floor is deterministic.

Figure 17: Normalized predicted (SAE, dashed gray) vs. observed (distillation, solid red) floors. Both decrease monotonically; the widening gap reflects the constant baseline $B$ dominating at larger widths (see Table 5 ).

Figure 18: Distillation summary. Top left: eval curves with floor estimates. Top right: per-token KL floor vs. width. Bottom left: normalized observed vs. predicted floors. Bottom right: seed variance at $d_{\mathrm{S}}=128$ .

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 819 papers

Teacher Supervision over Representation Equivalence Classes
Jul 3, 2026Sang Il HanRepresentation Identifiability Language Model Distillation
Korea University of Technology and Education · Biomonster Co.
Knowledge Distillation Must Account for What It Loses
Apr 28, 2026Wenshuo WangTeacher-Student Learning Knowledge Distillation
School of Future Technology, South China University of Technology, China
What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2
Oct 27, 2026Nabeelah Faumi, Deshan SumanathilakaLanguage Model Distillation Knowledge Distillation
Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka

Jul 3, 2026J/K move · Enter open · S save

Teacher Supervision over Representation Equivalence Classes

Sang Il Han

Korea University of Technology and Education · Biomonster Co.

Representation Identifiability Language Model Distillation Representation Geometry in Language Models Knowledge Distillation Representation Alignment

Open paper AskPDF arXiv

Knowledge distillation is usually framed as a choice of what to match in the teacher - its logits, hidden features, or sample relations - which presupposes that the teacher's representation has absolute coordinates to match. It does not: a pretrained representation is identifiable only up to an orthogonal-and-isotropic-scaling equivalence class, so a student should learn the teacher's equivalence class, not its features. The organizing fact is that capability is the teacher's output function, a class invariant that factors through the quotient by the class action, so an objective recovers capability exactly when it is defined there. This makes absolute feature matching ill-posed, and admissible supervision a matter of targeting class invariants (Gram structure, CKA, principal subspaces) or aligning coordinates first, unifying feature matching, relational distillation, alignment, and grafting in one geometric account. We validate our framework on Qwen2.5 and Llama-3.1. A restoration study recovers a corrupted model's representation (CKA ~ 0.99) but not its capability, and an ablation isolates the cause: output-function (logit) matching drives capability, while matching hidden representations aligns geometry without restoring function. Recovery is confined to the corpus-covered region, and a graft study confirms that boundary overlap predicts transplant success but is necessary, not sufficient.

Abstract

Knowledge Distillation

Explore similar work

Teacher Supervision over Representation Equivalence Classes

Knowledge Distillation Must Account for What It Loses

What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2