Function routing -- selecting the correct API call from a fixed catalog given a natural-language request -- is a deployment problem where small students are attractive but knowledge distillation gains are typically reported single-seed, at scales where seed variance is unknown. On a 740-instance healthcare API routing task with a 1.5B Qwen student and a 20B teacher, we compare eight KD variants against supervised cross-entropy, using three to six seeds for key configurations. We find: (i) per-seed standard deviation ranges from 2.8 to 48.7 percentage points, swallowing every claimed KD gain below five points; (ii) three of seven KD variants exhibit bimodal collapse, with at least one in three to five seeds falling below 55% accuracy while the others train normally, and a fourth showing elevated variance; (iii) collapse has distinct modes -- wrong-function selection for ce_kd and ce_paraphrase, and a previously undocumented output-truncation mode for reasoning_kd, where the model emits reasoning but terminates before producing a function name (0.9% accuracy); (iv) only progressive_kd and rank_kd avoid collapse across observed seeds, with sigma <= 3.9 pp; (v) a naive cross-split +3.78 pp gain from input enrichment reverses to -2.70 pp under controlled within-split multi-seed re-testing. Single-seed evaluation is therefore unable to detect central failure modes in small-model KD.
Figures & tables
Method
n
Mean ±σ
Collapse
ce_only
6
82.6±4.8
—
progressive_kd
4
82.5±2.8
none
rank_kd
4
80.9±3.9
none
full
4
78.2±6.6
none
ce_paraphrase
5
75.7±13.4
1/5
gated_kd
4
73.9±10.5
none
Table 1: Multi-seed V1 test accuracies (%), names-only prompts, ranked by mean. Collapse = number of seeds with test accuracy <55 %. Three of seven KD configurations exhibit collapse; a fourth ( gated_kd ) shows elevated variance without a sub-55% seed. Only progressive_kd and rank_kd are stable across all observed seeds. Per-seed values in Appendix A . † reasoning_kd seed 7 collapses to 0.9% via a qualitatively distinct output-truncation mode (Section 4.2 ), not wrong-function selection.
Figure 1: Two qualitatively distinct collapse modes. (a) Wrong-function collapse in ce_Knowledge Distillation seed 456 (31.5% accuracy): the model emits valid function names but consistently aliases related intents (each row: gold → predicted, two test instances per row). (b) Output-truncation collapse in reasoning_kd seed 7 (0.9% accuracy): 107 of 111 test predictions return None because the output terminates on a learned single-character EOS surrogate (Unicode codepoint U+622A, meaning “cut/intercept”) before the model commits to a function name.
Method
Acc
F1
PC
ce_only (baseline)
88.5
79.2
90.9
ce_Knowledge Distillation
85.9
72.9
72.7
ce_paraphrase
84.6
69.4
81.8
full
69.2
52.5
45.5
Table 2: V2 condition results, single seed ( n=78 test). Given the seed variance documented for V1, single-point V2 numbers should be read as samples from high-variance distributions rather than stable per-method effects.
Condition
Acc
F1
PC
V1 ce_only (names)
84.7
74.0
78.6
V2 ce_only (descriptions)
88.5
79.2
90.9
Δ (V2 − V1), naive
+3.78
+5.20
+12.34
Table 3: Naive cross-split comparison. The apparent +3.78 pp gain confounds prompt format with subset selection (V2 is a description-available subset of V1).
Arm
Acc (mean ±σ )
Δ
V1, names-only
82.9±6.3
—
V1, descriptions
80.2±10.9
−2.70
Table 4: Controlled within-split prompt-format comparison, 3 seeds per arm (per-seed values in Appendix A ). Descriptions show a −2.70 pp mean difference, with both arms exhibiting standard deviations larger than the difference. The naive +3.78 pp finding (Table 3 ) does not survive.
Figure 2: The cross-split reversal. (a) A naive single-seed comparison across two splits suggests a +3.78 pp gain from adding function descriptions to the student prompt. (b) The same intervention re-tested under a controlled within-split protocol with three seeds per arm reverses sign ( −2.70 pp). Dots are individual seed runs; bars and error bars show arm mean ± one standard deviation. Both arms’ standard deviations are larger than the difference between their means: the apparent effect was a subset-selection artifact, and the within-split data do not support a description effect of either sign at this scale.
Figure 3: Per-seed test accuracy (%) for the multi-seed V1 conditions in Table 1 . Dots are individual seed runs; vertical ticks show means. Sub-55% seeds (the collapse region, dashed line) appear in three of seven KD configurations; reasoning_kd ’s 0.9% point reflects the output-truncation mode (Section 4.2 ), not wrong-function selection. Only progressive_kd and rank_kd keep their full seed range above the baseline’s worst seed.
Figure 4: Mean test accuracy versus per-seed standard deviation across the eight multi-seed conditions. Dotted reference lines mark the baseline ( ce_only ). The stability-accuracy tradeoff is monotone in our data: methods that match the baseline mean ( progressive_kd , rank_kd ) sit at or below the baseline’s variance; methods with elevated variance ( gated_kd , ce_paraphrase ) lose accuracy; methods with high variance lose substantially more ( ce_Knowledge Distillation , reasoning_kd ). No method achieves both higher mean and lower variance than the baseline.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Method
Per-seed test accuracy (%)
ce_only
87.4, 85.6, 75.7, 85.6, 83.8, 77.5
progressive_kd
85.6, 81.1, 83.8, 79.3
rank_kd
83.8, 77.5, 77.5, 84.7
full
85.6, 75.7, 70.3, 81.1
ce_paraphrase
77.5, 80.2, 52.3, 85.6, 82.9
gated_kd
74.8, 64.9, 67.6, 88.3
Appendix
Table 5: Per-seed accuracies for all multi-seed conditions. Seed assignments use the base set 42, 123, 7; additional seeds (456, 789, 1729, 2024, 2025) were added incrementally.
Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined. On the BanSum Bangla summarization benchmark, we find that standard KD improves ROUGE-L by only +0.0003 over a cross-entropy baseline, and that approximately 51.3% of training samples are estimated to actively harm student validation loss under standard KD. We propose two complementary reliability-aware distillation methods. CHAD (Counterfactual Harm-Aware Distillation) measures per-sample KD usefulness via gradient alignment with the validation loss direction and trains a lightweight gate that generalizes this counterfactual judgment to the full training set. EWAD+CPDP combines token-level entropy-weighted adaptive distillation with a capacity-proportional geometric constraint from a second, vocabulary-incompatible teacher. On BanSum, both methods substantially outperform standard KD: CHAD by +0.0173 ROUGE-L and EWAD+CPDP by +0.0219 ROUGE-L, where standard KD itself improves ROUGE-L by only +0.0003; despite using only 60M parameters, both outperform a fine-tuned Qwen 2.5-3B model (50x larger). We further evaluate the stronger method, EWAD+CPDP, across 15 typologically diverse XL-Sum languages organised into three sets, beating the CE-only baseline on 10/15 languages; gains are most reliable where the two teachers contribute complementary signal, and weakest where they have saturated or jointly weak target-language coverage. We release code and trained models to support reproducibility and further research on selective distillation.
Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela +4
Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD). This recovery step largely decides the final quality, yet it is expensive. We present a practitioner's study of how to make distillation training efficient, organised around two systems contributions. First, we show that offline KD (caching the teacher's top-K logits once and training the student against the cache) matches online distillation at near-identical training loss while removing the teacher from memory, running about 29% faster per iteration, and reaching up to 41% higher throughput on a single H200 GPU. Second, we introduce a fused, chunked KL loss that never materialises the full vocabulary-sized logit tensor, making peak memory linear in the sequence length. This removes the memory spike that otherwise caps context length and lets us train at four times the context (32{,}768 tokens) on a single GPU. A separate output-head-only toy benchmark isolates the loss kernel and confirms its memory and iteration-rate scaling from 4K to 256K tokens. Together these make large-scale healing and hundreds of ablations affordable. We also report supporting ablations on loss design and sequence packing. We release our chunked-loss implementation: https://github.com/CompactifAI/Full-Chunked-KL-Loss.
Bakbergen Ryskulov, Iker García-Ferrero, David Montero +5
Knowledge distillation is an established technique for improving the capabilities of small, efficient student models by training them with the representations of larger, more capable teacher models. Much of the recent work in the distillation of large language models (LLMs) has focused on distilling abilities learned during post-training, such as instruction following, chain-of-thought reasoning, and tool usage. This has left a large research gap in general knowledge distillation for LLMs, which is essential for developing efficient and private systems suitable for deployment on edge devices. We take a first-principles approach, evaluating previous lessons from prior works and conducting new explorations to develop a distillation methodology suitable for modern LLMs. We present KDFP, a novel methodology for white-box general knowledge distillation in LLMs. We demonstrate that KDFP outperforms existing methods by 1.6% − 4.9% across 9 benchmarks while increasing training efficiency by up to 99.1% through ephemeral parameter reduction.
Ryan Swift, Konstantinos Psounis
Thomas Lord Department of Computer Science University of Southern California Los Angeles, CA 90007, USA