Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation

Dipto Sumit; Sakib Ul Haque; Farig Sadeque

cs.CLAug 27, 2026

Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation

Authors: Dipto Sumit, Sakib Ul Haque, Farig Sadeque

Organizations: Department of Computer Science and Engineering BRAC University, Bangladesh

Abstract

Function routing -- selecting the correct API call from a fixed catalog given a natural-language request -- is a deployment problem where small students are attractive but knowledge distillation gains are typically reported single-seed, at scales where seed variance is unknown. On a 740-instance healthcare API routing task with a 1.5B Qwen student and a 20B teacher, we compare eight KD variants against supervised cross-entropy, using three to six seeds for key configurations. We find: (i) per-seed standard deviation ranges from 2.8 to 48.7 percentage points, swallowing every claimed KD gain below five points; (ii) three of seven KD variants exhibit bimodal collapse, with at least one in three to five seeds falling below 55% accuracy while the others train normally, and a fourth showing elevated variance; (iii) collapse has distinct modes -- wrong-function selection for ce_kd and ce_paraphrase, and a previously undocumented output-truncation mode for reasoning_kd, where the model emits reasoning but terminates before producing a function name (0.9% accuracy); (iv) only progressive_kd and rank_kd avoid collapse across observed seeds, with sigma <= 3.9 pp; (v) a naive cross-split +3.78 pp gain from input enrichment reverses to -2.70 pp under controlled within-split multi-seed re-testing. Single-seed evaluation is therefore unable to detect central failure modes in small-model KD.

Figures & tables

Method	$n$	Mean $\pm\sigma$	Collapse
ce_only	6	$82.6\pm 4.8$	—
progressive_kd	4	$82.5\pm 2.8$	none
rank_kd	4	$80.9\pm 3.9$	none
full	4	$78.2\pm 6.6$	none
ce_paraphrase	5	$75.7\pm 13.4$	1/5
gated_kd	4	$73.9\pm 10.5$	none

Table 1: Multi-seed V1 test accuracies (%), names-only prompts, ranked by mean. Collapse = number of seeds with test accuracy

<55

%. Three of seven KD configurations exhibit collapse; a fourth ( gated_kd ) shows elevated variance without a sub-55% seed. Only progressive_kd and rank_kd are stable across all observed seeds. Per-seed values in Appendix A . † reasoning_kd seed 7 collapses to 0.9% via a qualitatively distinct output-truncation mode (Section 4.2 ), not wrong-function selection.

Figure 1: Two qualitatively distinct collapse modes. (a) Wrong-function collapse in ce_Knowledge Distillation seed 456 (31.5% accuracy): the model emits valid function names but consistently aliases related intents (each row: gold

\to

predicted, two test instances per row). (b) Output-truncation collapse in reasoning_kd seed 7 (0.9% accuracy): 107 of 111 test predictions return None because the output terminates on a learned single-character EOS surrogate (Unicode codepoint U+622A, meaning “cut/intercept”) before the model commits to a function name.

Method	Acc	F1	PC
ce_only (baseline)	88.5	79.2	90.9
ce_Knowledge Distillation	85.9	72.9	72.7
ce_paraphrase	84.6	69.4	81.8
full	69.2	52.5	45.5

Table 2: V2 condition results, single seed (

n=78

test). Given the seed variance documented for V1, single-point V2 numbers should be read as samples from high-variance distributions rather than stable per-method effects.

Condition	Acc	F1	PC
V1 ce_only (names)	84.7	74.0	78.6
V2 ce_only (descriptions)	88.5	79.2	90.9
$\Delta$ (V2 $-$ V1), naive	+3.78	+5.20	+12.34

Table 3: Naive cross-split comparison. The apparent +3.78 pp gain confounds prompt format with subset selection (V2 is a description-available subset of V1).

Arm	Acc (mean $\pm\sigma$ )	$\Delta$
V1, names-only	$82.9\pm 6.3$	—
V1, descriptions	$80.2\pm 10.9$	$-2.70$

Table 4: Controlled within-split prompt-format comparison, 3 seeds per arm (per-seed values in Appendix A ). Descriptions show a

-2.70

pp mean difference, with both arms exhibiting standard deviations larger than the difference. The naive

+3.78

pp finding (Table 3 ) does not survive.

Figure 2: The cross-split reversal. (a) A naive single-seed comparison across two splits suggests a

+3.78

pp gain from adding function descriptions to the student prompt. (b) The same intervention re-tested under a controlled within-split protocol with three seeds per arm reverses sign (

-2.70

pp). Dots are individual seed runs; bars and error bars show arm mean

\pm

one standard deviation. Both arms’ standard deviations are larger than the difference between their means: the apparent effect was a subset-selection artifact, and the within-split data do not support a description effect of either sign at this scale.

Figure 3: Per-seed test accuracy (%) for the multi-seed V1 conditions in Table 1 . Dots are individual seed runs; vertical ticks show means. Sub-55% seeds (the collapse region, dashed line) appear in three of seven KD configurations; reasoning_kd ’s 0.9% point reflects the output-truncation mode (Section 4.2 ), not wrong-function selection. Only progressive_kd and rank_kd keep their full seed range above the baseline’s worst seed.

Figure 4: Mean test accuracy versus per-seed standard deviation across the eight multi-seed conditions. Dotted reference lines mark the baseline ( ce_only ). The stability-accuracy tradeoff is monotone in our data: methods that match the baseline mean ( progressive_kd , rank_kd ) sit at or below the baseline’s variance; methods with elevated variance ( gated_kd , ce_paraphrase ) lose accuracy; methods with high variance lose substantially more ( ce_Knowledge Distillation , reasoning_kd ). No method achieves both higher mean and lower variance than the baseline.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Method	Per-seed test accuracy (%)
ce_only	87.4, 85.6, 75.7, 85.6, 83.8, 77.5
progressive_kd	85.6, 81.1, 83.8, 79.3
rank_kd	83.8, 77.5, 77.5, 84.7
full	85.6, 75.7, 70.3, 81.1
ce_paraphrase	77.5, 80.2, 52.3, 85.6, 82.9
gated_kd	74.8, 64.9, 67.6, 88.3

Appendix

Table 5: Per-seed accuracies for all multi-seed conditions. Seed assignments use the base set 42, 123, 7; additional seeds (456, 789, 1729, 2024, 2025) were added incrementally.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 821 papers

When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization
Jul 22, 2026Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela +4Selective Knowledge Distillation Text Summarization
BRAC University, Dhaka, Bangladesh
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
Aug 4, 2026Bakbergen Ryskulov, Iker García-Ferrero, David Montero +5Online Knowledge Distillation Efficient Language Model Training
Multiverse Computing
KDFP: A first-principles approach to knowledge distillation in large language models
Oct 7, 2026Ryan Swift, Konstantinos PsounisLanguage Model Distillation Knowledge Distillation
Thomas Lord Department of Computer Science University of Southern California Los Angeles, CA 90007, USA

Jul 22, 2026J/K move · Enter open · S save

When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela, Atia Haque Asha+3

BRAC University, Dhaka, Bangladesh

Selective Knowledge Distillation Text Summarization Multi-Teacher Knowledge Distillation Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined. On the BanSum Bangla summarization benchmark, we find that standard KD improves ROUGE-L by only +0.0003 over a cross-entropy baseline, and that approximately 51.3% of training samples are estimated to actively harm student validation loss under standard KD. We propose two complementary reliability-aware distillation methods. CHAD (Counterfactual Harm-Aware Distillation) measures per-sample KD usefulness via gradient alignment with the validation loss direction and trains a lightweight gate that generalizes this counterfactual judgment to the full training set. EWAD+CPDP combines token-level entropy-weighted adaptive distillation with a capacity-proportional geometric constraint from a second, vocabulary-incompatible teacher. On BanSum, both methods substantially outperform standard KD: CHAD by +0.0173 ROUGE-L and EWAD+CPDP by +0.0219 ROUGE-L, where standard KD itself improves ROUGE-L by only +0.0003; despite using only 60M parameters, both outperform a fine-tuned Qwen 2.5-3B model (50x larger). We further evaluate the stronger method, EWAD+CPDP, across 15 typologically diverse XL-Sum languages organised into three sets, beating the CE-only baseline on 10/15 languages; gains are most reliable where the two teachers contribute complementary signal, and weakest where they have saturated or jointly weak target-language coverage. We release code and trained models to support reproducibility and further research on selective distillation.

Abstract

Knowledge Distillation

Explore similar work

When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss

KDFP: A first-principles approach to knowledge distillation in large language models