Knowledge distillation transfers deep learning representations from a teacher model to a student model, typically reducing model size or inference cost. However, knowledge distillation tooling has not kept pace with the breadth of research methods: existing toolkits typically support only a limited subset of techniques and use cases. We present SilverSpoon-KD, an open-source PyTorch toolkit that unifies different distillation paradigms and architectural relationships across workloads and applications behind a single API. Moreover, SilverSpoon-KD offers the first general-purpose blockwise knowledge distillation implementation with unique scalability features. We validate its versatility across vision, masked language model, and causal language models on tasks spanning homogeneous, heterogeneous, and quantization distillation across paradigms. We also evaluate SilverSpoon-KD by comparing its performance, including throughput and GPU memory usage, with existing toolkits. Despite extensive support for convenience, metric calculation, and tracking, we have found SilverSpoon-KD's performance to be on par with or better than existing toolkits.
Figures & tables
Figure 1: Knowledge distillation at a glance. KD spans diverse workloads, architectural relationships, and paradigms; existing tooling covers narrow slices of this combinatorial space, motivating a general-purpose toolkit. Panel (d) shows the four paradigms used throughout the paper: response-based (ResKD), feature-based (FeatKD), holistic (HKD), blockwise (BKD), and relation-based (RelKD).
Feature
SilverSpoon-KD
torchdistill
TextBrewer
torchtune
DistillKit
EasyDistill
Distillation paradigms
Response-based KD
✓
✓
✓
✓
✓
✓
Holistic feature KD
✓
✓
✓
✓
Blockwise feature KD
✓
Relation-based KD
✓
✓
Workload support
Table 1: Feature comparison of knowledge distillation toolkits. SilverSpoon-KD provides the broadest feature surface across the five compared toolkits, including the only general-purpose blockwise KD implementation and FSDP-sharded teacher placement.
Figure 4: Experimental coverage matrix. The eight distillation-quality experiments above the dashed line span vision, encoder LM, and decoder LM workloads under homogeneous, heterogeneous, and quantization-based settings, exercising all four paradigms; the toolkit-performance benchmark below the line tests SilverSpoon-KD against four peers under matched workloads. Pills: Res response-based, H holistic, B blockwise, Rel relation-based KD.
Figure 5: Per-experiment results summary. Colored bars are SilverSpoon-KD runs: single-stage bars use paradigm colors ( BKD , HKD , ResKD , RelKD ), and multi-stage chains use green , with the chain (e.g., BKD→HKD→LoRA) shown in the row label. Gray bars are reference points (not SilverSpoon-KD): teacher (dark), no-distillation student baseline (light), and external comparison from a competing toolkit or reference paper (medium). FT = full fine-tuning; LoRA = low-rank adapters, base frozen.
Table 2: Head-to-head toolkit benchmark (fixed batch size, single GPU unless noted). SilverSpoon-KD wins peak memory in all 5 settings and wall time in 3/5. Δ columns are SilverSpoon-KD relative to comparison; negative is better. Bold = winner per metric.
Teacher
Toolkit
Batch
Tput (tok/s)
Peak Mem (MB)
Qwen3-4B (8 GB)
DistillKit
4
15 184
37 967
TextBrewer
5
15 786
35 510
SilverSpoon-KD
6
14 700
37 333
Qwen3-8B (16 GB)
DistillKit
2
11 161
33 027
TextBrewer
3
11 514
34 119
SilverSpoon-KD
4
9 975
34 842
Table 3: Capability frontier as teacher size grows on 2 × A100 ( ∼40 GB per-GPU), Qwen3-0.6B student, seq=1024. SilverSpoon-KD is the only setup that scales to Qwen3-14B with batch > 1; DistillKit OOM s at 14B. Bold = best per metric per teacher.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Standard training from hard labels (no teacher). The student’s learnable parameters θ are updated via backpropagation to minimize the loss between its prediction and the ground-truth labels.
Figure 7: response-based KD: a teacher model produces soft-labels that the student learns to match. Only the student’s parameters θS are updated; the teacher is frozen.
Figure 8: holistic KD, a FeatKD strategy. The student performs a full end-to-end forward pass, and intermediate block outputs are compared with the corresponding teacher block outputs. Losses L1…N all contribute gradients to θ1S (since all subsequent outputs depend on Block 1), losses L2…N update θ2S , while θNS only receives gradients from LN .
Figure 9: blockwise KD, a FeatKD strategy. Each student block receives the same input as the corresponding teacher block, and loss is calculated by comparing the resulting outputs. Since each student block is only used to calculate its own output and is not used for subsequent student blocks, only the configuration of student learnable parameters θiS influences the loss Li , meaning backpropagation does not propagate to non-involved earlier student blocks.
Figure 10: Multi-stage distillation pipeline (BKD → HKD → LoRA, the headline GPT-2 Compression configuration of Appendix C.5 ). Each stage reuses the same frozen teacher T and the trained student from the prior stage, switching distiller class to apply a different paradigm; the trained student weights θ(k)S from stage k become the initialization for stage k+1 . Stage 1 (BKD) trains all student blocks under per-block teacher-forced losses with no cross-block gradient flow; Stage 2 (HKD) keeps Stage 1’s per-layer alignments but propagates gradients end-to-end through the full student chain; Stage 3 freezes the post-Stage-2 student weights, attaches LoRA adapters ΔiS on top, and trains only the adapters under a single response-based loss (no feature alignments). The optimizer state is reset at every stage boundary. Per-paradigm internals (forward chains, parameter-gradient annotations, alignment-loss formulae) are detailed in Figures 7 – 9 .
Method
Test Acc. (%)
Peak Mem (MB)
Steps/s
Time (h)
Baseline (no teacher)
82.69
692
29.09
0.37
BKD
72.49
9 991
32.60
0.33
BKD → Fine-tune
88.92
692
28.45
0.23
BKD → HKD
87.97
1 200
14.90
0.73
ResKD
84.35
720
20.16
0.54
BKD → ResKD
90.42
1 048
18.16
0.60
Appendix
Table 4: VGG Compression on CIFAR-10. BKD → ResKD wins accuracy at 90.42% (+3.16pp over teacher). Bold = best per column among non-teacher rows.
Method
Ours (%)
Park et al. (%)
Peak Mem (MB)
Steps/s
Time (h)
Teacher (ResNet-50)
77.81
—
4 212
7.89
2.75
Baseline (VGG-11-BN, no teacher)
65.16
71.26
750
32.20
0.67
ResKD ( T=16 )
73.65
74.26
1 190
13.20
1.64
HKD (MSE)
72.78
—
1 190
12.97
1.67
HKD ( relkd distance)
70.40
72.27
1 192
12.75
1.70
HKD ( relkd angle)
71.01
—
1 192
12.33
1.76
Appendix
Table 5: VGG Relational KD on CIFAR-100 (ResNet-50 → VGG-11-BN). ResKD leads our results at 73.65%; relation-based variants span 70.40–71.61%. All paradigms reachable through configuration-only changes. Bold = best per column among non-teacher rows (within Ours column).
Student
Method
Align Loss
MLM PPL
MNLI (%)
Peak Mem
Steps/s
Time
(Stage 1)
(Stage 1)
(Stage 2)
(MB)
(h)
T6
Baseline
—
14.08
77.29
6 588
6.45
0.86
T6
BKD
0.077
297.02
79.13
12 334
3.31
1.68
T6
HKD
0.408
40.57
80.33
13 774
2.45
2.27
T6
ResKD
1.661
36.51
79.01
9 746
1.94
2.86
T4-tiny
Baseline
—
24.42
66.92
5 458
12.97
0.43
Appendix
Table 6: BERT distillation (pre-training + downstream MNLI). HKD wins MNLI at both compression levels (T6: 80.33%, +3.04pp; T4-tiny: 70.44%, +3.52pp); the paradigm spread fans from ∼ 1.3pp at T6 to 5.2pp at T4-tiny, where standalone BKD turns harmful. Bold = best per column within each student-size group; Teacher row excluded from competition.
Run
MNLI-m (%)
MNLI-mm (%)
Peak Mem (MB)
Steps/s
Time (h)
Teacher (BERT-base)
83.59
83.76
—
—
—
T6 ( ≈59% of teacher)
Baseline (no distillation)
78.47
78.99
—
—
—
TextBrewer
80.09
80.48
—
—
—
SilverSpoon HKD → FT
83.47
83.93
6 154
3.57
7.16
T4-tiny ( ≈13% of teacher)
Appendix
Table 7: BERT Downstream MNLI: SilverSpoon-KD vs. TextBrewer. HKD → FT matches the BERT-base teacher within 0.12pp at T6 (83.47% vs. 83.59%) and beats TextBrewer by 3.38pp; at T4-tiny TextBrewer narrowly leads (81.20% vs. 80.87%). Reference-only rows (Teacher, baselines, TextBrewer) lack our system metrics. The T4-tiny HKD → FT row’s MNLI-mm cell is empty by design. 6 6 6 The bert_T4_tiny student uses a pruned architecture (hidden=312 with the teacher’s 768-dim Q/K/V projections retained); stock BertModel.from_pretrained silently reinitializes the mismatched layers, so post-hoc evaluation cannot recover MNLI-mm without reconstructing the architecture via silverspoon_kd.utils.prune_model . The matched value comes from the in-training trainer.evaluate() call. MNLI-mm typically tracks MNLI-m within ±0.5 pp on the BERT family. Bold = best per column within each student-size group; Teacher and TextBrewer rows excluded from competition.
Method
Wiki PPL
LAMBADA (%)
HellaSwag (%)
Peak Mem (MB)
Steps/s
Time (h)
Baseline (no teacher)
107.15
15.25
27.24
20 848
1.78
9.36
BKD
245,281.09
0.00
25.48
27 304
4.53
3.68
HKD
97.69
17.74
26.59
23 288
1.15
28.21
ResKD
133.99
6.48
26.74
25 598
1.52
21.32
BKD → HKD
109.56
16.90
27.12
23 210
1.95
16.68
BKD → HKD → LoRA
93.39
19.56
27.21
21 720
1.75
9.52
Appendix
Table 8: GPT-2 Compression. The three-stage BKD → HKD → LoRA pipeline rescues standalone-BKD divergence (PPL 245k → 93) and yields the best LAMBADA (19.56%, +28% over baseline). Benchmarks: LAMBADA [ 33 ] , HellaSwag [ 39 ] . Bold = best per column among non-teacher rows.
Method
Wiki PPL
LAMBADA (%)
HellaSwag (%)
Peak Mem (MB)
Steps/s
Time (h)
Baseline (no teacher)
117.78
16.34
27.48
22 484
2.57
6.48
HKD
246.44
0.87
26.29
33 978
7.29
2.29
ResKD
127.14
6.02
26.47
39 804
0.38
43.82
HKD → Fine-tune
118.49
15.06
27.67
17 706
2.60
6.41
ResKD → Fine-tune
117.28
14.96
27.65
22 484
2.60
6.41
Teacher (GPT-2 Small)
37.37
32.56
31.14
—
—
—
Appendix
Table 9: Cross-Architecture Distillation (GPT-2 → DeepSeek-V3, no shared architectural lineage). Standalone KD degrades LAMBADA below baseline; FT-recovery (HKD → FT, ResKD → FT) brings results back near baseline. Bold = best per column among non-teacher rows.
Method
Wiki PPL
LAMBADA (%)
HellaSwag (%)
Peak Mem (MB)
Steps/s
Time (h)
LoLCATs baseline (no KD)
53.49
30.55
31.00
25 408
2.31
7.22
BKD
235.37
30.89
31.26
25 428
2.19
7.61
HKD
320.55
30.91
31.03
42 620
1.01
16.44
BKD → HKD
285.53
30.86
31.34
42 622
0.64
12.94
ResKD
441.36
30.95
30.85
44 130
3.62
4.61
BKD → ResKD
267.22
30.95
31.29
43 066
0.56
37.56
Appendix
Table 10: Attention Linearization (softmax → LoLCATs hybrid attention). Only BKD → FT (PPL 52.01) beats the LoLCATs no-KD floor (53.49); standalone KDs push PPL above 230. Granular sub-module targeting makes this attention-only alignment accessible. Bold = best per column among non-teacher, non-LoLCATs-baseline rows.
Method
MMLU
ARC-E
ARC-C(N)
HS(N)
Wiki PPL
Peak Mem
Steps/s
Time(h)
bf16 pretrained (ceiling)
55.55
72.26
43.09
60.35
21.05
—
—
—
INT4 PTQ (floor)
40.71
58.46
32.42
54.26
37.20
—
—
—
INT4 QAT (no teacher)
41.50
58.96
32.85
54.53
36.15
29 144 MB
1.76
3.16
INT4 QAT + BKD
42.67
59.51
33.87
54.41
33.70
29 144 MB
1.76
3.16
INT4 QAT + HKD
48.26
65.57
36.69
54.66
29.98
35 162 MB
0.80
6.97
INT4 QAT + ResKD
43.75
63.85
35.07
53.94
31.11
32 436 MB
0.92
6.02
Appendix
Table 11: Qwen3-1.7B INT4/INT5/INT6 QAT with distillation. HKD dominates at INT4 (+6.76pp MMLU vs. QAT-alone, +9.42pp ARC-E); paradigms converge at INT5 and INT6 where PTQ already approaches the bf16 ceiling. Benchmarks: ARC-E/C [ 45 ] , HS = HellaSwag, (N) = normalized accuracy. Bold = best per column within each precision group; bf16 ceiling and PTQ floor rows excluded from competition.
Knowledge distillation is an established technique for improving the capabilities of small, efficient student models by training them with the representations of larger, more capable teacher models. Much of the recent work in the distillation of large language models (LLMs) has focused on distilling abilities learned during post-training, such as instruction following, chain-of-thought reasoning, and tool usage. This has left a large research gap in general knowledge distillation for LLMs, which is essential for developing efficient and private systems suitable for deployment on edge devices. We take a first-principles approach, evaluating previous lessons from prior works and conducting new explorations to develop a distillation methodology suitable for modern LLMs. We present KDFP, a novel methodology for white-box general knowledge distillation in LLMs. We demonstrate that KDFP outperforms existing methods by 1.6% − 4.9% across 9 benchmarks while increasing training efficiency by up to 99.1% through ephemeral parameter reduction.
Ryan Swift, Konstantinos Psounis
Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, CA 90007, USA
Knowledge distillation (KD) is widely used to compress and post-train large language models (LLMs), yet many existing frameworks execute teacher inference with the same training-oriented backend as student optimization, leading to suboptimal efficiency. In this paper, we propose KDFlow, a novel framework for LLM distillation that features a decoupled architecture and employs SGLang for teacher inference. KDFlow combines SGLang for teacher inference with PyTorch FSDP2 for student optimization, allowing each model to run on a backend tailored to its workload. To enable efficient full-vocabulary distillation in this decoupled architecture, KDFlow transfers the teacher's final hidden states via Ray's object store and recomputes teacher logits on each student worker using a frozen copy of the teacher's output head. Furthermore, our framework supports both off-policy and on-policy distillation and incorporates cross-tokenizer algorithms through highly extensible and user-friendly APIs. Experiments show that KDFlow achieves a 1.44× to 6.36× speedup over MS-SWIFT in off-policy distillation and a 1.43× to 1.75× speedup over verl in on-policy distillation. KDFlow further scales to 64 GPUs, achieving 3.68× and 2.52× strong-scaling speedups in two representative model configurations. The code and documentation are publicly available.
Songming Zhang, Xue Zhang, Tong Zhang +3
School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education · Weixin AI
Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD). This recovery step largely decides the final quality, yet it is expensive. We present a practitioner's study of how to make distillation training efficient, organised around two systems contributions. First, we show that offline KD (caching the teacher's top-K logits once and training the student against the cache) matches online distillation at near-identical training loss while removing the teacher from memory, running about 29% faster per iteration, and reaching up to 41% higher throughput on a single H200 GPU. Second, we introduce a fused, chunked KL loss that never materialises the full vocabulary-sized logit tensor, making peak memory linear in the sequence length. This removes the memory spike that otherwise caps context length and lets us train at four times the context (32{,}768 tokens) on a single GPU. A separate output-head-only toy benchmark isolates the loss kernel and confirms its memory and iteration-rate scaling from 4K to 256K tokens. Together these make large-scale healing and hundreds of ablations affordable. We also report supporting ablations on loss design and sequence packing. We release our chunked-loss implementation: https://github.com/CompactifAI/Full-Chunked-KL-Loss.
Bakbergen Ryskulov, Iker García-Ferrero, David Montero +5