Knowledge distillation transfers deep learning representations from a teacher model to a student model, typically reducing model size or inference cost. However, knowledge distillation tooling has not kept pace with the breadth of research methods: existing toolkits typically support only a limited subset of techniques and use cases. We present SilverSpoon-KD, an open-source PyTorch toolkit that unifies different distillation paradigms and architectural relationships across workloads and applications behind a single API. Moreover, SilverSpoon-KD offers the first general-purpose blockwise knowledge distillation implementation with unique scalability features. We validate its versatility across vision, masked language model, and causal language models on tasks spanning homogeneous, heterogeneous, and quantization distillation across paradigms. We also evaluate SilverSpoon-KD by comparing its performance, including throughput and GPU memory usage, with existing toolkits. Despite extensive support for convenience, metric calculation, and tracking, we have found SilverSpoon-KD's performance to be on par with or better than existing toolkits.
Figures & tables
Figure 1: Knowledge distillation at a glance. KD spans diverse workloads, architectural relationships, and paradigms; existing tooling covers narrow slices of this combinatorial space, motivating a general-purpose toolkit. Panel (d) shows the four paradigms used throughout the paper: response-based (ResKD), feature-based (FeatKD), holistic (HKD), blockwise (BKD), and relation-based (RelKD).
Feature
SilverSpoon-KD
torchdistill
TextBrewer
torchtune
DistillKit
EasyDistill
Distillation paradigms
Response-based KD
✓
✓
✓
✓
✓
✓
Holistic feature KD
✓
✓
✓
✓
Blockwise feature KD
✓
Relation-based KD
✓
✓
Workload support
Table 1: Feature comparison of knowledge distillation toolkits. SilverSpoon-KD provides the broadest feature surface across the five compared toolkits, including the only general-purpose blockwise KD implementation and FSDP-sharded teacher placement.
Figure 4: Experimental coverage matrix. The eight distillation-quality experiments above the dashed line span vision, encoder LM, and decoder LM workloads under homogeneous, heterogeneous, and quantization-based settings, exercising all four paradigms; the toolkit-performance benchmark below the line tests SilverSpoon-KD against four peers under matched workloads. Pills: Res response-based, H holistic, B blockwise, Rel relation-based KD.
Figure 5: Per-experiment results summary. Colored bars are SilverSpoon-KD runs: single-stage bars use paradigm colors ( BKD , HKD , ResKD , RelKD ), and multi-stage chains use green , with the chain (e.g., BKD→HKD→LoRA) shown in the row label. Gray bars are reference points (not SilverSpoon-KD): teacher (dark), no-distillation student baseline (light), and external comparison from a competing toolkit or reference paper (medium). FT = full fine-tuning; LoRA = low-rank adapters, base frozen.
Table 2: Head-to-head toolkit benchmark (fixed batch size, single GPU unless noted). SilverSpoon-KD wins peak memory in all 5 settings and wall time in 3/5. Δ columns are SilverSpoon-KD relative to comparison; negative is better. Bold = winner per metric.
Teacher
Toolkit
Batch
Tput (tok/s)
Peak Mem (MB)
Qwen3-4B (8 GB)
DistillKit
4
15 184
37 967
TextBrewer
5
15 786
35 510
SilverSpoon-KD
6
14 700
37 333
Qwen3-8B (16 GB)
DistillKit
2
11 161
33 027
TextBrewer
3
11 514
34 119
SilverSpoon-KD
4
9 975
34 842
Table 3: Capability frontier as teacher size grows on 2 × A100 ( ∼40 GB per-GPU), Qwen3-0.6B student, seq=1024. SilverSpoon-KD is the only setup that scales to Qwen3-14B with batch > 1; DistillKit OOM s at 14B. Bold = best per metric per teacher.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Standard training from hard labels (no teacher). The student’s learnable parameters θ are updated via backpropagation to minimize the loss between its prediction and the ground-truth labels.
Figure 7: response-based KD: a teacher model produces soft-labels that the student learns to match. Only the student’s parameters θS are updated; the teacher is frozen.
Figure 8: holistic KD, a FeatKD strategy. The student performs a full end-to-end forward pass, and intermediate block outputs are compared with the corresponding teacher block outputs. Losses L1…N all contribute gradients to θ1S (since all subsequent outputs depend on Block 1), losses L2…N update θ2S , while θNS only receives gradients from LN .
Figure 9: blockwise KD, a FeatKD strategy. Each student block receives the same input as the corresponding teacher block, and loss is calculated by comparing the resulting outputs. Since each student block is only used to calculate its own output and is not used for subsequent student blocks, only the configuration of student learnable parameters θiS influences the loss Li , meaning backpropagation does not propagate to non-involved earlier student blocks.
Figure 10: Multi-stage distillation pipeline (BKD → HKD → LoRA, the headline GPT-2 Compression configuration of Appendix C.5 ). Each stage reuses the same frozen teacher T and the trained student from the prior stage, switching distiller class to apply a different paradigm; the trained student weights θ(k)S from stage k become the initialization for stage k+1 . Stage 1 (BKD) trains all student blocks under per-block teacher-forced losses with no cross-block gradient flow; Stage 2 (HKD) keeps Stage 1’s per-layer alignments but propagates gradients end-to-end through the full student chain; Stage 3 freezes the post-Stage-2 student weights, attaches LoRA adapters ΔiS on top, and trains only the adapters under a single response-based loss (no feature alignments). The optimizer state is reset at every stage boundary. Per-paradigm internals (forward chains, parameter-gradient annotations, alignment-loss formulae) are detailed in Figures 7 – 9 .
Method
Test Acc. (%)
Peak Mem (MB)
Steps/s
Time (h)
Baseline (no teacher)
82.69
692
29.09
0.37
BKD
72.49
9 991
32.60
0.33
BKD → Fine-tune
88.92
692
28.45
0.23
BKD → HKD
87.97
1 200
14.90
0.73
ResKD
84.35
720
20.16
0.54
BKD → ResKD
90.42
1 048
18.16
0.60
Appendix
Table 4: VGG Compression on CIFAR-10. BKD → ResKD wins accuracy at 90.42% (+3.16pp over teacher). Bold = best per column among non-teacher rows.
Method
Ours (%)
Park et al. (%)
Peak Mem (MB)
Steps/s
Time (h)
Teacher (ResNet-50)
77.81
—
4 212
7.89
2.75
Baseline (VGG-11-BN, no teacher)
65.16
71.26
750
32.20
0.67
ResKD ( T=16 )
73.65
74.26
1 190
13.20
1.64
HKD (MSE)
72.78
—
1 190
12.97
1.67
HKD ( relkd distance)
70.40
72.27
1 192
12.75
1.70
HKD ( relkd angle)
71.01
—
1 192
12.33
1.76
Appendix
Table 5: VGG Relational KD on CIFAR-100 (ResNet-50 → VGG-11-BN). ResKD leads our results at 73.65%; relation-based variants span 70.40–71.61%. All paradigms reachable through configuration-only changes. Bold = best per column among non-teacher rows (within Ours column).
Student
Method
Align Loss
MLM PPL
MNLI (%)
Peak Mem
Steps/s
Time
(Stage 1)
(Stage 1)
(Stage 2)
(MB)
(h)
T6
Baseline
—
14.08
77.29
6 588
6.45
0.86
T6
BKD
0.077
297.02
79.13
12 334
3.31
1.68
T6
HKD
0.408
40.57
80.33
13 774
2.45
2.27
T6
ResKD
1.661
36.51
79.01
9 746
1.94
2.86
T4-tiny
Baseline
—
24.42
66.92
5 458
12.97
0.43
Appendix
Table 6: BERT distillation (pre-training + downstream MNLI). HKD wins MNLI at both compression levels (T6: 80.33%, +3.04pp; T4-tiny: 70.44%, +3.52pp); the paradigm spread fans from ∼ 1.3pp at T6 to 5.2pp at T4-tiny, where standalone BKD turns harmful. Bold = best per column within each student-size group; Teacher row excluded from competition.
Run
MNLI-m (%)
MNLI-mm (%)
Peak Mem (MB)
Steps/s
Time (h)
Teacher (BERT-base)
83.59
83.76
—
—
—
T6 ( ≈59% of teacher)
Baseline (no distillation)
78.47
78.99
—
—
—
TextBrewer
80.09
80.48
—
—
—
SilverSpoon HKD → FT
83.47
83.93
6 154
3.57
7.16
T4-tiny ( ≈13% of teacher)
Appendix
Table 7: BERT Downstream MNLI: SilverSpoon-KD vs. TextBrewer. HKD → FT matches the BERT-base teacher within 0.12pp at T6 (83.47% vs. 83.59%) and beats TextBrewer by 3.38pp; at T4-tiny TextBrewer narrowly leads (81.20% vs. 80.87%). Reference-only rows (Teacher, baselines, TextBrewer) lack our system metrics. The T4-tiny HKD → FT row’s MNLI-mm cell is empty by design. 6 6 6 The bert_T4_tiny student uses a pruned architecture (hidden=312 with the teacher’s 768-dim Q/K/V projections retained); stock BertModel.from_pretrained silently reinitializes the mismatched layers, so post-hoc evaluation cannot recover MNLI-mm without reconstructing the architecture via silverspoon_kd.utils.prune_model . The matched value comes from the in-training trainer.evaluate() call. MNLI-mm typically tracks MNLI-m within ±0.5 pp on the BERT family. Bold = best per column within each student-size group; Teacher and TextBrewer rows excluded from competition.
Method
Wiki PPL
LAMBADA (%)
HellaSwag (%)
Peak Mem (MB)
Steps/s
Time (h)
Baseline (no teacher)
107.15
15.25
27.24
20 848
1.78
9.36
BKD
245,281.09
0.00
25.48
27 304
4.53
3.68
HKD
97.69
17.74
26.59
23 288
1.15
28.21
ResKD
133.99
6.48
26.74
25 598
1.52
21.32
BKD → HKD
109.56
16.90
27.12
23 210
1.95
16.68
BKD → HKD → LoRA
93.39
19.56
27.21
21 720
1.75
9.52
Appendix
Table 8: GPT-2 Compression. The three-stage BKD → HKD → LoRA pipeline rescues standalone-BKD divergence (PPL 245k → 93) and yields the best LAMBADA (19.56%, +28% over baseline). Benchmarks: LAMBADA [ 33 ] , HellaSwag [ 39 ] . Bold = best per column among non-teacher rows.
Method
Wiki PPL
LAMBADA (%)
HellaSwag (%)
Peak Mem (MB)
Steps/s
Time (h)
Baseline (no teacher)
117.78
16.34
27.48
22 484
2.57
6.48
HKD
246.44
0.87
26.29
33 978
7.29
2.29
ResKD
127.14
6.02
26.47
39 804
0.38
43.82
HKD → Fine-tune
118.49
15.06
27.67
17 706
2.60
6.41
ResKD → Fine-tune
117.28
14.96
27.65
22 484
2.60
6.41
Teacher (GPT-2 Small)
37.37
32.56
31.14
—
—
—
Appendix
Table 9: Cross-Architecture Distillation (GPT-2 → DeepSeek-V3, no shared architectural lineage). Standalone KD degrades LAMBADA below baseline; FT-recovery (HKD → FT, ResKD → FT) brings results back near baseline. Bold = best per column among non-teacher rows.
Method
Wiki PPL
LAMBADA (%)
HellaSwag (%)
Peak Mem (MB)
Steps/s
Time (h)
LoLCATs baseline (no KD)
53.49
30.55
31.00
25 408
2.31
7.22
BKD
235.37
30.89
31.26
25 428
2.19
7.61
HKD
320.55
30.91
31.03
42 620
1.01
16.44
BKD → HKD
285.53
30.86
31.34
42 622
0.64
12.94
ResKD
441.36
30.95
30.85
44 130
3.62
4.61
BKD → ResKD
267.22
30.95
31.29
43 066
0.56
37.56
Appendix
Table 10: Attention Linearization (softmax → LoLCATs hybrid attention). Only BKD → FT (PPL 52.01) beats the LoLCATs no-KD floor (53.49); standalone KDs push PPL above 230. Granular sub-module targeting makes this attention-only alignment accessible. Bold = best per column among non-teacher, non-LoLCATs-baseline rows.
Method
MMLU
ARC-E
ARC-C(N)
HS(N)
Wiki PPL
Peak Mem
Steps/s
Time(h)
bf16 pretrained (ceiling)
55.55
72.26
43.09
60.35
21.05
—
—
—
INT4 PTQ (floor)
40.71
58.46
32.42
54.26
37.20
—
—
—
INT4 QAT (no teacher)
41.50
58.96
32.85
54.53
36.15
29 144 MB
1.76
3.16
INT4 QAT + BKD
42.67
59.51
33.87
54.41
33.70
29 144 MB
1.76
3.16
INT4 QAT + HKD
48.26
65.57
36.69
54.66
29.98
35 162 MB
0.80
6.97
INT4 QAT + ResKD
43.75
63.85
35.07
53.94
31.11
32 436 MB
0.92
6.02
Appendix
Table 11: Qwen3-1.7B INT4/INT5/INT6 QAT with distillation. HKD dominates at INT4 (+6.76pp MMLU vs. QAT-alone, +9.42pp ARC-E); paradigms converge at INT5 and INT6 where PTQ already approaches the bf16 ceiling. Benchmarks: ARC-E/C [ 45 ] , HS = HellaSwag, (N) = normalized accuracy. Bold = best per column within each precision group; bf16 ceiling and PTQ floor rows excluded from competition.
School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education · Weixin AI