cs.LGNov 17, 2026

SilverSpoon-KD: A General-Purpose Toolkit for Knowledge Distillation

Authors: Xaver R. Davey, David Broman

Organizations: KTH Royal Institute of Technology, Stockholm, Sweden

Abstract

Knowledge distillation transfers deep learning representations from a teacher model to a student model, typically reducing model size or inference cost. However, knowledge distillation tooling has not kept pace with the breadth of research methods: existing toolkits typically support only a limited subset of techniques and use cases. We present SilverSpoon-KD, an open-source PyTorch toolkit that unifies different distillation paradigms and architectural relationships across workloads and applications behind a single API. Moreover, SilverSpoon-KD offers the first general-purpose blockwise knowledge distillation implementation with unique scalability features. We validate its versatility across vision, masked language model, and causal language models on tasks spanning homogeneous, heterogeneous, and quantization distillation across paradigms. We also evaluate SilverSpoon-KD by comparing its performance, including throughput and GPU memory usage, with existing toolkits. Despite extensive support for convenience, metric calculation, and tracking, we have found SilverSpoon-KD's performance to be on par with or better than existing toolkits.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. KDFP: A first-principles approach to knowledge distillation in large language models

    Oct 7, 2026Ryan Swift, Konstantinos PsounisLanguage Model DistillationKnowledge Distillation

  2. KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models

    Mar 2, 2026Songming Zhang, Xue Zhang, Tong Zhang +3Cross-Tokenizer Knowledge DistillationLanguage Model Distillation