Language Model Distillation

Overview

Language model distillation studies how to transfer a teacher model’s knowledge and behavior into a student that is cheaper to train or deploy, while retaining useful capabilities. Common approaches train on teacher-generated answers or reasoning traces, match teacher and student token probabilities, align hidden representations, or compress model weights; newer on-policy methods also use feedback on student-generated text. Researchers evaluate these methods on reasoning, coding, instruction-following, multilingual, and agent tasks, often comparing capability retention against training and inference costs. A central challenge is that teacher guidance can be noisy or poorly matched to a student’s abilities, so improving transfer efficiency must be balanced against stability, generalization, diversity, and loss of existing capabilities.

Latest papers

CardsList

562 papers67 papers in the last four weeks, down 50% on the four weeks before. 0.5% of all new papers.

  1. DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation

    Feb 25, 2026Duc Trung Vu, Pham Khanh Chi, Dat Phi Van +3Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

  2. Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation

    Feb 24, 2026Sayantan Dasgupta, Trevor Cohn, Timothy BaldwinLanguage Model DistillationKnowledge Distillation

  3. Aligning Language Models from User Interactions

    Feb 18, 2026Thomas Kleine Buening, Jonas Hübotter, Barna Pásztor +3LLM AlignmentPersonalized Language Model Alignment

  4. Task-Specific Knowledge Distillation via Intermediate Probes

    Feb 18, 2026Ryan Brown, Chris RussellLanguage Model DistillationKnowledge Distillation

  5. Semantic Self-Distillation for Language Model Uncertainty

    Feb 4, 2026Edward Phillips, Sean Wu, Fredrik K. Gustafsson +2Uncertainty QuantificationConfidence Estimation in Language Models

  6. Distilling LLM Reasoning into Graph of Concept Predictors

    Feb 3, 2026Ziyang Yu, Liang ZhaoCoT DistillationActive Learning

  7. Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models

    Feb 2, 2026Hao Wang, Hao Gu, Hongming Piao +6Supervised Fine-TuningLanguage Model Distillation

  8. Distilling Token-Trained Models into Byte-Level Models

    Feb 1, 2026Zishuo Bao, Jiaqi Leng, Junxiong Wang +2LLM Fine-TuningByte-Level Language Model

  9. OVD: On-policy Verbal Distillation

    Jan 29, 2026Jing Xiong, Hui Shen, Shansan Gong +7Black-Box Knowledge DistillationLanguage Model Distillation

  10. Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation

    Oct 13, 2025Hengyuan Zhang, Shiping Yang, Xiao Liang +8Synthetic Data GenerationLanguage Model Distillation

  11. Base Models Know How to Reason, Thinking Models Learn When

    Oct 8, 2025Constantin Venhoff, Iván Arcuschin, Philip Torr +2LLM InterpretabilityRL for Language Model Reasoning

  12. Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners

    Sep 30, 2025Xin Xu, Clive Bai, Kai Yang +4Efficient Language Model ReasoningRL for Language Model Reasoning

  13. Circuit Distillation

    Sep 29, 2025Somin Wadhwa, Silvio Amir, Byron C. WallaceLanguage Model DistillationMechanistic Interpretability

  14. Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation

    Sep 26, 2025Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El Haddad +2CoT DistillationLLM Fine-Tuning

  15. Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

    Aug 13, 2025Xiaojun Wu, Xiaoguang Jiang, Huiyang Li +11CoT DistillationLanguage Model Distillation

  16. A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

    May 19, 2025Jitai Hao, Qiang Huang, Hao Liu +3Language Model PretrainingLLM Pruning

  17. KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation

    Apr 26, 2025Jiabin Fan, Guoqing Luo, Michael Bowling +1RL for Knowledge DistillationLanguage Model Distillation

  18. LLMR: Knowledge Distillation with a Large Language Model-Induced Reward

    Sep 19, 2024Dongheng Li, Yongchang Hao, Lili MouRL for Knowledge DistillationLanguage Model Distillation