Language Model Distillation

Ask about this topic

Overview

Language model distillation studies how to transfer a teacher model’s knowledge and behavior into a student that is cheaper to train or deploy, while retaining useful capabilities. Common approaches train on teacher-generated answers or reasoning traces, match teacher and student token probabilities, align hidden representations, or compress model weights; newer on-policy methods also use feedback on student-generated text. Researchers evaluate these methods on reasoning, coding, instruction-following, multilingual, and agent tasks, often comparing capability retention against training and inference costs. A central challenge is that teacher guidance can be noisy or poorly matched to a student’s abilities, so improving transfer efficiency must be balanced against stability, generalization, diversity, and loss of existing capabilities.

Latest papers

CardsList

522 papers135 papers in the last four weeks, up 160% on the four weeks before. 0.8% of all new papers.

  1. Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatch

    Mar 23, 2026Stella Eva Tsiapali, Cong-Thanh Do, Kate KnillCross-Tokenizer Knowledge DistillationLanguage Model Distillation

    University of Cambridge, Department of Engineering, Cambridge, United Kingdom · Toshiba Europe Limited, Cambridge, United Kingdom

  2. Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guarantees

    Mar 22, 2026Alberlucia Rafael Soarez, Daniel Kim, Mariana Costa +1Language Model DistillationKnowledge Distillation

    Department of Computer Science, University of Brasilia

  3. Grounded Chess Reasoning in Language Models via Master Distillation

    Mar 20, 2026Zhenwei Tang, Qianfeng Wen, Seth Grief-Albert +4LLM GroundingCoT Distillation

    Department of Computer Science, University of Toronto · Queen’s University · Coolwei AI Lab

  4. Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation

    Mar 19, 2026Zhuolin Yang, Zihan Liu, Yang Chen +14RL for Language Model ReasoningRL for LLM Agents

  5. Effective Distillation to Hybrid xLSTM Architectures

    Mar 16, 2026Lukas Hauzenberger, Niklas Schmidinger, Thomas Schmied +7LLM CompressionLanguage Model Distillation

    1NXAI · 2Johannes Kepler University Linz

  6. Retrieval-Feedback-Driven Distillation and Preference Alignment for Efficient LLM-based Query Expansion

    Mar 14, 2026Minghan Li, Guodong ZhouLanguage Model DistillationEfficient Language Model Inference

    School of Computer Science and Technology, Soochow University, China

  7. Knowledge Distillation for Large Language Models

    Mar 14, 2026Alejandro Paredes La Torre, Barbara Flores, Diego RodriguezLanguage Model DistillationLLM Compression

    Duke University

  8. PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence

    Mar 11, 2026Yuanda Xu, Hejian Sang, Zhengze Zhou +2Language Model DistillationAdaptive Knowledge Distillation

    LinkedIn Corporation

  9. From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers

    Mar 11, 2026Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar +1Cross-Modal Knowledge DistillationKnowledge Distillation

    Indian Institute of Technology Delhi, India · Indraprastha Institute of Information Technology Delhi, India

  10. HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation

    Mar 11, 2026Wenjing Zhang, Jiangze Yan, Jieyun Huang +7CoT DistillationLanguage Model Distillation

    Data Science & AI Research Institute, China Unicom · Unicom Data Intelligence, China Unicom

  11. HoloByte: Continuous Hyperspherical Distillation for Tokenizer-Free Modeling

    Mar 10, 2026Vladimer KhasiaByte-Level Language ModelLanguage Modeling

    Independent Researcher

  12. LinearARD: Linear-Memory Attention Distillation for RoPE Restoration

    Mar 9, 2026Ning Yang, Hengyu Zhong, Wentao Wang +4Attention-Guided DistillationMemory-Efficient Attention

    Institute of Automation, Chinese Academy of Sciences, Beijing, China · Southwest University, Chongqing, China · Dalian University of Technology, Dalian, Liaoning, China +3

  13. DistillGuard: Evaluating Defenses Against LLM Knowledge Distillation

    Mar 8, 2026Bo JiangLanguage Model DistillationModel Extraction Attacks

  14. Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Training

    Mar 7, 2026Chuxue Cao, Honglin Lin, Zhanping Zhong +5Financial Language Model EvaluationLLM Training

    Shanghai Artificial Intelligence Laboratory, OpenDataLab, OpenDataArena · Hong Kong University of Science and Technology

  15. Entropy-Aware On-Policy Distillation of Language Models

    Mar 7, 2026Woogyeol Jin, Taywon Min, Yongjin Yang +5Language Model DistillationOn-Policy Distillation

    KAIST AI, Seoul, South Korea · University of Toronto, Ontario, Canada · Vector Institute, Ontario, Canada +1

  16. Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision

    Mar 7, 2026Shreyas Gopal, Donghang Wu, Ashutosh Anshul +5Speech Language ModelsLanguage Model Distillation

    College of Computing and Data Science, Nanyang Technological University, Singapore · AI Singapore, National University of Singapore · Institute for Infocomm Research (I2R), A∗STAR, Singapore

  17. KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models

    Mar 2, 2026Songming Zhang, Xue Zhang, Tong Zhang +3Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

    School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education · Weixin AI

  18. Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion

    Feb 26, 2026Zhen Tan, Chengshuai Zhao, Song Wang +3Language Model DistillationReasoning Distillation

    Arizona State University · University of Virginia · University of North Carolina at Chapel Hill

  19. Decoder-based Sense Knowledge Distillation

    Feb 25, 2026Qitong Wang, Mohammed J. Zaki, Georgios Kollias +1Language Model DistillationKnowledge Distillation

    Rensselaer Polytechnic Institute Troy, New York, USA · IBM Research Yorktown Heights, New York, USA

  20. D-COT: Disciplined Chain-of-Thought Learning for Efficient Reasoning in Small Language Models

    Feb 25, 2026Shunsuke UbukataEfficient Language Model ReasoningCoT Distillation

    Toyo University

  21. Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation

    Feb 25, 2026Minsang Kim, Seung Jun BaekLanguage Model DistillationSelective Knowledge Distillation

    Korea University · SK Telecom

  22. DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation

    Feb 25, 2026Duc Trung Vu, Pham Khanh Chi, Dat Phi Van +3Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

    Hanoi University of Science and Technology, Vietnam · University of Monash, Australia

  23. Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning

    Feb 24, 2026Sanket Badhe, Deep ShahEfficient Language Model ReasoningLanguage Model Distillation

    Google Mountain View, California, USA

  24. Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation

    Feb 24, 2026Sayantan Dasgupta, Trevor Cohn, Timothy BaldwinLanguage Model DistillationKnowledge Distillation

    School of Computing and Information Systems, University of Melbourne, Melbourne, Australia.

  25. Flow Map Language Models: One-step Language Modeling via Continuous Denoising

    Feb 18, 2026Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal +6Continuous Diffusion Language ModelsOne-Step Generative Modeling

    KAIST · Carnegie Mellon University

  26. Aligning Language Models from User Interactions

    Feb 18, 2026Thomas Kleine Buening, Jonas Hübotter, Barna Pásztor +3LLM AlignmentPersonalized Language Model Alignment

    ETH Zurich · MIT · University of Zurich

  27. Task-Specific Knowledge Distillation via Intermediate Probes

    Feb 18, 2026Ryan Brown, Chris RussellLanguage Model DistillationKnowledge Distillation

    Oxford Internet Institute, University of Oxford, Oxford, United Kingdom.

  28. Fast and Effective On-policy Distillation from Reasoning Prefixes

    Feb 16, 2026Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani +6Language Model DistillationOn-Policy Distillation

    Optum AI

  29. Semantic Self-Distillation for Language Model Uncertainty

    Feb 4, 2026Edward Phillips, Sean Wu, Fredrik K. Gustafsson +2Uncertainty QuantificationConfidence Estimation in Language Models

    Department of Engineering Science, University of Oxford · Oxford Suzhou Centre for Advanced Research, University of Oxford, Suzhou

  30. Distilling LLM Reasoning into Graph of Concept Predictors

    Feb 3, 2026Ziyang Yu, Liang ZhaoCoT DistillationActive Learning

    Department of Computer Science Emory University Atlanta, USA