Language Model Distillation

Ask about this topic

Overview

Language model distillation studies how to transfer a teacher model’s knowledge and behavior into a student that is cheaper to train or deploy, while retaining useful capabilities. Common approaches train on teacher-generated answers or reasoning traces, match teacher and student token probabilities, align hidden representations, or compress model weights; newer on-policy methods also use feedback on student-generated text. Researchers evaluate these methods on reasoning, coding, instruction-following, multilingual, and agent tasks, often comparing capability retention against training and inference costs. A central challenge is that teacher guidance can be noisy or poorly matched to a student’s abilities, so improving transfer efficiency must be balanced against stability, generalization, diversity, and loss of existing capabilities.

Latest papers

CardsList

522 papers135 papers in the last four weeks, up 160% on the four weeks before. 0.8% of all new papers.

  1. Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation

    Oct 29, 2026Yifu Luo, Chuang Ma, Zeyu Chen +3On-Policy Self-DistillationLanguage Model Distillation

    TU München · Munich Center for Machine Learning (MCML) · relAI – Konrad Zuse School of Excellence in Reliable AI +4

  2. Teaching a Tree to Recover: Online Distillation of an LLM Supervisor with Counterfactual Replay

    Oct 29, 2026Kristian Ceder, Kilian Freitag, Knut ÅkessonRobot Failure RecoveryRobot Navigation

    Department of Electrical Engineering, Chalmers University of Technology, Sweden

  3. Distilling the Shape of Uncertainty: What On-Policy Distillation Learns and How to Predict It

    Oct 28, 2026Shiping Yang, Chuanqi Tang, Hossein Hajimirsadeghi +3Language Model DistillationOn-Policy Distillation

    Simon Fraser University · Borealis AI

  4. On-Policy Self-Distillation with Teacher Evolution

    Oct 28, 2026Shufan Shen, Zhongni Hou, Chao Bi +3On-Policy Self-DistillationLanguage Model Distillation

    State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Meituan

  5. How Much Do Teacher Bridges Matter in Interleaved On-Policy Distillation?

    Oct 28, 2026Leran Chen, Zhiyuan Yao, Lingnan KongLanguage Model DistillationOn-Policy Distillation

    The University of Sydney · University of California San Diego

  6. Multi-Token Prediction Done Right

    Oct 28, 2026Jayden TeohMulti-Token PredictionLanguage Model Pretraining

    MIT

  7. MASS: Multi-mask Selective Self-Distillation for Leakage-Resilient Reasoning

    Oct 28, 2026Xiuqi Liu, Oveys Delafrooz Noroozi, Upamanyu MadhowLanguage Model DistillationSelective Knowledge Distillation

    Chongqing Bashu Secondary School · University of California, Santa Barbara

  8. Preserving Safety During Sequential Fine-Tuning: A Factorial and Geometric Analysis of Replay

    Oct 27, 2026George Ruellan, Diana Benavides-PradoSafety-Preserving Fine-Tuning of Language ModelsSafety-Utility Trade-Off in Language Models

    School of Electronic Engineering and Computer Science, Queen Mary University of London, London, United Kingdom

  9. Watermark Basins: Black-Box Detection of Distilled LLMs

    Oct 27, 2026Hongyan Chang, Hady Elsahar, Tom Sander +10LLM WatermarkingLanguage Model Auditing

    FAIR, Meta Superintelligence Labs

  10. What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2

    Oct 27, 2026Nabeelah Faumi, Deshan SumanathilakaLanguage Model DistillationKnowledge Distillation

    Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka

  11. Proposing Better Rollouts for On-Policy Distillation

    Oct 27, 2026Sirui Chen, Yunzhe Qi, Haobo Xu +4Language Model DistillationOn-Policy Distillation

    University of Illinois Urbana-Champaign · Stanford University

  12. From Solver Trajectories to Teaching Trajectories: Cognition-Aligned Reasoning Distillation

    Oct 27, 2026Yashuo Luo, Qianren Mao, Ziqi Qin +6Reasoning DistillationLanguage Model Distillation

    National College for Excellent Engineers, Beihang University · Zhongguancun Laboratory · Beihang University +3

  13. SRA: Selective Residual Adaptation for LLM Reasoning Enhancement

    Oct 25, 2026Zhenxing Dou, Kun He, Mingyang Yi +1Language Model DistillationSelective Knowledge Distillation

    Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Information, Renmin University of China, Beijing, China

  14. A Distribution-Matching Perspective on LLM Training Algorithms

    Oct 24, 2026Yanxing Huang, Bingzhang Yan, Zihan Tang +3LLM TrainingLanguage Model Distillation

    Qiuzhen College, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Weiyang College, Tsinghua University, Beijing, China +2

  15. When to Follow the Teacher: Advantage-Based On-Policy Distillation

    Oct 22, 2026Mahdi Sabbaghi, George Pappas, Hamed Hassani +1Language Model DistillationOn-Policy Distillation

    University of Pennsylvania · University of Southern California

  16. DIAL: Steerable Language Models with Distilled Instruction-Adherence Levels

    Oct 22, 2026Thanh Q. Tran, John Russell Himawan, Arun Verma +4Language Model SteeringInstruction Following

    School of Computing, National University of Singapore, Singapore · Singapore-MIT Alliance for Research and Technology Centre, Singapore · CSAIL, Massachusetts Institute of Technology, USA +1

  17. Conflict-Aware Logit Adapters for Utility-Preserving Anti-Distillation

    Oct 22, 2026Quoc Phong Dao, Binh Anh Nguyen, Linh Ngo Van +3Language Model Distillation

    Hanoi University of Science and Technology · Center for AI Research (CAIR), VinUniversity · Monash University

  18. Tracing Subliminal Learning Through Pretraining

    Oct 22, 2026Angela Tang, Vishesh Anand, Gonçalo PauloLanguage Model DistillationOn-Policy Distillation

    EleutherAI

  19. Reverse-KL Policy Distillation: Statistical Guarantees and Limits

    Oct 22, 2026Haolin Liu, Chen-Yu Wei, Julian ZimmertLanguage Model DistillationKnowledge Distillation

    University of Virginia · Google Research · National Taiwan University

  20. GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation

    Oct 21, 2026Quang Anh Kiet Tran, Nam Anh Tran, Manh Hung Trinh +3Small Language ModelsLanguage Modeling

    UIT, VNU-HCM · TwoHearts Technologies Co., Ltd., Ho Chi Minh City, Vietnam · IU, VNU-HCM +3

  21. TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation

    Oct 21, 2026Linfei Ji, Jiayun Li, Haozhao Wang +5Language Model DistillationOn-Policy Self-Distillation

    Huazhong University of Science and Technology · Tsinghua University · Harbin Institute of Technology +1

  22. Target Regeneration Shrinks the Gains of Draft-Training Objectives in Speculative Decoding

    Oct 21, 2026Jungseob Lee, Hogun Seong, Euijun Choi +1Speculative DecodingLanguage Model Distillation

    Korea University · Yonsei University Mirae Campus

  23. CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

    Oct 20, 2026Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge +3Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

  24. Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation

    Oct 20, 2026Guangyu Liu, Siyuan Gan, Shilong Zhou +3Language Model DistillationOn-Policy Distillation

    State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China

  25. On-Policy Transition Distillation

    Oct 20, 2026Runyang You, Yongqi Li, Tiezheng Yu +1Language Model DistillationOn-Policy Distillation

    The Hong Kong Polytechnic University · Huawei Technologies Ltd.

  26. SAPD: Learning from Agent Experience without Environment Interaction

    Oct 20, 2026Juncheng Diao, Haibin Wang, Wei Shi +4LLM Agent TrainingLanguage Model Distillation

    Meituan, LongCat Team · University of Chinese Academy of Sciences

  27. Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets

    Oct 20, 2026Shripad V. Deshmukh, Christian Bock, Kangrui Ruan +4Language Model DistillationOnline Knowledge Distillation

    University of Massachusetts · AWS AI Labs

  28. Ask to Distill: Learning to Synthesize Data for On-Policy Distillation

    Oct 20, 2026Jiashuo Sun, Hao Zhang, Xueqiang Xu +9Language Model DistillationOn-Policy Distillation

    University of Illinois Urbana-Champaign, Urbana, IL, USA · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Texas A&M University, College Station, TX, USA

  29. Selective Transfer through Sequence Distillation: From Benchmark Knowledge to Backdoor Behaviour

    Oct 19, 2026Jonibek Mansurov, Sholpan Bolatzhanova, Filbert Aurelian Tjiaranta +2Language Model DistillationKnowledge Distillation

    Mohamed bin Zayed University of Artificial Intelligence, UAE · University of Indonesia, Indonesia · Binus University, Indonesia

  30. Nash-MOPD: Optimizing Task Trade-offs in Multi-Teacher On-Policy Distillation

    Oct 19, 2026Siyuan Gan, Bo Yang, Linjian Meng +4Language Model DistillationMulti-Teacher Knowledge Distillation

    State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Yili Normal University, Yining, China