Language Model Distillation

Overview

Language model distillation studies how to transfer a teacher model’s knowledge and behavior into a student that is cheaper to train or deploy, while retaining useful capabilities. Common approaches train on teacher-generated answers or reasoning traces, match teacher and student token probabilities, align hidden representations, or compress model weights; newer on-policy methods also use feedback on student-generated text. Researchers evaluate these methods on reasoning, coding, instruction-following, multilingual, and agent tasks, often comparing capability retention against training and inference costs. A central challenge is that teacher guidance can be noisy or poorly matched to a student’s abilities, so improving transfer efficiency must be balanced against stability, generalization, diversity, and loss of existing capabilities.

Latest papers

CardsList

562 papers67 papers in the last four weeks, down 50% on the four weeks before. 0.5% of all new papers.

  1. Which Tokens, in What Order, and How Fast? A Theoretical Analysis of On-Policy Distillation Objectives

    Nov 17, 2026Hubert Strauss, Liam Fowl, Noam RazinLanguage Model DistillationOn-Policy Distillation

  2. SilverSpoon-KD: A General-Purpose Toolkit for Knowledge Distillation

    Nov 17, 2026Xaver R. Davey, David BromanKnowledge DistillationLanguage Model Distillation

  3. Staged Semantic Distillation: Preserving Free-Form Semantics in a Low-Compute Non-LLM Surrogate of a Staged LLM Agent

    Nov 14, 2026Zhiyang Niu, Huayuan Shen, Chuanjie Li +2Language Model DistillationLLM Agent Training

  4. Folding Language Models into Themselves

    Nov 13, 2026Ziyang Zhang, Yubin Jing, Yuyao Li +2Language Model DistillationModel Compression

  5. Beyond Passive Imitation: On-Policy Mid-Training Distillation for Language Models

    Nov 10, 2026Ruize Zheng, Xuyang Liu, Siyu Ren +5Mid-Training for Language ModelsLanguage Model Distillation

  6. Bootstrapped On-Policy Distillation

    Nov 10, 2026Yue Wu, Xueyu Zhou, Yangrong Hu +1Language Model DistillationOn-Policy Distillation

  7. Speculative Self-Distillation enables Efficient Knowledge Internalization

    Nov 9, 2026Shayan Talaei, Agam Bhatia, Arshia Soltani Moakhar +3Language Model DistillationSelf-Distillation

  8. Perturbing Inputs Improves Pretraining Distillation

    Nov 8, 2026Ziyang Cai, Yihe Dong, Dimitris Papailiopoulos +1Language Model DistillationKnowledge Distillation

  9. How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?

    Nov 8, 2026Cong Yu, Ziting Wang, Songlin Jiang +5Training-Inference MismatchOn-Policy Distillation

  10. Failure Profiling and Reachable Trajectory Selection for Reasoning Distillation

    Nov 6, 2026Yijun Zhu, Jianxin Wang, Chengchao ShenReasoning DistillationLanguage Model Distillation

  11. Learning Dynamics of On-Policy Distillation

    Nov 5, 2026Guan Zhe Hong, Sumeet Ramesh Motwani, Liang Chen +5On-Policy DistillationLanguage Model Distillation

  12. Compression Through a Context Lens: Evaluating Adaptation, Quantization, and Distillation in Arabic Legal LLMs

    Nov 5, 2026Md. Rafiul Biswas, George Mikros, Wajdi ZaghouaniLegal Language Model EvaluationLLM Compression

  13. CoRD: Coupled Rollouts Fill the Blind Spot of On-Policy Distillation

    Nov 5, 2026Guhan Chen, Bohan Li, Jun Rao +8Language Model DistillationKnowledge Distillation

  14. O2^2PD: Overlap-Gated On-Policy Distillation on Reliable Prefix

    Nov 5, 2026Shuaishuai Zu, Jihao Zhao, Pengfei Li +5Language Model DistillationOn-Policy Distillation

  15. Reinforcing What the Student Has Learned: Teacher-Guided On-Policy Distillation

    Nov 5, 2026Bozhen Fang, Linhao Zhong, Xuehao Wang +3Language Model DistillationOn-Policy Distillation

  16. SKIM: Pruning Large Language Model Agents via Selective Knowledge Informed Masking

    Nov 5, 2026Moonseok Choi, Jongwon Jeong, Minki Kang +2LLM PruningLanguage Model Distillation

  17. D2D^2-OPD: Separating Direction and Distance in Extrapolative On-Policy Distillation

    Nov 4, 2026Bowen Fang, Dingmin Wang, Hengzhi PeiLanguage Model DistillationOn-Policy Distillation

  18. EvoOPD: Evolving Privileged Information Strategies for Black-Box On-Policy Distillation

    Nov 4, 2026Ximo Zhu, Ruiqi Liu, Yi Han +6Black-Box Knowledge DistillationLanguage Model Distillation

  19. On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation

    Nov 3, 2026Yichen Wang, Yufa Zhou, Siqi Zhu +4Language Model DistillationOn-Policy Distillation

  20. P2^2-MOPD: Accelerating Multi-Teacher On-Policy Distillation via Parameter Priors

    Nov 2, 2026Sinuo Wang, Jiacheng Lu, Zeyu Zhang +8Language Model DistillationMulti-Teacher On-Policy Distillation

  21. ZeroGraph: Compositional Blockwise Training via Materialized Boundary Supervision

    Nov 1, 2026Omar Ramadan, Sam Siavoshian, Amir Kashif Saeed +3Language Model Distillation

  22. Order-Only On-Policy Distillation

    Nov 1, 2026Guangzeng Han, Yu Wu, Xiaolei HuangLanguage Model DistillationOn-Policy Distillation