Language Model Distillation

Follow this topic Ask about this topic

Language model distillation studies how to transfer a teacher model’s knowledge and behavior into a student that is cheaper to train or deploy, while retaining useful capabilities. Common approaches train on teacher-generated answers or reasoning traces, match teacher and student token probabilities, align hidden representations, or compress model weights; newer on-policy methods also use feedback on student-generated text. Researchers evaluate these methods on reasoning, coding, instruction-following, multilingual, and agent tasks, often comparing capability retention against training and inference costs. A central challenge is that teacher guidance can be noisy or poorly matched to a student’s abilities, so improving transfer efficiency must be balanced against stability, generalization, diversity, and loss of existing capabilities.

Latest papers

522 papers135 papers in the last four weeks, up 160% on the four weeks before. 0.8% of all new papers.

OPSDL: On-Policy Self-Distillation for Long-Context Language Models
Apr 19, 2026Xinsen Zhang, Zhenkai Ding, Tianjun Pan +4Long-Context Language Modeling On-Policy Self-Distillation
Baidu Inc.
Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting
Apr 17, 2026Chi Liu, Xin Chen, Xu Zhou +2LLM Compression Language Model Distillation
PayPal AI
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
Apr 17, 2026Yao Chen, Jiawei Sheng, Wenyuan Zhang +1CoT Distillation Small Language Models
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences
Subliminal Transfer of Unsafe Behaviors in AI Agent Distillation
Apr 16, 2026Jacob Dang, Brian Y. Xie, Omar G. YounisSubliminal Learning Language Model Distillation
UCLA · Santa Monica College · Mila, Silverstream AI
Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
Apr 15, 2026Tatsuya Ichinose, Youmi Ma, Masanari Oi +2Instruction Tuning Synthetic Data Generation for Language Models
Department of Computer Science, School of Computing, Institute of Science Tokyo · National Institute of Advanced Industrial Science and Technology · Research and Development Center for Large Language Models, NII
Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
Apr 14, 2026Yaxuan Li, Yuxin Zuo, Bingxiang He +8Language Model Distillation On-Policy Distillation
Tsinghua University · ShanghaiTech University · University of Illinois Urbana-Champaign +1
Validity-Calibrated Reasoning Distillation
Apr 14, 2026Khouloud Saadi, Di WangReasoning Distillation Language Model Distillation
Department of Computer Science, KAUST, Thuwal, Saudi Arabia
BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation
Apr 14, 2026Sihang Guo, Chenlin Zhou, Jiaqi Wang +3Spiking Language Models Spiking Neural Networks
Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China
Disposition Distillation at Small Scale: A Three-Arc Negative Result
Apr 13, 2026Hari SadasivanLanguage Model Distillation
Tinman Lab
MedThink: Enhancing Diagnostic Accuracy in Small Models via Teacher-Guided Reasoning Correction
Apr 9, 2026Xinchun Su, Chunxu Luo, Lipeng Ma +2Language Model Distillation Knowledge Distillation
School of Computer Science, Fudan University, Shanghai, China
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Apr 9, 2026Feng Luo, Yu-Neng Chuang, Guanchu Wang +4Repetition in Language Generation Language Model Distillation
Department of Computer Science, Rice University, Houston, USA · Department of Computer Science, University of North Carolina at Charlotte, Charlotte, USA · Department of Computer Science, Johns Hopkins University, Baltimore, USA +1
On-Policy Distillation of Language Models for Autonomous Vehicle Motion Planning
Apr 9, 2026Amirhossein Afsharrad, Amirhesam Abedsoltan, Ahmadreza Moradipari +1Language Model Distillation Autonomous Driving Planning
Stanford University · University of California, San Diego · University of California, Santa Barbara
Structured Distillation of Web Agent Capabilities Enables Generalization
Apr 9, 2026Xing Han Lù, Siva ReddyLLM Agent Training Web Agents
Mila – Quebec AI Institute McGill University
Cross-Tokenizer LLM Distillation through a Byte-Level Interface
Apr 8, 2026Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba +2Cross-Tokenizer Knowledge Distillation Language Model Distillation
MediaTek Research, Cambridge (United Kingdom)
State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation
Apr 7, 2026Navan Preet Singh, Anurag Garikipati, Ahmed Abulkhair +6Arabic Language Models Arabic NLP
Forta, Houston, TX · Incept Labs, Houston, TX · Titan Holdings, San Francisco, CA
Short Data, Long Context: Distilling Positional Knowledge in Transformers
Apr 7, 2026Patrick Huber, Ernie Chang, Chinnadhurai Sankar +4Long-Context Language Modeling LLM Context Management
Meta Reality Labs
SODA: Semi On-Policy Black-Box Distillation for Large Language Models
Apr 4, 2026Xiwen Chen, Jingjing Wang, Wenhui Zhu +7Language Model Distillation Knowledge Distillation
Clemson University · LinkedIn · Washington University in St. Louis +2
Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization
Apr 3, 2026Zihe Liu, Yulong Mao, Jinan Xu +2Language Model Distillation Knowledge Distillation
Key Laboratory of Big Data & Artificial Intelligence in Transportation · School of Computer Science and Technology, Beijing Jiaotong University
Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selection
Apr 3, 2026Chaoqun He, Yingfa Chen, Chaojun Xiao +2Language Model Distillation CoT Distillation
Tsinghua University
Embarrassingly Simple Self-Distillation Improves Code Generation
Apr 1, 2026Ruixiang Zhang, Richard He Bai, Huangjie Zheng +3Language Model Decoding Language Model Distillation
Apple
Attention to Mamba: A Recipe for Cross-Architecture Distillation
Apr 1, 2026Abhinav Moudgil, Ningyuan Huang, Eeshan Gunesh Dhekane +3Cross-Architecture Knowledge Distillation Language Model Distillation
Apple · MILA Research Institute · Flat Iron Institute
A Survey of On-Policy Distillation for Large Language Models
Apr 1, 2026Mingyang Song, Mao ZhengExposure Bias Language Model Distillation
Large Language Model Department Tencent, China
Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation
Mar 31, 2026Hoang-Chau Luong, Dat Ba Tran, Lingwei ChenLanguage Model Distillation Knowledge Distillation
Rochester Institute of Technology, Rochester, NY, USA · Rowan University, Glassboro, NJ, USA
Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models
Mar 31, 2026Gabriel Loiseau, Damien Sileo, Damien Riquet +2Privacy-Preserving ML Language Model Distillation
Hornetsecurity, Hem, France · Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000 Lille, France
When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
Mar 27, 2026Juan Gabriel Kostelec, Qinghai GuoLLM Inference Efficiency LLM Evaluation
Huawei Switzerland · ACS Lab, Huawei Technologies
Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
Mar 26, 2026Yuqian Fu, Haohuan Huang, Kaiwen Jiang +4Language Model Distillation On-Policy Distillation
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Fudan University +1
TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models
Mar 25, 2026Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan +3Language Model Distillation Cross-Architecture Knowledge Distillation
Department of Computer Science, University of Toronto · Vector Institute, Toronto, Canada
Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
Mar 25, 2026Jeonghye Kim, Xufang Luo, Minbeom Kim +5Self-Distillation Language Model Distillation
Microsoft Research · KAIST · SK hynix +1
Self-Distillation for Multi-Token Prediction
Mar 25, 2026Guoliang Zhao, Ruobing Xie, An Wang +3Multi-Token Prediction LLM Inference Acceleration
Large Language Model Department, Tencent
Gumbel Distillation for Parallel Text Generation
Mar 23, 2026Chi Zhang, Xixi Hu, Bo Liu +1Non-Autoregressive Text Generation Language Model Distillation
Department of Computer Science The University of Texas at Austin

Apr 19, 2026J/K move · Enter open · S save

OPSDL: On-Policy Self-Distillation for Long-Context Language Models

Xinsen Zhang, Zhenkai Ding, Tianjun Pan, Run Yang+3

Baidu Inc.

Long-Context Language Modeling On-Policy Self-Distillation Language Model Distillation Language Model Post-Training

Open paper AskPDF arXiv

Extending the effective context length of large language models (LLMs) remains a central challenge for real-world applications. While recent post-training methods have made progress in long-context scaling, they either rely on high-quality supervision data or sparse sequence-level rewards, leading to unstable and inefficient optimization. We propose OPSDL, an On-Policy Self-Distillation method for enhancing the Long-context capabilities of LLMs. Unlike other recent self-distillation methods that inject privileged information and rely on the model's in-context learning ability to act as a teacher, OPSDL leverages the model's own inherently strong short-context capability as a self-teacher to supervise its own generation in long-context scenarios. The model first generates responses conditioned on the full long-context, then the self-teacher provides per-token supervision signals via point-wise reverse KL divergence under the relevant extracted short-context. This dense token-level signal encourages faithful use of relevant evidence and mitigates hallucinations induced by irrelevant context. We evaluate OPSDL on long-context benchmarks across a range of models from 7B to 32B parameters. Results show consistent and substantial improvements across varying context lengths, outperforming standard post-training approaches such as SFT and DPO with higher sample efficiency. Notably, these gains are achieved without degrading general short-context performance. These findings highlight the effectiveness of OPSDL as a scalable and stable approach for long-context learning.