Language Model Distillation

Follow this topic Ask about this topic

Momentum

69 papers in the last four weeks, up 200% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 350

All topics

OPSDL: On-Policy Self-Distillation for Long-Context Language Models
Apr 19, 2026Xinsen Zhang, Zhenkai Ding, Tianjun Pan +4Long-Context Language Modeling On-Policy Self-Distillation
Baidu Inc.
Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting
Apr 17, 2026Chi Liu, Xin Chen, Xu Zhou +2LLM Compression Language Model Distillation
PayPal AI
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
Apr 17, 2026Yao Chen, Jiawei Sheng, Wenyuan Zhang +1CoT Distillation Small Language Models
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences
Subliminal Transfer of Unsafe Behaviors in AI Agent Distillation
Apr 16, 2026Jacob Dang, Brian Y. Xie, Omar G. YounisSubliminal Learning Language Model Distillation
UCLA · Santa Monica College · Mila, Silverstream AI
Embarrassingly Simple Self-Distillation Improves Code Generation
Apr 1, 2026Ruixiang Zhang, Richard He Bai, Huangjie Zheng +3Language Model Decoding Language Model Distillation
Apple
When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
Mar 27, 2026Juan Gabriel Kostelec, Qinghai GuoLLM Inference Efficiency LLM Evaluation
Huawei Switzerland · ACS Lab, Huawei Technologies
Grounded Chess Reasoning in Language Models via Master Distillation
Mar 20, 2026Zhenwei Tang, Qianfeng Wen, Seth Grief-Albert +4LLM Grounding CoT Distillation
Department of Computer Science, University of Toronto · Queen’s University · Coolwei AI Lab
Effective Distillation to Hybrid xLSTM Architectures
Mar 16, 2026Lukas Hauzenberger, Niklas Schmidinger, Thomas Schmied +7LLM Compression Language Model Distillation
1NXAI · 2Johannes Kepler University Linz
HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation
Mar 11, 2026Wenjing Zhang, Jiangze Yan, Jieyun Huang +7CoT Distillation Language Model Distillation
Data Science & AI Research Institute, China Unicom · Unicom Data Intelligence, China Unicom
KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models
Mar 2, 2026Songming Zhang, Xue Zhang, Tong Zhang +3Cross-Tokenizer Knowledge Distillation Language Model Distillation
School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education · Weixin AI
Semantic Self-Distillation for Language Model Uncertainty
Feb 4, 2026Edward Phillips, Sean Wu, Fredrik K. Gustafsson +2Uncertainty Quantification Confidence Estimation in Language Models
Department of Engineering Science, University of Oxford · Oxford Suzhou Centre for Advanced Research, University of Oxford, Suzhou
Distilling LLM Reasoning into Graph of Concept Predictors
Feb 3, 2026Ziyang Yu, Liang ZhaoCoT Distillation Active Learning
Department of Computer Science Emory University Atlanta, USA
Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models
Feb 2, 2026Hao Wang, Hao Gu, Hongming Piao +6Supervised Fine-Tuning Language Model Distillation
1City University of Hong Kong · 2The Hong Kong University of Science and Technology · 3The Chinese University of Hong Kong
Distilling Token-Trained Models into Byte-Level Models
Feb 1, 2026Zishuo Bao, Jiaqi Leng, Junxiong Wang +2LLM Fine-Tuning Byte-Level Language Model
Fuzhou University · NYU Shanghai · Fudan University +2
OVD: On-policy Verbal Distillation
Jan 29, 2026Jing Xiong, Hui Shen, Shansan Gong +7Language Model Distillation On-Policy Distillation
The University of Hong Kong, Hong Kong, China · Nanjing University, Nanjing, China · Huawei Technologies, China
Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation
Oct 13, 2025Hengyuan Zhang, Shiping Yang, Xiao Liang +8Synthetic Data Generation Language Model Distillation
The University of Hong Kong · Simon Fraser University · University of California, Los Angeles +3
Base Models Know How to Reason, Thinking Models Learn When
Oct 8, 2025Constantin Venhoff, Iván Arcuschin, Philip Torr +2LLM Interpretability RL for Language Model Reasoning
University of Oxford, UK · MATS · Poseidon Research.
Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation
Sep 26, 2025Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El Haddad +2CoT Distillation LLM Fine-Tuning
1Diabolocom · 5MICS, CentraleSupélec, Université Paris-Saclay · 2Artefact Research Center +2
Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
Aug 13, 2025Xiaojun Wu, Xiaoguang Jiang, Huiyang Li +11CoT Distillation Language Model Distillation
1Zhongxing Telecom Equipment(ZTE), China · 2China Mobile Communications Group Co Ltd
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
May 19, 2025Jitai Hao, Qiang Huang, Hao Liu +3Language Model Pretraining LLM Pruning
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen · Baidu Inc. · Leiden University +1

Apr 19, 2026, cs.CLJ/K move · Enter open · S save

OPSDL: On-Policy Self-Distillation for Long-Context Language Models

Xinsen Zhang, Zhenkai Ding, Tianjun Pan, Run Yang+3

Baidu Inc.

Long-Context Language Modeling On-Policy Self-Distillation Language Model Distillation Language Model Post-Training

Open paper AskPDF arXiv

Extending the effective context length of large language models (LLMs) remains a central challenge for real-world applications. While recent post-training methods have made progress in long-context scaling, they either rely on high-quality supervision data or sparse sequence-level rewards, leading to unstable and inefficient optimization. We propose OPSDL, an On-Policy Self-Distillation method for enhancing the Long-context capabilities of LLMs. Unlike other recent self-distillation methods that inject privileged information and rely on the model's in-context learning ability to act as a teacher, OPSDL leverages the model's own inherently strong short-context capability as a self-teacher to supervise its own generation in long-context scenarios. The model first generates responses conditioned on the full long-context, then the self-teacher provides per-token supervision signals via point-wise reverse KL divergence under the relevant extracted short-context. This dense token-level signal encourages faithful use of relevant evidence and mitigates hallucinations induced by irrelevant context. We evaluate OPSDL on long-context benchmarks across a range of models from 7B to 32B parameters. Results show consistent and substantial improvements across varying context lengths, outperforming standard post-training approaches such as SFT and DPO with higher sample efficiency. Notably, these gains are achieved without degrading general short-context performance. These findings highlight the effectiveness of OPSDL as a scalable and stable approach for long-context learning.

Momentum

Related topics

Latest papers 350