Language Model Distillation

Follow this topic Ask about this topic

Language model distillation studies how to transfer a teacher model’s knowledge and behavior into a student that is cheaper to train or deploy, while retaining useful capabilities. Common approaches train on teacher-generated answers or reasoning traces, match teacher and student token probabilities, align hidden representations, or compress model weights; newer on-policy methods also use feedback on student-generated text. Researchers evaluate these methods on reasoning, coding, instruction-following, multilingual, and agent tasks, often comparing capability retention against training and inference costs. A central challenge is that teacher guidance can be noisy or poorly matched to a student’s abilities, so improving transfer efficiency must be balanced against stability, generalization, diversity, and loss of existing capabilities.

Latest papers

522 papers135 papers in the last four weeks, up 160% on the four weeks before. 0.8% of all new papers.

Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models
Feb 2, 2026Hao Wang, Hao Gu, Hongming Piao +6Supervised Fine-Tuning Language Model Distillation
1City University of Hong Kong · 2The Hong Kong University of Science and Technology · 3The Chinese University of Hong Kong
Distilling Token-Trained Models into Byte-Level Models
Feb 1, 2026Zishuo Bao, Jiaqi Leng, Junxiong Wang +2LLM Fine-Tuning Byte-Level Language Model
Fuzhou University · NYU Shanghai · Fudan University +2
OVD: On-policy Verbal Distillation
Jan 29, 2026Jing Xiong, Hui Shen, Shansan Gong +7Language Model Distillation On-Policy Distillation
The University of Hong Kong, Hong Kong, China · Nanjing University, Nanjing, China · Huawei Technologies, China
Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation
Oct 13, 2025Hengyuan Zhang, Shiping Yang, Xiao Liang +8Synthetic Data Generation Language Model Distillation
The University of Hong Kong · Simon Fraser University · University of California, Los Angeles +3
Base Models Know How to Reason, Thinking Models Learn When
Oct 8, 2025Constantin Venhoff, Iván Arcuschin, Philip Torr +2LLM Interpretability RL for Language Model Reasoning
University of Oxford, UK · MATS · Poseidon Research.
Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
Sep 30, 2025Xin Xu, Clive Bai, Kai Yang +4Efficient Language Model Reasoning RL for Language Model Reasoning
LLM Department, Tencent · The Hong Kong University of Science and Technology · The University of Hong Kong
Circuit Distillation
Sep 29, 2025Somin Wadhwa, Silvio Amir, Byron C. WallaceLanguage Model Distillation Mechanistic Interpretability
Khoury College of Computer Sciences, Northeastern University, Boston, MA 02115, USA
Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation
Sep 26, 2025Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El Haddad +2CoT Distillation LLM Fine-Tuning
1Diabolocom · 5MICS, CentraleSupélec, Université Paris-Saclay · 2Artefact Research Center +2
Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
Aug 13, 2025Xiaojun Wu, Xiaoguang Jiang, Huiyang Li +11CoT Distillation Language Model Distillation
1Zhongxing Telecom Equipment(ZTE), China · 2China Mobile Communications Group Co Ltd
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
May 19, 2025Jitai Hao, Qiang Huang, Hao Liu +3Language Model Pretraining LLM Pruning
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen · Baidu Inc. · Leiden University +1
KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation
Apr 26, 2025Jiabin Fan, Guoqing Luo, Michael Bowling +1RL for Knowledge Distillation Language Model Distillation
Dept. Computing Science, Alberta Machine Intelligence Institute (Amii), University of Alberta, Canada · Canada CIFAR AI Chair, Amii
LLMR: Knowledge Distillation with a Large Language Model-Induced Reward
Sep 19, 2024Dongheng Li, Yongchang Hao, Lili MouRL for Knowledge Distillation Language Model Distillation
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta · Canada CIFAR AI Chair, Amii

Feb 2, 2026J/K move · Enter open · S save

Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models

Hao Wang, Hao Gu, Hongming Piao, Kaixiong Gong+5

1City University of Hong Kong · 2The Hong Kong University of Science and Technology · 3The Chinese University of Hong Kong

Supervised Fine-Tuning Language Model Distillation Self-Distillation Large Reasoning Models

Open paper AskPDF arXiv

The standard post-training recipe for large reasoning models, supervised fine-tuning followed by reinforcement learning (SFT-then-RL), may limit the benefits of the RL stage: while SFT imitates expert demonstrations, it often causes overconfidence and reduces generation diversity, leaving RL with a narrowed solution space to explore. Adding entropy regularization during SFT is not a cure-all; it tends to flatten token distributions toward uniformity, increasing entropy without improving meaningful exploration capability. In this paper, we propose CurioSFT, an entropy-preserving SFT method designed to enhance exploration capabilities through intrinsic curiosity. It consists of (a) Self-Exploratory Distillation, which distills the model toward a self-generated, temperature-scaled teacher to encourage exploration within its capability; and (b) Entropy-Guided Temperature Selection, which adaptively adjusts distillation strength to mitigate knowledge forgetting by amplifying exploration at reasoning tokens while stabilizing factual tokens. Extensive experiments on mathematical reasoning tasks demonstrate that, in SFT stage, CurioSFT outperforms the vanilla SFT by 2.5 points on in-distribution tasks and 2.9 points on out-of-distribution tasks. We also verify that exploration capabilities preserved during SFT successfully translate into concrete gains in RL stage, yielding an average improvement of 5.0 points.