Language Model Distillation

Follow this topic Ask about this topic

Language model distillation studies how to transfer a teacher model’s knowledge and behavior into a student that is cheaper to train or deploy, while retaining useful capabilities. Common approaches train on teacher-generated answers or reasoning traces, match teacher and student token probabilities, align hidden representations, or compress model weights; newer on-policy methods also use feedback on student-generated text. Researchers evaluate these methods on reasoning, coding, instruction-following, multilingual, and agent tasks, often comparing capability retention against training and inference costs. A central challenge is that teacher guidance can be noisy or poorly matched to a student’s abilities, so improving transfer efficiency must be balanced against stability, generalization, diversity, and loss of existing capabilities.

Latest papers

522 papers135 papers in the last four weeks, up 160% on the four weeks before. 0.8% of all new papers.

Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibility
May 28, 2026Jiahao Huang, Fei Cheng, Junfeng Jiang +1Reasoning Distillation Training Data Selection
University of Tokyo · Kyoto University · National Institute of Informatics
Apertus LLM Family Expansion via Distillation and Quantization
May 27, 2026Andrei Panferov, Davit Melikidze, Martin Jaggi +1LLM Quantization LLM Compression
ISTA · EPFL · ETH Zürich +1
ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation
May 27, 2026Kun Liang, Chenming Tang, Clive Bai +3Language Model Distillation On-Policy Distillation
School of Computer Science, Peking University · 2National Key Laboratory for Multimedia Information Processing, Peking University · 3LLM Department, Tencent
Context Distillation as Latent Memory Management
May 27, 2026Ziyang Zheng, Zeju Li, Xiangyu Wen +5Memory-Augmented Language Models Language Model Distillation
The Chinese University of Hong Kong · Huawei Noah’s Ark Lab
Beyond Imitation: Reflective On-Policy Self-Distillation for LLM Reasoning
May 27, 2026Ziqi Zhao, Xinyu Ma, Liu Yang +6On-Policy Self-Distillation Language Model Distillation
The Hong Kong Polytechnic University · Baidu Inc. · Shandong University +1
Gradient Transformer: Learning to Generate Updates for LLMs
May 26, 2026Binh-Nguyen Nguyen, Khang Tran, NhatHai Phan +1Privacy-Preserving Language Models LLM Fine-Tuning
Department of Data Science, New Jersey Institute of Technology, Newark, NJ, USA · Qatar Computing Research Institute, HBKU, Doha, Qatar
Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generation
May 26, 2026Ning Wu, Rui Liu, Xinkun Lin +5Medical Report Generation Long-Form Document Generation
UNSW Sydney, Sydney, Australia · University of Technology Sydney, Sydney, Australia · School of Computer Science and Engineering, Sun Yat-sen University, China +2
MAIGO: Mitigating Lost-in-Conversation with History-Cleaned On-Policy Self-Distillation
May 26, 2026Haoyu Zheng, Yun Zhu, Shu Yuan +5Instruction Following Data Contamination in Language Models
1Zhejiang University · 2Shanghai AI Laboratory · 3Huazhong University of Science and Technology +2
Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation
May 26, 2026Tianlei Chen, Jiao Ou, Ziyuan Liu +3Language Model Distillation Multi-Teacher Knowledge Distillation
Kuaishou Technology, Beijing, China
Less is More: Early Stopping Rollout for On-Policy Distillation
May 26, 2026Zhou Ziheng, Jiaqi Li, Huacong Tang +2Language Model Distillation On-Policy Distillation
University of California, Los Angeles · Beijing Institute of General Artificial Intelligence
When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection
May 26, 2026Zhengyu Hu, Zheyuan Xiao, Linxin Song +10Language Model Distillation LLM Post-Training
University of Washington · University of Texas at Austin · University of Southern California +6
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
May 26, 2026Yuanyi Wang, Su Lu, Yanggan Gu +6Selective Knowledge Distillation Language Model Distillation
The Hong Kong Polytechnic University, PolyU · Hong Kong Polytechnic University Daya Bay Technology and Innovation Research Institute
EmoDistill: Offline Emotion Skill Distillation for Language Model Agents in Adversarial Negotiation
May 26, 2026Yunbo Long, Haolang Zhao, Lukas Beckenbauer +2Multi-Agent Negotiation Automated Negotiation
University of Cambridge · Technical University of Munich · Exiger LLC +1
The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works
May 25, 2026Guanghui Wang, Kaiwen Lv Kacuila, Zhiyong Yang +5LLM Compression Language Model Distillation
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Alibaba Group, Hangzhou, China · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China +2
Extreme Region Policy Distillation
May 25, 2026Changyu Chen, Xiting Wang, Rui YanKL-Regularized RL RL for Language Model Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · Wuhan University
Decision-Making with Lightweight Confidence-Aware Language Model for Autonomous Driving
May 25, 2026Ruoyu Yao, Ruiguo Zhong, Pei Liu +3Confidence Estimation in Language Models Language Model Distillation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China
Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
May 23, 2026Tianlang Chen, Shirley Wu, Jure LeskovecLanguage Model Distillation Teacher-Student Learning
Stanford
Strong Teacher Not Needed? On Distillation in LLM Pretraining
May 22, 2026Taiming Lu, Zhuang LiuLanguage Model Pretraining Language Model Distillation
Princeton University
The Distillation Game: Adaptive Evaluations & Efficient Defenses
May 21, 2026Youssef Allouah, Mahdi Haghifam, Sanmi Koyejo +1Model Extraction Attacks Adversarial Attacks
Toyota Technological Institute at Chicago · National University of Singapore
Self-Policy Distillation via Capability-Selective Subspace Projection
May 21, 2026Guangya Hao, Hanxue Liang, Yitong Shang +2Self-Training for Language Models Language Model Distillation
University of Cambridge · HKUST · University of Chicago
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
May 20, 2026Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning
May 20, 2026Xiaogeng Liu, Xinyan Wang, Yingzi Ma +2CoT Distillation On-Policy Self-Distillation
1Johns Hopkins University · University of Wisconsin–Madison · 3Nanyang Technological University
Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline
May 20, 2026Tony Lee, Percy LiangLanguage Model Self-Verification Self-Training for Language Models
Stanford University
AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals
May 20, 2026Duy Nguyen, Hanqi Xiao, Archiki Prasad +7On-Policy Self-Distillation Language Model Distillation
UNC Chapel Hill · Capital One · The University of Texas at Austin
What the Abstention Option Measures: Refusal Loss, Sampling Churn, and Option Capture in Distilled Small Language Models
May 19, 2026Plawan Kumar RathRefusal Behavior in Language Models LLM Abstention
Meta
Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
May 19, 2026Bing Wang, Shaotian Yan, Chen Shen +7CoT Distillation Language Model Distillation
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin University · Tongyi Lab, Alibaba Group +3
AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment
May 18, 2026Zhenlin Wei, Pu Jian, Yingzhuo Deng +6Token-Level Credit Assignment RL for Language Model Reasoning
Meituan Beijing, China · University of Chinese Academy of Sciences
FedSDR: Federated Self-Distillation with Rectification
May 18, 2026Ziheng Ren, Zhanming Shen, Hao Wang +2LLM Fine-Tuning Hallucination in Language Models
Beijing University of Aeronautics and Astronautics · Zhejiang University · Stevens Institute of Technology +1
$\boldsymbol{f}$ -OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control
May 18, 2026Xianwei Chen, Shimin Zhang, Jibin WuLanguage Model Distillation On-Policy Distillation
The Hong Kong Polytechnic University
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
May 15, 2026Shuaiyi Li, Zhisong Zhang, Yan Wang +5Long-Context Modeling LLM Inference Acceleration
The Chinese University of Hong Kong · City University of Hong Kong · Tencent +2

May 28, 2026J/K move · Enter open · S save

Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibility

Jiahao Huang, Fei Cheng, Junfeng Jiang, Akiko Aizawa

University of Tokyo · Kyoto University · National Institute of Informatics

Reasoning Distillation Training Data Selection Language Model Distillation Curriculum Learning

Open paper AskPDF arXiv

Reasoning distillation transfers complex reasoning abilities from large language models (LLMs) to smaller ones, yet its success depends on how well the training data align with the student model. This paper introduces the Data-Model Compatibility (DMC) metric, which can be used to assess the suitability of a dataset for reasoning distillation on a student model. DMC provides an assessment by jointly considering data quality, relative difficulty, and student capability. We validated the effectiveness of DMC from two perspectives: (1) DMC exhibits a strong correlation with reasoning distillation performance; and (2) using DMC as the criterion for data selection leads to improved reasoning distillation performance. Both findings are consistently demonstrated across multiple student models and tasks. Moreover, since the DMC of each dataset dynamically changes during training, our experiments demonstrate that dynamically selecting datasets based on DMC can further enhance performance.