Language Model Distillation

Follow this topic Ask about this topic

Language model distillation studies how to transfer a teacher model’s knowledge and behavior into a student that is cheaper to train or deploy, while retaining useful capabilities. Common approaches train on teacher-generated answers or reasoning traces, match teacher and student token probabilities, align hidden representations, or compress model weights; newer on-policy methods also use feedback on student-generated text. Researchers evaluate these methods on reasoning, coding, instruction-following, multilingual, and agent tasks, often comparing capability retention against training and inference costs. A central challenge is that teacher guidance can be noisy or poorly matched to a student’s abilities, so improving transfer efficiency must be balanced against stability, generalization, diversity, and loss of existing capabilities.

Latest papers

522 papers135 papers in the last four weeks, up 160% on the four weeks before. 0.8% of all new papers.

Distribution Corrected Offline Data Distillation for Large Language Models
May 13, 2026Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake +1CoT Distillation Online Knowledge Distillation
George Mason University, Fairfax, VA, USA
Towards Resource-Efficient LLMs: End-to-End Energy Accounting of Distillation Pipelines
May 13, 2026Katherine Lambert, Sasha LuccioniEnergy-Efficient ML Language Model Distillation
University of Toronto · 2Sustainable AI Group
Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation
May 13, 2026Kaiyuan Liu, Ziyuan Zhuang, Yang Bai +3Selective Knowledge Distillation Language Model Distillation
College of Computer Science and Technology, Zhejiang University · Meituan LongCat Team, China · College of Computer Science and Technology, Jilin University
Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
May 13, 2026Xinyu Liu, Kechen Jiao, Chunyang Xiao +10RL for Language Model Reasoning Policy Optimization
School of Computer Science and Engineering, Northeastern University, China · Tsinghua University · Meituan +2
GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning
May 13, 2026Kasidit Sermsri, Teerapong PanboonyuenCoT Distillation Language Model Distillation
†Chulalongkorn University · ‡MARSAIL · §PBYAIL (Panboonyuen AI Lab)
Multi-Rollout On-Policy Distillation via Peer Successes and Failures
May 12, 2026Weichen Yu, Xiaomin Li, Yizhou Zhao +8RL for Language Model Reasoning Language Model Distillation
1Carnegie Mellon University · 2Microsoft · 3Purdue University
Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training
May 12, 2026Yuanda Xu, Hejian Sang, Zhengze Zhou +3Reinforcement Learning RL for Language Model Reasoning
OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning
May 12, 2026Yuxiao Yang, Xiaoyun Wang, Weitong ZhangLanguage Model Calibration Language Model Distillation
UNC Chapel Hill · NVIDIA
OmniThoughtVis: A Scalable Distillation Pipeline for Deployable Multimodal Reasoning Models
May 12, 2026Yuanhao Yue, Chengyu Wang, Yuanjie Lyu +2Multimodal CoT Reasoning Multimodal Reasoning
Alibaba Group, Hangzhou, China
A Study on Hidden Layer Distillation for Large Language Model Pre-Training
May 12, 2026Maxime Guigon, Lucas Dixon, Michaël E. SanderLanguage Model Pretraining Decoder-Only Language Models
Google DeepMind
Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning
May 12, 2026Zihao Han, Tiangang Zhang, Huaibin Wang +1Adaptive Knowledge Distillation On-Policy Self-Distillation
ByteDance Douyin
SOMA: Efficient Multi-turn LLM Serving via Small Language Model
May 11, 2026Xueqi Cheng, Qiong Wu, Zhengyi Zhou +3LLM Serving Small Language Models
Florida State University · AT&T Chief Data Office · Louisiana State University +1
ReAD: Reinforcement-Guided Capability Distillation for Large Language Models
May 11, 2026Xueqi Cheng, Xugui Zhou, Tyler Derr +1LLM Compression Language Model Distillation
Florida State University · Louisiana State University · Vanderbilt University
Curriculum Learning-Guided Progressive Distillation in Large Language Models
May 11, 2026Jincheng Cao, Fanzhi Zeng, Leqi Liu +1Reasoning Distillation Language Model Distillation
The University of Texas at Austin · Google Research
Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
May 11, 2026Mohammadreza Armandpour, Fatih Ilhan, David Harrison +6On-Policy Self-Distillation Language Model Distillation
Apple
Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR
May 11, 2026Jeonghye Kim, Jiwon Jeon, Dongsheng Li +1RL for Language Model Reasoning Language Model Distillation
Microsoft Research · KAIST
Identified-Set Geometry of Distributional Model Extraction under Top- $K$ Censored API Access
May 11, 2026Wenhua Nie, ZiCheng Zhu, Jianan Wu +3Parameter Identifiability Partial Identification
National Taiwan University
On-Policy Distillation with Best-of-N Teacher Rollout Selection
May 10, 2026Ke Zhang, Yunjie Tian, Dongdi Zhao +4CoT Distillation Best-of-N Sampling
Johns Hopkins University · TikTok · University of California, San Diego +1
Crosslingual On-Policy Self-Distillation for Multilingual Reasoning
May 10, 2026Yihong Liu, Raoyuan Zhao, Michael A. Hedderich +1On-Policy Self-Distillation Cross-Lingual Reasoning in Language Models
Center for Information and Language Processing, LMU Munich · Munich Center for Machine Learning (MCML)
Test-Time Speculation
May 10, 2026Avinash Kumar, Sujay Sanghavi, Poulami DasTest-Time Adaptation Test-Time Optimization
The University of Texas at Austin
Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
May 10, 2026Yuxuan Jiang, Runchao Li, Shubhashis Roy Dipta +2Language Model Distillation On-Policy Distillation
University of Maryland, Baltimore County · Case Western Reserve University · Arizona State University +1
Reasoning Compression with Mixed-Policy Distillation
May 9, 2026Han Yang, Mingyan Wu, Bailan He +4Language Model Distillation Efficient Language Model Inference
Technical University of Munich · GESIS – Leibniz Institute for the Social Sciences · Northeastern University +5
Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning
May 9, 2026Zhengyang Zhao, Lu Ma, Wentao ZhangOn-Policy Self-Distillation Language Model Distillation
Peking University
Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation
May 8, 2026Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade +3Flow Matching Diffusion Language Model Inference
1The Ohio State University · 2Apple
KL for a KL: On-Policy Distillation with Control Variate Baseline
May 8, 2026Minjae Oh, Sangjun Song, Gyubin Choi +2Language Model Distillation Policy Gradient Methods
Graduate School of Data Science Seoul National University
Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning
May 8, 2026Zhicheng Yang, Zhijiang Guo, Yifan Song +5RL for Language Model Reasoning Language Model Distillation
1The Hong Kong University of Science and Technology (Guangzhou) · 2The Hong Kong University of Science and Technology · 3MBZUAI +2
Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models
May 8, 2026Boyu Shi, YiCheng Jiang, Chang Liu +3Small Language Models Cross-Architecture Knowledge Distillation
School of Computer Science and Engineering, Southeast University, Nanjing, China
SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
May 8, 2026Jie Sun, Mao Zheng, Mingyang Song +6Cross-Tokenizer Knowledge Distillation Language Model Distillation
University of Science and Technology of China · Shanghai Innovation Institute · Large Language Model Department, Tencent +1
LiteGUI: Lightweight GUI Agents via Multi-Solution Guided Distillation and Dual-Level Reinforcement Learning
May 8, 2026Yubin Wu, Zicheng Cai, Liping Ning +4RL for GUI Agents GUI Agents
Moore Threads AI
Rubric-based On-policy Distillation
May 8, 2026Junfeng Fang, Zhepei Hong, Mao Zheng +7LLM Alignment Language Model Distillation
National University of Singapore · University of Science and Technology of China · Tencent

May 13, 2026J/K move · Enter open · S save

Distribution Corrected Offline Data Distillation for Large Language Models

Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake, Zhuangdi Zhu

George Mason University, Fairfax, VA, USA

CoT Distillation Online Knowledge Distillation Efficient Language Model Reasoning Language Model Distillation Exposure Bias Teacher-Student Learning+1

Open paper AskPDF arXiv

Distilling reasoning traces from strong large language models into smaller ones is a promising route to improve intelligence in resource-constrained settings. Existing approaches face a fundamental trade-off: offline distillation from teacher-generated traces provides high-quality, sample-efficient supervision but suffers from distributional drift: during training, the student model conditions on teacher-generated prefixes, whereas during inference the student autoregresses on self-generated prefixes, leading to compounding errors over long reasoning trajectories. Meanwhile, on-policy or self-distillation methods better match the student's inference-time distribution, but require costly online sampling and often produce low-quality traces in early training. We propose a principled offline reasoning distillation framework that preserves the efficiency and supervision quality of offline teacher-generated data while correcting teacher-student distribution drift. It adaptively emphasizes teacher supervision that is better aligned with the student's on-policy distribution. Evaluations on mathematical reasoning benchmarks of GSM8K, MATH, MATH500, and harder held-out competition-style tasks, including AMC, AIME, and OlympiadBench, show that our method improves reasoning accuracy over prior offline distillation algorithms and yields more stable reasoning traces while preserving instruction-following capabilities. Our work shows that lightweight, distribution-correction-aware training can substantially strengthen offline reasoning distillation without online rollouts.