Language Model Distillation

Follow this topic Ask about this topic

Momentum

69 papers in the last four weeks, up 200% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 350

All topics

Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization
May 28, 2026Junlin He, Yihong Tang, Tong Nie +5LLM Pruning LLM Compression
The Hong Kong Polytechnic University, Hong Kong SAR, China · McGill University, Montreal, QC, Canada
Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibility
May 28, 2026Jiahao Huang, Fei Cheng, Junfeng Jiang +1Reasoning Distillation Training Data Selection
University of Tokyo · Kyoto University · National Institute of Informatics
Apertus LLM Family Expansion via Distillation and Quantization
May 27, 2026Andrei Panferov, Davit Melikidze, Martin Jaggi +1LLM Quantization LLM Compression
ISTA · EPFL · ETH Zürich +1
ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation
May 27, 2026Kun Liang, Chenming Tang, Clive Bai +3Language Model Distillation On-Policy Distillation
School of Computer Science, Peking University · 2National Key Laboratory for Multimedia Information Processing, Peking University · 3LLM Department, Tencent
Context Distillation as Latent Memory Management
May 27, 2026Ziyang Zheng, Zeju Li, Xiangyu Wen +5Memory-Augmented Language Models Language Model Distillation
The Chinese University of Hong Kong · Huawei Noah’s Ark Lab
Beyond Imitation: Reflective On-Policy Self-Distillation for LLM Reasoning
May 27, 2026Ziqi Zhao, Xinyu Ma, Liu Yang +6On-Policy Self-Distillation Language Model Distillation
The Hong Kong Polytechnic University · Baidu Inc. · Shandong University +1
Gradient Transformer: Learning to Generate Updates for LLMs
May 26, 2026Binh-Nguyen Nguyen, Khang Tran, NhatHai Phan +1Privacy-Preserving Language Models LLM Fine-Tuning
Department of Data Science, New Jersey Institute of Technology, Newark, NJ, USA · Qatar Computing Research Institute, HBKU, Doha, Qatar
Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generation
May 26, 2026Ning Wu, Rui Liu, Xinkun Lin +5Medical Report Generation Long-Form Document Generation
UNSW Sydney, Sydney, Australia · University of Technology Sydney, Sydney, Australia · School of Computer Science and Engineering, Sun Yat-sen University, China +2
MAIGO: Mitigating Lost-in-Conversation with History-Cleaned On-Policy Self-Distillation
May 26, 2026Haoyu Zheng, Yun Zhu, Shu Yuan +5Instruction Following Data Contamination in Language Models
1Zhejiang University · 2Shanghai AI Laboratory · 3Huazhong University of Science and Technology +2
Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation
May 26, 2026Tianlei Chen, Jiao Ou, Ziyuan Liu +3Language Model Distillation Multi-Teacher Knowledge Distillation
Kuaishou Technology, Beijing, China
Less is More: Early Stopping Rollout for On-Policy Distillation
May 26, 2026Zhou Ziheng, Jiaqi Li, Huacong Tang +2Language Model Distillation On-Policy Distillation
University of California, Los Angeles · Beijing Institute of General Artificial Intelligence
When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection
May 26, 2026Zhengyu Hu, Zheyuan Xiao, Linxin Song +10Language Model Distillation LLM Post-Training
University of Washington · University of Texas at Austin · University of Southern California +6
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
May 26, 2026Yuanyi Wang, Su Lu, Yanggan Gu +6Selective Knowledge Distillation Language Model Distillation
The Hong Kong Polytechnic University, PolyU · Hong Kong Polytechnic University Daya Bay Technology and Innovation Research Institute
EmoDistill: Offline Emotion Skill Distillation for Language Model Agents in Adversarial Negotiation
May 26, 2026Yunbo Long, Haolang Zhao, Lukas Beckenbauer +2Multi-Agent Negotiation Automated Negotiation
University of Cambridge · Technical University of Munich · Exiger LLC +1
The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works
May 25, 2026Guanghui Wang, Kaiwen Lv Kacuila, Zhiyong Yang +5LLM Compression Language Model Distillation
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Alibaba Group, Hangzhou, China · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China +2
Extreme Region Policy Distillation
May 25, 2026Changyu Chen, Xiting Wang, Rui YanKL-Regularized RL RL for Language Model Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · Wuhan University
Decision-Making with Lightweight Confidence-Aware Language Model for Autonomous Driving
May 25, 2026Ruoyu Yao, Ruiguo Zhong, Pei Liu +3Confidence Estimation in Language Models Language Model Distillation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China
Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
May 23, 2026Tianlang Chen, Shirley Wu, Jure LeskovecLanguage Model Distillation Teacher-Student Learning
Stanford
Strong Teacher Not Needed? On Distillation in LLM Pretraining
May 22, 2026Taiming Lu, Zhuang LiuLanguage Model Pretraining Language Model Distillation
Princeton University
The Distillation Game: Adaptive Evaluations & Efficient Defenses
May 21, 2026Youssef Allouah, Mahdi Haghifam, Sanmi Koyejo +1Model Extraction Attacks Adversarial Attacks
Toyota Technological Institute at Chicago · National University of Singapore
Self-Policy Distillation via Capability-Selective Subspace Projection
May 21, 2026Guangya Hao, Yitong Shang, Yunbo Long +2Self-Training for Language Models Language Model Distillation
University of Cambridge · 2HKUST · University of Chicago
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
May 20, 2026Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning
May 20, 2026Xiaogeng Liu, Xinyan Wang, Yingzi Ma +2CoT Distillation On-Policy Self-Distillation
1Johns Hopkins University · University of Wisconsin–Madison · 3Nanyang Technological University
Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline
May 20, 2026Tony Lee, Percy LiangSelf-Training for Language Models Language Model Self-Assessment
Stanford University
AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals
May 20, 2026Duy Nguyen, Hanqi Xiao, Archiki Prasad +7On-Policy Self-Distillation Language Model Distillation
UNC Chapel Hill · Capital One · The University of Texas at Austin
Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
May 19, 2026Bing Wang, Shaotian Yan, Chen Shen +7CoT Distillation Language Model Distillation
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin University · Tongyi Lab, Alibaba Group +3
AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment
May 18, 2026Zhenlin Wei, Pu Jian, Yingzhuo Deng +6Token-Level Credit Assignment RL for Language Model Reasoning
Meituan Beijing, China · University of Chinese Academy of Sciences
FedSDR: Federated Self-Distillation with Rectification
May 18, 2026Ziheng Ren, Zhanming Shen, Hao Wang +2LLM Fine-Tuning Hallucination in Language Models
Beijing University of Aeronautics and Astronautics · Zhejiang University · Stevens Institute of Technology +1
$\boldsymbol{f}$ -OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control
May 18, 2026Xianwei Chen, Shimin Zhang, Jibin WuLanguage Model Distillation On-Policy Distillation
The Hong Kong Polytechnic University
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
May 15, 2026Shuaiyi Li, Zhisong Zhang, Yan Wang +5Long-Context Modeling LLM Inference Acceleration
The Chinese University of Hong Kong · City University of Hong Kong · Tencent +2

May 28, 2026, cs.CLJ/K move · Enter open · S save

Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization

Junlin He, Yihong Tang, Tong Nie, Guilong Li+4

The Hong Kong Polytechnic University, Hong Kong SAR, China · McGill University, Montreal, QC, Canada

LLM Pruning LLM Compression Language Model Distillation LLM Reasoning

Open paper AskPDF arXiv

Efficient Distillation (EDistill) compresses large language models (LLMs) by structured pruning parameters and tuning lightweight modules with high training efficiency. Although these EDistilled LLMs achieve state-of-the-art (SOTA) performance on general ability benchmarks relative to similarly sized LLMs, we identify a severe degradation in their multi-step reasoning ability, which we term reasoning collapse. We systematically analyze the geometric origins of reasoning collapse and show that the SOTA EDistill method based on width-reducing projection matrices suffers from eRank collapse, in which the effective rank (eRank) of hidden representations drops. We theoretically explain how singular values of randomly initialized projection matrices become unevenly distributed, leading to eRank collapse and thus token indistinguishability. To address this issue, we propose RED (Reasoning-preserved Efficient Distillation) for LLMs, which introduces activation-aware initialization to initialize projection matrices as channel-selection matrices, thus theoretically mitigating eRank collapse. Experiments on Llama and Qwen series demonstrate that RED substantially recovers reasoning while maintaining high training efficiency and SOTA general ability.

Momentum

Related topics

Latest papers 350