Language Model Distillation

Follow this topic Ask about this topic

Momentum

69 papers in the last four weeks, up 200% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 350

All topics

K2P: Label-Free Knowledge to Prompt Distillation
Sep 30, 2026Yingchuan Zhang, Haoran Lu, Wenxuan Zhong +1Prompt Learning Data-Free Knowledge Distillation
Department of Statistics University of Georgia Athens, GA 30602
Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
Sep 30, 2026Ilgee Hong, Changlong Yu, Zhenghao Xu +5LLM-as-a-Judge Language Model Distillation
Georgia Institute of Technology · Amazon · UC San Diego
On the Off-Policy Teacher in On-Policy Distillation
Sep 29, 2026Langlin Huang, Hao Liu, Mononito Goswami +5Language Model Distillation On-Policy Distillation
Washington University in St. Louis · AWS AI Labs · Carnegie Mellon University +1
Activation-Conditioned Self-Distillation
Sep 29, 2026Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati +2Numerical Reasoning in Language Models On-Policy Self-Distillation
Rensselaer Polytechnic Institute · IBM Research
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
Sep 29, 2026Zhenyu Wang, Tianze Wang, Linjun Zhang +1Language Model Distillation Bilevel Optimization
Department of Statistics, Rutgers University
From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation
Sep 29, 2026Yuhao Wang, Ruiyang Ren, Yinan Zhang +3Language Model Distillation On-Policy Distillation
Nanyang Technological University, Singapore · Baidu Inc.
Beyond Prompt Count: How Data Shapes Transfer in On-Policy Distillation
Sep 29, 2026Jiaxuan Wang, Jiafei Lyu, Yuchen Cai +8Training Data Selection Language Model Distillation
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · LLM Department, Tencent +1
Interpolated Policy Distillation: A Controllable Continuum Between Off-Policy and On-Policy Distillation
Sep 29, 2026Youxu Shi, Yifan Sun, Dacheng Yin +5Language Model Distillation On-Policy Distillation
University of Science and Technology of China · WeChat Vision, Tencent Inc.
Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
Sep 29, 2026Zheng Zhang, Xinyue Tan, Lufei Li +3On-Policy Self-Distillation Language Model Distillation
School of Information Science and Technology, ShanghaiTech University · State Key Laboratory of General Artificial Intelligence, BIGAI · Hefei University of Technology
Beyond Compression: Diagnosing How Post-Training Changes Mathematical Reasoning
Sep 29, 2026Hongyang Li, Yiming Zhu, Xiao Li +3Language Model Distillation RL Post-Training
University of Luxembourg · Seafill Open-Source Community · Université Paris-Saclay
Learning from Think-Mode Advantage via On-Policy Distillation
Sep 29, 2026Wanqi Ren, Jianxiang Wang, Danxuan Liu +2CoT Distillation Language Model Distillation
ByteDance, China
Beam Search as Test-Time Self-Distillation via Counterfactual Contexts
Sep 29, 2026Su Ee Tan, Xiaotong Ji, Rasul Tutunov +2Language Model Distillation Self-Distillation
Huawei Noah’s Ark Lab · UCL Centre for AI
Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillation
Sep 29, 2026Rui Wang, Ruijie Wang, Bo Chen +2On-Policy Self-Distillation Selective Knowledge Distillation
Peking University · University of Oxford · The University of Hong Kong
SAKI: Maximal-Coupling-Routed Teacher Supervision for On-Policy Distillation
Sep 29, 2026Miteto Wei, Xiaohan Wang, Zehao Chen +7Language Model Distillation On-Policy Distillation
Meituan · KTH Royal Institute of Technology
Interactive-Policy Distillation with Bidirectional Propose-and-Verify
Sep 29, 2026Shutong Wu, Xiwen Chen, Brendan Rappazzo +4Language Model Distillation On-Policy Distillation
Department of Computer Sciences, University of Wisconsin–Madison · Machine Learning Research, Morgan Stanley · Department of Electrical and Computer Engineering, Rutgers University–New Brunswick
Learning from Teacher Continuations at Student States
Sep 28, 2026Haojin Wang, Dylan Zhang, Huaibo Chen +8Language Model Distillation On-Policy Distillation
University of Illinois at Urbana-Champaign · Massachusetts Institute of Technology · University of Pennsylvania +2
Distillation Defenses Easily Break After Reinforcement Learning
Sep 28, 2026Shidan Javaheri, Alexander Panfilov, Oliver Britton +2LLM Security Language Model Distillation
University of Oxford · ELLIS Institute Tübingen & MPI for Intelligent Systems
Reward-Aligned Reweighting for On-Policy Distillation
Sep 28, 2026Haofeng Xu, Junwei Su, Lansong Diao +2Language Model Distillation On-Policy Distillation
The University of Hong Kong · Alibaba Group · University of Science and Technology of China
An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning
Sep 28, 2026Shangzhe Li, Yuxiao Yang, Tianrun Yu +4RL for Language Model Reasoning Language Model Distillation
University of North Carolina at Chapel Hill · Brigham Young University · NVIDIA
Inductive Feedback for Mixed-Policy Distillation
Sep 28, 2026Amir Moeini, Huaijiang Zhu, Daniel Havir +1Language Model Distillation On-Policy Distillation
Alquist Robotics · University of Virginia
Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
Sep 28, 2026Xin Li, Hao Jiang, Xin Gao +6Language Model Distillation Multi-Teacher Knowledge Distillation
Nanyang Technological University · Yale University · University of Manchester
Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agents
Sep 28, 2026Tong Zhang, Zhou Liu, Yihao Liu +8Language Model Distillation On-Policy Distillation
Qwen Large Model Application Team, Alibaba · Peking University · University of Chinese Academy of Sciences +1
On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics
Sep 28, 2026Julianna Piskorz, Antonin Berthon, Mihaela van der SchaarLanguage Model Distillation On-Policy Distillation
University of Cambridge Cambridge, UK
No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation
Sep 28, 2026Seonghyeon Kim, Chaeyun Jang, Noah Lee +2Task Arithmetic Language Model Distillation
KAIST AI · Kakao
PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation
Sep 28, 2026Youzhi Liu, Ruobing Zheng, Boyuan Tong +5Gradient Interference Multi-Task Learning
Ant Group
Unbiased Top- $k$ Estimation for On-Policy Distillation
Sep 28, 2026Linjian Meng, Siyuan Gan, YuHan Li +5Language Model Distillation On-Policy Distillation
Shanghai Artificial Intelligence Laboratory, Shanghai, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China
Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation
Sep 28, 2026Yongliang Miao, Shuang Liu, Yanguang Liu +2Language Model Distillation On-Policy Distillation
The Chinese University of Hong Kong, Shenzhen · Carnegie Mellon University · New Jersey Institute of Technology +1
DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
Sep 28, 2026Julian Boesch, Andrew Wee, Alexander StranzlIn-Context Retrieval Recurrent Transformers
Purdue University · Obit Research · State University of New York at Stony Brook.
Dual-Vocabulary Language Model for Cross-Tokenizer Distillation
Sep 27, 2026Kedi Chen, Chen Lin, Yutao Sun +1Cross-Tokenizer Knowledge Distillation Language Model Distillation
East China Normal University · Shanghai Innovation Institute · Tsinghua University
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Sep 27, 2026Wenze Lin, Jiyuan Long, Jiale Zhao +11Data-Free Knowledge Distillation Language Model Distillation
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +3

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

K2P: Label-Free Knowledge to Prompt Distillation

Yingchuan Zhang, Haoran Lu, Wenxuan Zhong, Ping Ma

Department of Statistics University of Georgia Athens, GA 30602

Prompt Learning Data-Free Knowledge Distillation Language Model Distillation Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation can transfer reasoning from stronger teachers to frozen students through reusable prompts, but avoiding weight updates does not eliminate supervision. Without ground-truth answers, teacher solutions are unverified, and agreement with the teacher can reward shared mistakes. We introduce Knowledge-to-Prompt (K2P) for label-free knowledge distillation to prompts. K2P synthesizes reusable instructions from teacher solutions, refines them using paired teacher and student responses, and guides search and selection with answer agreement. It retains candidates that adaptive search may undervalue and selects on reserved questions. Deployment uses only the frozen student and selected prompt. Our theory separates generation and selection gaps and gives conditions under which agreement-guided construction yields accuracy guarantees despite imperfect teacher references. Across reasoning tasks and students, K2P outperforms label-free alternatives overall and remains competitive with supervised prompt optimization. Ablations and archive diagnostics assess the contributions of teacher solutions and refinement, while revealing the limits of agreement-guided selection.

Momentum

Related topics

Latest papers 350