Multi-Teacher Knowledge Distillation

Follow this topic Ask about this topic

Multi-teacher knowledge distillation studies how to transfer complementary knowledge from several models into a smaller student or a single model with broader capabilities. Common approaches combine teacher outputs or features, route examples or tokens to reliable specialists, and progressively or collaboratively distill teachers; on-policy methods give feedback on the student’s own generated responses or actions. The methods are used in settings from language-model reasoning and multimodal agents to vision, speech, healthcare, and scientific prediction, with evaluation on task benchmarks and measures of efficiency or retained capability. A central challenge is balancing complementary expertise against unreliable or conflicting guidance, uneven teacher influence, and the training cost of more elaborate methods.

Latest papers

108 papers34 papers in the last four weeks, up 143% on the four weeks before. 0.2% of all new papers.

Curriculum Learning-Guided Progressive Distillation in Large Language Models
May 11, 2026Jincheng Cao, Fanzhi Zeng, Leqi Liu +1Reasoning Distillation Language Model Distillation
The University of Texas at Austin · Google Research
LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
May 11, 2026Nikolaos Gkalelis, Vasileios MezarisVision-Language Models Efficient VLM Inference
CERTH-ITI, Thessaloniki, Greece, 57001
ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design
May 11, 2026Yulin Zhang, He Cao, Zihao Jiang +6Protein Language Models Multi-Teacher Knowledge Distillation
Tsinghua University · International Digital Economy Academy · Hong Kong University of2026 Science and Technology (Guangzhou) +1
Evolving Knowledge Distillation for Lightweight Neural Machine Translation
May 11, 2026Xuewen Zhang, Haixiao Zhang, Xinlong HuangMulti-Teacher Knowledge Distillation Knowledge Distillation
Department of Content Generation Li Auto Beijing, China
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
May 5, 2026Wenjin Hou, Shangpin Peng, Weinong Wang +13Language Model Distillation Multi-Teacher Knowledge Distillation
1Zhejiang University · 2Shenzhen Loop Area Institute · 3LLM Department, Tencent
Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding
May 4, 2026Taewon Yun, Jisu Shin, Jeonghwan Choi +2CoT Distillation Efficient Language Model Reasoning
KAIST · UNIST
MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate
May 2, 2026Jianze Wang, Ying Liu, Jinlong Chen +7Multi-Agent Debate Multi-Teacher Knowledge Distillation
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · Alibaba Group
Co-Evolving Policy Distillation
Apr 29, 2026Naibin Gu, Chenxu Yang, Qingyi Si +7Reinforcement Learning with Verifiable Rewards Multi-Teacher Knowledge Distillation
Institute of Information Engineering, CAS, 2School of Cyber Security, UCAS, 3JD.COM
Reliability Gated Multi-Teacher Distillation for Low Resource Abstractive Summarization
Apr 3, 2026Dipto Sumit, Ankan Kumar Roy, Sadia Khair Rodela +4Text Summarization Multi-Teacher Knowledge Distillation
BRAC University
Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement
Mar 25, 2026Xin Zhang, Jianyang Xu, Hao Peng +5Knowledge Distillation Multi-Teacher Knowledge Distillation
Hangzhou Dianzi University
Efficient Universal Perception Encoder
Mar 23, 2026Chenchen Zhu, Saksham Suri, Cijo Jose +8Visual Representation Learning Vision Foundation Models
Meta Reality Labs · FAIR at Meta
CTFS : Collaborative Teacher Framework for Forward-Looking Sonar Image Semantic Segmentation with Extremely Limited Labels
Mar 22, 2026Ping Guo, Chengzhou Li, Guanchen Meng +6Image Segmentation Semantic Segmentation
School of Software Technology, Dalian University of Technology, Dalian, China
Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understanding
Mar 12, 2026Songlin Li, Xin Zhu, Zechao Guan +2VLM Distillation Knowledge Distillation
University of Electronic Science and Technology of China · Robotics Center, XPeng Motors
Standing on the Shoulders of Giants: Rethinking EEG Foundation Model Pretraining via Multi-Teacher Distillation
Mar 4, 2026Chenqi Li, Yu Liu, Shuo Zhang +2EEG Foundation Models Multi-Teacher Knowledge Distillation
Department of Engineering Science, University of Oxford.
Brewing Stronger Features: Dual-Teacher Distillation for Multispectral Earth Observation
Feb 23, 2026Filip Wolf, Blaž Rolih, Luka Čehovin ZajcRemote Sensing Image Understanding Visual Representation Learning
University of Ljubljana, Faculty of Computer and Information Science, Slovenia
LLM-Guided Knowledge Distillation for Temporal Knowledge Graph Reasoning
Feb 16, 2026Wang Xing, Wei Song, Siyu Lin +2Knowledge Distillation Temporal KGs
School of Computer Science and Technology, Xidian University · School of Computing and Artificial Intelligence, Southwest Jiaotong University · School of Information Science and Engineering, Chongqing Jiaotong University
Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation
Oct 13, 2025Hengyuan Zhang, Shiping Yang, Xiao Liang +8Synthetic Data Generation Language Model Distillation
The University of Hong Kong · Simon Fraser University · University of California, Los Angeles +3
Collaborative Knowledge Distillation via a Learning-by-Education Node Community
Sep 30, 2024Anestis Kaimakamidis, Ioannis Mademlis, Ioannis PitasOnline Knowledge Distillation Continual Learning
Aristotle University of Thessaloniki, Thessaloniki, 54124, Greece

May 11, 2026J/K move · Enter open · S save

Curriculum Learning-Guided Progressive Distillation in Large Language Models

Jincheng Cao, Fanzhi Zeng, Leqi Liu, Aryan Mokhtari

The University of Texas at Austin · Google Research

Reasoning Distillation Language Model Distillation Multi-Teacher Knowledge Distillation Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation is a key technique for transferring the capabilities of large language models (LLMs) into smaller, more efficient student models. Existing distillation approaches often overlook two critical factors: the learning order of training data and the capacity mismatch between teacher and student models. This oversight limits distillation performance, as manifested by the counter-intuitive phenomenon where stronger teachers fail to produce better students. In this work, we propose Curriculum Learning-Guided Progressive Distillation (CLPD), a unified framework that explicitly accounts for both factors by aligning data difficulty with teacher strength. CLPD constructs an explicit curriculum by organizing training examples from easy to hard, while simultaneously applying an implicit curriculum over supervision signals by progressively scheduling teachers of increasing capacity. Our framework is modular and can be integrated into standard distillation algorithms with minimal overhead. Empirical results on the reasoning benchmarks demonstrate that CLPD consistently outperforms standard distillation, data ordering alone, and teacher scheduling alone across multiple settings. These findings highlight the importance of jointly considering data ordering and teacher capacity when distilling reasoning abilities into small language models.