Multi-Teacher Knowledge Distillation

Follow this topic Ask about this topic

Latest papers 85

29 papers in the last four weeks, up 164% on the four weeks before. 0.2% of all new papers.

All topics

Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning
Sep 14, 2026Xun Xu, Zaixi ZhangRL for Language Model Reasoning Multi-Teacher Knowledge Distillation
1Fudan University · 2Hong Kong University of Science and Technology
Decision Shifts, Lost Label Functionality, and an Inconclusive Grounding Audit in Correctness-Gated Multi-Teacher Distillation
Sep 9, 2026Xiaofei FengLanguage Model Safety Evaluation Multi-Teacher Knowledge Distillation
University of Illinois Urbana-Champaign
Video-MOPD: Multi-Teacher On-Policy Distillation for Video Understanding
Sep 8, 2026Zhenxin Qin, Peng Shi, Cong Han +3Temporal Video Grounding Video Understanding
Tongji University · Bilibili Inc.
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Sep 8, 2026Youngrok Park, Sangmin Bae, Hojung Jung +6Policy Gradient Weak-to-Strong Generalization
KAIST AI · Microsoft · University of Toronto +3
Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs
Sep 2, 2026Xixiang He, Xingming Li, Baiqi Wu +4Multi-Teacher Knowledge Distillation Self-Distillation
National University of Defense Technology · Zhejiang University · Intelligent Game and Decision Lab
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
Aug 30, 2026Huiyi Zhang, Zijian Li, Xiaocheng Feng +4Cross-Tokenizer Knowledge Distillation Cross-Architecture Knowledge Distillation
1Harbin Institute of Technology · 2Peng Cheng Laboratory
Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation
Aug 27, 2026Ziyuan Liu, Jiao Ou, Jian Liang +2Uncertainty-Aware Knowledge Distillation Multi-Teacher Knowledge Distillation
Kuaishou Technology Beijing, China
Socialized Detector Learning: Trajectory-Guided and Reciprocal Distillation for Heterogeneous Object Detectors
Aug 26, 2026Weihao Li, Yunqi Zhu, Zhihe Fan +5Object Detection Knowledge Distillation
School of New Media and Communication, Tianjin University, Tianjin, China · School of Computer Science and Engineering, University of New South Wales, Sydney, Australia · School of Sports Training, Tianjin University of Sport, Tianjin, China +5
D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
Aug 25, 2026Zechen Sun, Zhiwei Zhang, Fei Zhao +7Adaptive Sampling Language Model Distillation
AllSpark Team
IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves
Aug 25, 2026Feyza Yavuz, Mert Bülent Sarıyıldız, Diane LarlusData-Free Knowledge Distillation Multi-Teacher Knowledge Distillation
NAVER LABS Europe
Who Should Teach? Confidence-Aware Dual-Teacher Learning for Few-Shot Node Classification on Text-Attributed Graphs
Aug 22, 2026Hojin Kim, Sujin Yoon, Sungsu Lim +2Text-Attributed Graph Learning Node Classification
Chungbuk National University Cheongju, South Korea · Chungnam National University Daejeon, South Korea · The Pennsylvania State University University Park, PA, USA
Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation
Aug 19, 2026Huan-ang Gao, Haohan Chi, Yong Yan +7Multi-Teacher Knowledge Distillation On-Policy Distillation
SIA-Lab of Tsinghua AIR and ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University
Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models
Aug 17, 2026Zhaoyi Li, Deyang Kong, Yuan Wei +13Language Model Distillation On-Policy Distillation
University of Science and Technology of China · IQuest Research · Peking University +2
REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation
Aug 12, 2026Yang Sun, Lichao Ma, Houyuan Qin +5Multi-Teacher Knowledge Distillation On-Policy Distillation
Shanghai Artificial Intelligence Laboratory · Peking University · Southwest Jiaotong University +5
DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models
Aug 10, 2026Mingfeng Lin, Chengfei Cai, Lin Xu +2Flow Matching RL for Image Generation
Harbin Institute of Technology (Shenzhen) · Zhejiang University · Harbin Institute of Technology
Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Models
Aug 5, 2026Siming Fu, Haojun Xu, Ruizhe He +9Diffusion Model Distillation Multi-Teacher Knowledge Distillation
1Joy Future Academy · 2Zhejiang University · 3Beihang University
Language-Specialized Multi-Teacher On-Policy Distillation for Multilingual LLM-Based ASR
Aug 4, 2026Yuan Xie, Jiaqi Song, Xianliang Wang +3Multilingual Language Models Multi-Teacher Knowledge Distillation
Advanced Intelligent Systems Group, NIO, Beijing, China
MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation
Jul 31, 2026Hang Yan, Zhangxuan GU, Beitong Zhou +5Multi-Teacher Knowledge Distillation LLM Agent Training
1Xi’an Jiaotong University · 2Ant Group · 3Shanghai Jiao Tong University
Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold
Jul 30, 2026Songshuo Lu, Zhi Chen, Yaohua TangRL for Language Model Reasoning Multi-Teacher Knowledge Distillation
Moore Threads AI
Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers
Jul 30, 2026Vincent Ryusuke Takahashi, Yoshinari Takeishi, Jun'ichi Takeuchi +1Adversarial Training Neural Network Robustness
Joint Graduate School of Mathematics for Innovation Kyushu University · Faculty of Information Science and Electrical Engineering Kyushu University · Université Savoie Mont Blanc
OrthKD: Extracting Generalized Clinical Knowledge from Heterogeneous Teachers for Lightweight Deployment
Jul 28, 2026Yi Xu, Cheng Chen, Mufan CaoDiabetic Retinopathy Grading Selective Knowledge Distillation
Tongji University, Shanghai, China
AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition
Jul 28, 2026Yuqi Li, Yi-Cheng Lin, Xianglong Wang +5Relational Knowledge Distillation Speech Emotion Recognition
Department of Electrical Engineering, The City College of New York · National Taiwan University · Wyze Inc. +1
OPOD: On-Policy Omni Distillation
Jul 23, 2026Tong Zhao, Yuyang Hu, Yutao Zhu +5Cross-Modal Knowledge Distillation Unified Multimodal Models
1Renmin University of China · 2Tencent
When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization
Jul 22, 2026Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela +4Selective Knowledge Distillation Text Summarization
BRAC University, Dhaka, Bangladesh
Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models
Jul 16, 2026Mingxi Fu, Jiawen Li, Renao Yan +4Multi-Teacher Knowledge Distillation Multiple Instance Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University · University of Washington · City University of Hong Kong (Dongguan) +2
ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts
Jul 10, 2026Jiawen Li, Tian Guan, Huijuan Shi +5Computational Pathology Unified Multimodal Models
Institute of Biopharmaceutical and Health Engineering, Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Department of Pathology, The First Affiliated Hospital of Sun Yat-sen University, Guangzhou, China · Department of Engineering Science, University of Oxford, Oxford, UK +1
Compete Then Collaborate: Frontier AI Teachers Build a Verifiable Curriculum to Improve a Coding Student Beyond Imitation
Jul 9, 2026Miseong Shawn KimCode Generation Curriculum RL
Genesis Cortex AI Inc.
Diagnosing and Calibrating Tool-Call Boundary Drift in Multi-Teacher On-Policy Distillation
Jul 8, 2026Jiabin Shen, Guang Chen, Chengjun MaoLLM Tool Use Multi-Teacher Knowledge Distillation
UI-MOPD: Multi-Platform On-Policy Distillation for Continual GUI Agent Learning
Jul 5, 2026Niu Lian, Alan Chen, Zhehao Yu +8Continual Learning for LLM Agents GUI Agents
Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen · Zhejiang University +2
H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation
Jul 1, 2026Qixiang Yin, Huanjin Yao, Yuchen Cai +4Cross-Modal Knowledge Distillation VLM Distillation
Beijing University of Posts and Telecommunications · Zhongguancun Academy, Beijing, China · HKUST +2

Sep 14, 2026, cs.AIJ/K move · Enter open · S save

Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning

Xun Xu, Zaixi Zhang

1Fudan University · 2Hong Kong University of Science and Technology

RL for Language Model Reasoning Multi-Teacher Knowledge Distillation On-Policy Distillation Scientific Reasoning in Language Models Knowledge Distillation

Open paper AskPDF arXiv

Multi-teacher on-policy distillation (OPD) is becoming the standard way to integrate specialist capabilities into one model: train experts with RL, then distill them into the student on its own rollouts. Existing recipes assign supervision at the sequence level - each prompt goes to one domain teacher and every token receives the same weight - which implicitly assumes that a teacher is uniformly useful across a response. We find instead that useful teacher signal is sparse and heterogeneous along a reasoning trajectory, which raises a finer question: who should teach which token? Verifier-Gated Multi-Expert On-Policy Distillation (VG-OPD) answers it by verification: the counterfactual gain of an expert on a specific answer criterion licenses that expert to teach, its disagreement with the student localizes the supervision, and criterion importance sets its weight; the gated KL enters GRPO as an additive token-level advantage. Instantiated for scientific reasoning with RL-trained capability experts, VG-OPD attains the best overall performance on seven benchmarks for 4B and 8B students, ranking first on five at both scales, with the largest gains on knowledge-intensive scientific reasoning tasks. Further analysis shows that the gains come from localizing verified supervision rather than from adding teachers or distillation loss: misplacing the same supervision budget is the single most damaging change, and indiscriminate distillation drags RL below its own floor where gated distillation lifts it.