Multi-Teacher Knowledge Distillation

Latest papers 85

29 papers in the last four weeks, up 164% on the four weeks before. 0.2% of all new papers.

Holonic Active Distillation for Scalable Multi-Agent Learning in Multi-Sensor Systems
Jun 30, 2026Dani Manjah, Tim Bary, Benoît Macq +1Multi-Agent Systems Multi-Teacher Knowledge Distillation
Institute of Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM), UCLouvain, 1348 Louvain-la-Neuve, Belgium · Université de Technologie de Belfort Montbéliard, UTBM, CIAD UR 7533, F-90010 Belfort cedex, France
Regime-Aware Peer Specialization for Robust RAG under Heterogeneous Knowledge Conflicts
Jun 29, 2026Bo Wang, Heyan Huang, Yaolin Li +5Retrieval-Augmented Generation Knowledge Conflicts in Language Models
School of Computer Science, Beijing Institute of Technology
MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
Jun 29, 2026Wenhan Ma, Jianyu Wei, Liang Zhao +10Multi-Teacher Knowledge Distillation On-Policy Distillation
Peking University · Xiaomi · LLM Core, Xiaomi +2
Heterogeneous and Adept Snapshot Distillation for 3D Semantic Segmentation
Jun 24, 2026Xiaopei Wu, Yuenan Hou, Junkai Xu +7Cross-Modal Knowledge Distillation 3D Semantic Segmentation
1Shanghai AI Laboratory · 2Zhejiang University · Hangzhou Dianzi University
The Professor: Multi-Teacher Unsupervised Prompt Distillation for Vision-Language Models
Jun 22, 2026Ahmad Algadhi, Ahmed Alzuhair, Omar Alkhulaif +1VLM Distillation Multi-Teacher Knowledge Distillation
Department of Information and Computer Science King Fahd University of Petroleum and Minerals
UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning
Jun 18, 2026Wenhao Chi, Arkaprava Sinha, Dominick Reilly +2Cross-Modal Knowledge Distillation Multi-View Learning
University of North Carolina at Charlotte
HumP-KD: A Hybrid Uncertainty-Aware Multi-Stage Progressive Knowledge Distillation Framework for Efficient Fire Classification
Jun 12, 2026Mohammed Arif Mainuddin, Najifa Tabassum, Omar Ibne Shahid +1Efficient Neural Network Inference Cross-Architecture Knowledge Distillation
Electrical and Computer Engineering North South University Dhaka, Bangladesh
When to Trust, How to Distill: Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecasting
Jun 10, 2026Rupasree Dey, Abdul Matin, Nathan Orwick +3Multivariate Time Series Forecasting Time Series Forecasting
Department of Computer Science Colorado State University Fort Collins, Colorado, USA · Department of Soil and Crop Sciences Colorado State University Fort Collins, Colorado, USA
FedMTFI: Feature Importance Based Optimized Multi Teacher Knowledge Distillation in Heterogeneous Federated Learning Environment
Jun 1, 2026Nazmus Shakib Shadin, Aaron Cummings, Xinyue Zhang +1Non-IID Federated Learning Multi-Teacher Knowledge Distillation
Department of Computer Science, Kennesaw State University, Marietta, GA, 30060 USA
Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors
May 27, 2026Luyang Fang, Yongkai Chen, Jiazhang Cai +2Uncertainty Quantification Uncertainty-Aware Knowledge Distillation
Department of Statistics, University of Georgia · Department of Statistics, Harvard University
Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation
May 26, 2026Tianlei Chen, Jiao Ou, Ziyuan Liu +3Language Model Distillation Multi-Teacher Knowledge Distillation
Kuaishou Technology, Beijing, China
CollectionLoRA: Collecting 50 Effects in 1 LoRA via Multi-Teacher On-Policy Distillation
May 25, 2026Fangtai Wu, Hailong Guo, Shijie Huang +7Diffusion Model Distillation Low-Rank Adaptation
Zhejiang University · Qwen Applications Business Group of Alibaba · Xi’an Jiaotong University
Cross-Paradigm Knowledge Distillation: A Comprehensive Study of Bidirectional Transfer Between Random Forests and Deep Neural Networks for Big Data Applications
May 19, 2026Mahdi Naser MoghadasiMulti-Teacher Knowledge Distillation Knowledge Distillation
BrightMind AI Research Seattle, Washington, USA
Distilling Tabular Foundation Models for Structured Health Data
May 18, 2026Aditya Tanna, Nassim Bouarour, Mohamed Bouadi +2Tabular Foundation Models Efficient Neural Network Inference
Lexsi Labs
Curriculum Learning-Guided Progressive Distillation in Large Language Models
May 11, 2026Jincheng Cao, Fanzhi Zeng, Leqi Liu +1Reasoning Distillation Language Model Distillation
The University of Texas at Austin · Google Research
LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
May 11, 2026Nikolaos Gkalelis, Vasileios MezarisVision-Language Models Efficient VLM Inference
CERTH-ITI, Thessaloniki, Greece, 57001
ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design
May 11, 2026Yulin Zhang, He Cao, Zihao Jiang +6Protein Language Models Multi-Teacher Knowledge Distillation
Tsinghua University · International Digital Economy Academy · Hong Kong University of2026 Science and Technology (Guangzhou) +1
Evolving Knowledge Distillation for Lightweight Neural Machine Translation
May 11, 2026Xuewen Zhang, Haixiao Zhang, Xinlong HuangMulti-Teacher Knowledge Distillation Knowledge Distillation
Department of Content Generation Li Auto Beijing, China
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
May 5, 2026Wenjin Hou, Shangpin Peng, Weinong Wang +13Language Model Distillation Multi-Teacher Knowledge Distillation
1Zhejiang University · 2Shenzhen Loop Area Institute · 3LLM Department, Tencent
Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding
May 4, 2026Taewon Yun, Jisu Shin, Jeonghwan Choi +2CoT Distillation Efficient Language Model Reasoning
KAIST · UNIST
MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate
May 2, 2026Jianze Wang, Ying Liu, Jinlong Chen +7Multi-Agent Debate Multi-Teacher Knowledge Distillation
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · Alibaba Group
Co-Evolving Policy Distillation
Apr 29, 2026Naibin Gu, Chenxu Yang, Qingyi Si +7Reinforcement Learning with Verifiable Rewards Multi-Teacher Knowledge Distillation
Institute of Information Engineering, CAS, 2School of Cyber Security, UCAS, 3JD.COM
Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation
Oct 13, 2025Hengyuan Zhang, Shiping Yang, Xiao Liang +8Synthetic Data Generation Language Model Distillation
The University of Hong Kong · Simon Fraser University · University of California, Los Angeles +3
Collaborative Knowledge Distillation via a Learning-by-Education Node Community
Sep 30, 2024Anestis Kaimakamidis, Ioannis Mademlis, Ioannis PitasContinual Learning Decentralized Learning
Aristotle University of Thessaloniki, Thessaloniki, 54124, Greece
DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models
Date pendingQuanhao Li, Junqiu Yu, Kaixun Jiang +7Diffusion Model Distillation Multi-Task RL
Fudan University, China · Alibaba Group, China

Jun 30, 2026, cs.MAJ/K move · Enter open · S save

Holonic Active Distillation for Scalable Multi-Agent Learning in Multi-Sensor Systems

Dani Manjah, Tim Bary, Benoît Macq, Stéphane Galland

Institute of Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM), UCLouvain, 1348 Louvain-la-Neuve, Belgium · Université de Technologie de Belfort Montbéliard, UTBM, CIAD UR 7533, F-90010 Belfort cedex, France

Multi-Agent Systems Multi-Teacher Knowledge Distillation Knowledge Distillation

Open paper AskPDF arXiv

The rapid expansion of sensor-based networks introduces major challenges in scalability, adaptability, and knowledge transfer, especially in open environments where new subsystems can dynamically join or leave. In this work, we propose a Holonic Active Distillation architecture within a Holonic Multi-Agent System (HMAS) to address these issues. Our approach integrates Clustered Stream-Based Active Distillation (CSBAD), a framework in which specialized student models collect local data, query pseudo-labels from teacher models, and cluster into groups of similar sensors. Results show that the holonic organization balances local specialization with global generalization, while efficiently adapting to sensor departures and re-integrations. We also analyzed trade-offs among incremental model updates, system reorganization, and scalability limits. Our findings highlight the advantages of holonic learning for multi-sensor systems while identifying key challenges related to model drift and long-term adaptation.