Multi-Teacher Knowledge Distillation

Latest papers 84

All topics
CardsList
  1. Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts

    Oct 7, 2026Hejian Sang, Zhengze Zhou, Shayan Mohajer Hamidi +3Multi-Teacher Knowledge DistillationOn-Policy Distillation

  2. From Probabilities to Decisions: Search and Multi-Teacher Distillation with Jev

    Oct 6, 2026Mohamad Yazan Sadoun, Sarah Sharif, Yaser Mike BanadLanguage Model DistillationMulti-Teacher Knowledge Distillation

  3. UP-MOPD: Update Projection in Multi-Teacher On-Policy Distillation

    Oct 6, 2026Taojie Zhu, Jing Jin, Yuan Xia +8Language Model DistillationGradient Projection

  4. Progressive Multi-Ancestor Bit-Depth Distillation

    Oct 2, 2026Adil Mubashir Chaudhry, Osama Ahmad, Zubair Khalid +1Model CompressionNeural Network Quantization

  5. GRAFT: Growing Agglomerative Foundation Models via Continual Teacher Distillation

    Oct 1, 2026Zhenghao Zhao, Chi Zhang, Qingshuang Chen +1Multi-Teacher Knowledge DistillationVision Foundation Models

  6. Latent-MOPD: Latent Multi-Teacher On-Policy Distillation

    Oct 1, 2026Zhengyu Fang, Seoyeon Hong, Jie Yang +4Language Model DistillationMulti-Teacher Knowledge Distillation

  7. Lexicographic Multi-Objective On-Policy Distillation

    Oct 1, 2026Doseok Jang, Jon Ander Campos, Youran QiMulti-Objective Reinforcement LearningMulti-Objective Optimization

  8. Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservation

    Oct 1, 2026Xiaofei Yin, Tong Chu, Jiyuan Fu +3On-Policy DistillationMulti-Teacher Knowledge Distillation

  9. From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

    Oct 1, 2026Siqi Zhu, Suozhi Huang, Kaixuan Zhang +4Policy GradientLanguage Model Distillation

  10. Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning

    Sep 30, 2026Hongwei Zhao, Rui Liu, Yansong Liu +2Class-Incremental LearningLow-Rank Adaptation

  11. Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives

    Sep 29, 2026Qiwei Di, Xuheng Li, Kaixuan Ji +3Multi-Teacher Knowledge DistillationOn-Policy Distillation

  12. Med-RADIO: Reducing All Medical Domains Into One via Multi-Teacher Distillation

    Sep 29, 2026Chu Zhang, Haoyu Jiang, Hongyuan Zhang +2Medical Imaging Foundation ModelsMedical Image Classification

  13. LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation

    Sep 29, 2026Jaeyun Shin, Hangeol Chang, Jong Chul YeVLM DistillationMulti-Teacher Knowledge Distillation

  14. Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

    Sep 28, 2026Xin Li, Hao Jiang, Xin Gao +6Language Model DistillationMulti-Teacher Knowledge Distillation

  15. CapField-OPD: Learning Continuous Capability Fields via Joint-Anchored Multi-Teacher On-Policy Distillation for Flow Models

    Sep 28, 2026Pengyang Ling, Jiazi Bu, Yujie Zhou +6Generative Flow NetworksMulti-Teacher Knowledge Distillation

  16. ChemOPD: Multi-Teacher On-Policy Distillation for Multi-Task Chemical Reasoning

    Sep 27, 2026Yaoyao Xu, Xinjian Zhao, Xiaozhuang Song +2Reasoning DistillationMulti-Teacher Knowledge Distillation

  17. Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

    Sep 27, 2026Wenze Lin, Jiyuan Long, Jiale Zhao +11Data-Free Knowledge DistillationLanguage Model Distillation

  18. SoFT: Soft Targets for Generalizable LLM Fine-Tuning

    Sep 26, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +7LLM Fine-TuningFine-Tuning

  19. MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation

    Sep 25, 2026Tianze Xu, Yanzhao Zheng, Zhentao Zhang +10Multi-Teacher Knowledge DistillationOn-Policy Distillation

  20. Distillation for Efficient Multitask Manipulation Policies via Conditional Flow Matching

    Sep 23, 2026Shreya Deshmukh, Imen Mahdi, Nick Heppert +1Conditional Flow MatchingMulti-Task Robotic Manipulation

  21. DTKDP: A Dual Teacher Knowledge Distillation and Pruning Framework for Lightweight Oriented SAR Ship Detection

    Sep 21, 2026Yuming Li, Fan Zhang, Alin M. AchimMulti-Teacher Knowledge DistillationObject Detection

  22. ACLArena: Agent Continue Learning in Multi-stage Post-training

    Sep 21, 2026Haixin Wang, Xiaoxuan Wang, Junkai Zhang +9Continual Learning for LLM AgentsRL Post-Training

  23. Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation

    Sep 20, 2026Jie Sun, Mao Zheng, Mingyang Song +8Language Model DistillationMulti-Teacher Knowledge Distillation

  24. Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding

    Sep 16, 2026Benedikt Mayrhofer, Enrique Orozco Olivares, Franz Pernkopf +2Audio Representation LearningStreaming ASR