Multi-Teacher Knowledge Distillation

Multi-teacher knowledge distillation studies how to transfer complementary knowledge from several models into a smaller student or a single model with broader capabilities. Common approaches combine teacher outputs or features, route examples or tokens to reliable specialists, and progressively or collaboratively distill teachers; on-policy methods give feedback on the student’s own generated responses or actions. The methods are used in settings from language-model reasoning and multimodal agents to vision, speech, healthcare, and scientific prediction, with evaluation on task benchmarks and measures of efficiency or retained capability. A central challenge is balancing complementary expertise against unreliable or conflicting guidance, uneven teacher influence, and the training cost of more elaborate methods.