Teacher Model
Teacher models are large, pre-trained models used in knowledge distillation to train smaller, more efficient student models while preserving performance. Current research focuses on improving the accuracy and efficiency of this knowledge transfer, exploring techniques like data augmentation, loss function optimization (e.g., MSE loss), and novel architectures such as multi-teacher and online distillation frameworks. This work is significant because it addresses the computational cost and resource limitations associated with deploying large language and vision models, enabling broader accessibility and application in various fields including object detection, natural language processing, and ecological monitoring.
63papers
Papers
April 2, 2025
February 10, 2025
January 29, 2025
December 13, 2024
Multi-Surrogate-Teacher Assistance for Representation Alignment in Fingerprint-based Indoor Localization
Son Minh Nguyen, Linh Duy Tran, Duc Viet Le, Paul J.M HavingaCan Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias
Jianhua Zhang, Yi Gao, Ruyu Liu, Xu Cheng, Houxiang Zhang, Shengyong Chen
November 19, 2024
October 27, 2024
October 7, 2024