Knowledge Distillation
Knowledge distillation is a machine learning technique that transfers knowledge from a large, complex "teacher" model to a smaller, more efficient "student" model, aiming to improve the student's performance and reduce computational costs. Current research focuses on improving distillation methods for various model architectures, including convolutional neural networks, transformers, and large language models, often incorporating techniques like parameter-efficient fine-tuning, multi-task learning, and data augmentation to enhance knowledge transfer. This approach is significant because it enables the deployment of high-performing models on resource-constrained devices and addresses challenges related to model size, training time, and privacy in diverse applications such as image captioning, speech processing, and medical diagnosis.
Papers - Page 5
CleanPose: Category-Level Object Pose Estimation via Causal Learning and Knowledge Distillation
Xiao Lin, Yun Peng, Liuyi Wang, Xianyou Zhong, Minghao Zhu, Jingwei Yang, Chengju Liu, Qijun ChenA Framework for Double-Blind Federated Adaptation of Foundation Models
Nurbek Tastan, Karthik NandakumarMIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks
Alejandro Guerra-Manzanares, Farah E. Shamout
FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation
Wenzheng Jiang, Ji Wang, Xiongtao Zhang, Weidong Bao, Cheston Tan, Flint Xiaofeng FanRole of Mixup in Topological Persistence Based Knowledge Distillation for Wearable Sensor Data
Eun Som Jeon, Hongjun Choi, Matthew P. Buman, Pavan Turaga