Mutual Distillation
Mutual distillation is a machine learning technique focused on improving model efficiency and robustness by transferring knowledge between multiple models. Current research emphasizes its application in diverse areas, including model compression (e.g., pruning and knowledge distillation of large language and vision models), federated learning (addressing data heterogeneity and bias), and dataset condensation (creating smaller, representative datasets). This approach offers significant potential for reducing computational costs, enhancing model generalization, and improving the privacy and security of machine learning systems across various applications.
Papers
Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers
Diana-Nicoleta Grigore, Mariana-Iuliana Georgescu, Jon Alvarez Justo, Tor Johansen, Andreea Iuliana Ionescu, Radu Tudor Ionescu
FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning
Changlin Song, Divya Saxena, Jiannong Cao, Yuqing Zhao