Representation Distillation
Representation distillation is a machine learning technique that transfers knowledge from a complex, high-performing "teacher" model to a simpler, more efficient "student" model. Current research focuses on improving knowledge transfer efficiency through various methods, including information-theoretic approaches, the use of multiple teachers, and incorporating "explanations" from the teacher model beyond simple predictions. This technique is significant because it allows for deploying powerful models on resource-constrained devices and improving the performance of smaller models across diverse applications, such as image recognition, natural language processing, and robotic manipulation.
Papers
September 10, 2024
March 20, 2024
March 1, 2024
October 4, 2023
May 9, 2023
February 11, 2023
January 18, 2022
December 1, 2021