Representation Distillation

Representation distillation is a machine learning technique that transfers knowledge from a complex, high-performing "teacher" model to a simpler, more efficient "student" model. Current research focuses on improving knowledge transfer efficiency through various methods, including information-theoretic approaches, the use of multiple teachers, and incorporating "explanations" from the teacher model beyond simple predictions. This technique is significant because it allows for deploying powerful models on resource-constrained devices and improving the performance of smaller models across diverse applications, such as image recognition, natural language processing, and robotic manipulation.

Papers