Teacher Student
Teacher-student learning frameworks are a burgeoning area of research focusing on improving the efficiency and performance of machine learning models, particularly large language models (LLMs). Current research emphasizes techniques like knowledge distillation, where a complex "teacher" model trains a smaller, faster "student" model, and curriculum learning, which strategically sequences training data. These methods are proving valuable across diverse applications, including educational technology (simulating classrooms, generating teacher-student dialogues), visualizations, and various signal processing tasks (speech separation, acoustic scene classification), by enabling efficient model deployment and improved accuracy with limited data.
Papers
Semi-Supervised Learning of Dynamical Systems with Neural Ordinary Differential Equations: A Teacher-Student Model Approach
Yu Wang, Yuxuan Yin, Karthik Somayaji Nanjangud Suryanarayana, Jan Drgona, Malachi Schram, Mahantesh Halappanavar, Frank Liu, Peng Li
How a student becomes a teacher: learning and forgetting through Spectral methods
Lorenzo Giambagli, Lorenzo Buffoni, Lorenzo Chicchi, Duccio Fanelli