Feature-Based Knowledge Distillation
Overview
Feature-based knowledge distillation trains a compact student to learn from a teacher’s internal representations, adding information beyond the teacher’s final predictions. Common approaches align intermediate activations or attention maps, or transfer relationships between examples, selected tokens, channels, or representation subspaces; some combine these signals with logit matching. The methods are used in settings including image recognition and segmentation, object detection, language and speech models, graph learning, and robotics, where they can reduce inference costs or transfer knowledge across domains and model types. Results depend strongly on student capacity, architecture, and task: extra feature supervision can improve data-scarce or cross-domain learning, but can also overfit or conflict with other objectives, so its benefits require careful, compute-aware comparisons.
Latest papers
49 papers18 papers in the last four weeks, up 350% on the four weeks before. 0.1% of all new papers.