Relational Knowledge Distillation

Relational knowledge distillation studies how to teach a smaller or differently structured model by preserving relationships in the teacher’s representations, rather than matching only individual outputs or features. Common methods align pairwise similarities, distances, class prototypes, or feature differences, sometimes using attention, optimal transport, or trajectory-based objectives. The approach is used in image classification, detection, segmentation, vision-language models, language modeling, and continual learning, where preserving geometry can help students generalize or retain knowledge. A central challenge is deciding which relations are reliable and task-relevant, since noisy or mismatched relationships can dilute supervision and students may not have enough capacity to reproduce the teacher’s structure.