Relational Knowledge Distillation

Overview

Relational knowledge distillation studies how to teach a smaller or differently structured model by preserving relationships in the teacher’s representations, rather than matching only individual outputs or features. Common methods align pairwise similarities, distances, class prototypes, or feature differences, sometimes using attention, optimal transport, or trajectory-based objectives. The approach is used in image classification, detection, segmentation, vision-language models, language modeling, and continual learning, where preserving geometry can help students generalize or retain knowledge. A central challenge is deciding which relations are reliable and task-relevant, since noisy or mismatched relationships can dilute supervision and students may not have enough capacity to reproduce the teacher’s structure.

Latest papers

CardsList

29 papers3 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

  1. Heterogeneous Asymmetric Contrastive Distillation

    Nov 7, 2026Qi Wang, Jinjia ZhouContrastive Knowledge DistillationKnowledge Distillation

  2. Learning from high-resolution 2D images to reconstruct 3D microstructures

    Nov 1, 2026Wenbo Sui, Lars P. Mikkelsen, Jens W. Andreasen +33D ReconstructionCross-Modal Knowledge Distillation

  3. Relational Geometry Forcing for World Action Models

    Oct 21, 2026Yuan Xu, Yixiang Chen, Qisen Ma +10World Models for RoboticsWorld Action Models

  4. SimOPD: Relational Knowledge Transfer for On-Policy Distillation

    Oct 16, 2026Kai Tang, Xing Zheng, Haobo Wang +6Language Model DistillationOn-Policy Distillation

  5. Representational Difference Distillation

    Oct 11, 2026Abdulkarim Mugisha, Divin IrakizaAdaptive Knowledge DistillationKnowledge Distillation

  6. Difference Feature Map Distillation: Transferring Inter-Sample Relational Knowledge Towards Efficient Transformer-Based Tracking

    Oct 5, 2026Zhicheng Ding, Xinyu Chu, Qing TianVisual Object TrackingAutonomous Driving Perception

  7. Gromov-Wasserstein Distillation for Inductive Multi-View Embedding

    Sep 30, 2026Rafael Pereira Eufrazio, Eduardo Fernandes Montesuma, Charles Casimiro CavalcanteRepresentation LearningMulti-View Learning

  8. Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition

    Sep 11, 2026Yanjiang Shi, Peng Zhao, Nan Qi +1Video UnderstandingHuman Activity Recognition

  9. ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models

    Sep 3, 2026Quang Hoang Trung, Quang Huu Hieu, Nguyen Van Hoang Phuc +1Language Model PretrainingAutoregressive Language Modeling

  10. Relational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision

    Aug 28, 2026Yuria Shimizu, Soh Takahashi, Takato Horii +1Relational Knowledge Distillation

  11. CloSeR: Unified Relational Distillation from Closed-Set Teachers for Category Discovery

    Aug 26, 2026Yuanpei Liu, Zhenqi He, Jialu Tang +1Generalized Category DiscoveryRelational Knowledge Distillation

  12. RAPID: Reliability-Aware Pair Importance Distillation

    Aug 23, 2026Ali Mahdavi, Azadeh Zamanifar, Amirfarhad Farhadi +1Relational Knowledge DistillationImportance Sampling

  13. FD-CanKD: Frequency-Decoupled Cross-Attention Distillation as a Refinement Prior for Compact Object Detectors

    Aug 19, 2026YoungJae Cheong, Jhonghyun AnKnowledge DistillationEfficient Object Detection

  14. Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

    Aug 14, 2026Dawen Jiang, Zhishu Shen, Zeyu Liu +1Knowledge DistillationRelational Knowledge Distillation

  15. Flow-Map Distillation on Relation Manifolds for Image Restoration

    Aug 6, 2026Zihao He, Songhua LiuFlow MatchingFlow Map Learning

  16. AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition

    Jul 28, 2026Yuqi Li, Yi-Cheng Lin, Xianglong Wang +5Relational Knowledge DistillationSpeech Emotion Recognition

  17. Privileged Lesion-Context Relational Distillation for Mask-Free Skin Lesion Classification

    Jul 21, 2026Abu Mukaddim Rahi, Md Mithun Hossain, Md Zulficar Hasan Joy +2Skin Lesion ClassificationMedical Image Classification

  18. Displacement Preserving Relational Distillation for Robust Medical Segmentation

    Jul 6, 2026Zhicheng Ding, Xinyu Chu, Jung Im Choi +5Model CompressionRelational Knowledge Distillation

  19. ContiStain: Cross-Domain Relation-Preserving Distillation for Continual Multi-Domain Virtual IHC Staining

    Jul 4, 2026Fuqiang Chen, Yifeng Wang, Hongpeng Wang +1Computational PathologyVirtual Staining

  20. PHF: Privileged Hidden Flow for On-Policy Self-Distillation

    Jun 28, 2026Yuhan Li, Mingxu Zhang, Dazhong Shen +1On-Policy Self-DistillationLanguage Model Distillation

  21. Geometry-Aware Distillation for Prompt Tuning Biomedical Vision-Language Models

    Jun 3, 2026Tran Dinh Tien, Zhiqiang ShenVLM AdaptationVLM Distillation

  22. Lightweight SAR Ship Detection via Contrastive Distillation

    May 27, 2026Surendar Devasundaram, Banafsheh Saber Latibari, Abhijit MahalanobisKnowledge DistillationRelational Knowledge Distillation

  23. Cross-Sample Relational Fusion: Unifying Domain Generalization and Class-Incremental Learning

    May 9, 2026Zhen-Hao Xie, Yan Wang, Hao Sun +3Class-Incremental LearningContinual Learning

  24. SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models

    May 8, 2026Jiesong Lian, Zixiang Zhou, Ruizhe Zhong +6Diffusion Model AlignmentVideo Diffusion Models

  25. MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation

    May 2, 2026Pham Khanh Chi, Quoc Phong Dao, Thuat Nguyen +3Representation LearningLanguage Model Distillation

  26. CLIP-RD: Relational Distillation for Efficient CLIP Knowledge Distillation

    Mar 26, 2026Jeannie Chung, Hanna Jang, Ingyeong Yang +2Cross-Modal LearningCross-Modal Knowledge Distillation

  27. Relational Representation Distillation

    Jul 16, 2024Nikos Giakoumoglou, Tania StathakiContrastive LearningRelational Knowledge Distillation