Knowledge Distillation

Ask about this topic

Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

  1. From Foundation ECG Models to NISQ Learners: Distilling ECGFounder into a VQC Student

    Mar 28, 2026Giovanni dos Santos Franco, Felipe Mahlow, Ellison Fernando Cardoso +1Electrocardiogram ClassificationKnowledge Distillation

    Faculty of Sciences, S˜ao Paulo State University (Unesp), Bauru, Brazil. · Hospital Israelita Albert Einstein, S˜ao Paulo, Brazil. · Laboratory of Magnetic Resonance in Neuroradiology (LIM-44), University of S˜ao Paulo Faculty of Medicine Clinics Hospital, S˜ao Paulo, Brazil.

  2. Distilling Genomic Models for Efficient mRNA Representation Learning via Embedding Matching

    Mar 27, 2026Rasched Haidari, Sam Martin, Maxime AllardKnowledge DistillationMolecular Representation Learning

    Helical London, UK

  3. GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

    Mar 27, 2026Xujing Tao, Chuxin Wang, Yubo Ai +8Open-Vocabulary 3D Segmentation3D Semantic Segmentation

    University of Science and Technology of China · National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory

  4. Knowledge Distillation for Efficient Transformer-Based Reinforcement Learning in Hardware-Constrained Energy Management Systems

    Mar 27, 2026Pascal Henrich, Jonas Sievers, Maximilian Beichter +3Policy DistillationTransformer-Based RL

    Karlsruhe Institute of Technology, Karlsruhe, Germany

  5. 4DRaL: Bridging 4D Radar with LiDAR for Place Recognition using Knowledge Distillation

    Mar 27, 2026Ningyuan Huang, Zhiheng Li, Zheng FangMillimeter-Wave RadarKnowledge Distillation

    The authors are all with the Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China.

  6. HAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learning

    Mar 27, 2026Xuerui Zhang, Xuehao Wang, Zhan Zhuang +5Knowledge DistillationDense Prediction

    Southern University of Science and Technology · Technical University of Munich · Zhejiang University +2

  7. Dual-Stage Invariant Continual Learning under Extreme Visual Sparsity

    Mar 27, 2026Rangya Zhang, Jiaping Xiao, Lu Bai +2Continual Object DetectionIncremental Learning

    School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798, Singapore

  8. Efficient Few-Shot Learning for Edge AI via Knowledge Distillation on MobileViT

    Mar 27, 2026Shuhei Tsuyuki, Reda Bensaid, Jérémy Morlier +4Few-Shot Image ClassificationEdge Computing

    Research Institute of Electrical Communication, Tohoku University, Japan · Graduate School of Engineering, Tohoku University, Japan · IMT Atlantique, Lab-STICC, UMR CNRS 6285, F-29238 Brest, France

  9. Learnable Instance Attention Filtering for Adaptive Detector Distillation

    Mar 27, 2026Chen Liu, Qizhen Lan, Zhicheng Ding +2Knowledge DistillationFeature-Based Knowledge Distillation

    Dept. of CS, University of Alabama at Birmingham · Dept. of CS, Bowling Green State University

  10. Training the Knowledge Base through Evidence Distillation and Write-Back Enrichment

    Mar 26, 2026Yuxing Lu, Xukai Zhao, Wei Wu +1Retrieval-Augmented GenerationKnowledge Augmentation for Language Models

    Peking University · Georgia Institute of Technology · Tsinghua University

  11. Neural Network Conversion of Machine Learning Pipelines

    Mar 26, 2026Man-Ling Sung, Jan Silovsky, Man-Hung Siu +2Transfer LearningKnowledge Distillation

    Raytheon BBN Technologies, 10 Moulton Street, Cambridge, MA 02138

  12. Residual-as-Teacher: Mitigating Bias Propagation in Student--Teacher Estimation

    Mar 26, 2026Kakei Yamamoto, Martin J. WainwrightStatistical Learning TheoryKnowledge Distillation

    Lab for Information and Decision Systems; Statistics and Data Science Center; EECS; Massachusetts Institute of Technology · Lab for Information and Decision Systems; Statistics and Data Science Center; EECS and Mathematics; Massachusetts Institute of Technology

  13. Multimodal Dataset Distillation via Phased Teacher Models

    Mar 26, 2026Shengbin Guo, Hang Zhao, Senqiao Yang +5Dataset DistillationKnowledge Distillation

    Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong

  14. CLIP-RD: Relational Distillation for Efficient CLIP Knowledge Distillation

    Mar 26, 2026Jeannie Chung, Hanna Jang, Ingyeong Yang +2Cross-Modal LearningCross-Modal Knowledge Distillation

    Ewha Womans University, Seoul 03760, South Korea

  15. A Public Theory of Distillation Resistance via Constraint-Coupled Reasoning Architectures

    Mar 26, 2026Peng Wei, Wesley ShuKnowledge Distillation

    College of Plant Protection, Southwest University, China · The Institute of Energetic Paradigm

  16. Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement

    Mar 25, 2026Xin Zhang, Jianyang Xu, Hao Peng +5Knowledge DistillationMulti-Teacher Knowledge Distillation

    Hangzhou Dianzi University

  17. CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning

    Mar 25, 2026Hieu Hoang, Dung Trung Tran, Hong Nguyen +1Temporal Action DetectionHuman Activity Recognition

    Viettel Group, Vietnam · School of Electrical and Electronics Engineering, Hanoi University of Science and Technology, Vietnam · Signal Analysis and Interpretation Lab, University of Southern California, United States

  18. TED: Training-Free Experience Distillation for Multimodal Reasoning

    Mar 25, 2026Shuozhi Yuan, Jinqing Wang, Zihao Liu +5Context DistillationKnowledge Distillation

    China Telecom Digital Intelligence Technology Co.,Ltd., Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

  19. MemCollab: Cross-Model Memory Collaboration via Contrastive Trajectory Distillation

    Mar 24, 2026Yurui Chang, Yiran Wu, Qingyun Wu +1Memory-Augmented Language Model AgentsMulti-Agent Collaboration

    Pennsylvania State University · AG2AI

  20. Adaptive Dual-Teacher Distillation with Subnetwork Rectification for Bridging Semantic Gaps in Black-Box Domain Adaptation

    Mar 24, 2026Zhe Zhang, Jing Li, Wanli Xue +4Source-Free Domain AdaptationKnowledge Distillation

    School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China. · School of Artificial Intelligence, Tianjin University, Tianjin 300350, China

  21. Transformers Trained via Gradient Descent Can Provably Learn a Class of Teacher Models

    Mar 24, 2026Chenyang Zhang, Qingyue Zhao, Quanquan Gu +1TransformerStatistical Learning Theory

    The University of Hong Kong · University of California, Los Angeles

  22. Clinical Graph-Mediated Distillation for Unpaired MRI-to-CFI Hypertension Prediction

    Mar 23, 2026Dillan Imans, Phuoc-Nguyen Bui, Duc-Tai Le +1Cross-Modal Knowledge DistillationClinical Prediction

    Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, South Korea · Convergence Research Institute, Sungkyunkwan University, Suwon, South Korea · Department of AI Systems Engineering, Sungkyunkwan University, Suwon, South Korea +1

  23. Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models

    Mar 22, 2026Jingchen Sun, Shaobo Han, Deep Patel +3Uncertainty-Aware Knowledge DistillationKnowledge Distillation

    NEC Laboratories America, Inc., USA · University at Buffalo, SUNY · Rutgers University

  24. Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guarantees

    Mar 22, 2026Alberlucia Rafael Soarez, Daniel Kim, Mariana Costa +1Language Model DistillationKnowledge Distillation

    Department of Computer Science, University of Brasilia

  25. VSD-MOT: End-to-End Multi-Object Tracking in Low-Quality Video Scenes Guided by Visual Semantic Distillation

    Mar 21, 2026Jun DuMulti-Object TrackingVisual Object Tracking

  26. EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation

    Mar 19, 2026Longfei Liu, Yongjie Hou, Yang Li +7Dense PredictionEfficient ViTs

    Intellindust AI Lab

  27. Collaborative Adaptive Curriculum for Progressive Knowledge Distillation

    Mar 19, 2026Jing Liu, Zhenchao Ma, Han Yu +5Federated Knowledge DistillationKnowledge Distillation

    Fudan University · The University of British Columbia · Duke Kunshan University +2

  28. HSI Image Enhancement Classification Based on Knowledge Distillation: A Study on Forgetting

    Mar 19, 2026Songfeng ZhuClass-Incremental LearningHyperspectral Image Classification

    School of Software Engineering, Henan Polytechnic University, Henan 454000, china

  29. Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting

    Mar 17, 2026Yu-Chen Den, Kuan-Yu Chen, Kendro Vincent +1Time Series ForecastingFinancial Time Series

    SinoPac Holdings, Taipei, Taiwan · National Chengchi University, Taipei, Taiwan

  30. CD-FKD: Cross-Domain Feature Knowledge Distillation for Robust Single-Domain Generalization in Object Detection

    Mar 17, 2026Junseok Lee, Sungho Shin, Seongju Lee +1Object DetectionCross-Domain Generalization

    Junseok Lee is with the Advanced Robotics Lab, LG Electronics, Seoul, Republic of Korea. · Sungho Shin is with the Place AI Flatform, Naver, Gyeonggi-do, Republic of Korea · Seongju Lee and Kyoobin Lee are with the Department of AI Convergence, Gwangju Institute of Science and Technology (GIST), Gwangju, Republic of Korea.