Knowledge Distillation

Ask about this topic

Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

  1. Edge AI for Automotive Vulnerable Road User Safety: Deployable Detection via Knowledge Distillation

    Apr 29, 2026Akshay Karjol, Darrin M. HannaEdge InferenceAutonomous Driving Perception

    Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309 USA

  2. GaitKD: A Universal Decoupled Distillation Framework for Efficient Gait Recognition

    Apr 29, 2026Yuqi Li, Qian Zhou, Huiran Duan +5Gait AnalysisGait Recognition

    The City University of New York · Wuhan University · Beijing Jiaotong University +2

  3. Privileged Foresight Distillation: Zero-Cost Future Correction for World Action Models

    Apr 28, 2026Pengcheng Fang, Hongli Chen, Xiaohao CaiWorld Model LearningAction-Conditioned World Models

    The University of Southampton · The University of Queensland

  4. Improving Diversity in Black-box Few-shot Knowledge Distillation

    Apr 28, 2026Tri-Nhan Vo, Dang Nguyen, Kien Do +1Synthetic Data AugmentationFew-Shot Learning

    Applied Artificial Intelligence Institute (A2I2), Deakin University, Australia

  5. Diverse Image Priors for Black-box Data-free Knowledge Distillation

    Apr 28, 2026Tri-Nhan Vo, Dang Nguyen, Trung Le +2Synthetic Data AugmentationData-Free Knowledge Distillation

    Applied Artificial Intelligence Initiative, Deakin University, Geelong, Australia · Department of Data Science & AI, Monash University, Melbourne, Australia

  6. Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experiment

    Apr 28, 2026Chayanon Kitkana, Shivam AroraSubliminal LearningTeacher-Student Learning

    Independent · Equivariant labs

  7. The Surprising Effectiveness of Canonical Knowledge Distillation for Semantic Segmentation

    Apr 28, 2026Muhammad Ali, Kevin Alexander Laube, Madan Ravi Ganesh +3Image SegmentationFeature-Based Knowledge Distillation

    University of Freiburg · Bosch Center for Artificial Intelligence · Aleph Alpha Research +1

  8. Knowledge Distillation Must Account for What It Loses

    Apr 28, 2026Wenshuo WangTeacher-Student LearningKnowledge Distillation

    School of Future Technology, South China University of Technology, China

  9. S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models

    Apr 27, 2026Mohammed Ali El Adlouni, Aurian Quelennec, Pierre Chouteau +2Model CompressionAudio Representation Learning

    LTCI, T´el´ecom Paris, Institut Polytechnique de Paris, Palaiseau, France

  10. Selective Correlation Based Knowledge Distillation for Ground Reaction Force Estimation

    Apr 27, 2026Eun Som Jeon, Jisoo Lee, Huisu Lim +3Gait AnalysisWearable Sensing

    Department of Computer Science and Engineering, Seoul National University of Science and Technology, Seoul, 01811, Republic of Korea · Geometric Media Lab, School of Computing and Augmented Intelligence, Arizona State University, Tempe, 85281, AZ, USA · School for Engineering of Matter, Transport and Energy, Arizona State University, Tempe, 82587, AZ, USA +1

  11. Hiding in Plain Sight: Detectability-Aware Antidistillation of Reasoning Models

    Apr 25, 2026Max Hartman, Vidhata Jayaraman, Moulik Choraria +1CoT DistillationTeacher-Student Learning

    Stony Brook University

  12. Distilling Vision Transformers for Distortion-Robust Representation Learning

    Apr 24, 2026Konstantinos Alexis, Giorgos Giannopoulos, Dimitrios GunopulosRepresentation LearningVisual Representation Learning

    Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Greece · National Observatory of Athens, Greece

  13. Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation

    Apr 23, 2026Nikita Severin, Danil Kartushov, Vladislav Urzhumov +8Sequential RecommendationLanguage Model Distillation

    Independent researcher, Belgrade, Serbia · ITMO University, Saint Petersburg, Russia · Innopolis University, Innopolis, Russia +3

  14. VARestorer: One-Step VAR Distillation for Real-World Image Super-Resolution

    Apr 23, 2026Yixuan Zhu, Shilin Ma, Haolin Wang +6Autoregressive Image Super-ResolutionAutoregressive Image Generation

    Tsinghua University

  15. When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors

    Apr 23, 2026Chenghao Yang, Yuning Zhang, Zhoufutu Wen +4Tool-Augmented Language Model AgentsLLM Agent Evaluation

    School of Cyber Science and Technology, USTC · Anhui Province Key Laboratory of Digital Security · 3M-A-P +1

  16. Hybrid Policy Distillation for LLMs

    Apr 22, 2026Wenhong Zhu, Ruobing Xie, Rui Wang +1Language Model DistillationOn-Policy Distillation

    Department of Computer Science and Engineering, Shanghai Jiao Tong University · Shanghai Innovation Institute · Large Language Department, Tencent

  17. Weighted Knowledge Distillation for Semi-Supervised Segmentation of Maxillary Sinus in Panoramic X-ray Images

    Apr 22, 2026Juha Park, Jiho Choi, Jong Pil Yun +4Medical ImagingDental Imaging

    Division of Electronics and Information Engineering, College of Engineering, Jeonbuk National University, 567, Baekje-daero, Deokjin-gu, 54896, Jeonju, Republic of Korea · Manufacturing AI Research Center, Korea Institute of Industrial Technology (KITECH), Incheon 21999, Republic of Korea · KITECH School, University of Science and Technology, Cheonan 31056, Republic of Korea +2

  18. From Skeletons to Pixels: Few-Shot Precise Event Spotting via Representation and Prediction Distillation

    Apr 21, 2026Zhong Han Ervin Yeoh, Jiang KanCross-Modal Knowledge DistillationFew-Shot Learning

    Department of Business Analytics School of Computing National University of Singapore · Department of Computer Science School of Computing National University of Singapore

  19. Distillation Traps and Guards: A Calibration Knob for LLM Distillability

    Apr 21, 2026Weixiao Zhan, Yongcheng Jing, Leszek Rutkowski +1Language Model CalibrationLanguage Model Distillation

    Generative AI Lab, College of Computing and Data Science Nanyang Technological University, Singapore 639798 · Systems Research Institute of the Polish Academy of Sciences, AGH University of Krakow, 30-059 Kraków, and the SAN University, 90-113, Łód´z, Poland

  20. Rethinking Dataset Distillation: Hard Truths about Soft Labels

    Apr 20, 2026Priyam Dey, Aditya Sahdev, Sunny Bhati +2Training Data SelectionDataset Distillation

    Vision and AI Lab, Indian Institute of Science, Bangalore · University of Maryland, Baltimore County · IIT Hyderabad

  21. SemLT3D: Semantic-Guided Expert Distillation for Camera-only Long-Tailed 3D Object Detection

    Apr 20, 2026Hao Vo, Khoa Vo, Thinh Phan +5Autonomous Driving Perception3D Object Detection

    AICV Lab, University of Arkansas, USA · University of Utah, USA · University of Houston, USA +1

  22. Sharpening Lightweight Models for Generalized Polyp Segmentation: A Boundary Guided Distillation from Foundation Models

    Apr 20, 2026Shivanshu Agnihotri, Snehashis Majhi, Deepak Ranjan NayakImage SegmentationVision Foundation Model Adaptation

    Dept. of Computer Science and Engineering Malaviya National Institute of Technology Jaipur, India

  23. Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information

    Apr 17, 2026Yao Chen, Jiawei Sheng, Wenyuan Zhang +1CoT DistillationSmall Language Models

    Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences

  24. Weak-to-Strong Knowledge Distillation Accelerates Visual Learning

    Apr 16, 2026Baiang Li, Wenhao Chai, Felix HeideTeacher-Student LearningKnowledge Distillation

    Princeton University

  25. DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models

    Apr 16, 2026Jingyuan Wang, Zhihao Jia, Chenyu Liu +7Efficient Neural Network InferenceEEG Decoding

    1Xiamen University · 2Nanyang Technological University · Institute of Automation, Chinese Academy of Sciences +1

  26. Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models

    Apr 16, 2026Haoyi Sun, Xiaoxiao Wang, Ning Mao +5Cross-Modal Knowledge DistillationVision-Language Models

    Li Auto Inc.

  27. TIP: Token Importance in On-Policy Distillation

    Apr 15, 2026Yuanda Xu, Hejian Sang, Zhengze Zhou +3Knowledge DistillationOn-Policy Distillation

  28. Granularity-Aware Transfer for Tree Instance Segmentation in Synthetic and Real Forests

    Apr 15, 2026Pankaj Deoli, Atef Tej, Anmol Ashri +2Instance SegmentationSynthetic-to-Real Domain Adaptation

    Robotics Research Lab, University of Kaiserslautern-Landau, Kaiserslautern, Germany · University of Kaiserslautern-Landau, Kaiserslautern, Germany

  29. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

    Apr 14, 2026Yaxuan Li, Yuxin Zuo, Bingxiang He +8Language Model DistillationOn-Policy Distillation

    Tsinghua University · ShanghaiTech University · University of Illinois Urbana-Champaign +1

  30. On the Distillation Loss Functions of Speech VAE for Unified Reconstruction, Understanding, and Generation

    Apr 14, 2026Changhao Cheng, Wei Wang, Wangyou Zhang +4Variational AutoencodersAudio Representation Learning

    School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China · Auditory Cognition and Computational Acoustics Lab, MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · ByteDance Seed, China