Knowledge Distillation

Ask about this topic

Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

  1. ASDA: Automated Skill Distillation and Adaptation for Financial Reasoning

    Mar 17, 2026Tik Yu Yim, Wenting Tan, Sum Yee Chan +2Domain Adaptation for LLMsAgent Skill Learning

    The University of Hong Kong, Hong Kong SAR, China

  2. Longitudinal Risk Prediction in Mammography with Privileged History Distillation

    Mar 16, 2026Banafsheh Karimian, Soufiane Belharbi, Alexis Guichemerre +3Breast CancerMedical Imaging

    LIVIA, Dept. of Systems Engineering, ETS Montreal, Canada · Goodman Cancer Institute, Dept. of Oncology, McGill University, Canada

  3. TabKD: Tabular Knowledge Distillation through Interaction Diversity of Learned Feature Bins

    Mar 16, 2026Shovon Niverd Pereira, Krishna Khadka, Yu LeiFeature Interaction ModelingData-Free Knowledge Distillation

    Department of Computer Science and Engineering, The University of Texas at Arlington

  4. Photonic Quantum-Enhanced Knowledge Distillation

    Mar 16, 2026Kuan-Cheng Chen, Shang Yu, Chen-Yu Liu +10Knowledge DistillationQuantum Machine Learning

    Department of Electrical and Electronic Engineering, Imperial College London, South Kensington, London, SW7 2AZ, England, United Kingdom. · Imperial Centre for Quantum Engineering, Science and Technology, Imperial College London, South Kensington, London, SW7 2AZ, England, United Kingdom. · JIJ, Rutherford Appleton Laboratory, Harwell Campus, Didcot, OX11 0QX, United Kingdom. +9

  5. Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoders

    Mar 15, 2026Jiaming Chu, Tao Wang, Lei JinPosition ExtrapolationVariational Autoencoders

    School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China

  6. Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation

    Mar 13, 2026Boxuan Lyu, Haiyue Song, Zhi QuMBR DecodingLLM-Assisted Annotation

    Institute of Science Tokyo · National Institute of Information and Communications Technology, Japan

  7. Can Decision Trees Teach Large Language Models? Distilling Verbalized Knowledge for Molecular Property Prediction

    Mar 12, 2026Khiem Le, Sreejata Dey, Marcos Martínez Galindo +4Molecular Property PredictionKnowledge Distillation

    University of Notre Dame, IN, USA · IBM Research

  8. Gaussian Process Regression-based Knowledge Distillation Framework for Simultaneous Prediction of Physical and Mechanical Properties of Epoxy Polymers

    Mar 12, 2026Sindu B. S., Jan HamaekersMaterials Property PredictionPolymer Property Prediction

    Special and Multifunctional Structures Laboratory, CSIR-Structural Engineering Research Centre, Taramani, Chennai, Tamil Nadu, India - 600113. · Fraunhofer Institute for Algorithms and Scientific Computing SCAI, Schloss Birlinghoven, 53757 Sankt Augustin, Germany

  9. Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understanding

    Mar 12, 2026Songlin Li, Xin Zhu, Zechao Guan +2VLM DistillationKnowledge Distillation

    University of Electronic Science and Technology of China · Robotics Center, XPeng Motors

  10. PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence

    Mar 11, 2026Yuanda Xu, Hejian Sang, Zhengze Zhou +2Language Model DistillationAdaptive Knowledge Distillation

    LinkedIn Corporation

  11. From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers

    Mar 11, 2026Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar +1Cross-Modal Knowledge DistillationKnowledge Distillation

    Indian Institute of Technology Delhi, India · Indraprastha Institute of Information Technology Delhi, India

  12. Distilling LLM Semantic Priors into Encoder-Only Multi-Talker ASR with Talker-Count Routing

    Mar 11, 2026Hao Shi, Yusuke Fujita, Roman Koshkin +4Multi-Speaker ASRConnectionist Temporal Classification

    SB Intuitions, Tokyo, Japan

  13. A Multi-Prototype-Guided Federated Knowledge Distillation Approach in AI-RAN Enabled Multi-Access Edge Computing System

    Mar 10, 2026Luyao Zou, Hayoung Oh, Chu Myaet Thwal +3Federated Knowledge DistillationFederated Learning

    College of Computing and Informatics, Sungkyunkwan University, Republic of Korea · Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do, 17104, Republic of Korea · Department of Information and Communication Engineering, Noakhali Science and Technology University, Noakhali-3814, Bangladesh +2

  14. Decoder-Free Distillation for Quantized Image Restoration

    Mar 10, 2026S. M. A. Sharif, Abdur Rehman, Seongwan Kim +1Image RestorationQuantization-Aware Distillation

    Opt-AI Inc., Seoul, South Korea

  15. EvoDriveVLA: Evolving Driving VLA Models via Collaborative Perception-Planning Distillation

    Mar 10, 2026Jiajun Cao, Xiaoan Zhang, Xiaobao Wei +10End-to-End Autonomous DrivingAutonomous Driving Planning

    State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · XPeng Motors

  16. EventVGGT: Exploring Cross-Modal Distillation for Consistent Event-based Depth Estimation

    Mar 10, 2026Yinrui Ren, Jinjing Zhu, Kanghao Chen +8Event-Based VisionDepth Estimation

    Hong Kong University of Science and Technology (Guangzhou) · South China Normal University · Chinese University of Hong Kong

  17. LinearARD: Linear-Memory Attention Distillation for RoPE Restoration

    Mar 9, 2026Ning Yang, Hengyu Zhong, Wentao Wang +4Attention-Guided DistillationMemory-Efficient Attention

    Institute of Automation, Chinese Academy of Sciences, Beijing, China · Southwest University, Chongqing, China · Dalian University of Technology, Dalian, Liaoning, China +3

  18. SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning

    Mar 9, 2026Kaushik Roy, Giovanni D'urso, Nicholas Lawrance +2Imitation LearningRobot Imitation Learning

    CSIRO Robotics, DATA61, CSIRO, Australia. · Queensland University of Technology (QUT), Brisbane, Australia.

  19. Reverse Distillation: Consistently Scaling Protein Language Model Representations

    Mar 8, 2026Darius Catrina, Christian Bepler, Samuel Sledzieski +1Protein Language ModelsProtein Representation Learning

    Department of Computer Science, Duke University, Durham, NC, USA · Depts of Biostats. and Bioinformatics; and Cell Biology, Duke University, Durham, NC, USA

  20. PDD: Manifold-Prior Diverse Distillation for Medical Anomaly Detection

    Mar 7, 2026Xijun Lu, Hongying Liu, Fanhua Shang +2Medical Image Anomaly DetectionKnowledge Distillation

    Medical School, Tianjin University, Tianjin, China · College of Intelligence and Computing, Tianjin University, Tianjin, China

  21. Efficient Personalized Reranking with Semi-Autoregressive Generation and Online Knowledge Distillation

    Mar 7, 2026Kai Cheng, Hao Wang, Wei Guo +4Online Knowledge DistillationRecommender Systems

    University of Science and Technology of China, Hefei, China · Huawei, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China +1

  22. DLRMamba: Distilling Low-Rank Mamba for Edge Multispectral Fusion Object Detection

    Mar 6, 2026Qianqian Zhang, Leon Tabaro, Ahmed M. Abdelmoniem +1Efficient Object DetectionState Space Models

    National Space Science Center, Chinese Academy of Sciences, Beijing, 101499, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, 100049, China · School of Electronic Engineering and Computer Science, Queen Mary University of London, London, E1 4NS, UK +1

  23. BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations

    Mar 6, 2026Thomas Monninger, Shaoyuan Xie, Qi Alfred Chen +1VLMs for Autonomous DrivingEnd-to-End Autonomous Driving

    Mercedes-Benz Research & Development North America, San Jose, USA · University of California, Irvine, USA

  24. TaPD: Temporal-adaptive Progressive Distillation for Observation-Adaptive Trajectory Forecasting in Autonomous Driving

    Mar 6, 2026Mingyu Fan, Yi Liu, Hao Zhou +3Trajectory ReconstructionTrajectory Prediction

    College of Information and Intelligent Science, Donghua University, Shanghai 201620 China. · School of Computing and Information Technology, Great Bay Institute for Advanced Study/Great Bay University, Dongguan, China, and also with the Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · Chongqing Chang’an Technology Co., Ltd, Chongqing 401120 China. +1

  25. DARK: Diagonal-Anchored Repulsive Knowledge Distillation for Vision-Language Models under Extreme Compression

    Mar 5, 2026Numan Saeed, Asif Hanif, Fadillah Adamsyah Maani +2VLM DistillationKnowledge Distillation

    Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

  26. Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation

    Mar 5, 2026Kabir ThayaniKnowledge DistillationContrastive Learning

    Independent Researcher · India

  27. A Benchmark Study of Neural Network Compression Methods for Hyperspectral Image Classification

    Mar 5, 2026Sai ShiNeural Network CompressionNeural Network Quantization

    Department of Computer and Information Science, Temple University, Philadelphia, USA

  28. Scaling Dense Event-Stream Pretraining from Visual Foundation Models

    Mar 4, 2026Zhiwen Chen, Junhui Hou, Zhiyu Zhu +2Event-Based VisionRepresentation Learning

    City University of Hong Kong · Xidian University

  29. Vector-Quantized Soft Label Compression for Dataset Distillation

    Mar 4, 2026Ali Abbasi, Ashkan Shahbazi, Hamed Pirsiavash +1Dataset DistillationVector Quantization

    Vanderbilt University · University of California, Davis

  30. Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation

    Mar 3, 2026Chonghua Lv, Dong Zhao, Shuang Wang +4Domain Generalization for Semantic SegmentationKnowledge Distillation

    School of Artificial Intelligence, Xidian University, China · Department of Information Engineering and Computer Science, University of Trento, Italy · Department of Automation, Tsinghua University, China +1