Knowledge Distillation

Ask about this topic

Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

830 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

  1. Open-SWE-Traces: Advancing Dual-Mode Multilingual Distillation for Software Engineering Agents

    Jun 14, 2026Wasi Uddin Ahmad, Nikolai Ludwig, Somshubra Majumdar +1Software Engineering AgentsLLM Agent Training

    NVIDIA

  2. On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

    Jun 14, 2026Gengsheng Li, Mao Zheng, Mingyang Song +8Language Model DistillationLLM Agent Training

    Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Large Language Model Department, Tencent +4

  3. Mutual Distillation of Dual-Foundation Models for Semi-Supervised PET/CT Segmentation

    Jun 14, 2026Fuyou Mao, Beining Wu, Yanfeng Jiang +5Cross-Modality Medical Image SegmentationMedical Imaging Foundation Models

    Central South University, Changsha 410083, China · Hangzhou Dianzi University, Hangzhou 310018, China · Communication University of Zhejiang, Hangzhou 310018, China +1

  4. SPARK: Spatial Policy-driven Adaptive Reinforcement learning for Knowledge distillation

    Jun 13, 2026Mohamed Jismy Aashik Rasool, Shabir Ahmad, Gisong Oh +1RL for Knowledge DistillationQuantization-Aware Training

    Department of IT Convergence Engineering, Gachon University, Sujeong-gu, Seongnam-si, Gyeonggi-do 461-701, Republic of Korea

  5. Distilling latent electrostatics from foundation machine learning interatomic potentials

    Jun 12, 2026Xiaoyu Wang, Bingqing ChengInfrared SpectroscopyMachine Learning Interatomic Potentials

    Department of Chemistry, UC Berkeley, California 94720, United States · Bakar Institute of Digital Materials for the Planet, UC Berkeley, California 94720, United States · Chemical Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, California, 94720, United States

  6. Improved Knowledge Distillation for Land-Use Image Classification

    Jun 12, 2026Arundhuti Sur, Abhiroop Chatterjee, Susmita Ghosh +1Cross-Architecture Knowledge DistillationKnowledge Distillation

    Jadavpur University, India · Rochester Institute of Technology, USA

  7. HumP-KD: A Hybrid Uncertainty-Aware Multi-Stage Progressive Knowledge Distillation Framework for Efficient Fire Classification

    Jun 12, 2026Mohammed Arif Mainuddin, Najifa Tabassum, Omar Ibne Shahid +1Efficient Neural Network InferenceCross-Architecture Knowledge Distillation

    Electrical and Computer Engineering North South University Dhaka, Bangladesh

  8. Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback

    Jun 12, 2026Woohyeon Byeon, Jiwon Jeon, Jeonghye Kim +1Language Model DistillationOn-Policy Distillation

    KAIST

  9. Visual Place Recognition in Forests with Depth-Aware Distillation

    Jun 11, 2026Walter Nedov, Saimunur Rahman, Kavindie Katuwandeniya +3Visual Place RecognitionKnowledge Distillation

    CSIRO Robotics, Brisbane, Australia · University of Queensland, Brisbane, Australia · Queensland University of Technology, Brisbane, Australia

  10. Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents

    Jun 10, 2026Tianyu Ding, Jianhong Xin, Juan Pablo De la Cruz WeinsteinToken-Level Credit AssignmentGroup Relative Policy Optimization

    Amazon Web Services

  11. MARD: Mirror-Augmented Reasoning Distillation for Mechanism-Level Drug-Drug Interaction Prediction

    Jun 10, 2026Mohammadreza Riyazat, Vian Lelo, Rameen Jafri +2Knowledge DistillationDrug Discovery

    University of Guelph, Canada · York University, Canada · Vector Institute, Canada

  12. Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers

    Jun 10, 2026MohammadHossein Rezaei, Anas Mahmoud, Zihao Wang +6Rubric-Based RLLanguage Model Distillation

    Scale AI

  13. Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions

    Jun 10, 2026José Medina, Paul Honeine, Abdelaziz Bensrhair +1Data AugmentationModel Calibration

    ITS Lab, Institute of Computer Science, University of Tartu, Estonia. · LITIS, Universit´e de Rouen, France. · LITIS, INSA de Rouen, France.

  14. Fast Speech Foundation Model Distillation Using Interleaved Stacking

    Jun 10, 2026Eungbeom Kim, Kyogu LeeSpeech Foundation ModelsSpeech Processing

    1IPAI, 2AIIS, 3Dept. of Intelligence and Information Seoul National University, Seoul, Republic of Korea

  15. RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

    Jun 10, 2026Leyi Pan, Shuchang Tao, Yunpeng Zhai +5RL for Language Model ReasoningLanguage Model Distillation

    1Tsinghua University · 2Tongyi Lab , Alibaba Group · 3Peking University

  16. When to Trust, How to Distill: Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecasting

    Jun 10, 2026Rupasree Dey, Abdul Matin, Nathan Orwick +3Multivariate Time Series ForecastingTime Series Forecasting

    Department of Computer Science Colorado State University Fort Collins, Colorado, USA · Department of Soil and Crop Sciences Colorado State University Fort Collins, Colorado, USA

  17. Physics-Distilled Neural Network enabled by Large Language Models for Manufacturing Process-Property Predictive Modeling

    Jun 10, 2026Ge Song, Kiarash Naghavi Khanghah, Anandkumar Patel +2Neural Surrogate ModelingMaterials Property Prediction

    School of Mechanical, Aerospace and Manufacturing Engineering, University of Connecticut, Storrs, CT 06269 · Department of Mechanical & Aerospace Engineering, Rutgers, the State University of New Jersey, Piscataway, NJ 08854

  18. FreqKD: Frequency-Decoupled Cross-Modal Knowledge Distillation for Infrared Object Detection

    Jun 10, 2026Keval Thaker, Venkatraman Narayanan, Abdalmalek Aburaddaha +1Cross-Modal Knowledge DistillationMultispectral Object Detection

    University of Michigan-Dearborn 4901 Evergreen Rd Dearborn, MI 48128, USA

  19. FADA: Accessible fetal ultrasound interpretation and annotation with a selectively distilled unified vision-language model

    Jun 9, 2026Mahmood Alzubaidi, Uzair Shah, Raden Muaz +6Ultrasound Image SegmentationSelective Knowledge Distillation

    College of Science and Engineering , Hamad Bin Khalifa University,2026 Doha, Qatar. · Center for Clinical Precision Medicine and Genomics, HMC, Doha, Qatar.Jun · 3Advanced AlRazi Diagnostic Center, Al-Hodeidah, Yemen. +1

  20. Quantifying Subliminal Behavioral Transfer Ratios in Language Model Distillation

    Jun 9, 2026Uwe König, Hamza Kazmi, Ruizhe Li +1Subliminal LearningLanguage Model Distillation

    1Technical University of Munich · 2Algoverse · University of Aberdeen +1

  21. PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning

    Jun 9, 2026Xinyue Peng, Yi Qian, Jiaojiao Lin +2Mixture-of-Experts Language ModelsExpert Routing

    Intel Corporation, China · Zhejiang University, China

  22. Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

    Jun 8, 2026Yifan Niu, Han Xiao, Dongyi Liu +4Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

    The Hong Kong University of Science and Technology (Guangzhou) · Tencent · The Hong Kong University of Science and Technology

  23. Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions

    Jun 8, 2026Xin Jin, Huanqia Cai, Zhen Li +9Reward ModelingReasoning Distillation

    Z-Image Team, Alibaba Group · VCIP, CS, Nankai University

  24. Heterophily-Aware Adaptive Knowledge Distillation for Hypergraph Neural Networks

    Jun 8, 2026Joohee Cho, David Yoon Suk Kang, Yunyong KoEfficient Neural Network InferenceHypergraph Neural Networks

    Chung-Ang University · Seoul, South Korea · Chungbuk National University +1

  25. Trajectory-Refined Distillation

    Jun 7, 2026Li Jiang, Haoran Xu, Yichuan Ding +1On-Policy DistillationLLM Post-Training

    McGill University · Mila Quebec AI Institute · UT Austin

  26. TLRD: Teaching LLMs to Reason over Tabular Data with Tri-Level Rationale Distillation

    Jun 6, 2026Tianyuan Liang, Xuwei Tan, Lei Shi +6LLM GroundingKnowledge Distillation

    The Ohio State University · Stevens Institute of Technology

  27. Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation

    Jun 5, 2026Mohammadreza Sadeghi, Sareh Soleimani, Zihan Wang +1ClusteringUnsupervised Clustering

    Department of Electrical and Computer Engineering, McGill University, Montreal, QC H3A 0E9, Canada · Mila – Quebec AI Institute, Montreal, QC H2S 3H1, Canada

  28. LRMIL: Efficient Low-Resolution Multiple Instance Learning via High-Resolution Knowledge Distillation for Whole Slide Image Classification

    Jun 5, 2026Yonghan Shin, Won-Ki JeongHistopathology Image ClassificationKnowledge Distillation

    Department of Computer Science and Engineering, Korea University, Seoul, Korea

  29. Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces

    Jun 5, 2026Debjyoti Saha RoyMulti-Label ClassificationCoT Distillation

    Khoury College of Computer Sciences, Northeastern University

  30. USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding

    Jun 4, 2026Heng-Jui Chang, Alexander H. Liu, Saurabhchand Bhati +4Audio Representation LearningAudio Understanding

    MIT CSAIL, USA · Amazon, USA