Knowledge Distillation

Ask about this topic

Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

  1. A Unified Revisit of Temperature in Classification-Based Knowledge Distillation

    Mar 2, 2026Logan Frank, Jim DavisKnowledge Distillation

    Ohio State University

  2. KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models

    Mar 2, 2026Songming Zhang, Xue Zhang, Tong Zhang +3Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

    School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education · Weixin AI

  3. Dual Distillation for Few-Shot Anomaly Detection

    Mar 2, 2026Le Dong, Qinzhong Tan, Chunlei Li +5Few-Shot Anomaly DetectionAnomaly Detection

    Xidian University, China · MedAI Technology (Wuxi) Co. Ltd., China · Technical University of Munich, Germany

  4. Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention

    Mar 2, 2026Wenye Lin, Kai HanLLM Post-TrainingLLM Reasoning

    The University of Hong Kong

  5. Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion

    Feb 26, 2026Zhen Tan, Chengshuai Zhao, Song Wang +3Language Model DistillationReasoning Distillation

    Arizona State University · University of Virginia · University of North Carolina at Chapel Hill

  6. Spectrally Distilled Representations Aligned with Instruction-Augmented LLMs for Satellite Imagery

    Feb 26, 2026Minh Kha Do, Wei Xiang, Kang Han +5Remote Sensing VLMsRemote Sensing Image Understanding

    La Trobe University, Melbourne, VIC 3086, Australia · Cisco Research, San Jose, CA, USA

  7. Reinforcement-aware Knowledge Distillation for LLM Reasoning

    Feb 26, 2026Zhaoyang Zhang, Shuli Jiang, Yantao Shen +6Knowledge DistillationRL for Knowledge Distillation

    AWS Agentic AI

  8. Decoder-based Sense Knowledge Distillation

    Feb 25, 2026Qitong Wang, Mohammed J. Zaki, Georgios Kollias +1Language Model DistillationKnowledge Distillation

    Rensselaer Polytechnic Institute Troy, New York, USA · IBM Research Yorktown Heights, New York, USA

  9. Structure and Redundancy in Large Language Models: A Spectral Study via Random Matrix Theory

    Feb 25, 2026Davide EttoriLLM CompressionRepresentation Geometry in Language Models

  10. Distilling Deep Reinforcement Learning into Interpretable Fuzzy Rules: An Explainable AI Framework

    Feb 24, 2026Sanup S. Araballi, Simon Khan, Chilukuri K. MohanInterpretability in RLKnowledge Distillation

    Dept. of EECS, Syracuse University, NY 13244, USA · Air Force Research Laboratory, Rome, NY 13441, USA

  11. Momentum Memory for Knowledge Distillation in Computational Pathology

    Feb 24, 2026Yongxin Guo, Hao Lu, Onur C. Koyun +3Knowledge DistillationCross-Modal Knowledge Distillation

    Wake Forest University School of Medicine, Winston-Salem, NC, USA

  12. Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation

    Feb 24, 2026Sayantan Dasgupta, Trevor Cohn, Timothy BaldwinLanguage Model DistillationKnowledge Distillation

    School of Computing and Information Systems, University of Melbourne, Melbourne, Australia.

  13. GATES: Self-Distillation under Privileged Context with Consensus Gating

    Feb 24, 2026Alex Stein, Furong Huang, Tom GoldsteinKnowledge DistillationOn-Policy Self-Distillation

    University of Maryland, College Park

  14. On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inference

    Feb 23, 2026Moritz A. Zanger, Yijun Wu, Pascal R. Van der Vaart +2Uncertainty QuantificationBayesian Inference

    Delft University of Technology, Delft, 2628 XE, The Netherlands

  15. Federated Reasoning Distillation Framework with Model Learnability-Aware Data Allocation

    Feb 21, 2026Wei Guo, Siyuan Lu, Xiangdong Ran +8Reasoning DistillationFederated Knowledge Distillation

    Beihang University, Beijing 100191, China · Heilongjiang University, Harbin 150080, China · Beijing University of Civil Engineering and Architecture, Beijing 100044, China +3

  16. ColBERT-Zero: To Pre-train Or Not To Pre-train ColBERT models

    Feb 18, 2026Antoine Chaffin, Luca Arnaboldi, Amélie Chatelain +1Multi-Vector RetrievalKnowledge Distillation

    LightOn, France · Ecole Polytechnique Fédérale de Lausanne (EPFL), IdePHICS Lab, CH-1015 Lausanne, Switzerland

  17. Task-Specific Knowledge Distillation via Intermediate Probes

    Feb 18, 2026Ryan Brown, Chris RussellLanguage Model DistillationKnowledge Distillation

    Oxford Internet Institute, University of Oxford, Oxford, United Kingdom.

  18. jina-embeddings-v5-text: Task-Targeted Embedding Distillation

    Feb 17, 2026Mohammad Kalim Akram, Saba Sturua, Nastia Havriushenko +4Text Embedding ModelsKnowledge Distillation

    Jina by Elastic

  19. Logit Distance Bounds Representational Similarity

    Feb 17, 2026Beatrix M. G. Nielsen, Emanuele Marconato, Luigi Gresele +2Representational Similarity AnalysisRepresentation Learning

    IT University of Copenhagen, Denmark · University of Trento, Italy · University of Copenhagen, Denmark +2

  20. Protecting Language Models Against Unauthorized Distillation through Trace Rewriting

    Feb 16, 2026Xinhang Ma, William Yeoh, Ning Zhang +1LLM WatermarkingKnowledge Distillation

    Washington University in St. Louis

  21. LLM-Guided Knowledge Distillation for Temporal Knowledge Graph Reasoning

    Feb 16, 2026Wang Xing, Wei Song, Siyu Lin +2Knowledge DistillationTemporal KGs

    School of Computer Science and Technology, Xidian University · School of Computing and Artificial Intelligence, Southwest Jiaotong University · School of Information Science and Engineering, Chongqing Jiaotong University

  22. Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation

    Feb 10, 2026Joseph Attieh, Timothee Mickus, Anne-Laure Ligozat +2Energy-Efficient MLKnowledge Distillation

    University of Helsinki · Université Paris-Saclay, CNRS, LISN

  23. Distill3R: A Pipeline for Democratizing 3D Foundation Models on Commodity Hardware

    Jan 31, 2026Brandon Leblanc, Charalambos Poullis3D Foundation ModelsKnowledge Distillation

    Immersive and Creative Technologies Lab Concordia University Montreal, Canada

  24. Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

    Jan 30, 2026Yanlong Chen, Amirhossein Habibian, Luca Benini +1Cross-Modal Knowledge DistillationVision-Language Models

    Department of Information Technology and Electrical Engineering, ETH Zurich, Zurich, Switzerland · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Qualcomm AI Research, Amsterdam, the Netherlands +1

  25. OVD: On-policy Verbal Distillation

    Jan 29, 2026Jing Xiong, Hui Shen, Shansan Gong +7Language Model DistillationOn-Policy Distillation

    The University of Hong Kong, Hong Kong, China · Nanjing University, Nanjing, China · Huawei Technologies, China

  26. SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines

    Jan 4, 2026Itai Morad, Nir Shlezinger, Yonina C. EldarStatistical Learning TheoryTeacher-Student Learning

    School of ECE, Ben-Gurion University, Be’er Sheva, Israel · Faculty of Math and CS, Weizmann Institute of Science, Rehovot, Israel

  27. Long-Range Distillation: Distilling 10,000 Years of Simulated Climate into Long Timestep AI Weather Models

    Dec 28, 2025Scott A. Martin, Noah Brenowitz, Dale Durran +1Weather ForecastingSpatiotemporal Forecasting

    NVIDIA Research, Santa Clara, CA, USA · School of Oceanography, University of Washington, Seattle, WA, USA · Department of Atmospheric & Climate Science, University of Washington, Seattle, WA, USA

  28. Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation

    Dec 24, 2025Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi +5CoT DistillationSelective Knowledge Distillation

    1The University of British Columbia · 2LinkedIn · 3Canada Research Chair in NLP and ML

  29. Generative Manifold Distillation: Aligning Restoration Trajectories with Natural Image Prior

    Dec 11, 2025Yuyang Hu, Mojtaba Sahraee-Ardakan, Arpit Bansal +4Source-Free Domain AdaptationImage Restoration

    Google · Washington University in St. Louis

  30. Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safety

    Dec 8, 2025Max Zhang, Derek Liu, Kai Zhang +2Adversarial Attacks on LLMsLLM Safety Alignment