Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

887 papers87 papers in the last four weeks, down 49% on the four weeks before. 0.7% of all new papers.

  1. Streamline pathology foundation model by cross-magnification distillation

    Sep 27, 2025Ziyu Su, Abdul Rehman Akbar, Usama Sajjad +2Pathology Foundation ModelsComputational Pathology

  2. Resource-Limited Joint Multimodal Sentiment Reasoning and Classification via Chain-of-Thought Enhancement and Distillation

    Aug 7, 2025Haonan Shangguan, Xiaocui Yang, Shi Feng +3Efficient Multimodal InferenceSentiment Analysis

  3. Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation

    Jul 24, 2025Lakshmana Sri Harsha Nemani, P. K. Srijith, Tomasz KuśmierczykLLM Uncertainty EstimationUncertainty-Aware Knowledge Distillation

  4. A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

    May 19, 2025Jitai Hao, Qiang Huang, Hao Liu +3Language Model PretrainingLLM Pruning

  5. Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning

    May 17, 2025Chen Shu, Boyu Fu, Yiman Li +5Multiple Instance LearningHistopathology Image Classification

  6. How to Backdoor Image Knowledge Distillation

    Apr 30, 2025Qian Ma, Chen Wu, Prasenjit Mitra +1Adversarial AttacksData Poisoning Attacks

  7. KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation

    Apr 26, 2025Jiabin Fan, Guoqing Luo, Michael Bowling +1RL for Knowledge DistillationLanguage Model Distillation

  8. UINO-FSS: Unifying Representation Learning and Few-shot Segmentation via Hierarchical Distillation and Mamba-HyperCorrelation

    Apr 22, 2025Wei Zhuo, Zhiyue Tang, Wufeng Xue +3Few-Shot Semantic SegmentationMultimodal Foundation Models

  9. Information-Geometric Inverse Distillation for Enhancing Adversarial Transferability

    Feb 24, 2025Wenyuan Wu, Yuan Sun, Yingke Chen +4Transfer-Based Adversarial AttacksKnowledge Distillation

  10. Tuned Reverse Distillation: Enhancing Multimodal Industrial Anomaly Detection with Crossmodal Tuners

    Dec 12, 2024Xinyue Liu, Jianyuan Wang, Biao Leng +1Cross-Modal Knowledge DistillationIndustrial Anomaly Detection

  11. Collaborative Knowledge Distillation via a Learning-by-Education Node Community

    Sep 30, 2024Anestis Kaimakamidis, Ioannis Mademlis, Ioannis PitasOnline Knowledge DistillationContinual Learning

  12. LLMR: Knowledge Distillation with a Large Language Model-Induced Reward

    Sep 19, 2024Dongheng Li, Yongchang Hao, Lili MouRL for Knowledge DistillationLanguage Model Distillation

  13. Relational Representation Distillation

    Jul 16, 2024Nikos Giakoumoglou, Tania StathakiContrastive LearningRelational Knowledge Distillation

  14. Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework

    Mar 2, 2024Junxian Li, Bin Shi, Erfei Cui +2Graph Positional EncodingKnowledge Distillation