Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

887 papers87 papers in the last four weeks, down 49% on the four weeks before. 0.7% of all new papers.

  1. Spectrally Distilled Representations Aligned with Instruction-Augmented LLMs for Satellite Imagery

    Feb 26, 2026Minh Kha Do, Wei Xiang, Kang Han +5Remote Sensing VLMsRemote Sensing Image Understanding

  2. Decoder-based Sense Knowledge Distillation

    Feb 25, 2026Qitong Wang, Mohammed J. Zaki, Georgios Kollias +1Language Model DistillationKnowledge Distillation

  3. Distilling Deep Reinforcement Learning into Interpretable Fuzzy Rules: An Explainable AI Framework

    Feb 24, 2026Sanup S. Araballi, Simon Khan, Chilukuri K. MohanInterpretability in RLKnowledge Distillation

  4. Momentum Memory for Knowledge Distillation in Computational Pathology

    Feb 24, 2026Yongxin Guo, Hao Lu, Onur C. Koyun +3Knowledge DistillationCross-Modal Knowledge Distillation

  5. Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation

    Feb 24, 2026Sayantan Dasgupta, Trevor Cohn, Timothy BaldwinLanguage Model DistillationKnowledge Distillation

  6. GATES: Self-Distillation under Privileged Context with Consensus Gating

    Feb 24, 2026Alex Stein, Furong Huang, Tom GoldsteinKnowledge DistillationOn-Policy Self-Distillation

  7. On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inference

    Feb 23, 2026Moritz A. Zanger, Yijun Wu, Pascal R. Van der Vaart +2Uncertainty QuantificationBayesian Inference

  8. Federated Reasoning Distillation Framework with Model Learnability-Aware Data Allocation

    Feb 21, 2026Wei Guo, Siyuan Lu, Xiangdong Ran +8Reasoning DistillationFederated Knowledge Distillation

  9. ColBERT-Zero: To Pre-train Or Not To Pre-train ColBERT models

    Feb 18, 2026Antoine Chaffin, Luca Arnaboldi, Amélie Chatelain +1Multi-Vector RetrievalKnowledge Distillation

  10. Task-Specific Knowledge Distillation via Intermediate Probes

    Feb 18, 2026Ryan Brown, Chris RussellLanguage Model DistillationKnowledge Distillation

  11. jina-embeddings-v5-text: Task-Targeted Embedding Distillation

    Feb 17, 2026Mohammad Kalim Akram, Saba Sturua, Nastia Havriushenko +4Text Embedding ModelsKnowledge Distillation

  12. Logit Distance Bounds Representational Similarity

    Feb 17, 2026Beatrix M. G. Nielsen, Emanuele Marconato, Luigi Gresele +2Representational Similarity AnalysisRepresentation Learning

  13. Protecting Language Models Against Unauthorized Distillation through Trace Rewriting

    Feb 16, 2026Xinhang Ma, William Yeoh, Ning Zhang +1LLM WatermarkingKnowledge Distillation

  14. LLM-Guided Knowledge Distillation for Temporal Knowledge Graph Reasoning

    Feb 16, 2026Wang Xing, Wei Song, Siyu Lin +2Knowledge DistillationTemporal KGs

  15. Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation

    Feb 10, 2026Joseph Attieh, Timothee Mickus, Anne-Laure Ligozat +2Energy-Efficient MLKnowledge Distillation

  16. Distill3R: A Pipeline for Democratizing 3D Foundation Models on Commodity Hardware

    Jan 31, 2026Brandon Leblanc, Charalambos Poullis3D Foundation ModelsKnowledge Distillation

  17. Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

    Jan 30, 2026Yanlong Chen, Amirhossein Habibian, Luca Benini +1Cross-Modal Knowledge DistillationVision-Language Models

  18. OVD: On-policy Verbal Distillation

    Jan 29, 2026Jing Xiong, Hui Shen, Shansan Gong +7Black-Box Knowledge DistillationLanguage Model Distillation

  19. Advancing Model Refinement: Muon-Optimized Distillation and Quantization for LLM Deployment

    Jan 14, 2026Jacob Sander, Brian Jalaian, Venkat R. DasariLLM CompressionLocal LLM Deployment

  20. SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines

    Jan 4, 2026Itai Morad, Nir Shlezinger, Yonina C. EldarStatistical Learning TheoryTeacher-Student Learning

  21. Long-Range Distillation: Distilling 10,000 Years of Simulated Climate into Long Timestep AI Weather Models

    Dec 28, 2025Scott A. Martin, Noah Brenowitz, Dale Durran +1Weather ForecastingSpatiotemporal Forecasting

  22. Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation

    Dec 24, 2025Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi +5CoT DistillationSelective Knowledge Distillation

  23. Generative Manifold Distillation: Aligning Restoration Trajectories with Natural Image Prior

    Dec 11, 2025Yuyang Hu, Mojtaba Sahraee-Ardakan, Arpit Bansal +4Source-Free Domain AdaptationImage Restoration

  24. LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection

    Dec 5, 2025Johannes Meier, Jonathan Michel, Oussema Dhaouadi +7Efficient Object DetectionReal-Time Object Detection

  25. From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answering

    Nov 14, 2025Yu Zhao, Ying Zhang, Xuhui Sui +4Visual Question AnsweringCoT Distillation

  26. RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models

    Oct 29, 2025Zijun Liao, Yian Zhao, Xin Shan +5Real-Time Object DetectionEfficient Object Detection

  27. Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge

    Oct 22, 2025Penghao Wang, Yuhao Zhou, Mengxuan Wu +3Cross-Architecture Knowledge DistillationKnowledge Distillation