Knowledge Distillation

Ask about this topic

Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

  1. Soil Moisture Estimation in Precision Agriculture: A Knowledge Distillation Approach

    Oct 29, 2026Mohammed Rakib, Vrinda Nagula, Arunkumar BagavathiPrecision AgricultureAgricultural Remote Sensing

    Oklahoma State University · Aurora High School

  2. Distilling Large-Scale Machine Learning Interatomic Potentials to Predict Long-Timescale Electrolyte Transport Properties

    Oct 29, 2026Ishan Amin, Yue Jian, Nitesh Kumar +10Machine Learning Interatomic PotentialsMaterials Property Prediction

    Department of Computer Science, University of California, Berkeley, Berkeley, California 94720, USA · Department of Chemical and Biomolecular Engineering, University of California, Berkeley, Berkeley, California 94720, USA · Materials Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, California 94720, USA +3

  3. YOLOF-MLD: A Multi-Level Distillation Framework for Efficient Object Detection

    Oct 29, 2026Xiaolong Wu, Liming Zheng, Zitong Liu +1Efficient Object DetectionObject Detection

    The Chinese University of Hong Kong (Shenzhen), Shenzhen, Guangdong, 518172, P.R. China · The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China

  4. Distilling the Shape of Uncertainty: What On-Policy Distillation Learns and How to Predict It

    Oct 28, 2026Shiping Yang, Chuanqi Tang, Hossein Hajimirsadeghi +3Language Model DistillationOn-Policy Distillation

    Simon Fraser University · Borealis AI

  5. Information-Theoretic Reliability--Safety Bounds and Finite-Alignment Dynamics in AI Knowledge Distillation

    Oct 28, 2026Miao-Miao Yi, Zhen-Yu Cai, Y -M. Du +1Trustworthy AIKnowledge Distillation

    Graduate School of China Academy of Engineering Physics, Beijing, 100193, China

  6. What Can Firms Do in the Face of Model Distillation?

    Oct 28, 2026Diptangshu Sen, Jibang Wu, Juba ZianiModel Extraction AttacksKnowledge Distillation

    Georgia Institute of Technology · New York University Shanghai

  7. You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces

    Oct 27, 2026Hamza Etcibasi, Ramazan Gokberk CinbisKnowledge DistillationFeature-Based Knowledge Distillation

    Department of Computer Engineering, Middle East Technical University, Ankara, Türkiye · ASELSAN Inc., Ankara, Türkiye · ROMER Robotics and AI Center, Middle East Technical University, Ankara, Türkiye

  8. What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2

    Oct 27, 2026Nabeelah Faumi, Deshan SumanathilakaLanguage Model DistillationKnowledge Distillation

    Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka

  9. Ground-to-Satellite Knowledge Distillation for Crop Classification under Temporal and Geographic Shift

    Oct 27, 2026Miro Miranda, Rocco Sedona, Valerio Marsocci +2Agricultural Remote SensingKnowledge Distillation

    Jülich Supercomputing Centre, Forschungszentrum Jülich, Germany · European Space Agency (ESA/ESRIN), Italy · University of Iceland, Iceland +1

  10. How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning

    Oct 26, 2026Johnny Yu, Cyril GorllaSelf-DistillationKnowledge Distillation

    University of California, Berkeley Berkeley, California, USA · CTGT San Francisco, California, USA

  11. Joint Latent Reasoning and Observation for Agentic Video Understanding

    Oct 26, 2026Junzhang Liu, Hao Yuan, Yiming Wu +3Video UnderstandingKnowledge Distillation

    Hong Kong University of Science and Technology · Shanghai Artificial Intelligence Laboratory · The University of Hong Kong +1

  12. MuSID-KD: Multigranularity Knowledge Distillation for Generative Recommendation with Semantic IDs

    Oct 26, 2026Fengshi Zhang, Xiyan Ding, Shuxiao Xie +1Knowledge DistillationGenerative Recommendation

    East China Normal University Shanghai, China · Fudan University Shanghai, China

  13. DAC-Loop: Distillation-Aware Compression for Remote Sensing CNN

    Oct 26, 2026Rico Staedeli, Lorena Raichle, Konstantinos Tzevelekakis +2Neural Network CompressionNeural Network Pruning

    AIML Lab, School of Computer Science, University of St. Gallen

  14. RSKD: Grounding Semantic Consistency in Driving VLMs via Distillation

    Oct 26, 2026Chuchu Xie, Yang Zhong, Kangan Qian +3VLM DistillationKnowledge Distillation

    School of Vehicle and Mobility, Tsinghua University, Beijing, China · Automotive and Robotics, Xiaomi Corporation, Beijing, China

  15. Knowledge distillation across teacher architectures for compact diabetic retinopathy grading

    Oct 25, 2026Mossa Manisa, Habiba Jahan Priya, A. K. M. Fakhrul HossainKnowledge DistillationCross-Architecture Knowledge Distillation

    Department of Computer Science and Engineering, Shahjalal University of Science and Technology, Sylhet, Bangladesh

  16. TAD: Distilling Vision-Language-Action Policies from Teacher Trajectories Alone

    Oct 24, 2026Yun Zhu, Da WangVisuomotor Policy LearningKnowledge Distillation

    Independent Researcher, China · School of Computer and Information Technology, Shanxi University, Taiyuan, China

  17. Knowledge Distillation across Item and Code Spaces for Parallel Generative Recommendation

    Oct 23, 2026Yixuan Xu, Hailin Liu, Linxiang Peng +1Semantic ID RecommendationGenerative Recommendation

    The Hong Kong University of Science and Technology, Hong Kong, China · University of Science and Technology of China, Hefei, China

  18. Yalda-Embedding: Open Development of a Competitive Persian Retrieval Model

    Oct 23, 2026Yalda Taheri, Mohammad Hassan Heydari, Armon RasooliText Embedding ModelsInformation Retrieval

    Faculty of Engineering, Azad University · Faculty of Computer Engineering, University of Isfahan · Department of Electrical Engineering, Iran University of Science and Technology

  19. Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?

    Oct 22, 2026Ya Gao, Zhengtao Zou, Pekka MarttinenOn-Policy DistillationKnowledge Distillation

    Aalto University

  20. HERD: Harness-Evolving Recursive Distillation for Continual Learning

    Oct 22, 2026Xun Sun, Thanh Thi Nguyen, Thuy-Trang VuContinual LearningContinual Learning for LLM Agents

    Monash University · University of the Sunshine Coast

  21. One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation

    Oct 22, 2026Khouloud Saadi, Di WangOn-Policy Self-DistillationKnowledge Distillation

    Department of Computer Science, King Abdullah University of Science and Technology, KAUST, Thuwal, Saudi Arabia

  22. Reverse-KL Policy Distillation: Statistical Guarantees and Limits

    Oct 22, 2026Haolin Liu, Chen-Yu Wei, Julian ZimmertLanguage Model DistillationKnowledge Distillation

    University of Virginia · Google Research · National Taiwan University

  23. GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL

    Oct 22, 2026Shijie Li, Yilin Gao, Tieyuan Chen +3RL for Language Model ReasoningLLM Reasoning

    Shanghai Jiao Tong University · Tencent

  24. CODI: Improving Self-Supervised Vision Transformers with Cross Distillation

    Oct 22, 2026Siran Dai, Qianqian Xu, Peisong Wen +3Vision TransformerSelf-Supervised Visual Representation Learning

    Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, CAS +3

  25. Benchmarking Model Compression Techniques Across Heterogeneous Edge Platforms: A Predictive Maintenance Use Case for Rotating Machinery

    Oct 21, 2026Matheus Calheira Guimarães de Oliveira, Marco Aurélio da Silva Cruz, Regis Cardoso +9Model CompressionEdge Inference

    SENAI CIMATEC University, Salvador, BA, Brazil · Surrey Institute for People-Centred AI, School of Computer Science and Electronic Engineering, Faculty of Engineering and Physical Sciences, University of Surrey, Guildford, UK · SENAI Institute of Innovation in Embedded Systems, Florianópolis, SC, Brazil +6

  26. CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

    Oct 21, 2026Wenhao Yu, Zhengcheng Wang, Jiahong Liu +7Gradient InterferenceSelf-Distillation

    The Chinese University of Hong Kong · Tencent · Peking University

  27. How Far Can Foundation Models Be Distilled for Efficient Jet Tagging?

    Oct 21, 2026Jay Agarwal, Siddharth Khare, Dhruv KumarJet TaggingKnowledge Distillation

    BITS Pilani, KEK · BITS Pilani

  28. D-Skill: Distilling Teacher Knowledge through Agent Skills

    Oct 20, 2026Shuo Yuan, Dacheng Li, Joseph E. GonzalezSkill DistillationKnowledge Distillation

    University of California, Berkeley

  29. CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

    Oct 20, 2026Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge +3Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

  30. De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control

    Oct 20, 2026Shaoyi Zheng, Zhongtao Jiang, Yue Peng +4Efficient Inference for World Action ModelsWorld Models for VLA Models

    ByteDance. · New York University (NYU).