Knowledge Distillation

Ask about this topic

Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

830 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

  1. Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learning

    Jul 8, 2026Maximilian Andreas Hoefler, Karsten Mueller, Wojciech SamekNon-IID Federated LearningFederated Learning

    Fraunhofer Heinrich Hertz Institute · BIFOLD Technical University Berlin

  2. Enhancing deep learning models for time series classification via knowledge distillation

    Jul 7, 2026Javidan Abdullayev, Maxime Devanne, Jonathan Weber +1Time Series ClassificationKnowledge Distillation

    IRIMAS, University of Haute Alsace, Frères Lumière, Mulhouse 68200, France · Faculty of IT, Monash University, Wellington Road, Melbourne 3800, Australia

  3. Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation

    Jul 7, 2026Cemil-Andrei Dilmac, Florinel-Alin Croitoru, Radu Tudor IonescuTraining Data SelectionCoreset Selection

    Department of Computer Science, University of Bucharest, 15G Iuliu Maniu, Bucharest 061075, Romania

  4. Patch Knowledge Transfer for Efficient AI-Generated Image Quality Assessment

    Jul 6, 2026Jiquan YuanImage Quality AssessmentImage Generation Evaluation

    School of Software and Microelectronics, Peking University, Beijing, China

  5. Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models

    Jul 6, 2026Tim Lenz, Maurice Heide, Marco Gustav +2Computational PathologyPathology Foundation Models

    EKFZ for Digital Health, TU Dresden, Dresden, Germany · Pathology, University of Augsburg, Augsburg, Germany · Bavarian Cancer Research Center (BZKF), Augsburg, Germany +3

  6. Trust Region Policy Distillation

    Jul 6, 2026Zhengpeng Xie, Li Lyna Zhang, Zeke Xie +1On-Policy DistillationPolicy Learning

  7. Displacement Preserving Relational Distillation for Robust Medical Segmentation

    Jul 6, 2026Zhicheng Ding, Xinyu Chu, Jung Im Choi +5Model CompressionRelational Knowledge Distillation

    Bowling Green State University, Bowling Green, OH, USA · University of Alabama at Birmingham, Birmingham, AL, USA · The University of Findlay, Findlay, OH, USA +3

  8. Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation

    Jul 5, 2026Kargi Chauhan, Aditya ShahMechanistic InterpretabilityKnowledge Distillation

    University of California, Santa Cruz · 2Google

  9. SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation

    Jul 5, 2026Omer Tariq, Syed Muhammad Raza, Jeongbae SonLow-Rank AdaptationLow-Rank Compression

    Neubility Inc, Seoul, South Korea

  10. XS-VLA: Teaching Tiny Vision-Language-Action Models with Spatial Supervision and Demonstration Conditioning

    Jul 5, 2026Iok Tong Lei, Ying Jie Yap, Wei Huang +5Robot Policy LearningLanguage-Conditioned Robot Manipulation

    Department of Computer Science and Technology, Tsinghua University, Beijing, China. · National College for Excellent Engineers, Beihang University, Beijing, China. · Independent Researcher +1

  11. ContiStain: Cross-Domain Relation-Preserving Distillation for Continual Multi-Domain Virtual IHC Staining

    Jul 4, 2026Fuqiang Chen, Yifeng Wang, Hongpeng Wang +1Computational PathologyVirtual Staining

    School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, China · School of Computer Science and Technology, Tsinghua University, Beijing, China

  12. Teacher Supervision over Representation Equivalence Classes

    Jul 3, 2026Sang Il HanRepresentation IdentifiabilityLanguage Model Distillation

    Korea University of Technology and Education · Biomonster Co.

  13. ComplexMimic: Human-Scene Interaction Imitation in Complex 3D Environments

    Jul 2, 2026Lu Pan, Hongwei ZhaoHuman Behavior SimulationHuman Motion Generation

    College of Computer Science and Technology, Jilin University, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China

  14. SFKD: Spatial--Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction

    Jul 2, 2026Cuipeng Wang, Haipeng WangCross-Architecture Knowledge DistillationKnowledge Distillation

    Key Laboratory for Information Science of Electromagnetic Waves, Ministry of Education, Fudan University, Shanghai, China · Discipline and Technology Center of Microwave Vision Intelligent Sensing, Fudan University, Shanghai, China

  15. Geometric Foundation Model Distillation for Efficient Lunar 3D Reconstruction

    Jul 2, 2026Clémentine Grethen, Florient Chouteau, Géraldine Morin +13D Foundation Models3D Reconstruction

    1 1 IRIT, University of Toulouse, France · 2 2 Airbus Defence and Space, Toulouse, France

  16. C2E: Boosting Ego-Only 3D Object Detection via Multi-Teacher Contrastive Knowledge Distillation

    Jul 2, 2026Jinlong Wang, Xun Huang, Qiming Xia +2Lidar-Based 3D Object Detection3D Object Detection

    Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University, China · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China · Zhongguancun Academy, China

  17. Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation

    Jul 1, 2026Jaehyun Jang, Eunseop Yoon, Hee Suk Yoon +3Cross-Modal Knowledge DistillationVLM Distillation

    Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea · University of Illinois Urbana-Champaign, Champaign, USA

  18. OnPoint: Offline-to-Online Multi-Level Distillation for Point-Supervised Online Temporal Action Localization

    Jul 1, 2026Sakib Reza, Gauri Jagatap, Mohsen Moghaddam +2Knowledge DistillationTemporal Video Understanding

    Northeastern University, Boston, MA 02115, USA · Georgia Institute of Technology, Atlanta, GA 30332, USA · Dolby Laboratories +1

  19. TallyTrain: Communication-Efficient Federated Distillation

    Jun 30, 2026Radhakrishna Achanta, Will ReedFederated Learning AggregationCommunication-Efficient Distributed Training

    Cisco Systems Inc.

  20. Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification

    Jun 30, 2026Aizierjiang AiersilanData LeakagePoint Cloud Classification

    University of Macau

  21. Improving Certified Robustness via Adversarial Distillation

    Jun 30, 2026Matteo Melis, Jesus Martinez Del Rincon, Vishal SharmaNeural Network RobustnessCertified Robustness

    School of EEECS Queen’s University Belfast

  22. Holonic Active Distillation for Scalable Multi-Agent Learning in Multi-Sensor Systems

    Jun 30, 2026Dani Manjah, Tim Bary, Benoît Macq +1Multi-Agent SystemsMulti-Teacher Knowledge Distillation

    Institute of Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM), UCLouvain, 1348 Louvain-la-Neuve, Belgium · Université de Technologie de Belfort Montbéliard, UTBM, CIAD UR 7533, F-90010 Belfort cedex, France

  23. Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition

    Jun 30, 2026Gaurab Baral, Aaditya Khanal, Yangyang Tao +1CoT DistillationMathematical Reasoning

    Northern Kentucky University, Highland Heights, KY 41099, USA

  24. Staged Hybridisation for Visual Quantum Reinforcement Learning via Knowledge Distillation

    Jun 29, 2026Javier Lazaro, Juan-Ignacio Vazquez, Pablo Garcia-BringasHybrid Quantum-Classical MLVariational Quantum Circuits

    University of Deusto, Bilbao, Spain · FSAS International Quantum Center, Fujitsu

  25. DuoMem: Towards Capable On-Device Memory Agents via Dual-Space Distillation

    Jun 29, 2026Peyman Hosseini, Ondrej Bohdal, Ahmed Alajrami +6LLM AgentsLLM Agent Training

    Samsung R&D Institute UK · Queen Mary University of London

  26. ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation

    Jun 29, 2026Zilong Liu, Xuewen Zhang, Jinrui Xing +3RL for Knowledge DistillationLanguage Model Distillation

    Li Auto, Beijing, China · School of Software and Microelectronics, Peking University, China

  27. KbSD: Knowledge Boundary aware Self-Distillation for Behavioral Calibration in Agentic Search

    Jun 29, 2026Tao Feng, Xinke Jiang, Chao WuLanguage Model CalibrationAgentic RAG

    Zhejiang University, Hangzhou, China · Peking University, Beijing, China

  28. PHF: Privileged Hidden Flow for On-Policy Self-Distillation

    Jun 28, 2026Yuhan Li, Mingxu Zhang, Dazhong Shen +1On-Policy Self-DistillationLanguage Model Distillation

    1The Hong Kong University of Science and Technology (Guangzhou) · 2Nanjing University of Aeronautics and Astronautics · 3The 63rd Research Institute, National University of Defense Technology, Nanjing

  29. DistilledGemma: Balanced Efficiency-Accuracy for Person-Place Relation Extraction from Multilingual Historical Articles

    Jun 28, 2026Youssef Aboelwafa, Ahmed Samir, Nagwa Elmakky +1Relation ExtractionHistorical Document Analysis

    1Alexandria University, Egypt

  30. Labeling Training Data for Entity Matching Using Large Language Models

    Jun 27, 2026Aaron Steiner, Christian BizerLLM-Assisted AnnotationKnowledge Distillation

    Data and Web Science Group University of Mannheim Mannheim, Germany