Knowledge Distillation

Follow this topic Ask about this topic

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representations
May 8, 2026Yuan Zhang, Chenyi Li, Guoqing Ma +7Diffusion Model Distillation Diffusion Sampling
JD Explore Academy, China · The Hong Kong University of Science and Technology · Tsinghua University +1
Attention Transfer Is Not Universally Effective for Vision Transformers
May 8, 2026Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +4Self-Attention Vision Transformer
Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · School of Artificial Intelligence, Sichuan University · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University
Structural Rationale Distillation via Reasoning Space Compression
May 8, 2026Jialin Yang, Jiankun Wang, Jiajun Wu +3CoT Distillation Structured Reasoning
University of Calgary · Calgary, Canada · University of Michigan +1
How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignment
May 7, 2026Rui Zhu, Weiheng Bai, Qiushi Wu +3LLM Alignment KV-Cache Compression
Yale University · University of Minnesota Twin Cities · Indiana University Bloomington
Model-to-Data Distillation for Graph Neural Networks
May 7, 2026Debolina Halder Lina, Arlei SilvaGraph Structure Learning Graph Neural Networks
Department of Computer Science, Rice University, Houston, TX 77005, USA · Department of Computer Science & Ken Kennedy Institute, Rice University, Houston, TX 77005, USA
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
May 7, 2026Yiqiao Jin, Yiyang Wang, Lucheng Fu +7Language Model Distillation Self-Distillation
Georgia Institute of Technology · University of California, Los Angeles · Carnegie Mellon University +1
CKT-WAM: Parameter-Efficient Context Knowledge Transfer Between World Action Models
May 7, 2026Yuhua Jiang, Yijun Guo, Hongbing Yang +7AI Control Robotic Manipulation
Tsinghua University · LivsynRobotics · Shanghai AI Laboratory
TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models
May 7, 2026Bibin WilsonSelf-Supervised Learning Self-Supervised Pre-Training
Estimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation
May 7, 2026Huizi Cui, Huan Ma, Qilin Wang +2Language Model Distillation LLM Uncertainty Estimation
School of Computer Science and Technology, Tianjin University, China · School of Future Technology, Tianjin University, China · Georgia Tech Shenzhen Institute, Tianjin University, China +1
FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings
May 7, 2026Quang-Huy Nguyen, Jiaqi Wang, Wei-shinn KuFederated Knowledge Distillation Knowledge Distillation
Department of Computer Science and Software Engineering Auburn University Auburn, AL 36849
Learned Neighbor Trust for Collaborative Deployment in Model-Agnostic Decentralized Learning
May 6, 2026Michael Lanier, Luise Ge, Sastry Kompella +1Ensemble Learning Decentralized Learning
Department of Computer Science Washington University in St. Louis and Nexcepta
Deep Reprogramming Distillation for Medical Foundation Models
May 6, 2026Siyuan Du, Yuhang Zhou, Haolin Li +5Medical Imaging Foundation Models Vision Foundation Model Adaptation
School of Computer Science, Fudan University and with the Shanghai AI Laboratory, Shanghai, China · Shanghai AI Laboratory, Shanghai, China · Cooperative Medianet Innovation Center, Shanghai Jiao Tong University and with the Shanghai AI Laboratory, Shanghai, China +3
Real Image Denoising with Knowledge Distillation for High-Performance Mobile NPUs
May 5, 2026Faraz Kayani, Sarmad Kayani, Asad Ahmed +2Efficient Neural Network Inference Knowledge Distillation
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions
May 5, 2026Qingzhuo Wang, Ruiyang Qin, Zhenxin Qin +2Feature Interaction Modeling LLM Interpretability
School of Computer Science and Technology, Tongji University, Shanghai, China.
Dynamic Distillation and Gradient Consistency for Robust Long-Tailed Incremental Learning
May 5, 2026Taigo Sakai, Kazuhiro HottaClass-Incremental Learning Long-Tail Learning
Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan
Standing on the Shoulders of Giants: Stabilized Knowledge Distillation for Cross--Language Code Clone Detection
May 4, 2026Mohamad Khajezade, Fatemeh H. Fard, Mohamed Sami ShehataLanguage Model Distillation Knowledge Distillation
University of British Columbia, Okanagan Campus, Kelowna, Canada · University of British Columbia, Okanagan Campus, Kelowna, British Columbia, Canada
Combining Trained Models in Reinforcement Learning
May 4, 2026Ujjwal Patil, Javad GhofraniReinforcement Learning Knowledge Distillation
Department of Computer Science Hochschule Bonn-Rhein-Sieg (H-BRS) Sankt Augustin, Germany
EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer
May 3, 2026Hao Zhang, Zhibin Zhang, Guangxin Wu +3LLM Compression Language Model Distillation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Multi-Dataset Cross-Domain Knowledge Distillation for Unified Medical Image Segmentation, Classification, and Detection
May 2, 2026Ceausescu Ciprian-Mihai, Anghelina Ion-Marian, Alexe Dumitru-BogdanMedical Image Object Detection Multi-Task Learning
Faculty of Mathematics and Computer Science, University of Bucharest, Str. Academiei, 14, Bucharest, 010014, Romania · Gheorghe Mihoc-Caius Iacob Institute of Mathematical Statistics and Applied Mathematics of the Romanian Academy, Romania, Calea 13 Septembrie, 13, Bucharest, 050711, Romania
MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation
May 2, 2026Pham Khanh Chi, Quoc Phong Dao, Thuat Nguyen +3Representation Learning Language Model Distillation
1Hanoi University of Science and Technology · 2Monash University · University of Oregon
AgriKD: Cross-Architecture Knowledge Distillation for Efficient Leaf Disease Classification
May 2, 2026Minh-Dung Le, Minh-Duc Hoang, Hoang-Vu Truong +1Efficient Neural Network Inference Plant Disease Classification
AIT laboratory, Faculty of Artificial Intelligence, FPT University, Da Nang, 550000, Viet Nam
SRA: Span Representation Alignment for Large Language Model Distillation
May 2, 2026Quoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
Hanoi University of Science and Technology · VNU University of Engineering and Technology · Monash University
LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
May 1, 2026Shashank Kapadia, Deep Naryan Mishra, Sujal Reddy Alugubelli +4Efficient Transformer Inference Dynamic Neural Networks
Walmart Inc. · Sam’s Club
AsymTalker: Identity-Consistent Long-Term Talking Head Generation via Asymmetric Distillation
May 1, 2026Yuxin Lu, Jiayang Sun, Guibo Zhu +1Video Diffusion Models Long Video Generation
Soochow University · Institute of Automation, Chinese Academy of Sciences
Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding
May 1, 2026Yan Zhang, Daiqing Wu, Huawen Shen +2GUI Grounding On-Policy Distillation
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · VCIP & TMCC & DISSec, College of Computer Science, Nankai University
Learning from Compressed CT: Feature Attention Style Transfer and Structured Factorized Projections for Resource-Efficient Medical Image Analysis
May 1, 2026Shadid Yousuf, S. M. Mahbubur Rahman, Mohammed Imamul Hassan BhuiyanModel Compression Medical Image Analysis
Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology
Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL
Apr 30, 2026Sudong Wang, Weiquan Huang, Xiaomin Yu +9Multimodal Large Language Models Multimodal Reasoning
Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University · Tsinghua University +3
Energy-Efficient Plant Monitoring via Knowledge Distillation
Apr 29, 2026Ilyass Moummad, Reda Bensaid, Kawtar Zaher +5Plant Disease Classification Energy-Efficient ML
LIRMM, Univ Montpellier, CNRS, Inria, Montpellier, France · IMT Atlantique, Brest, France. Polytechnique Montréal, Canada · Institut National de l’Audiovisuel, Paris, France +2
Lightweight Distillation of SAM 3 and DINOv3 for Edge-Deployable Individual-Level Livestock Monitoring and Longitudinal Visual Analytics
Apr 29, 2026Haiyu Yang, Miel HostensAgricultural Image Analysis Edge ML
College of Agriculture and Life Sciences, Cornell University, Ithaca, NY 14853 USA
Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models
Apr 29, 2026Gongbo Zhang, Wen Wang, Ye Tian +1Cross-Tokenizer Knowledge Distillation Diffusion Model Distillation
Peking University · Zhejiang University

May 8, 2026J/K move · Enter open · S save

Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representations

Yuan Zhang, Chenyi Li, Guoqing Ma, Jiajun Zha+6

JD Explore Academy, China · The Hong Kong University of Science and Technology · Tsinghua University+1

Diffusion Model Distillation Diffusion Sampling Diffusion-Based Image Generation Feature-Based Knowledge Distillation Knowledge Distillation

Open paper AskPDF arXiv

Sampling from pretrained diffusion and flow-matching models typically requires many forward passes to generate diverse and high-fidelity images. Existing distillation methods often rely on multiple auxiliary networks, carefully designed training stages, or complex optimization pipelines. In this work, we revisit the recently proposed Drifting Model objective and show that a single drifting loss can be directly used to simplify one step distillation. A key observation is that the pretrained diffusion teacher itself already provides a strong representation space. Unlike the original Drifting Model, which relies on an additional pretrained feature extractor, we use intermediate hidden states of the pretrained teacher model as the feature representation. This removes the need for training or introducing an extra representation network while preserving a semantically meaningful feature geometry for drifting. Furthermore, we introduce a lightweight mode coverage loss to mitigate mode collapse during distillation and encourage the student generator to cover diverse teacher-supported regions. Extensive experiments on ImageNet and SDXL demonstrate that our method achieves efficient one step generation with competitive image quality and diversity, achieving FID scores of 1.58 on ImageNet-64

\times

64 and 18.4 on SDXL, while substantially simplifying the overall distillation framework.