Knowledge Distillation

Follow this topic Ask about this topic

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

Quantum-Gated Task-interaction Knowledge Distillation for Pre-trained Model-based Class-Incremental Learning
Apr 13, 2026Linjie Li, Huiyu Xiao, Jiarui Cao +2Class-Incremental Learning Knowledge Distillation
School of Information and Communication Engineering, BUPT, China · Engineering Research Center for Information Network, Ministry of Education, China
DIB-OD: Preserving the Invariant Core for Robust Heterogeneous Graph Adaptation via Decoupled Information Bottleneck and Online Distillation
Apr 13, 2026Yang Yan, Yunxuan Li, Qiuyan Wang +2Online Knowledge Distillation Graph Transfer Learning
School of Information Technology and Engineering, Tianjin University of Technology and Education, Tianjin, China · School of Computer Science and Technology, Tiangong University, Tianjin, China · Department of Computer Science at University of Exeter
LIDEA: Human-to-Robot Imitation Learning via Implicit Feature Distillation and Explicit Geometry Alignment
Apr 12, 2026Yifu Xu, Bokai Lin, Xinyu Zhan +4Human-to-Robot Motion Imitation Cross-Embodiment Robot Learning
Shanghai Jiao Tong University · Shanghai Innovation Institute · Noematrix Ltd.
MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
Apr 11, 2026Wenchang Duan, Zhenguo Gao, Jinguo Xian +1Model Compression Multi-Agent Trajectory Prediction
School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai 200240, China · Bio-X Institutes, Key Laboratory for the Genetics of Developmental and Neuropsychiatric Disorders, Shanghai Jiao Tong University, Shanghai 200030, China · Shanghai Key Laboratory of Psychotic Disorders, and Brain Science and Technology Research Center, Shanghai Jiao Tong University, Shanghai 200030, China
AsymLoc: Towards Asymmetric Feature Matching for Efficient Visual Localization
Apr 10, 2026Mohammad Omama, Gabriele Berton, Eric Foxlin +1Visual Place Recognition Edge Inference
The University of Texas at Austin · Work done as a part of a summer internship at Amazon. · Amazon
EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation
Apr 10, 2026Shu-Hao Zhang, Le-Tong Huang, Xiang-Sheng Deng +5Quantization-Aware Distillation LLM Quantization
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210063, China · School of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China · Microsoft AI, Beijing 100080, China +1
Continual Distillation of Teachers from Different Domains
Apr 10, 2026Nicolas Michel, Maorong Wang, Jiangpeng He +1Knowledge Distillation Catastrophic Forgetting
The University of Tokyo · Japanese-French Laboratory for Informatics, CNRS · National Institute of Informatics +1
Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation
Apr 10, 2026Yutong Zhang, Jiaxin Chen, Honglin Chen +5Memory-Efficient Fine-Tuning Efficient Neural Network Inference
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China · School of Computer Science and Engineering, Beihang University, China · College of Information Technology, United Arab Emirates University, United Arab Emirates
Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
Apr 10, 2026Tokio Kajitsuka, Ukyo Honda, Sho TakaseCoT Distillation Reasoning Distillation
The University of Tokyo, Tokyo, Japan · CyberAgent, Tokyo, Japan
MedThink: Enhancing Diagnostic Accuracy in Small Models via Teacher-Guided Reasoning Correction
Apr 9, 2026Xinchun Su, Chunxu Luo, Lipeng Ma +2Language Model Distillation Knowledge Distillation
School of Computer Science, Fudan University, Shanghai, China
Orion-Lite: Distilling LLM Reasoning into Efficient Vision-Only Driving Models
Apr 9, 2026Jing Gu, Niccolò Cavagnero, Gijs DubbelmanEnd-to-End Autonomous Driving VLM Distillation
Eindhoven University of Technology
KD-MARL: Resource-Aware Knowledge Distillation in Multi-Agent Reinforcement Learning
Apr 8, 2026Monirul Islam Pavel, Siyi Hu, Muhammad Anwar Masum +3Policy Distillation Knowledge Distillation
School of Computer Science and Information Technology, Adelaide University, Australia · School of Electrical Engineering, Computing and Mathematical Sciences, Curtin University, Australia · Systems Research Institute, Polish Academy of Sciences, Poland
A Parameter-Efficient Transfer Learning Approach through Multitask Prompt Distillation and Decomposition for Clinical NLP
Apr 8, 2026Cheng Peng, Mengxian Lyu, Ziyi Chen +1Parameter-Efficient Fine-Tuning Transfer Learning
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA · Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, Florida, USA
Purify-then-Align: Towards Robust Human Sensing under Modality Missing with Knowledge Distillation from Noisy Multimodal Teacher
Apr 7, 2026Pengcheng Weng, Yanyu Qian, Yangxin Xu +1Missing-Modality Learning Multimodal Robustness
School of Software Engineering, Xi’an Jiaotong University, China · Institute of Computer Science, Universität Bern, Switzerland · College of Computing and Data Science, Nanyang Technological University, Singapore
TORA: Topological Representation Alignment for 3D Shape Assembly
Apr 5, 2026Nahyuk Lee, Zhiang Chen, Marc Pollefeys +1Flow Matching 3D Shape Generation
Independent Researcher · ETH Zurich · ETH AI Center
Geometric Limits of Knowledge Distillation: A Minimum-Width Theorem via Superposition Theory
Apr 5, 2026Nilesh Sarkar, Dawar Jyoti DekaKnowledge Distillation Neural Representation Geometry
Erd˝os AI Labs
SODA: Semi On-Policy Black-Box Distillation for Large Language Models
Apr 4, 2026Xiwen Chen, Jingjing Wang, Wenhui Zhu +7Language Model Distillation Knowledge Distillation
Clemson University · LinkedIn · Washington University in St. Louis +2
Training a Student Expert via Semi-Supervised Foundation Model Distillation
Apr 4, 2026Pardis Taghavi, Tian Liu, Renjie Li +2Knowledge Distillation Instance Segmentation
Texas A&M University
Reliability Gated Multi-Teacher Distillation for Low Resource Abstractive Summarization
Apr 3, 2026Dipto Sumit, Ankan Kumar Roy, Sadia Khair Rodela +4Text Summarization Multi-Teacher Knowledge Distillation
BRAC University
PRISM: LLM-Guided Semantic Clustering for High-Precision Topics
Apr 3, 2026Connor Douglas, Utkucan Balci, Joseph Aylett-BullockSemantic Clustering Topic Modeling
New York University New York, NY, USA · Binghamton University Binghamton, NY, USA · United Nations New York, NY, USA
DSBD: Dual-Aligned Structural Basis Distillation for Graph Domain Adaptation
Apr 3, 2026Yingxu Wang, Kunyu Zhang, Jiaxin Huang +4Graph Domain Adaptation Domain Adaptation
Zhengzhou University · The Hebei University of Technology · California State Polytechnic University +1
Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization
Apr 3, 2026Zihe Liu, Yulong Mao, Jinan Xu +2Language Model Distillation Knowledge Distillation
Key Laboratory of Big Data & Artificial Intelligence in Transportation · School of Computer Science and Technology, Beijing Jiaotong University
Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selection
Apr 3, 2026Chaoqun He, Yingfa Chen, Chaojun Xiao +2Language Model Distillation CoT Distillation
Tsinghua University
Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge
Apr 3, 2026Yiyang Shen, Lifu Tu, Weiran WangRL for Language Model Reasoning Knowledge Distillation
University of Iowa · Oracle AI
Rapidly deploying on-device eye tracking by distilling visual foundation models
Apr 2, 2026Cheng Jiang, Jogendra Kundu, David Colmenares +4Gaze Estimation Vision Foundation Model Adaptation
Meta Reality Labs, USA
Diff-KD: Diffusion-based Knowledge Distillation for Collaborative Perception under Corruptions
Apr 2, 2026Pengcheng Lyu, Chaokun Zhang, Gong Chen +2Cooperative Perception Autonomous Driving Perception
School of Future Technology, Tianjin University, Tianjin, China · School of Cybersecurity, Tianjin University, Tianjin, China · School of Computer Science and Technology, Tianjin University, Tianjin, China
A Survey of On-Policy Distillation for Large Language Models
Apr 1, 2026Mingyang Song, Mao ZhengExposure Bias Language Model Distillation
Large Language Model Department Tencent, China
Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation
Mar 31, 2026Hoang-Chau Luong, Dat Ba Tran, Lingwei ChenLanguage Model Distillation Knowledge Distillation
Rochester Institute of Technology, Rochester, NY, USA · Rowan University, Glassboro, NJ, USA
Quantization with Unified Adaptive Distillation to enable multi-LoRA based one-for-all Generative Vision Models on edge
Mar 31, 2026Sowmya Vajrala, Aakash Parmar, Prasanna R +4Quantization-Aware Distillation On-Device Inference
Samsung Research Institute Bangalore, India
JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inference
Mar 31, 2026Bo Ma, Jinsong Wu, Weiqi Yan +2Cross-Modal Knowledge Distillation Knowledge Distillation
J. Wu is with the Guilin University of Electronic Technology, Guilin, China. · H. Wei is with Hikvision Technology Co., Ltd, Hangzhou, China. · K. Liu is with the Hebei University of Technology, Tianjin 300401, China.

Apr 13, 2026J/K move · Enter open · S save

Quantum-Gated Task-interaction Knowledge Distillation for Pre-trained Model-based Class-Incremental Learning

Linjie Li, Huiyu Xiao, Jiarui Cao, Zhenyu Wu+1

School of Information and Communication Engineering, BUPT, China · Engineering Research Center for Information Network, Ministry of Education, China

Class-Incremental Learning Knowledge Distillation

Open paper AskPDF arXiv

Class-incremental learning (CIL) aims to continuously accumulate knowledge from a stream of tasks and construct a unified classifier over all seen classes. Although pretrained models (PTMs) have shown promising performance in CIL, they still struggle with the entanglement of multi-task subspaces, leading to catastrophic forgetting when task routing parameters are poorly calibrated or task-level representations are rigidly fixed. To address this issue, we propose a novel Quantum-Gated Task-interaction Knowledge Distillation (QKD) framework that leverages quantum gating to guide inter-task knowledge transfer. Specifically, we introduce a quantum-gated task modulation gating mechanism to model the relational dependencies among task embedding, dynamically capturing the sample-to-task relevance for both joint training and inference across streaming tasks. Guided by the quantum gating outputs, we perform task-interaction knowledge distillation guided by these task-embedding-level correlation weights from old to new adapters, enabling the model to bridge the representation gaps between independent task subspaces. Extensive experiments demonstrate that QKD effectively mitigates forgetting and achieves state-of-the-art performance.