Knowledge Distillation

Follow this topic Ask about this topic

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

SLAD : Shared LoRA Adapters for Task Specific Distillation
May 28, 2026Reda Bensaid, Yassir Bendou, Vincent Gripon +1Vision Foundation Model Adaptation Adapter Tuning
IMT Atlantique, Brest, France · Polytechnique Montr´eal, Montr´eal, Canada
Matching Rates and Optimal Allocation for Federated Probe-Logit Distillation under Heterogeneous Bandwidth Budgets
May 28, 2026Prasanjit Dubey, Xiaoming HuoFederated Learning Aggregation Communication-Efficient Distributed Training
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332, U.S.A.
GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering
May 28, 2026Xin Sun, Jianan Xie, Zhongqi Chen +6Agentic RL Knowledge Graph Question Answering
University of Science and Technology of China · 2NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences · 3ShanghaiTech University +1
LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation
May 28, 2026Hua Zheng, Shali Jiang, Boyang Liu +44Transfer Learning Recommender Systems
Lightweight SAR Ship Detection via Contrastive Distillation
May 27, 2026Surendar Devasundaram, Banafsheh Saber Latibari, Abhijit MahalanobisEfficient Object Detection Relational Knowledge Distillation
University of Arizona · Department of Electrical and Computer Engineering · Tucson, Arizona, USA
Skill-Conditioned Gated Self-Distillation for LLM Reasoning
May 27, 2026Jiazhen Huang, Xiao Chen, Xiao Luo +3Mathematical Reasoning Self-Distillation
Tsinghua University · University of Wisconsin-Madison · X-Humanoid +2
Deformable Gaussian Occupancy: Decoupling Rigid and Nonrigid Motion with Factorized Distillation
May 27, 2026Yang Gao, Wuyang Li, Po-Chien Luan +13D Gaussian Splatting Autonomous Driving Perception
´Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Switzerland
Dual-branch Distilled Transformer for Efficient Asymmetric UAV Tracking
May 27, 2026Hongtao Yang, Bineng Zhong, Qihua Liang +4Visual Object Tracking Efficient ViTs
Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin, 541004, China · Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, 541004, China · Nanjing University of Science and Technology +1
Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors
May 27, 2026Luyang Fang, Yongkai Chen, Jiazhang Cai +2Uncertainty Quantification Uncertainty-Aware Knowledge Distillation
Department of Statistics, University of Georgia · Department of Statistics, Harvard University
When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection
May 26, 2026Zhengyu Hu, Zheyuan Xiao, Linxin Song +10Language Model Distillation LLM Post-Training
University of Washington · University of Texas at Austin · University of Southern California +6
Generalist Graph Anomaly Detection via Prototype-Based Distillation
May 26, 2026Yiming Xu, Zihan Chen, Zhen Peng +4Unsupervised Anomaly Detection Zero-Shot Anomaly Detection
School of Computer Science and Technology, Xi’an Jiao-tong University, Xi’an, China · National Engineering Research Center for Visual Information and Applications, Xi’an, China · University of Virginia, Charlottesville, USA +3
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
May 26, 2026Yuanyi Wang, Su Lu, Yanggan Gu +6Selective Knowledge Distillation Language Model Distillation
The Hong Kong Polytechnic University, PolyU · Hong Kong Polytechnic University Daya Bay Technology and Innovation Research Institute
Detail Consistent Stage-Wise Distillation for Efficient 3D MRI Segmentation
May 25, 2026Mengchen Fan, Baocheng Geng, Xi Xiao +5Image Segmentation Medical Image Segmentation
University of Alabama at Birmingham, Birmingham, AL, USA · Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · UTHealth Houston, Houston, TX, USA
The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works
May 25, 2026Guanghui Wang, Kaiwen Lv Kacuila, Zhiyong Yang +5LLM Compression Language Model Distillation
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Alibaba Group, Hangzhou, China · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China +2
On-Policy Adversarial Flow Distillation for Autoregressive Video Generation
May 25, 2026Yang Luo, Shengju Qian, Xiaohang Tang +4Autoregressive Video Generation On-Policy Distillation
National Univesity of Singapore · LIGHTSPEED · University College London
SAM3-Assisted Training of Lightweight YOLO Models for Precision Pig Farming
May 25, 2026Marcos Vinicius Mendes Faria, Thiago Borges Pereira, Isabella C. F. S. Condotta +2Edge Inference Pseudo-Labeling
Coordenadoria de Inform´atica, Instituto Federal do Esp´ırito Santo (IFES), Serra, ES · Department of Animal Sciences, University of Illinois at Urbana-Champaign, USA
Extreme Region Policy Distillation
May 25, 2026Changyu Chen, Xiting Wang, Rui YanKL-Regularized RL RL for Language Model Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · Wuhan University
When Does Adaptive Guidance Help? Belief-Aware Privileged Distillation for Autonomous Driving Under Partial Observability
May 24, 2026Mehmet HaklidirPartially Observable RL Autonomous Driving
TUBITAK BILGEM Artificial Intelligence Institute, Turkey
PANDO: Efficient Multimodal AI Agents via Online Skill Distillation
May 24, 2026Yubo Li, Yidi Miao, Yuntian Shen +1LLM Agent Workflow Optimization LLM Agent Skill Learning
DisDop: Distillation with Domain Priors for Open-Vocabulary Aerial Object Detection
May 23, 2026Ruihao Xu, Yong Liu, Yansong Tang +6Open-Vocabulary Object Detection Knowledge Distillation
1Tsinghua Shenzhen International Graduate School, Tsinghua University · 2Tsinghua University
Strong Teacher Not Needed? On Distillation in LLM Pretraining
May 22, 2026Taiming Lu, Zhuang LiuLanguage Model Pretraining Language Model Distillation
Princeton University
Learning Through Noise: Why Subliminal Learning Works and When It Fails
May 22, 2026Vincent C. Brockers, Roman D. Ventzke, Valentin Neuhaus +2Subliminal Learning Cross-Architecture Knowledge Distillation
Max Planck Institute for Dynamics and Self-Organization, Göttingen · Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen
HARNESS-LM: A Three-Phase Training Recipe for Harnessing SLMs in Sponsored Search Retrieval
May 22, 2026Vipul Gupta, Shikhar Mohan, Lakshya Kumar +4Online Advertising Small Language Models
Microsoft AI India
One-Forcing: Towards Stable One-Step Autoregressive Video Generation
May 22, 2026Jiaqi Feng, Justin Cui, Yuanhao Ban +1Autoregressive Video Generation One-Step Generative Modeling
Tsinghua University · UCLA
TaxDistill: Improving Metagenomic Taxonomic Annotation via Distilled Genomic Foundation Models
May 22, 2026Rongye Ye, Lun Li, Zheng Luo +3Learning with Noisy Labels Knowledge Distillation
National Genomics Data Center, China National Center for Bioinformation, Beijing 100101, China · Beijing Institute of Genomics, Chinese Academy of Sciences, Beijing 100101, China · University of Chinese Academy of Sciences, Beijing 100049, China +1
Safeguarding Text-to-Image Generative Models Against Unauthorized Knowledge Distillation
May 21, 2026Yilan Gao, Sida Huang, Hongyuan Zhang +1Model Extraction Attacks T2I Generation
School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, P. R. China · Institute of Artificial Intelligence (TeleAI), China Telecom, P. R. China · The University of Hong Kong
Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
May 21, 2026Hongsin Lee, Hye Won ChungAdversarial Training Neural Network Robustness
School of Electrical Engineering, KAIST, Daejeon, Korea.
Visual-Advantage On-Policy Distillation for Vision-Language Models
May 21, 2026Ruiqi Liu, Xiaolei Lv, Gengsheng Li +8Vision-Language Models VLM Distillation
Institute of Automation, CAS · School of Advanced Interdisciplinary Sciences, UCAS · 3Hello +1
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
May 20, 2026Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning
May 20, 2026Xiaogeng Liu, Xinyan Wang, Yingzi Ma +2CoT Distillation On-Policy Self-Distillation
1Johns Hopkins University · University of Wisconsin–Madison · 3Nanyang Technological University

May 28, 2026J/K move · Enter open · S save

SLAD : Shared LoRA Adapters for Task Specific Distillation

Reda Bensaid, Yassir Bendou, Vincent Gripon, François Leduc-Primeau

IMT Atlantique, Brest, France · Polytechnique Montr´eal, Montr´eal, Canada

Vision Foundation Model Adaptation Adapter Tuning Low-Rank Adaptation Knowledge Distillation Representation Alignment

Open paper AskPDF arXiv

In the context of resource-constrained environments such as embedded systems, adapting reduced-size foundation models to downstream tasks has become increasingly popular. This has recently motivated the emerging setting of task-specific distillation, where a larger and a smaller version of the same foundation model are both adapted to the same downstream task, with the goal of transferring knowledge from the former to the latter. Recent work has demonstrated the benefits of using a larger version of the same foundation model to assist the adaptation of a smaller one. Typically, the larger model (teacher) is first adapted via fine-tuning or linear probing before its knowledge is distilled into the smaller model (student). While fine-tuning the teacher often increases its performance, recent work showed that probing it leads to better knowledge distillation to the student. Our findings show that this is mainly due to a mis-alignment in feature representation between the teacher and the student which occurs during the teacher's fine-tuning. Inspired by existing efforts to preserve previously learned knowledge, we first propose leveraging low-rank adaptation, resulting in better feature alignment and therefore better knowledge transfer. Drawing from this insight, we further enhance the feature alignment through a parameter-sharing strategy of the adapters between the two encoders during joint training. Our proposed method, SLAD, shows better feature alignment between the teacher and student, which results in increased performance for not only the student but also the teacher model, while being 2x faster to train than fine-tuning. Through extensive experiments on multiple classification and segmentation datasets, we demonstrate the improved accuracy and transfer efficiency of our method, achieving state-of-the-art performance in the task-specific distillation framework.