Knowledge Distillation

Ask about this topic

Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

834 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

  1. Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance

    Aug 29, 2026Kazuyuki Hara, Hideitsu HinoKnowledge DistillationModel Misspecification

    Nihon University · The Institute of Statistical Mathematics

  2. Temperature-Adaptive Transformed Teacher Matching

    Aug 29, 2026Hiroaki Aizawa, Yoshikazu HayashiKnowledge DistillationAdaptive Knowledge Distillation

    Graduate School of Advanced Science and Engineering Hiroshima University, Japan · Intelligent Production Technology Research & Development Center for Aerospace Gifu University, Japan

  3. Explainable Multi-Loss Distillation Framework for Efficient and Interpretable Shrimp Disease Text Classification

    Aug 29, 2026Anh Nguyen Quynh, Khang Nguyen Quoc, Luyl-Da QuachExplainable Text ClassificationText Classification

  4. VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation

    Aug 28, 2026Zewen Ding, Zezhong Wu, Zhou Tao +5On-Policy Self-DistillationTeacher-Student Learning

    University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence

  5. When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation

    Aug 28, 2026Siyuan Gan, Yuhan Li, Xiran Wang +5Language Model DistillationOn-Policy Distillation

    State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China

  6. SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models

    Aug 28, 2026Enqiao Lu, Xingrui Yu, Yiwei Fu +7Spiking Language ModelsSpiking Neural Networks

    The Chinese University of Hong Kong, Shenzhen · Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore +3

  7. Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation

    Aug 27, 2026Dipto Sumit, Sakib Ul Haque, Farig SadequeKnowledge DistillationLanguage Model Distillation

    Department of Computer Science and Engineering BRAC University, Bangladesh

  8. Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms

    Aug 27, 2026Siye Wu, Kai Yang, Yuchen Cai +8Reinforcement Learning with Verifiable RewardsLanguage Model Merging

    Fudan University · LLM Department, Tencent

  9. Video-OPSD: Exploiting Privileged Visual Evidence for On-Policy Self-Distillation in Video Large Language Models

    Aug 27, 2026Ziyue Wang, Shiqi Huang, Weiwen Xu +2Video-Language ModelsOn-Policy Self-Distillation

    School of Electrical and Electronic Engineering, Nanyang Technological University · The Chinese University of Hong Kong

  10. Scaling Model-Generated Distillation Data Can Make Latent Teacher Traits More Recoverable

    Aug 27, 2026Zhichen Dong, Zhixuan Liu, Yuyu Fan +3Language Model DistillationKnowledge Distillation

    Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Fudan University

  11. Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation

    Aug 27, 2026Ziyuan Liu, Jiao Ou, Jian Liang +2Uncertainty-Aware Knowledge DistillationMulti-Teacher Knowledge Distillation

    Kuaishou Technology Beijing, China

  12. SPEAR: Distilling Domain-Adaptive Reasoning Skeletons via Sequential Symbolic Alignment in Reinforcement Learning

    Aug 27, 2026Zhuochun Li, Yuelyu Ji, Yiming Zeng +1Process Reward ModelsRL for Language Model Reasoning

    University of Pittsburgh, Pittsburgh, USA · University of Connecticut, Storrs, USA

  13. DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation

    Aug 26, 2026Yutong Chen, Guangfu Guo, Zhichao Xu +1Adaptive Knowledge DistillationOn-Policy Self-Distillation

    Department of Computer Science, Clemson University, 105 Sikes Hall, Clemson, SC 29634, USA · Department of Computer Science, University of Utah

  14. Socialized Detector Learning: Trajectory-Guided and Reciprocal Distillation for Heterogeneous Object Detectors

    Aug 26, 2026Weihao Li, Yunqi Zhu, Zhihe Fan +5Object DetectionKnowledge Distillation

    School of New Media and Communication, Tianjin University, Tianjin, China · School of Computer Science and Engineering, University of New South Wales, Sydney, Australia · School of Sports Training, Tianjin University of Sport, Tianjin, China +5

  15. Why Does Graph Learning Fail to Fully Benefit from a Text Teacher?

    Aug 26, 2026Fumiaki Kimino, Ryoma SatoGraph Representation LearningText-Attributed Graph Learning

    SOKENDAI · National Institute of Informatics · National Institute of Informatics, SOKENDAI

  16. A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation

    Aug 26, 2026Bing Shao, Jiazheng Zhang, Long Ma +13Language Model DistillationOn-Policy Distillation

    College of Computer Science and Artificial Intelligence, Fudan University

  17. OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoning

    Aug 25, 2026Qinglin Ye, Zhiyuan Gu, Jingjie Xia +6On-Policy DistillationLLM Reasoning

    University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences +5

  18. PROOF-Gen: From Optimized Data to Better Distillation

    Aug 24, 2026Anh Ta, Junjie Zhu, Shahin ShayandehLLM Agent TrainingLLM Agent Trajectory Synthesis

    Apple

  19. Spatiotemporal Distillation via Recurrent Bottlenecks for Aortic Tracking

    Aug 24, 2026Dexter Wen Jie Teo, Nairouz Shehata, Herve LombaertMedical Image SegmentationCardiac MRI

    College of Computing and Data Science, Nanyang Technological University, Singapore · Department of Computer and Software Engineering, Polytechnique Montreal, Canada

  20. Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it

    Aug 24, 2026Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner +2Knowledge DistillationFeature-Based Knowledge Distillation

    Andalusian Research Institute in Data Science and Computational Intelligence, 18016 Granada, Spain · Lab for Uncertainty in Data and Decision Making (LUCID), School of Computer Science, University of Nottingham, NG7 2RD Nottingham, UK · Department of Computer Science and Artificial Intelligence, School of Computer Science and Telecommunications Engineering (ETSIIT), University of Granada, 18071 Granada, Spain

  21. Interpretable AI with Local Distillation

    Aug 24, 2026Erin Craig, Yiling Huang, Snigdha PanigrahiSurrogate ModelingInterpretable ML

    Department of Biostatistics, University of Michigan · Department of Statistics, University of Michigan

  22. Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Data

    Aug 24, 2026Yifei Song, Kun Efimov-Zhang, Claire GardentText GenerationCross-Domain Transfer Learning

    CNRS/LORIA and Université de Lorraine

  23. SelFusion: Self-distillation for Diffusion Language Models

    Aug 24, 2026Hyeongsoo Lim, Jinyoung Kim, Eunseo Seo +2Diffusion Language Model TrainingDiffusion Language Models

    Department of Artificial Intelligence, Chung-Ang University

  24. Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMs

    Aug 24, 2026Yan Zhou, Sara Kangaslahti, Jonathan Geuter +4Language Model DistillationKnowledge Distillation

    Harvard University · Kempner Institute · IST Austria

  25. DIAG: Diagnostic Iterative Alignment and Generation for Data-Efficient Mathematical Preference Distillation

    Aug 24, 2026Guhan Chen, Songtao Tian, Bohan Li +3LLM AlignmentPreference Optimization

    Tsinghua University · Kyoto University · Nanjing University

  26. DiD It in 87 Minutes: A Label-Free Softmax-to-Linear Adaptation of Vision Transformers for Object Detection

    Aug 23, 2026Huaiyuan Qin, Gabriel James Goenawan, Zihang Lin +2Efficient Object DetectionEfficient ViTs

    Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Nanyang Technological University

  27. More accurate behavioral predictions with hybrid Bayesian-connectionist models

    Aug 23, 2026Brenden M. Lake, Akshay K. Jagadish, Guangyuan JiangComputational Cognitive ModelingHuman Behavior Modeling

    Departments of Computer Science and Psychology, Princeton University · Princeton AI Lab, Princeton University · Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology

  28. Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation

    Aug 22, 2026Kailin Lyu, Zherui Zhang, Junhao Dong +11Data-Free Knowledge DistillationKnowledge Distillation

    Institute of Automation, Chinese Academy of Sciences, China · Zhongguancun Academy, China

  29. STAR-OPD: Structured Aspect-Cascade-Aware On-Policy Reward Distillation for ABSA Quadruple Extraction

    Aug 21, 2026Tong Sun, Mingyang Ma, Jiayang YuRL for Knowledge DistillationAspect-Based Sentiment Analysis

    Alibaba International Digital Commerce Group

  30. ForeTime-VLA: Causal Future-Token Distillation from a World Action Model for Conveyor-Belt Manipulation

    Aug 21, 2026Siyuan Ma, Yutian Zhang, Boshi Zhang +4Visuomotor Policy LearningVision-Language-Action Models

    Tsinghua University, Beijing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Harbin Institute of Technology, Harbin, China +1