Knowledge Distillation

Ask about this topic

Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

834 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

  1. OPOD: On-Policy Omni Distillation

    Jul 23, 2026Tong Zhao, Yuyang Hu, Yutao Zhu +5Cross-Modal Knowledge DistillationUnified Multimodal Models

    1Renmin University of China · 2Tencent

  2. Factor-Informed Uncertainty Distillation for Gaze Estimation

    Jul 22, 2026Mohammadreza Jamalifard, Yaxiong Lei, Javier Fumanal Idocin +3Gaze EstimationUncertainty Quantification

    University of Essex · Colchester, UK

  3. When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

    Jul 22, 2026Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela +4Selective Knowledge DistillationText Summarization

    BRAC University, Dhaka, Bangladesh

  4. Contrastive On-Policy Distillation

    Jul 21, 2026Jiacheng Ruan, Jun Tang, Wenzhen Yuan +5Language Model DistillationOn-Policy Distillation

    Shanghai Jiao Tong University · Alibaba Group

  5. Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?

    Jul 20, 2026Fan Yang, Rui Meng, Yuxin WenAgentic SearchOn-Policy Self-Distillation

    Chapman University, Orange, CA, USA · Lawrence Berkeley National Laboratory, Berkeley, CA, USA

  6. Distilled Reinforcement Learning for LLM Post-training

    Jul 19, 2026Chen Wang, Zhaochun Li, Jionghao Bai +4RL for Language Model ReasoningCross-Architecture Knowledge Distillation

    College of Elite Engineers, Nankai University · Zhongguancun Academy · Beijing Institute of Technology +4

  7. Staged Depth-Pruning Distillation of a Flow-Matching Text-to-Speech Teacher: A Compact Hindi Speech Synthesizer

    Jul 19, 2026Sivateja TrikutamTTS SynthesisLow-Resource TTS Synthesis

  8. Bridging the Information Gap: Semantic Densification and Hindsight Distillation for Cold-Start Prediction

    Jul 19, 2026Hao Duong Le, Yifei Gao, Huan Li +4Cold-Start RecommendationConversion Rate Prediction

    Tsinghua University Beijing, China · Meituan Beijing, China

  9. CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation

    Jul 18, 2026Satyam Kumar, Saurabh JhaCoT DistillationOn-Policy Distillation

  10. AgentBrew: Lifelong Knowledge Brewing from Strong Teachers to Weak LLM Agents

    Jul 18, 2026Yangqin Jiang, Chao HuangLLM Agent Self-ImprovementLLM Agent Skill Learning

    The University of Hong Kong

  11. On the Failure of Boundary-Seeking Distillation in Bottlenecked Generative Architectures

    Jul 17, 2026Mohamed Amine KinaSynthetic Data GenerationData-Free Knowledge Distillation

    Universität Bremen

  12. Geometric Distillation from Rectified Stereo: Leveraging Epipolar Cues for Monocular Depth

    Jul 17, 2026Jung-Hee Kim, Xiaoming LiuMetric Depth EstimationMonocular Depth Estimation

    Michigan State University, East Lansing, MI 48824, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC 27514, USA

  13. Symbiosis-Inspired Knowledge Distillation for Incremental Object Detection

    Jul 15, 2026Mingyue Zeng, De Cheng, Zhipeng Xu +3Class-Incremental LearningObject Detection

    State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China · School of Electronic Engineering, Xidian University, Xi’an, China

  14. Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations

    Jul 15, 2026Rui Wang, Hongru Wang, Yi Chen +4RL for Language Model ReasoningLanguage Model Distillation

    †The Chinese University of Hong Kong · ‡Tencent AI Lab

  15. Domain-Incremental Remote Sensing Change Detection via Difference-Guided Adaptation and Frequency-Decoupled Distillation

    Jul 14, 2026Daifeng Peng, Yaning Li, Haiyan GuanRemote Sensing Change DetectionDomain-Incremental Learning

    School of Remote Sensing and Geomatics Engineering, Nanjing University of Information Science and Technology, Nanjing 210044, China

  16. Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?

    Jul 14, 2026Kaiwen Zheng, Junchen Fu, Wenhao Deng +3Efficient Multimodal InferenceMultimodal Large Language Models

    University of Glasgow United Kingdom · Institute of Computing Technology, Chinese Academy of Sciences China · School of Artificial Intelligence, Shandong University China +1

  17. MAGE: Color-Invariant and Spatial Knowledge Distillation for Gastric Neoplasm Classification

    Jul 14, 2026Jiho Jun, Jeongwon Woo, Jaemin Song +6EndoscopyMedical Imaging

    Korea University, Korea · MEDAI, Korea · Seoul National University, Korea +3

  18. MobileSAM2: Lightweight Segment Anything for Spatial Intelligence

    Jul 14, 2026Kai Jiang, Jiaxing Huang, Jingyi Zhang +5Video Object SegmentationSegment Anything Models

    Hong Kong Polytechnic University, HK, China · Nanyang Technological University, Singapore · Xidian University, China +1

  19. Anatomy-Privileged Distillation with Token Routing for MRI-Based Prediction of Perineural Invasion

    Jul 13, 2026Hyunsu Go, Youngung Han, Kyeonghun Kim +15Perineural Invasion PredictionMedical Image Classification

    Seoul National University, Seoul, Republic of Korea · OUTTA, Seoul, Republic of Korea · Kyung Hee University, Seoul, Republic of Korea +5

  20. Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations

    Jul 13, 2026Xuyi Yu, Yaohua Liu, Ziming Song +3Data AugmentationKnowledge Distillation

    State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, China · Guangdong Institute of Intelligence Science and Technology, Zhuhai 519031, China · Institute of Collaborative Innovation, University of Macau, Taipa, Macau SAR, China +1

  21. LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Models

    Jul 13, 2026Gangsu Kim, Won-Ki JeongComputational PathologyAdaptive Knowledge Distillation

    Department of Computer Science and Engineering, College of Informatics, Korea University, Republic of Korea

  22. EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models

    Jul 13, 2026Jie Sun, Mao Zheng, Mingyang Song +7Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

    University of Science and Technology of China · Shanghai Innovation Institute · LLM Department, Tencent +1

  23. BucketKD: A Safety-Aware Bucket-Based Knowledge Distillation Framework for End-to-End Motion Planning

    Jul 12, 2026Md Nahidul Islam, Mohd Hasan Ali, Dipankar Dasgupta +1Safe Motion PlanningMotion Planning

    Department of Computer Science, University of Memphis, Memphis, TN, United States · Department of Electrical and Computer Engineering, University of Memphis, Memphis, TN, United States

  24. Hybrid Continual Learning for Low-Resource Australian Aboriginal Language Identification

    Jul 11, 2026Pravina Mylvaganam, Ting Dang, Eliathamby Ambikairajah +2Continual LearningSpeech Processing

    University of New South Wales, Australia · University of Melbourne, Australia · Massachusetts Institute of Technology, USA

  25. CVKD-UDA: Cross-View Knowledge Distillation for 3D Unsupervised Domain Adaptive Segmentation

    Jul 11, 2026Zhimin Yuan, Ming Cheng, Shangshu Yu +43D Point Cloud SegmentationUnsupervised Domain Adaptation

    School of Artificial Intelligence, Nanyang Normal University, Nanyang 473061, China · Fujian Key Laboratory of Sensing and Computing for Smart Cities, School of Informatics, Xiamen University, Xiamen 361005, China · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China

  26. SigLIP-HD by Fine-to-Coarse Supervision

    Jul 10, 2026Lihe Yang, Zhen Zhao, Hengshuang ZhaoEfficient VLM InferenceLarge Vision-Language Models

    The University of Hong Kong · Shanghai AI Laboratory

  27. Active rejection enables reliable generalization of universal machine-learning interatomic potentials

    Jul 10, 2026Mingxiang Luo, Xinnan Mao, Lu Wang +3Machine Learning Interatomic PotentialsSelective Prediction

    School of Future Information Innovation, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Suzhou Laboratory, Suzhou, China +1

  28. Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

    Jul 9, 2026Lu Dai, Ziyang Rao, Yili Wang +3Knowledge DistillationLLM Fine-Tuning

    HKUST · HKUST(GZ)

  29. Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment

    Jul 9, 2026Vinay Kumar ChagantiOn-Device Language Model InferenceLLM Information Extraction

    independent researcher

  30. SQuaD-SQL: Efficient Text-to-SQL with Small Language Models via LLM-Guided Knowledge Distillation

    Jul 9, 2026Wangyu Wu, Xiaojian Lin, Rong Fu +4Small Language ModelsText-to-SQL

    The University of Liverpool · Microsoft · Tsinghua University +4