Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

887 papers87 papers in the last four weeks, down 49% on the four weeks before. 0.7% of all new papers.

  1. Agents, Think Only When it Helps

    Nov 3, 2026Min-Seon Kim, Seungdong Yoa, Sanghyu Yoon +4CoT DistillationLLM Agent Efficiency

  2. TRUST-Stereo: Teacher-Assisted Robust Uncertainty-Guided Search and Transfer for Fast Stereo Matching

    Nov 3, 2026Pouya Sohrabipour, Yihong Feng, Chaitanya Kumar Reddy Pallerla +3Stereo MatchingStereo Depth Estimation

  3. All-For-One: Distilling Visual Foundation Models to a Fast, Memory-Efficient Visuomotor Policy

    Nov 3, 2026Byungkun Lee, Minho Park, Hojoon Lee +6Visuomotor Policy LearningRobot Learning

  4. Where Do Distilled Students Store Thinking?

    Nov 3, 2026Taeyeong Kim, Jaemin Jung, Ahhyun Kim +1CoT DistillationKnowledge Distillation

  5. RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods

    Nov 2, 2026Ryan Swift, Konstantinos PsounisRandom ProjectionKnowledge Distillation

  6. Contrastive Adapter Training and Consensus Knowledge Distillation for Multi-Source-Free Domain Adaptation in Skin Cancer Diagnosis

    Nov 2, 2026Ali Abedi, Q. M. Jonathan Wu, Ning Zhang +1Source-Free Domain AdaptationMulti-Source Domain Adaptation

  7. Meet Students Where They Are: Student-Prefix Teacher Adaptation for On-Policy Distillation

    Nov 2, 2026Xue Zhang, Songming Zhang, Yunlong Liang +3On-Policy DistillationKnowledge Distillation

  8. SurgPace: Complexity-Aware Progressive Distillation for Surgical Multimodal Large Language Models

    Nov 2, 2026Zhicheng He, Xiaohan Xing, Hongqiu Wang +7VLM DistillationMultimodal Large Language Models

  9. Distilling Literary Sentiment from LLM Annotations into an Ordinal BERT Model

    Nov 2, 2026Artem Suslov, Takehiro HashimotoSentiment AnalysisKnowledge Distillation

  10. Beyond Policy Patches: Distilling RL Specialists into Vision-Language-Action Policies

    Nov 2, 2026Zhennan Jiang, Tingying Xie, Shilong Ji +11Robotic ManipulationKnowledge Distillation

  11. Order-Only On-Policy Distillation

    Nov 1, 2026Guangzeng Han, Yu Wu, Xiaolei HuangLanguage Model DistillationOn-Policy Distillation

  12. Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens

    Nov 1, 2026Mingyu Cao, Yaokai Jin, Shiwei Liu +1Language Model DistillationOn-Policy Distillation

  13. Label Smoothing in Disguise? Gated Knowledge Distillation for Low-Resource Bangla Generation

    Oct 31, 2026Dipto Sumit, Zobeida Zaman Chowdhury, Syeda Fabiayee Jahan +1Language Model DistillationKnowledge Distillation

  14. ZOPGA-KD: Zeroth Order PGA Guided Data Free Knowledge Distillation

    Oct 31, 2026Tuhin Subhra Mukherjee, Tiya Rose Pulikunnel, Vaidhav Adit +2Data-Free Knowledge DistillationKnowledge Distillation

  15. DecisionRank: On-Policy Distillation for Decision-Model Rerankers

    Oct 30, 2026Yanran Wu, Dian Yu, Chongyang Bai +10Learning to RankLLM Reranking

  16. Stepwise-Querying: Distilling Reasoning without Access to Teacher Chain-of-Thought

    Oct 30, 2026Dicong Yu, Wenhua Wang, Tian WangBlack-Box Knowledge DistillationLanguage Model Distillation

  17. Distilling Knowledge from Search: Decoder Model Factory for Quantum LDPC Codes

    Oct 30, 2026Anil Bilgin, Wanda Hou, Pranav Srinivas Murali +1Quantum Error Correction DecodingQuantum LDPC Codes

  18. Task-Performance Routing for Multi-Teacher Distillation in CT Medical Image Segmentation

    Oct 30, 2026Jiaye Yang, Yao Liu, Die Dai +5Multi-Teacher Knowledge Distillation3D Medical Image Segmentation

  19. Soil Moisture Estimation in Precision Agriculture: A Knowledge Distillation Approach

    Oct 29, 2026Mohammed Rakib, Vrinda Nagula, Arunkumar BagavathiPrecision AgricultureAgricultural Remote Sensing

  20. Distilling Large-Scale Machine Learning Interatomic Potentials to Predict Long-Timescale Electrolyte Transport Properties

    Oct 29, 2026Ishan Amin, Yue Jian, Nitesh Kumar +10Machine Learning Interatomic PotentialsMaterials Property Prediction

  21. YOLOF-MLD: A Multi-Level Distillation Framework for Efficient Object Detection

    Oct 29, 2026Xiaolong Wu, Liming Zheng, Zitong Liu +1Efficient Object DetectionObject Detection

  22. Distilling the Shape of Uncertainty: What On-Policy Distillation Learns and How to Predict It

    Oct 28, 2026Shiping Yang, Chuanqi Tang, Hossein Hajimirsadeghi +3Language Model DistillationOn-Policy Distillation

  23. Information-Theoretic Reliability--Safety Bounds and Finite-Alignment Dynamics in AI Knowledge Distillation

    Oct 28, 2026Miao-Miao Yi, Zhen-Yu Cai, Y -M. Du +1Renormalization GroupTrustworthy AI

  24. What Can Firms Do in the Face of Model Distillation?

    Oct 28, 2026Diptangshu Sen, Jibang Wu, Juba ZianiModel Extraction AttacksKnowledge Distillation