Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

887 papers87 papers in the last four weeks, down 49% on the four weeks before. 0.7% of all new papers.

  1. GAD: Geometry Attention Distillation for Consistent Video Generation

    Oct 13, 2026Junfeng Ni, Ziyu Zhu, Fan Jiang +7Attention-Guided DistillationGeometric Consistency in Video Generation

  2. Does self-distillation improve reasoning?

    Oct 13, 2026Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5Self-DistillationOn-Policy Self-Distillation

  3. Imprinting: Knowledge Distillation Can Shift Student Behavior

    Oct 13, 2026Yu-Chien Tang, An-Zi YenLanguage Model DistillationKnowledge Distillation

  4. RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning

    Oct 13, 2026Kunwang Zheng, Xinyu Sun, Yu Cao +7Adaptive Knowledge DistillationRubric-Based RL

  5. From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction

    Oct 12, 2026Tomasz Gniazdowski, Mohammadreza Momenifar, Nadia Vase +6Information RetrievalKnowledge Distillation

  6. Overtaught, Underlearned: Learner-Centered Distillation from Multi-Agent Discussion

    Oct 12, 2026Mengtao Zhou, Jianxiao Wang, Xiupeng Chen +2Language Model DistillationKnowledge Distillation

  7. Representational Difference Distillation

    Oct 11, 2026Abdulkarim Mugisha, Divin IrakizaAdaptive Knowledge DistillationKnowledge Distillation

  8. Radioactive Feedback: Tracing Models Trained by Label-Only LLM Supervision

    Oct 11, 2026Yuejia Li, JunHeng Li, Hao Wu +2LLM WatermarkingLanguage Model Distillation

  9. The Gradient That Top-k Distillation Never Computes

    Oct 11, 2026Rasul Khanbayov, Hasan KurbanLanguage Model DistillationKnowledge Distillation

  10. VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning

    Oct 10, 2026Jiahe FengVLM DistillationVisually Grounded Reasoning

  11. Reward-Aligned Distillation

    Oct 10, 2026Yilan Chen, Zian Su, Manish Prajapat +4RL for Knowledge DistillationKnowledge Distillation

  12. CV-OPD: Contrastive Vision On-policy Distillation

    Oct 9, 2026Haowei Wang, Hao Lin, Qi Zhu +4Contrastive Knowledge DistillationVLM Distillation

  13. FPD-Drive: Future Policy Distillation for End-to-End Autonomous Driving

    Oct 9, 2026Li-Heng Chen, Qifeng Chen, Chengwei Wu +4End-to-End Autonomous DrivingAutonomous Driving Planning

  14. DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation

    Oct 8, 2026Anhao Zhao, Haoran Xin, Junlong Tong +5Language Model DistillationOn-Policy Distillation

  15. TAM: Task-Aware Memory Distillation for Efficient Spatiotemporal Prediction

    Oct 8, 2026Yuqi Li, Xiaoqin Feng, Fan Xu +4Spatiotemporal ForecastingKnowledge Distillation

  16. Efficient Multi-Granularity Knowledge Transfer for Radiology Report Generation

    Oct 8, 2026Xubin Zhong, Zheyu Zhang, Wenjian Qin +1Radiology Report GenerationRadiology VLMs

  17. PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation

    Oct 8, 2026Heng Li, Yong Zhang, Ning Cheng +5Few-Shot Text ClassificationAdaptive Knowledge Distillation

  18. CARE: Constrained Attention Refinement for Fine-Grained Visual Classification via Teacher-Student Distillation

    Oct 8, 2026Ruibo Wen, Hang Shao, Yiming LeiFine-Grained Image ClassificationExplainable Image Classification

  19. KDFP: A first-principles approach to knowledge distillation in large language models

    Oct 7, 2026Ryan Swift, Konstantinos PsounisLanguage Model DistillationKnowledge Distillation