Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

887 papers87 papers in the last four weeks, down 49% on the four weeks before. 0.7% of all new papers.

  1. You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces

    Oct 27, 2026Hamza Etcibasi, Ramazan Gokberk CinbisKnowledge DistillationFeature-Based Knowledge Distillation

  2. What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2

    Oct 27, 2026Nabeelah Faumi, Deshan SumanathilakaLanguage Model DistillationKnowledge Distillation

  3. Ground-to-Satellite Knowledge Distillation for Crop Classification under Temporal and Geographic Shift

    Oct 27, 2026Miro Miranda, Rocco Sedona, Valerio Marsocci +2Crop Type ClassificationAgricultural Remote Sensing

  4. Dynamic Merging Mutually-Distilled Multi-View Networks for RGB-Event-based Representation Learning

    Oct 27, 2026Futian Wang, Fan Zhang, Xiao Wang +4RGB-Event Sensor FusionRepresentation Learning

  5. How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning

    Oct 26, 2026Johnny Yu, Cyril GorllaSelf-DistillationKnowledge Distillation

  6. Joint Latent Reasoning and Observation for Agentic Video Understanding

    Oct 26, 2026Junzhang Liu, Hao Yuan, Yiming Wu +3Active Perception for Video UnderstandingVideo Understanding

  7. MuSID-KD: Multigranularity Knowledge Distillation for Generative Recommendation with Semantic IDs

    Oct 26, 2026Fengshi Zhang, Xiyan Ding, Shuxiao Xie +1Knowledge DistillationGenerative Recommendation

  8. DAC-Loop: Distillation-Aware Compression for Remote Sensing CNN

    Oct 26, 2026Rico Staedeli, Lorena Raichle, Konstantinos Tzevelekakis +2Neural Network CompressionNeural Network Pruning

  9. RSKD: Grounding Semantic Consistency in Driving VLMs via Distillation

    Oct 26, 2026Chuchu Xie, Yang Zhong, Kangan Qian +3VLM DistillationKnowledge Distillation

  10. Knowledge distillation across teacher architectures for compact diabetic retinopathy grading

    Oct 25, 2026Mossa Manisa, Habiba Jahan Priya, A. K. M. Fakhrul HossainKnowledge DistillationCross-Architecture Knowledge Distillation

  11. TAD: Distilling Vision-Language-Action Policies from Teacher Trajectories Alone

    Oct 24, 2026Yun Zhu, Da WangVisuomotor Policy LearningKnowledge Distillation

  12. Knowledge Distillation across Item and Code Spaces for Parallel Generative Recommendation

    Oct 23, 2026Yixuan Xu, Hailin Liu, Linxiang Peng +1Semantic ID RecommendationGenerative Recommendation

  13. Yalda-Embedding: Open Development of a Competitive Persian Retrieval Model

    Oct 23, 2026Yalda Taheri, Mohammad Hassan Heydari, Armon RasooliText Embedding ModelsInformation Retrieval

  14. HERD: Harness-Evolving Recursive Distillation for Continual Learning

    Oct 22, 2026Xun Sun, Thanh Thi Nguyen, Thuy-Trang VuContinual LearningContinual Learning for LLM Agents

  15. One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation

    Oct 22, 2026Khouloud Saadi, Di WangOn-Policy Self-DistillationKnowledge Distillation

  16. Reverse-KL Policy Distillation: Statistical Guarantees and Limits

    Oct 22, 2026Haolin Liu, Chen-Yu Wei, Julian ZimmertLanguage Model DistillationKnowledge Distillation

  17. GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL

    Oct 22, 2026Shijie Li, Yilin Gao, Tieyuan Chen +3RL for Language Model ReasoningLLM Reasoning

  18. CODI: Improving Self-Supervised Vision Transformers with Cross Distillation

    Oct 22, 2026Siran Dai, Qianqian Xu, Peisong Wen +3Vision TransformerSelf-Supervised Visual Representation Learning

  19. Benchmarking Model Compression Techniques Across Heterogeneous Edge Platforms: A Predictive Maintenance Use Case for Rotating Machinery

    Oct 21, 2026Matheus Calheira Guimarães de Oliveira, Marco Aurélio da Silva Cruz, Regis Cardoso +9Model CompressionEdge Inference

  20. CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

    Oct 21, 2026Wenhao Yu, Zhengcheng Wang, Jiahong Liu +7Gradient InterferenceSelf-Distillation

  21. How Far Can Foundation Models Be Distilled for Efficient Jet Tagging?

    Oct 21, 2026Jay Agarwal, Siddharth Khare, Dhruv KumarJet TaggingKnowledge Distillation

  22. D-Skill: Distilling Teacher Knowledge through Agent Skills

    Oct 20, 2026Shuo Yuan, Dacheng Li, Joseph E. GonzalezBlack-Box Knowledge DistillationSkill Distillation

  23. CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

    Oct 20, 2026Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge +3Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

  24. De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control

    Oct 20, 2026Shaoyi Zheng, Zhongtao Jiang, Yue Peng +4Efficient Inference for World Action ModelsWorld Models for VLAs

  25. On-Policy Transition Distillation

    Oct 20, 2026Runyang You, Yongqi Li, Tiezheng Yu +1Language Model DistillationOn-Policy Distillation

  26. Understanding the Benefits of Scaffolds through Computational Separations

    Oct 19, 2026Jingchu Gai, Adam Block, Andrej RisteskiKnowledge Distillation