Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

887 papers87 papers in the last four weeks, down 49% on the four weeks before. 0.7% of all new papers.

  1. Selective Transfer through Sequence Distillation: From Benchmark Knowledge to Backdoor Behaviour

    Oct 19, 2026Jonibek Mansurov, Sholpan Bolatzhanova, Filbert Aurelian Tjiaranta +2Language Model DistillationKnowledge Distillation

  2. FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction

    Oct 19, 2026Yifan Wang, Yifei DuCTR PredictionSelf-Distillation

  3. Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization

    Oct 19, 2026Suwan Yoon, Seongryong Jung, Changhee LeeLanguage Model Post-TrainingOn-Policy Self-Distillation

  4. MapSteal: Map-based Autonomous Knowledge Stealing Attack Against Large Language Models

    Oct 19, 2026Zhengao Li, Chuxu Zhang, Yushun DongKnowledge Augmentation for Language ModelsLLM Training

  5. CD-OPD: Contextual Digest for On-Policy Distillation

    Oct 18, 2026Mingyuan Li, Yucheng Li, Xiaxia Wang +1On-Policy DistillationKnowledge Distillation

  6. A Stronger Teacher Within: Strengthening MoE Models via On-Policy Bidirectional Distillation

    Oct 16, 2026Haoyu Chen, Jun Bai, Yang Liu +2Mixture-of-Experts Language ModelsLanguage Model Distillation

  7. Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

    Oct 16, 2026Heming Zou, Qi Wang, Yixiu Mao +7RL for LLM AgentsLong-Horizon RL

  8. GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation

    Oct 16, 2026Yushu Wang, Renkai Yang, Jingcao Xu +4On-Policy DistillationKnowledge Distillation

  9. PAC-KD: Pathology-Aware Compositional Knowledge Distillation for Respiratory Sound Classification

    Oct 16, 2026Li Xiao, Weiping Tu, Youqiang Zheng +1Respiratory Sound ClassificationKnowledge Distillation

  10. Training-Free On-Policy Distillation

    Oct 15, 2026Bangji Yang, Jiajun Fan, Zhaoyu Zhu +2On-Policy DistillationKnowledge Distillation

  11. Anvil: Scaling Embodied Spatial Intelligence through Knowledge Recirculation

    Oct 15, 2026Weilin Ruan, Bohan Zhou, Jinghao Li +12Embodied Spatial ReasoningEmbodied Reasoning

  12. Physics-Aware Distillation of a Grid Foundation Model for Real-Time Edge Inference

    Oct 15, 2026Vlad Alexandru Sănduleac, Shahariya Rasheed Rayamarakkarveettil Puthanpurayil, Noman Khan +1Knowledge DistillationPower Systems

  13. Condition-Contrastive Extrapolation for On-Policy Distillation of Language Models

    Oct 15, 2026Wenxuan Luo, Chenxi Li, Haoyue Liu +2Contrastive Knowledge DistillationLanguage Model Distillation

  14. ControlFlow3D: Distilling Multi-View Knowledge into Latent Flow Matching for Point Cloud Upsampling

    Oct 15, 2026Yuang Liu, Zhi Zuo, Zhengkai Zhao +4Point Cloud LearningFlow Matching

  15. Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

    Oct 14, 2026Kaixiang Zhao, Tianrun Yu, Shangzhe Li +4On-Policy DistillationKnowledge Distillation

  16. VIGOR-MPC: Vision-Guided Objective Regulation MPC

    Oct 14, 2026Jelena Trisovic, Rahel Rickenbach, Marek Landert +2Vision-Based Robot ControlModel Predictive Control

  17. Harness Policy Distillation: Aligning Agent Policies with Runtime Semantics

    Oct 14, 2026Junze Zhu, Weihao Chen, Zhiyang Zhou +2LLM Agent TrainingPolicy Distillation

  18. CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation

    Oct 14, 2026Zhi Li, Haowei Liu, Hongchen Yang +5Adversarial RobustnessAdversarial Knowledge Distillation

  19. Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents

    Oct 14, 2026Md Farhan Ishmam, Kenneth MarinoBlack-Box Knowledge DistillationLLM Agent Training

  20. ActionOPD: Distilling Decisions, Not Tokens

    Oct 14, 2026Huining Yuan, Xiangmin Yi, Zelai Xu +5LLM Agent TrainingKnowledge Distillation

  21. Learning Individual Scientific Taste with Privileged Hindsight Distillation

    Oct 14, 2026Hanwei Fan, Yanqiao Wang, Yiqi Zhu +3AI-Assisted Scientific ResearchLearning Using Privileged Information

  22. Divergence-Routed On-Policy Distillation for Language Models

    Oct 14, 2026Tiantian Peng, Guannan Lv, Hongyang Li +5On-Policy DistillationKnowledge Distillation

  23. Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?

    Oct 13, 2026Yiping Li, Fan Zhao, Zhibo Hou +2On-Policy DistillationTeacher-Student Learning