Overview

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

CardsList

887 papers87 papers in the last four weeks, down 49% on the four weeks before. 0.7% of all new papers.

  1. Posterior Matching Distillation

    Nov 19, 2026Hidir Yesiltepe, Julius Berner, Arash Vahdat +1Diffusion Model DistillationDiffusion-Based Image Generation

  2. Angular Conditional Distillation: Overcoming Classifier Geometry Mismatch

    Nov 18, 2026Xuefei Zhang, Ruoqi Wang, Lei CuiKnowledge DistillationCross-Architecture Knowledge Distillation

  3. Communicative On-Policy Distillation: Distilling Black-Box Teacher Capabilities through Communication for Long-Horizon Tasks

    Nov 18, 2026Lirui Luo, Kelong Mao, Xinwei Yang +9On-Policy DistillationKnowledge Distillation

  4. ResoLUT: Distilling Residual Networks into Lookup-Table Networks

    Nov 17, 2026Katarzyna Fojcik, Aidas Mikalauskas, Matthias Kampa +2Efficient Neural Network InferenceNeural Network Hardware Acceleration

  5. SilverSpoon-KD: A General-Purpose Toolkit for Knowledge Distillation

    Nov 17, 2026Xaver R. Davey, David BromanKnowledge DistillationLanguage Model Distillation

  6. SRED: Sparse Regional Evidence Distillation Across Magnifications in Whole-Slide Imaging

    Nov 17, 2026Zhiyuan Yang, Jiahao Cheng, Vincent Quoc-Huy Trinh +1Whole-Slide Image AnalysisHistopathology Image Classification

  7. Towards wearable-scale, transition-aware sleep staging for clinical populations

    Nov 16, 2026Suvadeep Maiti, Keya Sinha, Raju S. BapiSleep Stage ClassificationKnowledge Distillation

  8. InvarOrd: Self-Supervised Ordinal Knowledge Distillation with Rank-Invariant Alignment for Label-Efficient Cross-Species Hyperspectral Fish Freshness Assessment

    Nov 14, 2026Kazi Nabiul Alam, Omenuko Joseph Nwebonyi, Pooneh Bagheri Zadeh +1Few-Shot Domain AdaptationHyperspectral Imaging

  9. Program Distillation: Transferring LLM Capability from System 2 to System 0

    Nov 14, 2026Pujun Zheng, Wanying Ren, Guoxiu HePolicy DistillationKnowledge Distillation

  10. On-policy distillation as a Backdoor Propagation Channel

    Nov 14, 2026Zane Fu, Yanzhao Zheng, Ziting Qin +8LLM Backdoor AttacksOn-Policy Distillation

  11. Antidistillation is Cheap: Lightweight Distillation-Resistance via Stylistic Corruption

    Nov 13, 2026Anh Tu Ngo, Anupam ChattopadhyayKnowledge DistillationModel Extraction Attacks

  12. Two mathematical models of knowledge distillation

    Nov 12, 2026Audrey Xie, Ludwig Schmidt, John DuchiKnowledge DistillationLoss Landscape Geometry

  13. MutAtlas: A PDB-Wide Energy-Guided Atlas of Protein Mutation Effects

    Nov 12, 2026Ruihan Guo, Chaoran Cheng, Zhanghan Ni +3Protein Mutation Effect PredictionProtein Fitness Prediction

  14. Janus-MOPD: Tracing the Student Learning Dynamics in Multi-Teacher On-Policy Distillation

    Nov 11, 2026Zongji Yu, Ming Zhang, Dongcheng Zhang +7Multi-Teacher Knowledge DistillationKnowledge Distillation

  15. Serving-Aligned Entire-Space Conversion Modeling via Click-Through Rate Knowledge Distillation

    Nov 11, 2026Pavel Dolin, Xun Zou, Jiatian Wu +1Conversion Rate PredictionMulti-Task Learning

  16. SteerDistill: Quantizing LLM Agents via Closed-Loop Teacher Steering

    Nov 10, 2026Genming Cui, Xiangyu Li, Zixu Hao +3LLM QuantizationQuantization-Aware Distillation

  17. Which Successes Make Good References? Action-Sequence Alignment for Multi-Turn On-Policy Distillation

    Nov 9, 2026Mufei Li, Nikki Lijing Kuang, Zhuoer Wang +9On-Policy DistillationTool-Using Language Agents

  18. Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning

    Nov 9, 2026Haozhi Yuan, Handong Li, Yifan Zhang +1Multi-Teacher On-Policy DistillationRL Post-Training

  19. Perturbing Inputs Improves Pretraining Distillation

    Nov 8, 2026Ziyang Cai, Yihe Dong, Dimitris Papailiopoulos +1Language Model DistillationKnowledge Distillation

  20. Heterogeneous Asymmetric Contrastive Distillation

    Nov 7, 2026Qi Wang, Jinjia ZhouContrastive Knowledge DistillationKnowledge Distillation

  21. Failure Profiling and Reachable Trajectory Selection for Reasoning Distillation

    Nov 6, 2026Yijun Zhu, Jianxin Wang, Chengchao ShenReasoning DistillationLanguage Model Distillation

  22. Knowledge Distillation from Neural Networks to Differentiable Logic Networks

    Nov 6, 2026Hee Yun Suh, Niraj K. JhaKnowledge DistillationDifferentiable Logic Gate Networks

  23. CoRD: Coupled Rollouts Fill the Blind Spot of On-Policy Distillation

    Nov 5, 2026Guhan Chen, Bohan Li, Jun Rao +8Language Model DistillationKnowledge Distillation

  24. O2^2PD: Overlap-Gated On-Policy Distillation on Reliable Prefix

    Nov 5, 2026Shuaishuai Zu, Jihao Zhao, Pengfei Li +5Language Model DistillationOn-Policy Distillation

  25. VINO: Revisiting 3D Representation Learning with Volume Transformers

    Nov 4, 2026Adrian Kruse, Kadir Yilmaz, Daan de Geus +23D Representation LearningRepresentation Learning