Selective Knowledge Distillation

Ask about this topic

Overview

Selective knowledge distillation studies how to transfer useful teacher knowledge without teaching every example, token, feature, or reasoning step. Methods select or reweight supervision using confidence, uncertainty, teacher–student disagreement, task relevance, or estimates of whether a signal is learnable; they may transfer logits, features, or rationales. Much recent work applies these ideas to on-policy language-model reasoning, evaluated on math and coding benchmarks, while related approaches target vision, multimodal, and medical tasks. Selective supervision can reduce training and teacher-input costs or avoid harmful guidance, but its benefits depend on student capacity, selector quality, and optimization settings, and filtering too much can discard useful corrections.

Latest papers

CardsList

50 papers21 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

  1. OrthKD: Extracting Generalized Clinical Knowledge from Heterogeneous Teachers for Lightweight Deployment

    Jul 28, 2026Yi Xu, Cheng Chen, Mufan CaoDiabetic Retinopathy GradingSelective Knowledge Distillation

    Tongji University, Shanghai, China

  2. When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

    Jul 22, 2026Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela +4Selective Knowledge DistillationText Summarization

    BRAC University, Dhaka, Bangladesh

  3. UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning

    Jun 18, 2026Wenhao Chi, Arkaprava Sinha, Dominick Reilly +2Cross-Modal Knowledge DistillationMulti-View Learning

    University of North Carolina at Charlotte

  4. SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation

    Jun 17, 2026Yuhang Zhou, Lizhu Zhang, Yifan Wu +5Selective Knowledge DistillationLLM Agent Training

    Meta AI

  5. FADA: Accessible fetal ultrasound interpretation and annotation with a selectively distilled unified vision-language model

    Jun 9, 2026Mahmood Alzubaidi, Uzair Shah, Raden Muaz +6Ultrasound Image SegmentationSelective Knowledge Distillation

    College of Science and Engineering , Hamad Bin Khalifa University,2026 Doha, Qatar. · Center for Clinical Precision Medicine and Genomics, HMC, Doha, Qatar.Jun · 3Advanced AlRazi Diagnostic Center, Al-Hodeidah, Yemen. +1

  6. Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation

    Jun 1, 2026Yuying Li, Leqi Zheng, Yongzi Yu +6Selective Knowledge DistillationLanguage Model Distillation

    THU · Meituan · HKUST +2

  7. Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation

    May 26, 2026Yuanyi Wang, Su Lu, Yanggan Gu +6Selective Knowledge DistillationLanguage Model Distillation

    The Hong Kong Polytechnic University, PolyU · Hong Kong Polytechnic University Daya Bay Technology and Innovation Research Institute

  8. Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks

    May 14, 2026Kai Sun, Peibo Duan, Yongsheng Huang +4Selective Knowledge DistillationSpiking Neural Networks

    Faculty of Information Technology, Monash University, Australia · School of Software, Northeastern University, China · Department of Medicine, National University of Singapore, Singapore

  9. Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation

    May 13, 2026Kaiyuan Liu, Ziyuan Zhuang, Yang Bai +3Selective Knowledge DistillationLanguage Model Distillation

    College of Computer Science and Technology, Zhejiang University · Meituan LongCat Team, China · College of Computer Science and Technology, Jilin University

  10. TIP: Token Importance in On-Policy Distillation

    Apr 15, 2026Yuanda Xu, Hejian Sang, Zhengze Zhou +3Knowledge DistillationOn-Policy Distillation

  11. Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selection

    Apr 3, 2026Chaoqun He, Yingfa Chen, Chaojun Xiao +2Language Model DistillationCoT Distillation

    Tsinghua University

  12. JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inference

    Mar 31, 2026Bo Ma, Jinsong Wu, Weiqi Yan +2Cross-Modal Knowledge DistillationKnowledge Distillation

    J. Wu is with the Guilin University of Electronic Technology, Guilin, China. · H. Wei is with Hikvision Technology Co., Ltd, Hangzhou, China. · K. Liu is with the Hebei University of Technology, Tianjin 300401, China.

  13. Learnable Instance Attention Filtering for Adaptive Detector Distillation

    Mar 27, 2026Chen Liu, Qizhen Lan, Zhicheng Ding +2Knowledge DistillationFeature-Based Knowledge Distillation

    Dept. of CS, University of Alabama at Birmingham · Dept. of CS, Bowling Green State University

  14. TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models

    Mar 25, 2026Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan +3Language Model DistillationCross-Architecture Knowledge Distillation

    Department of Computer Science, University of Toronto · Vector Institute, Toronto, Canada

  15. HSI Image Enhancement Classification Based on Knowledge Distillation: A Study on Forgetting

    Mar 19, 2026Songfeng ZhuClass-Incremental LearningHyperspectral Image Classification

    School of Software Engineering, Henan Polytechnic University, Henan 454000, china

  16. SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning

    Mar 9, 2026Kaushik Roy, Giovanni D'urso, Nicholas Lawrance +2Imitation LearningRobot Imitation Learning

    CSIRO Robotics, DATA61, CSIRO, Australia. · Queensland University of Technology (QUT), Brisbane, Australia.

  17. Selective Transfer Learning of Cross-Modality Distillation for Monocular 3D Object Detection

    Mar 8, 2026Rui Ding, Meng Yang, Nanning ZhengMonocular 3D Object DetectionCross-Modal Knowledge Distillation

    Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, P.R.China.

  18. Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation

    Mar 3, 2026Chonghua Lv, Dong Zhao, Shuang Wang +4Domain Generalization for Semantic SegmentationKnowledge Distillation

    School of Artificial Intelligence, Xidian University, China · Department of Information Engineering and Computer Science, University of Trento, Italy · Department of Automation, Tsinghua University, China +1

  19. Reinforcement-aware Knowledge Distillation for LLM Reasoning

    Feb 26, 2026Zhaoyang Zhang, Shuli Jiang, Yantao Shen +6Knowledge DistillationRL for Knowledge Distillation

    AWS Agentic AI

  20. Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation

    Feb 25, 2026Minsang Kim, Seung Jun BaekLanguage Model DistillationSelective Knowledge Distillation

    Korea University · SK Telecom

  21. Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation

    Dec 24, 2025Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi +5CoT DistillationSelective Knowledge Distillation

    1The University of British Columbia · 2LinkedIn · 3Canada Research Chair in NLP and ML