Selective Knowledge Distillation

Overview

Selective knowledge distillation studies how to transfer useful teacher knowledge without teaching every example, token, feature, or reasoning step. Methods select or reweight supervision using confidence, uncertainty, teacher–student disagreement, task relevance, or estimates of whether a signal is learnable; they may transfer logits, features, or rationales. Much recent work applies these ideas to on-policy language-model reasoning, evaluated on math and coding benchmarks, while related approaches target vision, multimodal, and medical tasks. Selective supervision can reduce training and teacher-input costs or avoid harmful guidance, but its benefits depend on student capacity, selector quality, and optimization settings, and filtering too much can discard useful corrections.

Latest papers

CardsList

55 papers10 papers in the last four weeks, down 47% on the four weeks before. 0.1% of all new papers.

  1. Tiny Evidence Region Localization with Selective Contrastive On-Policy Self-Distillation

    Nov 7, 2026Yujuan Pang, Wenhao Xu, Yinglei TengVisual Evidence Sensitivity in VLMsFine-Grained Visual Perception

  2. Multi-Teacher Reasoning Distillation for LLM Recommendation with Self-Exploration

    Nov 2, 2026Liangbo Ning, Siyuan Liang, Yi Zhou +4Reasoning DistillationLarge Language Model-Based Recommendation

  3. MASS: Multi-mask Selective Self-Distillation for Leakage-Resilient Reasoning

    Oct 28, 2026Xiuqi Liu, Oveys Delafrooz Noroozi, Upamanyu MadhowLanguage Model DistillationSelective Knowledge Distillation

  4. RSKD: Grounding Semantic Consistency in Driving VLMs via Distillation

    Oct 26, 2026Chuchu Xie, Yang Zhong, Kangan Qian +3VLM DistillationKnowledge Distillation

  5. SRA: Selective Residual Adaptation for LLM Reasoning Enhancement

    Oct 25, 2026Zhenxing Dou, Kun He, Mingyang Yi +1Language Model DistillationSelective Knowledge Distillation

  6. GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL

    Oct 22, 2026Shijie Li, Yilin Gao, Tieyuan Chen +3RL for Language Model ReasoningLLM Reasoning

  7. Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation

    Oct 18, 2026Jike Zhong, Ming Li, Yuxiang Lai +9On-Policy DistillationAgentic RL

  8. Learning from Interaction Experience via Localized Context Distillation

    Oct 15, 2026Zhezheng Hao, Yongkang Yang, Xilin Xia +9Context DistillationContinual Learning for LLMs

  9. Divergence-Routed On-Policy Distillation for Language Models

    Oct 14, 2026Tiantian Peng, Guannan Lv, Hongyang Li +5On-Policy DistillationKnowledge Distillation

  10. Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?

    Oct 13, 2026Yiping Li, Fan Zhao, Zhibo Hou +2On-Policy DistillationTeacher-Student Learning

  11. RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning

    Oct 13, 2026Kunwang Zheng, Xinyu Sun, Yu Cao +7Adaptive Knowledge DistillationRubric-Based RL

  12. Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models

    Oct 12, 2026Tiago Martinelli, João Paulo Papa, Adriano José PereiraLanguage Model DistillationMedical LLMs

  13. BoundaryEvo: Data-Model Co-Evolution at the Capability Boundary for Long-Chain Reasoning Distillation

    Oct 9, 2026Jiaming Su, Suorong Yang, Xudong Li +2Language Model DistillationSelective Knowledge Distillation

  14. ReTeach: Building a Self-Teacher through Multi-Round Reflection and Retry

    Oct 8, 2026Yafeng Tang, Hao Li, Hongsheng Yu +1Language Model DistillationLLM Self-Refinement

  15. Learning What to Distill: Bilevel Top-K Token Selection for Self-Distillation in Large Language Models

    Oct 5, 2026Heng Liang, Xinwen Zhang, Hongchang GaoSelective Knowledge DistillationLanguage Model Distillation

  16. Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers

    Oct 5, 2026Xiaoyu Chen, Bo Shao, Tiangang Zhu +5RL for Language Model ReasoningSelective Knowledge Distillation

  17. KVE-KD: Key Visual Evidence-Guided Knowledge Distillation for Vision-Language Models

    Oct 2, 2026Jianbin Zhang, Xin Sun, Shanwen Wang +2VLM DistillationCross-Modal Knowledge Distillation

  18. Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation

    Oct 2, 2026Xiang Chen, Futao Su, Kong Wang +2Language Model DistillationOn-Policy Distillation

  19. Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

    Sep 29, 2026Ayan Sengupta, Vaibhav Seth, Tanmoy ChakrabortyLanguage Model DistillationSelective Knowledge Distillation

  20. Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillation

    Sep 29, 2026Rui Wang, Ruijie Wang, Bo Chen +2On-Policy Self-DistillationSelective Knowledge Distillation

  21. EOPSA: Efficient On-Policy Self-Distilled Safety Alignment

    Sep 28, 2026Qirui Liu, Yichen Sun, Yan Wang +5On-Policy Self-DistillationSelective Knowledge Distillation

  22. Cross-Modal Knowledge Distillation for Acoustic Pedestrian Detection

    Sep 24, 2026Yonghyun Kim, Chaeyeon Han, Sancho Gatungay +2Cross-Modal Knowledge DistillationSelective Knowledge Distillation

  23. Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD

    Sep 24, 2026Yibo Zhao, Zixuan Yang, Yunshi Lan +1Selective Knowledge DistillationOn-Policy Distillation