Selective Knowledge Distillation

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 19

All topics

Learning What to Distill: Bilevel Top-K Token Selection for Self-Distillation in Large Language Models
Oct 5, 2026Heng Liang, Xinwen Zhang, Hongchang GaoSelective Knowledge Distillation Language Model Distillation
Department of Computer and Information Sciences Temple University
Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers
Oct 5, 2026Xiaoyu Chen, Bo Shao, Tiangang Zhu +5RL for Language Model Reasoning Selective Knowledge Distillation
Institute of Software, Chinese Academy of Sciences · Microsoft · Work done during an internship at Microsoft +1
Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillation
Sep 29, 2026Rui Wang, Ruijie Wang, Bo Chen +2On-Policy Self-Distillation Selective Knowledge Distillation
Peking University · University of Oxford · The University of Hong Kong
EOPSA: Efficient On-Policy Self-Distilled Safety Alignment
Sep 28, 2026Qirui Liu, Yichen Sun, Yan Wang +5On-Policy Self-Distillation Selective Knowledge Distillation
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Ant Group
When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation
Sep 28, 2026Minjae Park, Taesun Yeom, Jaeho LeeSelective Knowledge Distillation Training Data Selection
Pohang University of Science and Technology (POSTECH)
Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD
Sep 24, 2026Yibo Zhao, Zixuan Yang, Yunshi Lan +1Selective Knowledge Distillation On-Policy Distillation
School of Data Science and Engineering East China Normal University · Hugging Face
CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation
Aug 13, 2026Enhan Li, Junhao He, Hongyang DuSelective Knowledge Distillation Language Model Distillation
The University of Hong Kong
BioKD: Selective Physiology-to-Video Knowledge Distillation via Reliability Gate for Emotion Recognition
Aug 6, 2026Bojing Hou, Ruohao Li, Yitong Zhu +3Cross-Modal Knowledge Distillation Selective Knowledge Distillation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · New York University, New York, NY, USA
Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance
Aug 1, 2026Zhuowen Han, Jinwei Xiao, Zhengxi Lu +9Self-Training for Language Models RL for Language Model Reasoning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University · Meituan Longcat Team
OrthKD: Extracting Generalized Clinical Knowledge from Heterogeneous Teachers for Lightweight Deployment
Jul 28, 2026Yi Xu, Cheng Chen, Mufan CaoDiabetic Retinopathy Grading Selective Knowledge Distillation
Tongji University, Shanghai, China
When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization
Jul 22, 2026Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela +4Selective Knowledge Distillation Text Summarization
BRAC University, Dhaka, Bangladesh
UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning
Jun 18, 2026Wenhao Chi, Arkaprava Sinha, Dominick Reilly +2Cross-Modal Knowledge Distillation Multi-View Learning
University of North Carolina at Charlotte
SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation
Jun 17, 2026Yuhang Zhou, Lizhu Zhang, Yifan Wu +5Selective Knowledge Distillation LLM Agent Training
Meta AI
FADA: Accessible fetal ultrasound interpretation and annotation with a selectively distilled unified vision-language model
Jun 9, 2026Mahmood Alzubaidi, Uzair Shah, Raden Muaz +6Ultrasound Image Segmentation Selective Knowledge Distillation
College of Science and Engineering , Hamad Bin Khalifa University,2026 Doha, Qatar. · Center for Clinical Precision Medicine and Genomics, HMC, Doha, Qatar.Jun · 3Advanced AlRazi Diagnostic Center, Al-Hodeidah, Yemen. +1
Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation
Jun 1, 2026Yuying Li, Leqi Zheng, Yongzi Yu +6Selective Knowledge Distillation Language Model Distillation
THU · Meituan · HKUST +2
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
May 26, 2026Yuanyi Wang, Su Lu, Yanggan Gu +6Selective Knowledge Distillation Language Model Distillation
The Hong Kong Polytechnic University, PolyU · Hong Kong Polytechnic University Daya Bay Technology and Innovation Research Institute
Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks
May 14, 2026Kai Sun, Peibo Duan, Yongsheng Huang +4Selective Knowledge Distillation Spiking Neural Networks
Faculty of Information Technology, Monash University, Australia · School of Software, Northeastern University, China · Department of Medicine, National University of Singapore, Singapore
Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation
May 13, 2026Kaiyuan Liu, Ziyuan Zhuang, Yang Bai +3Selective Knowledge Distillation Language Model Distillation
College of Computer Science and Technology, Zhejiang University · Meituan LongCat Team, China · College of Computer Science and Technology, Jilin University
Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
Dec 24, 2025Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi +5CoT Distillation Selective Knowledge Distillation
1The University of British Columbia · 2LinkedIn · 3Canada Research Chair in NLP and ML

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

Learning What to Distill: Bilevel Top-K Token Selection for Self-Distillation in Large Language Models

Heng Liang, Xinwen Zhang, Hongchang Gao

Department of Computer and Information Sciences Temple University

Selective Knowledge Distillation Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

Large language models have shown strong reasoning capabilities, but their high inference costs make knowledge distillation an important approach for transferring such capabilities to compact models in resource-constrained scenarios. On-policy self-distillation further reduces the reliance on external large teacher models while improving the reasoning ability of compact language models. However, existing methods typically either distill all token positions uniformly or select tokens using fixed heuristic criteria, assigning the same distillation strength to the selected positions rather than adaptively learning which tokens are most beneficial for distillation. To address these limitations, we propose BiToK-SD (Bilevel Top-K Token Selection for Self-Distillation), a bilevel-optimization-based token selection method that learns where distillation should be applied during on-policy self-distillation. Specifically, BiToK-SD is formulated as a bilevel optimization problem, where the lower-level problem models Top-K token selection as a differentiable threshold-based relaxation, allowing the selected positions to adapt as the student policy evolves, while the upper-level problem performs knowledge distillation on the selected positions. Experiments on mathematical reasoning benchmarks show that BiToK-SD achieves the best average performance among all compared methods while requiring only lightweight additional computation.

Momentum

Related topics

Latest papers 19