Adaptive Knowledge Distillation

Latest papers

13 papers2 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning
Oct 13, 2026Kunwang Zheng, Xinyu Sun, Yu Cao +7Adaptive Knowledge Distillation Rubric-Based RL
University of Science and Technology of China · Fudan University · Alibaba Group
Representational Difference Distillation
Oct 11, 2026Abdulkarim Mugisha, Divin IrakizaAdaptive Knowledge Distillation Knowledge Distillation
Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.
PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation
Oct 8, 2026Heng Li, Yong Zhang, Ning Cheng +5Few-Shot Text Classification Adaptive Knowledge Distillation
Ping An Technology (Shenzhen) Co., Ltd., China · University of Science and Technology of China
ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models
Sep 3, 2026Quang Hoang Trung, Quang Huu Hieu, Nguyen Van Hoang Phuc +1Language Model Pretraining Autoregressive Language Modeling
VJ Technologies, Da Nang, Vietnam · AJ Technologies, Nagoya, Japan · Vietnam National University, Ho Chi Minh City, Vietnam +1
Temperature-Adaptive Transformed Teacher Matching
Aug 29, 2026Hiroaki Aizawa, Yoshikazu HayashiKnowledge Distillation Adaptive Knowledge Distillation
Graduate School of Advanced Science and Engineering Hiroshima University, Japan · Intelligent Production Technology Research & Development Center for Aerospace Gifu University, Japan
DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation
Aug 26, 2026Yutong Chen, Guangfu Guo, Zhichao Xu +1Adaptive Knowledge Distillation On-Policy Self-Distillation
Department of Computer Science, Clemson University, 105 Sikes Hall, Clemson, SC 29634, USA · Department of Computer Science, University of Utah
Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
Aug 8, 2026Yongkang Yang, Zhezheng Hao, Hong Zhang +8Adaptive Knowledge Distillation On-Policy Self-Distillation
Tencent, Workbuddy Team · Zhejiang University · Independent Researcher
LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Models
Jul 13, 2026Gangsu Kim, Won-Ki JeongComputational Pathology Adaptive Knowledge Distillation
Department of Computer Science and Engineering, College of Informatics, Korea University, Republic of Korea
ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation
Jun 29, 2026Zilong Liu, Xuewen Zhang, Jinrui Xing +3Language Model Distillation Knowledge Distillation
Li Auto, Beijing, China · School of Software and Microelectronics, Peking University, China
RPM-Distill: Physiology-guided Adaptive Cross-modal Distillation for Robust Remote Physiological Measurement
Jun 26, 2026Jiyao Wang, Qingyong Hu, Duoxun Tang +3Remote Photoplethysmography Adaptive Knowledge Distillation
McGill University, Montreal, Canada · Hong Kong University of Science and Technology, Hong Kong, China · Tsinghua University, Shenzhen, China +1
BAHSD: Bridging the Long-tail Gap via Adaptive Distillation in Black-box Sequential Recommendation
Jun 2, 2026Xi Zhou, Famin Wu, Mingming Li +4Adaptive Knowledge Distillation Sequential Recommendation
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Beijing Institute for General Artificial Intelligence, Beijing, China
Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning
May 12, 2026Zihao Han, Tiangang Zhang, Huaibin Wang +1Adaptive Knowledge Distillation On-Policy Self-Distillation
ByteDance Douyin
PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence
Mar 11, 2026Yuanda Xu, Hejian Sang, Zhengze Zhou +2Language Model Distillation Adaptive Knowledge Distillation
LinkedIn Corporation

Oct 13, 2026J/K move · Enter open · S save

RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning

Kunwang Zheng, Xinyu Sun, Yu Cao, Shuocheng Li+6

University of Science and Technology of China · Fudan University · Alibaba Group

Adaptive Knowledge Distillation Rubric-Based RL Selective Knowledge Distillation On-Policy Distillation Knowledge Distillation Group Relative Policy Optimization

Open paper AskPDF arXiv

Rubric-based reinforcement learning grades a response against many written criteria but collapses them into one reward, which discards the detail the rubric carries and gives no GRPO reward gradient when a group's responses tie. A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO. RELAY uses advantages and rewards to select positive-advantage and low-scoring zero-advantage rollouts, and monitors the distillation loss to decide when to retire the teacher. Stage 1 combines selective distillation from a frozen teacher with GRPO; a plateau detector then removes the teacher, and Stage 2 continues with GRPO alone. Calibrated on one model-domain pair and reused unchanged in three others, the plateau detector fires between steps 122 and 171. RELAY scores highest in-domain among the compared methods in all four settings, ahead of GRPO on RaR-Medicine by 4.62 points with Qwen3-1.7B and 6.02 with Qwen3-4B. On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening or refreshing it. Distilling selected rollouts covering about half the response tokens also outperforms distilling all rollouts at a matched average coefficient.