Uncertainty-Aware Knowledge Distillation

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 15

All topics

Learning What to Imitate: Entropy-Aware Distribution Mixing
Oct 5, 2026Juan Garcia Giraldo, Matteo Santelmo, Eduard Durech +3Language Model Distillation Uncertainty-Aware Knowledge Distillation
EPFL · ETH AI Center · Allen Institute for AI
Learning from imperfect teachers for low-resource acoustic generalization
Oct 4, 2026Shuanglin Li, Ruxiao Qian, Jian Liu +3Uncertainty-Aware Knowledge Distillation Audio Classification
Outcome-Guided On-Policy Self-Distillation
Oct 4, 2026ZheXu Wang, Mao-Lin Luo, Yankun Hong +6On-Policy Self-Distillation Uncertainty-Aware Knowledge Distillation
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Huawei Noah’s Ark Lab +2
E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation
Oct 4, 2026Yifei Liu, Minghao Fang, Xinyu Gu +6On-Policy Self-Distillation Mathematical Reasoning
The Chinese University of Hong Kong · Zhejiang University · Shanghai Artificial Intelligence Laboratory +5
UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents
Sep 27, 2026Wenbo Zhang, Pengcheng Xu, Weizhi Du +2LLM Agents Uncertainty-Aware Knowledge Distillation
University of California, Irvine · University of Michigan, Ann Arbor
Hesitation-Aware On-Policy Distillation for Diffusion Language Models
Sep 27, 2026Jianguo Huang, Lipeng Wan, Yanchen Deng +1Diffusion Language Model Decoding Uncertainty-Aware Knowledge Distillation
Nanyang Technological University, Singapore · Xi’an Jiaotong University, China
Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation
Aug 27, 2026Ziyuan Liu, Jiao Ou, Jian Liang +2Uncertainty-Aware Knowledge Distillation Multi-Teacher Knowledge Distillation
Kuaishou Technology Beijing, China
HumP-KD: A Hybrid Uncertainty-Aware Multi-Stage Progressive Knowledge Distillation Framework for Efficient Fire Classification
Jun 12, 2026Mohammed Arif Mainuddin, Najifa Tabassum, Omar Ibne Shahid +1Efficient Neural Network Inference Cross-Architecture Knowledge Distillation
Electrical and Computer Engineering North South University Dhaka, Bangladesh
When to Trust, How to Distill: Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecasting
Jun 10, 2026Rupasree Dey, Abdul Matin, Nathan Orwick +3Multivariate Time Series Forecasting Time Series Forecasting
Department of Computer Science Colorado State University Fort Collins, Colorado, USA · Department of Soil and Crop Sciences Colorado State University Fort Collins, Colorado, USA
Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors
May 27, 2026Luyang Fang, Yongkai Chen, Jiazhang Cai +2Uncertainty Quantification Uncertainty-Aware Knowledge Distillation
Department of Statistics, University of Georgia · Department of Statistics, Harvard University
Cognitive-Uncertainty Guided Knowledge Distillation for Accurate Classification of Student Misconceptions
May 14, 2026Qirui Liu, Hao Chen, Weijie Shi +2Intelligent Tutoring Systems AI in Education
South China University of Technology · Tencent Financial Technology · The Hong Kong University of Science and Technology +2
Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning
May 13, 2026Junlong Ke, Zichen Wen, Weijia Li +2On-Policy Self-Distillation RL for Language Model Reasoning
2Tsinghua University · 1Shanghai Jiao Tong University · 3Shanghai AI Laboratory
GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning
May 13, 2026Kasidit Sermsri, Teerapong PanboonyuenCoT Distillation Language Model Distillation
†Chulalongkorn University · ‡MARSAIL · §PBYAIL (Panboonyuen AI Lab)
Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation
Jul 24, 2025Lakshmana Sri Harsha Nemani, P. K. Srijith, Tomasz KuśmierczykLLM Uncertainty Estimation Uncertainty-Aware Knowledge Distillation
Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad, India · Jagiellonian University, Faculty of Mathematics and Computer Science, Kraków, Poland
Uni-Light: An Ultra-Lightweight Framework via Uncertainty-Aware Knowledge Distillation for Brain Tumour Segmentation
Date pendingLibing Kuang, Soren Salehi, Ziling Wu +2Medical Image Segmentation 3D Medical Image Segmentation
School of Computer Science, University of Nottingham, Nottingham, UK · Electrical Engineering Department, Sharif University of Technology, Tehran, Iran · University of Pittsburgh, Pittsburgh, PA, USA

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

Learning What to Imitate: Entropy-Aware Distribution Mixing

Juan Garcia Giraldo, Matteo Santelmo, Eduard Durech, Imanol Schlag+2

EPFL · ETH AI Center · Allen Institute for AI

Language Model Distillation Uncertainty-Aware Knowledge Distillation On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

Small language models are often post-trained as students on reasoning traces from stronger teacher models to efficiently learn new skills. However, token-level imitation on traces that lie far outside the student's expected distribution often produces \textit{confident conflicts}, whereby the student is required to imitate a continuation that it deems unlikely (i.e., low-probability) despite being confident in a different continuation (i.e., in a low-entropy state). To mitigate the degradation in generalisation and catastrophic forgetting caused by these conflicts, we propose \textbf{Entropy-Aware Mixing}: a dynamic per-token interpolation of the student and teacher distributions, gated by the student's predictive entropy. We implement both convex and geometric interpolations for both offline trace generation (via speculative decoding, then SFT) and on-policy forward-KL distillation. Our results show that entropy-aware mixing stabilises distillation, improving in-distribution and out-of-distribution math reasoning while better preserving general capabilities than fixed-teacher supervision. Nonetheless, the optimal entropy schedule depends on the training source, with offline-generated traces favouring concave schedules (greater overall teacher influence) and on-policy training favouring linear or convex schedules (teacher concentrated in high-entropy states).

Momentum

Related topics

Latest papers 15