Uncertainty-Aware Knowledge Distillation

Ask about this topic

Overview

This field studies how to transfer a large model’s knowledge to a smaller one while accounting for uncertainty in the teacher’s advice, the student’s predictions, or both. Methods include entropy- and confidence-weighted losses, selective filtering or intervention, Bayesian and evidential outputs, and adaptive forward- or reverse-KL objectives. They are used in settings such as language-model reasoning and agents, multimodal question answering, vision, and forecasting, where teacher guidance can improve capability without the cost of deploying the teacher. A central tension is deciding when uncertain predictions contain useful learning signals and when they reflect unreliable supervision, while balancing accuracy, calibration, diversity, and training cost.

Latest papers

CardsList

28 papers8 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

  1. Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning

    Oct 13, 2026Jinhao Pan, Bowen Wei, Yuqing Zhou +1Language Model DistillationLLM Reasoning

    George Mason University

  2. Branching On-Policy Distillation

    Oct 9, 2026Dongyang Chen, Shudong Liu, Shubo Li +4On-Policy DistillationLanguage Model Distillation

    Tsinghua University · SenseTime Research · Peking University

  3. How Much Evidence Should a Coding Agent's Self-Correction Carry? Adaptive Dirichlet Evidence for Self-Distillation

    Oct 6, 2026Yunbo Long, Guangya Hao, Yuhan Liu +4LLM Self-CorrectionCoding Agents

    University of Cambridge · Western Sydney University · Guangdong University of Technology

  4. Learning What to Imitate: Entropy-Aware Distribution Mixing

    Oct 5, 2026Juan Garcia Giraldo, Matteo Santelmo, Eduard Durech +3Language Model DistillationUncertainty-Aware Knowledge Distillation

    EPFL · ETH AI Center · Allen Institute for AI

  5. Learning from imperfect teachers for low-resource acoustic generalization

    Oct 4, 2026Shuanglin Li, Ruxiao Qian, Jian Liu +3Uncertainty-Aware Knowledge DistillationAudio Classification

  6. Outcome-Guided On-Policy Self-Distillation

    Oct 4, 2026ZheXu Wang, Mao-Lin Luo, Yankun Hong +6On-Policy Self-DistillationUncertainty-Aware Knowledge Distillation

    School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Huawei Noah’s Ark Lab +2

  7. E2^2-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

    Oct 4, 2026Yifei Liu, Minghao Fang, Xinyu Gu +6On-Policy Self-DistillationMathematical Reasoning

    The Chinese University of Hong Kong · Zhejiang University · Shanghai Artificial Intelligence Laboratory +5

  8. Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

    Sep 29, 2026Ayan Sengupta, Vaibhav Seth, Tanmoy ChakrabortyLanguage Model DistillationSelective Knowledge Distillation

    Department of Electrical Engineering, Indian Institute of Technology Delhi · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi

  9. UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents

    Sep 27, 2026Wenbo Zhang, Pengcheng Xu, Weizhi Du +2LLM AgentsUncertainty-Aware Knowledge Distillation

    University of California, Irvine · University of Michigan, Ann Arbor

  10. Hesitation-Aware On-Policy Distillation for Diffusion Language Models

    Sep 27, 2026Jianguo Huang, Lipeng Wan, Yanchen Deng +1Diffusion Language Model DecodingUncertainty-Aware Knowledge Distillation

    Nanyang Technological University, Singapore · Xi’an Jiaotong University, China

  11. Uni-Light: An Ultra-Lightweight Framework via Uncertainty-Aware Knowledge Distillation for Brain Tumour Segmentation

    Sep 6, 2026Libing Kuang, Soren Salehi, Ziling Wu +2Medical Image Segmentation3D Medical Image Segmentation

    School of Computer Science, University of Nottingham, Nottingham, UK · Electrical Engineering Department, Sharif University of Technology, Tehran, Iran · University of Pittsburgh, Pittsburgh, PA, USA

  12. Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation

    Sep 6, 2026Jingang Zhou, Yuyi Zhou, Haiyang Guo +6Language Model DistillationOn-Policy Distillation

    School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Science · Ant Group

  13. Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation

    Aug 27, 2026Ziyuan Liu, Jiao Ou, Jian Liang +2Uncertainty-Aware Knowledge DistillationMulti-Teacher Knowledge Distillation

    Kuaishou Technology Beijing, China

  14. HumP-KD: A Hybrid Uncertainty-Aware Multi-Stage Progressive Knowledge Distillation Framework for Efficient Fire Classification

    Jun 12, 2026Mohammed Arif Mainuddin, Najifa Tabassum, Omar Ibne Shahid +1Efficient Neural Network InferenceCross-Architecture Knowledge Distillation

    Electrical and Computer Engineering North South University Dhaka, Bangladesh

  15. When to Trust, How to Distill: Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecasting

    Jun 10, 2026Rupasree Dey, Abdul Matin, Nathan Orwick +3Multivariate Time Series ForecastingTime Series Forecasting

    Department of Computer Science Colorado State University Fort Collins, Colorado, USA · Department of Soil and Crop Sciences Colorado State University Fort Collins, Colorado, USA

  16. Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors

    May 27, 2026Luyang Fang, Yongkai Chen, Jiazhang Cai +2Uncertainty QuantificationUncertainty-Aware Knowledge Distillation

    Department of Statistics, University of Georgia · Department of Statistics, Harvard University

  17. Cognitive-Uncertainty Guided Knowledge Distillation for Accurate Classification of Student Misconceptions

    May 14, 2026Qirui Liu, Hao Chen, Weijie Shi +2Intelligent Tutoring SystemsAI in Education

    South China University of Technology · Tencent Financial Technology · The Hong Kong University of Science and Technology +2

  18. Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning

    May 13, 2026Junlong Ke, Zichen Wen, Weijia Li +2On-Policy Self-DistillationRL for Language Model Reasoning

    2Tsinghua University · 1Shanghai Jiao Tong University · 3Shanghai AI Laboratory

  19. GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning

    May 13, 2026Kasidit Sermsri, Teerapong PanboonyuenCoT DistillationLanguage Model Distillation

    †Chulalongkorn University · ‡MARSAIL · §PBYAIL (Panboonyuen AI Lab)

  20. TIP: Token Importance in On-Policy Distillation

    Apr 15, 2026Yuanda Xu, Hejian Sang, Zhengze Zhou +3Knowledge DistillationOn-Policy Distillation

  21. SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting

    Apr 12, 2026Binbin Zheng, Xing Ma, Yiheng Liang +6On-Policy DistillationRL for Language Model Reasoning

    University of Science and Technology of China · Meituan LongCat Interaction Team · Nanjing University +2

  22. Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models

    Mar 22, 2026Jingchen Sun, Shaobo Han, Deep Patel +3Uncertainty-Aware Knowledge DistillationKnowledge Distillation

    NEC Laboratories America, Inc., USA · University at Buffalo, SUNY · Rutgers University

  23. Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understanding

    Mar 12, 2026Songlin Li, Xin Zhu, Zechao Guan +2VLM DistillationKnowledge Distillation

    University of Electronic Science and Technology of China · Robotics Center, XPeng Motors

  24. Selective Transfer Learning of Cross-Modality Distillation for Monocular 3D Object Detection

    Mar 8, 2026Rui Ding, Meng Yang, Nanning ZhengMonocular 3D Object DetectionCross-Modal Knowledge Distillation

    Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, P.R.China.

  25. Entropy-Aware On-Policy Distillation of Language Models

    Mar 7, 2026Woogyeol Jin, Taywon Min, Yongjin Yang +5Language Model DistillationOn-Policy Distillation

    KAIST AI, Seoul, South Korea · University of Toronto, Ontario, Canada · Vector Institute, Ontario, Canada +1

  26. DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation

    Feb 25, 2026Duc Trung Vu, Pham Khanh Chi, Dat Phi Van +3Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

    Hanoi University of Science and Technology, Vietnam · University of Monash, Australia

  27. Distill3R: A Pipeline for Democratizing 3D Foundation Models on Commodity Hardware

    Jan 31, 2026Brandon Leblanc, Charalambos Poullis3D Foundation ModelsKnowledge Distillation

    Immersive and Creative Technologies Lab Concordia University Montreal, Canada

  28. Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation

    Jul 24, 2025Lakshmana Sri Harsha Nemani, P. K. Srijith, Tomasz KuśmierczykLLM Uncertainty EstimationUncertainty-Aware Knowledge Distillation

    Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad, India · Jagiellonian University, Faculty of Mathematics and Computer Science, Kraków, Poland