Online Knowledge Distillation

Latest papers

10 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing
Oct 14, 2026Andrey Goncharov, Daniil Vyazhev, Simon Epanov +1Online Knowledge Distillation Language Model Distillation
Applied AI Institute
Learning from Teacher Continuations at Student States
Sep 28, 2026Haojin Wang, Dylan Zhang, Huaibo Chen +8Online Knowledge Distillation Language Model Distillation
University of Illinois at Urbana-Champaign · Massachusetts Institute of Technology · University of Pennsylvania +2
When and What to Teach: Budget-Aware Online Adaptation for Web Agents
Aug 31, 2026Jianwei Zhang, Sihan Cao, Pengcheng Zheng +7Online Knowledge Distillation LLM Agent Training
University of Electronic Science and Technology of China · Institute of Software Chinese Academy of Sciences · Xi’an University of Architecture and Technology
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
Aug 4, 2026Bakbergen Ryskulov, Iker García-Ferrero, David Montero +5Online Knowledge Distillation Efficient Language Model Training
Multiverse Computing
Distribution Corrected Offline Data Distillation for Large Language Models
May 13, 2026Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake +1CoT Distillation Online Knowledge Distillation
George Mason University, Fairfax, VA, USA
Private Speech Classification without Collapse: Stabilized DP Training and Offline Distillation
May 4, 2026Yadi Wen, Tianxin Li, Enji Liang +2Differentially Private Stochastic Gradient Descent Online Knowledge Distillation
Shanxi Key Laboratory of Industrial Internet Security, Taiyuan University of Technology, China · University of Michigan, Ann Arbor, USA
LIE: LiDAR-only HD Map Construction with Intensity Enhancement via Online Knowledge Distillation
May 2, 2026Kanak Mazumder, Fabian B. FlohrHD Map Construction Online Knowledge Distillation
Munich University of Applied Sciences, Intelligent Vehicles Lab (IVL), Munich, Germany
DIB-OD: Preserving the Invariant Core for Robust Heterogeneous Graph Adaptation via Decoupled Information Bottleneck and Online Distillation
Apr 13, 2026Yang Yan, Yunxuan Li, Qiuyan Wang +2Online Knowledge Distillation Graph Transfer Learning
School of Information Technology and Engineering, Tianjin University of Technology and Education, Tianjin, China · School of Computer Science and Technology, Tiangong University, Tianjin, China · Department of Computer Science at University of Exeter
Efficient Personalized Reranking with Semi-Autoregressive Generation and Online Knowledge Distillation
Mar 7, 2026Kai Cheng, Hao Wang, Wei Guo +4Online Knowledge Distillation Recommender Systems
University of Science and Technology of China, Hefei, China · Huawei, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China +1
Collaborative Knowledge Distillation via a Learning-by-Education Node Community
Sep 30, 2024Anestis Kaimakamidis, Ioannis Mademlis, Ioannis PitasOnline Knowledge Distillation Continual Learning
Aristotle University of Thessaloniki, Thessaloniki, 54124, Greece

Oct 14, 2026J/K move · Enter open · S save

Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing

Andrey Goncharov, Daniil Vyazhev, Simon Epanov, Alexey Zaytsev

Applied AI Institute

Online Knowledge Distillation Language Model Distillation Knowledge Distillation CoT Distillation Selective Knowledge Distillation

Open paper AskPDF arXiv

Distillation of reasoning chains from Large Language Models (LLMs) into their smaller siblings is a standard training approach. Frequently, researchers do not have full access to a teacher model and its internal representations, which limits the resulting performance. We propose a recursive offline distillation method combining two components: entropy-based sampling for selective distillation and answer mixing. Specifically, for three open models (3-4B), at each epoch, we sample reasoning chains weighted by entropy difference with an auxiliary medium model (70-72B) and, next, supplement thinking traces with single-token ground truth answers. Our approach consistently outperforms naive distillation with comparable data budgets (

57.15 \pm 1.01

vs 51.61 on average) and offers a much steeper learning curve. Additionally, we present a comparative analysis of methods to generate and normalize distillation traces.