Knowledge Distillation

Follow this topic Ask about this topic

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

REALM: Retrospective Encoder Alignment for LFP Modeling
May 14, 2026Peicheng Wu, Zhenyu Bu, Runze Ma +1Brain-Computer Interfaces Neural Decoding
Department of Biomedical Engineering, The Ohio State University, Columbus, OH, United States · Department of Information Technology, Monash University Malaysia, Subang Jaya, Malaysia · Department of Neurological Surgery, The Ohio State University, Columbus, OH, United States
Cognitive-Uncertainty Guided Knowledge Distillation for Accurate Classification of Student Misconceptions
May 14, 2026Qirui Liu, Hao Chen, Weijie Shi +2Intelligent Tutoring Systems AI in Education
South China University of Technology · Tencent Financial Technology · The Hong Kong University of Science and Technology +2
Learning with Semantic Priors: Stabilizing Point-Supervised Infrared Small Target Detection via Hierarchical Knowledge Distillation
May 14, 2026Yuanhang Yao, Ping Qian, Zhu Liu +2Infrared Small Target Detection Object Detection
School of Software Technology, Dalian University of Technology, China
Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks
May 14, 2026Kai Sun, Peibo Duan, Yongsheng Huang +4Selective Knowledge Distillation Spiking Neural Networks
Faculty of Information Technology, Monash University, Australia · School of Software, Northeastern University, China · Department of Medicine, National University of Singapore, Singapore
Distribution Corrected Offline Data Distillation for Large Language Models
May 13, 2026Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake +1CoT Distillation Online Knowledge Distillation
George Mason University, Fairfax, VA, USA
Towards Resource-Efficient LLMs: End-to-End Energy Accounting of Distillation Pipelines
May 13, 2026Katherine Lambert, Sasha LuccioniEnergy-Efficient ML Language Model Distillation
University of Toronto · 2Sustainable AI Group
Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation
May 13, 2026Kaiyuan Liu, Ziyuan Zhuang, Yang Bai +3Selective Knowledge Distillation Language Model Distillation
College of Computer Science and Technology, Zhejiang University · Meituan LongCat Team, China · College of Computer Science and Technology, Jilin University
Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning
May 13, 2026Junlong Ke, Zichen Wen, Weijia Li +2On-Policy Self-Distillation RL for Language Model Reasoning
2Tsinghua University · 1Shanghai Jiao Tong University · 3Shanghai AI Laboratory
On the Generalization of Knowledge Distillation: An Information-Theoretic View
May 13, 2026Bingying Li, Haiyun HeInformation-Theoretic Generalization Bounds Statistical Learning Theory
Internet of Things Thrust, Information Hub The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China
GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning
May 13, 2026Kasidit Sermsri, Teerapong PanboonyuenCoT Distillation Language Model Distillation
†Chulalongkorn University · ‡MARSAIL · §PBYAIL (Panboonyuen AI Lab)
Lossless Anti-Distillation Sampling
May 12, 2026Zibo Diao, Jingchu Gai, Xinyue Ai +3Knowledge Distillation
Tsinghua University · Carnegie Mellon University · Peking University
STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation
May 12, 2026Shuhan Ye, Yi Yu, Qixin Zhang +5Synthetic Data Generation Spiking Neural Networks
Nanyang Technological University · Jilin University · Wuhan University +1
Minimax Rates and Spectral Distillation for Tree Ensembles
May 12, 2026Binh Duc Vu, David S. WatsonModel Compression Gradient-Boosted Decision Trees
King’s College London
OmniThoughtVis: A Scalable Distillation Pipeline for Deployable Multimodal Reasoning Models
May 12, 2026Yuanhao Yue, Chengyu Wang, Yuanjie Lyu +2Multimodal CoT Reasoning Multimodal Reasoning
Alibaba Group, Hangzhou, China
Hindsight Hint Distillation: Scaffolded Reasoning for SWE Agents from CoT-free Answers
May 12, 2026Shengjie Wang, Guanghe Li, Zonghan Yang +1Software Engineering Agents Long-Horizon LLM Agents
1Tsinghua University
A Study on Hidden Layer Distillation for Large Language Model Pre-Training
May 12, 2026Maxime Guigon, Lucas Dixon, Michaël E. SanderLanguage Model Pretraining Decoder-Only Language Models
Google DeepMind
Uncovering Latent Pathological Signatures in Pulmonary CT via Cross-Window Knowledge Distillation
May 12, 2026Bo Peng, Wujian Xu, Kun Wang +8Medical Image Classification Computed Tomography
*These authors contributed equally to this work.
Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning
May 12, 2026Zihao Han, Tiangang Zhang, Huaibin Wang +1Adaptive Knowledge Distillation On-Policy Self-Distillation
ByteDance Douyin
Generative Diffusion Prior Distillation for Long-Context Knowledge Transfer
May 12, 2026Nilushika Udayangani, Kishor Nandakishor, Marimuthu PalaniswamiTime Series Classification Feature-Based Knowledge Distillation
Department of Electrical and Electronic Engineering University of Melbourne Parkville, VIC 3052, Australia
Curriculum Learning-Guided Progressive Distillation in Large Language Models
May 11, 2026Jincheng Cao, Fanzhi Zeng, Leqi Liu +1Reasoning Distillation Language Model Distillation
The University of Texas at Austin · Google Research
LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
May 11, 2026Nikolaos Gkalelis, Vasileios MezarisVision-Language Models Efficient VLM Inference
CERTH-ITI, Thessaloniki, Greece, 57001
Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks
May 11, 2026Minh-Toan Nguyen, Jean BarbierNeural Network Generalization Hierarchical Representation Learning
ICTP Trieste, Italy
Evolving Knowledge Distillation for Lightweight Neural Machine Translation
May 11, 2026Xuewen Zhang, Haixiao Zhang, Xinlong HuangMulti-Teacher Knowledge Distillation Knowledge Distillation
Department of Content Generation Li Auto Beijing, China
LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering
May 10, 2026Runze Ma, Shunbo Jia, Haonan Lyu +2CoT Distillation Healthcare
School of Information Technology Monash University Malaysia · Faculty of Innovation Engineering Macau University of Science and Technology · Department of Bioelectronics Faculty of Biomedical Engineering Shenzhen University of Advanced Technology +1
Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
May 10, 2026Yuxuan Jiang, Runchao Li, Shubhashis Roy Dipta +2Language Model Distillation On-Policy Distillation
University of Maryland, Baltimore County · Case Western Reserve University · Arizona State University +1
Evidence Over Plans: Online Trajectory Verification for Skill Distillation
May 9, 2026Yang Zhou, Zihan Dong, Zhenting Wang +7Skill Distillation Agent Evaluation
Rutgers University
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Mixture-of-Experts Pruning LLM Pruning
Qwen Team, Alibaba Inc. · MBZUAI · KAUST
SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models
May 8, 2026Jiesong Lian, Zixiang Zhou, Ruizhe Zhong +6Diffusion Model Alignment Video Diffusion Models
1Huazhong University of Science and Technology · 2Tencent Hunyuan · 3Shanghai Jiao Tong University +1
Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models
May 8, 2026Boyu Shi, YiCheng Jiang, Chang Liu +3Small Language Models Cross-Architecture Knowledge Distillation
School of Computer Science and Engineering, Southeast University, Nanjing, China
SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
May 8, 2026Jie Sun, Mao Zheng, Mingyang Song +6Cross-Tokenizer Knowledge Distillation Language Model Distillation
University of Science and Technology of China · Shanghai Innovation Institute · Large Language Model Department, Tencent +1

May 14, 2026J/K move · Enter open · S save

REALM: Retrospective Encoder Alignment for LFP Modeling

Peicheng Wu, Zhenyu Bu, Runze Ma, Lin Du

Department of Biomedical Engineering, The Ohio State University, Columbus, OH, United States · Department of Information Technology, Monash University Malaysia, Subang Jaya, Malaysia · Department of Neurological Surgery, The Ohio State University, Columbus, OH, United States

Brain-Computer Interfaces Neural Decoding Knowledge Distillation

Open paper AskPDF arXiv

Spike activity has been the dominant neural signal for behavior decoding because its high spatiotemporal resolution supports accurate decoding. However, as intracortical brain-computer interfaces (iBCIs) move toward higher channel counts and wireless operation, the high sampling rates required to record spikes create substantial power and bandwidth demands. Local field potentials (LFPs) offer complementary advantages, including greater long-term stability, lower energy consumption, and lower bandwidth requirements. However, LFP-based decoders often achieve lower accuracy and rely on non-causal architectures that cannot be used directly for real-time deployment. We propose REALM, a retrospective knowledge distillation (RKD) framework for causal LFP behavior decoding. Inspired by offline-to-online distillation in speech recognition, REALM transfers non-causal representational knowledge from a pretrained, multi-session bidirectional LFP teacher to a causal student model. We first pretrain a bidirectional Mamba-2 teacher across multiple recording sessions using continuous masked autoencoding (CMAE), and then distill its representation into a compact causal student using a combined objective of representation alignment and autoencoding. REALM achieves the highest mean accuracy among the compared decoders in both label-free and fine-tuned pipelines, with statistically significant improvements over each baseline, including the state-of-the-art CrossModalDistill. It does so using LFPs alone throughout pretraining, distillation, and decoding, with less than half the parameters of CrossModalDistill's published student and one-tenth of its pretraining time. These results show that a causal LFP-only model can achieve decoding accuracy competitive with a non-causal multi-modal model, offering a practical and scalable approach for next-generation wireless and implantable iBCIs.