Knowledge Distillation

Follow this topic Ask about this topic

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

834 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

Long-Range Distillation: Distilling 10,000 Years of Simulated Climate into Long Timestep AI Weather Models
Dec 28, 2025Scott A. Martin, Noah Brenowitz, Dale Durran +1Weather Forecasting Spatiotemporal Forecasting
NVIDIA Research, Santa Clara, CA, USA · School of Oceanography, University of Washington, Seattle, WA, USA · Department of Atmospheric & Climate Science, University of Washington, Seattle, WA, USA
Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
Dec 24, 2025Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi +5CoT Distillation Selective Knowledge Distillation
1The University of British Columbia · 2LinkedIn · 3Canada Research Chair in NLP and ML
Generative Manifold Distillation: Aligning Restoration Trajectories with Natural Image Prior
Dec 11, 2025Yuyang Hu, Mojtaba Sahraee-Ardakan, Arpit Bansal +4Source-Free Domain Adaptation Image Restoration
Google · Washington University in St. Louis
Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safety
Dec 8, 2025Max Zhang, Derek Liu, Kai Zhang +2Adversarial Attacks on LLMs LLM Safety Alignment
LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection
Dec 5, 2025Johannes Meier, Jonathan Michel, Oussema Dhaouadi +7Efficient Object Detection Real-Time Object Detection
DeepScenario · ETH Zurich · TU Munich +5
From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answering
Nov 14, 2025Yu Zhao, Ying Zhang, Xuhui Sui +4Visual Question Answering CoT Distillation
College of Computer Science, VCIP, DISSec, Nankai University, Tianjin, China · School of Software, Tiangong University, Tianjin, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China +1
RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models
Oct 29, 2025Zijun Liao, Yian Zhao, Xin Shan +5Real-Time Object Detection Efficient Object Detection
School of Electronic and Computer Engineering, Peking University, Shenzhen, China · Department of Automation and BNRist, Tsinghua University, Beijing, China
Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge
Oct 22, 2025Penghao Wang, Yuhao Zhou, Mengxuan Wu +3Cross-Architecture Knowledge Distillation Knowledge Distillation
National University of Singapore · The University of Texas at Austin
Streamline pathology foundation model by cross-magnification distillation
Sep 27, 2025Ziyu Su, Abdul Rehman Akbar, Usama Sajjad +2Pathology Foundation Models Computational Pathology
Department of Pathology, College of Medicine, The Ohio State University Wexner Medical Center, Columbus, OH,USA
Resource-Limited Joint Multimodal Sentiment Reasoning and Classification via Chain-of-Thought Enhancement and Distillation
Aug 7, 2025Haonan Shangguan, Xiaocui Yang, Shi Feng +3Efficient Multimodal Inference Sentiment Analysis
School of Computer Science and Engineering, Northeastern University, Shenyang, China
Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation
Jul 24, 2025Lakshmana Sri Harsha Nemani, P. K. Srijith, Tomasz KuśmierczykLLM Uncertainty Estimation Uncertainty-Aware Knowledge Distillation
Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad, India · Jagiellonian University, Faculty of Mathematics and Computer Science, Kraków, Poland
GenRecal: Generation after Recalibration from Large to Small Vision-Language Models
Jun 18, 2025Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro +2Cross-Modal Knowledge Distillation VLM Distillation
NVIDIA · KAIST
PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models
Jun 1, 2025Seongjae Kang, Dong Bok Lee, Hyungjoon Jang +2Active Learning Coreset Selection
VUNO Inc. · KAIST
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
May 19, 2025Jitai Hao, Qiang Huang, Hao Liu +3Language Model Pretraining LLM Pruning
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen · Baidu Inc. · Leiden University +1
Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning
May 17, 2025Chen Shu, Boyu Fu, Yiman Li +5Multiple Instance Learning Histopathology Image Classification
College of Computer Science, Sichuan University · Sichuan University-Pittsburgh Institute, Sichuan University · Department of Pathology West China Hospital, Sichuan University +1
Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
May 12, 2025Seongjae Kang, Dong Bok Lee, Hyungjoon Jang +1Knowledge Distillation VLM Distillation
VUNO Inc. · KAIST
How to Backdoor Image Knowledge Distillation
Apr 30, 2025Qian Ma, Chen Wu, Prasenjit Mitra +1Adversarial Attacks Data Poisoning Attacks
The Pennsylvania State University, State College, USA · Carnegie Mellon University Africa, Kigali, Rwanda
KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation
Apr 26, 2025Jiabin Fan, Guoqing Luo, Michael Bowling +1RL for Knowledge Distillation Language Model Distillation
Dept. Computing Science, Alberta Machine Intelligence Institute (Amii), University of Alberta, Canada · Canada CIFAR AI Chair, Amii
Information-Geometric Inverse Distillation for Enhancing Adversarial Transferability
Feb 24, 2025Wenyuan Wu, Yuan Sun, Yingke Chen +4Transfer-Based Adversarial Attacks Knowledge Distillation
College of Computer Science, Sichuan University, China · Department of Computer and Information Sciences, Northumbria University, UK · School of Artificial Intelligence, Sichuan University, China
Tuned Reverse Distillation: Enhancing Multimodal Industrial Anomaly Detection with Crossmodal Tuners
Dec 12, 2024Xinyue Liu, Jianyuan Wang, Biao Leng +1Cross-Modal Knowledge Distillation Industrial Anomaly Detection
School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Key Laboratory of Intelligent Bionic Unmanned Systems, Ministry of Education, and the School of Artificial Intelligence, University of Science and Technology Beijing, Beijing 100083, China · School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China
Collaborative Knowledge Distillation via a Learning-by-Education Node Community
Sep 30, 2024Anestis Kaimakamidis, Ioannis Mademlis, Ioannis PitasOnline Knowledge Distillation Continual Learning
Aristotle University of Thessaloniki, Thessaloniki, 54124, Greece
LLMR: Knowledge Distillation with a Large Language Model-Induced Reward
Sep 19, 2024Dongheng Li, Yongchang Hao, Lili MouRL for Knowledge Distillation Language Model Distillation
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta · Canada CIFAR AI Chair, Amii
Relational Representation Distillation
Jul 16, 2024Nikos Giakoumoglou, Tania StathakiContrastive Learning Relational Knowledge Distillation
Imperial College London London, UK, SW7 2AZ
Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework
Mar 2, 2024Junxian Li, Bin Shi, Erfei Cui +2Graph Positional Encoding Knowledge Distillation
Xi’an Jiaotong University · Arizona State University, Tempe, USA · Tongji University, Shanghai, China

Dec 28, 2025J/K move · Enter open · S save

Long-Range Distillation: Distilling 10,000 Years of Simulated Climate into Long Timestep AI Weather Models

Scott A. Martin, Noah Brenowitz, Dale Durran, Michael Pritchard

NVIDIA Research, Santa Clara, CA, USA · School of Oceanography, University of Washington, Seattle, WA, USA · Department of Atmospheric & Climate Science, University of Washington, Seattle, WA, USA

Weather Forecasting Spatiotemporal Forecasting Knowledge Distillation Synthetic Data Pretraining Probabilistic Forecasting

Open paper AskPDF arXiv

Accurate long-range weather forecasting remains a major challenge for AI models, both because errors accumulate over autoregressive rollouts and because reanalysis training datasets provide limited sampling of the slow modes of variability underpinning predictability. Most AI and all numerical weather prediction models are autoregressive, repeatedly applying short-lead forecasts to reach subseasonal-to-seasonal (S2S) or seasonal lead times, often causing instability and miscalibration. Long-timestep probabilistic models that generate long-range forecasts in a single step offer an alternative, but training on the 40-year reanalysis record leads to overfitting, suggesting orders of magnitude more data are required. We introduce long-range distillation, whereby a long-timestep probabilistic

student'' model is trained to forecast directly at long-range using a huge synthetic dataset generated by an autoregressive

teacher'' model. Using the Deep Learning Earth System Model (DLESyM) as teacher, we generate over 10,000 years of simulated climate to train student models across a range of timescales. In perfect-model experiments, the distilled models outperform climatology and approach their teacher's skill while replacing hundreds of autoregressive steps with a single timestep. In the real world, both a diffusion model student fine-tuned on ERA5 and a linear inverse model student with mixed DLESyM-ERA5 training achieve S2S forecast skill competitive with the ECMWF IFS ensemble, with synthetic pre-training providing improvements beyond ERA5-only training. This demonstrates the benefit of synthetic pre-training for both deep learning and classical data-driven methods, and represents the first demonstration that real-world skill can be improved by training on AI-generated synthetic data in volumes far exceeding the observational record.