Knowledge Distillation

Follow this topic Ask about this topic

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection
Dec 5, 2025Johannes Meier, Jonathan Michel, Oussema Dhaouadi +7Efficient Object Detection Real-Time Object Detection
DeepScenario · ETH Zurich · TU Munich +5
From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answering
Nov 14, 2025Yu Zhao, Ying Zhang, Xuhui Sui +4Visual Question Answering CoT Distillation
College of Computer Science, VCIP, DISSec, Nankai University, Tianjin, China · School of Software, Tiangong University, Tianjin, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China +1
RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models
Oct 29, 2025Zijun Liao, Yian Zhao, Xin Shan +5Real-Time Object Detection Efficient Object Detection
School of Electronic and Computer Engineering, Peking University, Shenzhen, China · Department of Automation and BNRist, Tsinghua University, Beijing, China
Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge
Oct 22, 2025Penghao Wang, Yuhao Zhou, Mengxuan Wu +3Cross-Architecture Knowledge Distillation Knowledge Distillation
National University of Singapore · The University of Texas at Austin
Streamline pathology foundation model by cross-magnification distillation
Sep 27, 2025Ziyu Su, Abdul Rehman Akbar, Usama Sajjad +2Pathology Foundation Models Computational Pathology
Department of Pathology, College of Medicine, The Ohio State University Wexner Medical Center, Columbus, OH,USA
Resource-Limited Joint Multimodal Sentiment Reasoning and Classification via Chain-of-Thought Enhancement and Distillation
Aug 7, 2025Haonan Shangguan, Xiaocui Yang, Shi Feng +3Efficient Multimodal Inference Sentiment Analysis
School of Computer Science and Engineering, Northeastern University, Shenyang, China
Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation
Jul 24, 2025Lakshmana Sri Harsha Nemani, P. K. Srijith, Tomasz KuśmierczykLLM Uncertainty Estimation Uncertainty-Aware Knowledge Distillation
Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad, India · Jagiellonian University, Faculty of Mathematics and Computer Science, Kraków, Poland
GenRecal: Generation after Recalibration from Large to Small Vision-Language Models
Jun 18, 2025Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro +2Cross-Modal Knowledge Distillation VLM Distillation
NVIDIA · KAIST
PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models
Jun 1, 2025Seongjae Kang, Dong Bok Lee, Hyungjoon Jang +2Active Learning Coreset Selection
VUNO Inc. · KAIST
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
May 19, 2025Jitai Hao, Qiang Huang, Hao Liu +3Language Model Pretraining LLM Pruning
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen · Baidu Inc. · Leiden University +1
Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
May 12, 2025Seongjae Kang, Dong Bok Lee, Hyungjoon Jang +1Knowledge Distillation VLM Distillation
VUNO Inc. · KAIST
How to Backdoor Image Knowledge Distillation
Apr 30, 2025Qian Ma, Chen Wu, Prasenjit Mitra +1Adversarial Attacks Data Poisoning Attacks
The Pennsylvania State University, State College, USA · Carnegie Mellon University Africa, Kigali, Rwanda
KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation
Apr 26, 2025Jiabin Fan, Guoqing Luo, Michael Bowling +1RL for Knowledge Distillation Language Model Distillation
Dept. Computing Science, Alberta Machine Intelligence Institute (Amii), University of Alberta, Canada · Canada CIFAR AI Chair, Amii
Information-Geometric Inverse Distillation for Enhancing Adversarial Transferability
Feb 24, 2025Wenyuan Wu, Yuan Sun, Yingke Chen +4Transfer-Based Adversarial Attacks Knowledge Distillation
College of Computer Science, Sichuan University, China · Department of Computer and Information Sciences, Northumbria University, UK · School of Artificial Intelligence, Sichuan University, China
Tuned Reverse Distillation: Enhancing Multimodal Industrial Anomaly Detection with Crossmodal Tuners
Dec 12, 2024Xinyue Liu, Jianyuan Wang, Biao Leng +1Cross-Modal Knowledge Distillation Industrial Anomaly Detection
School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Key Laboratory of Intelligent Bionic Unmanned Systems, Ministry of Education, and the School of Artificial Intelligence, University of Science and Technology Beijing, Beijing 100083, China · School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China
Collaborative Knowledge Distillation via a Learning-by-Education Node Community
Sep 30, 2024Anestis Kaimakamidis, Ioannis Mademlis, Ioannis PitasOnline Knowledge Distillation Continual Learning
Aristotle University of Thessaloniki, Thessaloniki, 54124, Greece
LLMR: Knowledge Distillation with a Large Language Model-Induced Reward
Sep 19, 2024Dongheng Li, Yongchang Hao, Lili MouRL for Knowledge Distillation Language Model Distillation
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta · Canada CIFAR AI Chair, Amii
Relational Representation Distillation
Jul 16, 2024Nikos Giakoumoglou, Tania StathakiContrastive Learning Relational Knowledge Distillation
Imperial College London London, UK, SW7 2AZ
Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework
Mar 2, 2024Junxian Li, Bin Shi, Erfei Cui +2Graph Positional Encoding Knowledge Distillation
Xi’an Jiaotong University · Arizona State University, Tempe, USA · Tongji University, Shanghai, China

Dec 5, 2025J/K move · Enter open · S save

LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection

Johannes Meier, Jonathan Michel, Oussema Dhaouadi, Yung-Hsu Yang+6

DeepScenario · ETH Zurich · TU Munich+5

Efficient Object Detection Real-Time Object Detection Knowledge Distillation Monocular 3D Object Detection

Open paper AskPDF arXiv

Real-time monocular 3D object detection remains challenging due to severe depth ambiguity, viewpoint shifts, and the high computational cost of 3D reasoning. Existing approaches either rely on LiDAR or geometric priors to compensate for missing depth or sacrifice efficiency to achieve competitive accuracy. We introduce LeAD-M3D, a monocular 3D detector that achieves state-of-the-art accuracy and real-time inference without extra modalities. Our method is enabled by three key components. Asymmetric Augmentation Denoising Distillation (A2D2) transfers geometric knowledge from a clean-image teacher to a MixUp-noised student via a quality- and importance-weighted depth-feature loss, enabling stronger depth reasoning without LiDAR. 3D-aware Consistent Matching (CM

_{\text{3D}}

) improves prediction-to-ground truth assignment by integrating 3D MGIoU into the matching score, yielding stable and precise supervision. Finally, Confidence-Gated 3D Inference (CGI

_{\text{3D}}

) accelerates inference by restricting expensive 3D regression to confident regions. Together, these contributions set a new Pareto frontier for monocular 3D detection: LeAD-M3D achieves state-of-the-art accuracy on KITTI and Waymo, and the best reported car AP on Rope3D, while running up to 3.6

\,\times

faster than prior high-accuracy models (e.g., MonoDiff). LeAD-M3D demonstrates that high fidelity and real-time monocular 3D detection is simultaneously attainable, without LiDAR, stereo, or strong geometric assumptions.