Knowledge Distillation

Follow this topic Ask about this topic

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

834 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity
Aug 11, 2026Junyong Choi, Cheolhyeon Park, Jaehoon ChoVision Transformer Cross-Architecture Knowledge Distillation
Hyundai Motor Company · School of Electronics and Avionics Engineering, Korea Aerospace University
SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features
Aug 11, 2026HyeonJun Lee, Hyeonsik Jo, Jinwoo Chung +1Quantization-Aware Training Quantization-Aware Distillation
Kookmin University
MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scale
Aug 11, 2026Yuhang Yao, Zeyu Wang, Wanyi Chen +8Continual Learning for LLM Agents LLM Fine-Tuning
Carnegie Mellon University · University of California, Los Angeles · Soochow University +3
Black-Box Knowledge Transfer across Distinct Feature Sets
Aug 10, 2026Oh-Ran Kwon, Daeyoung HamTransfer Learning Knowledge Distillation
Department of Statistics, The Ohio State University · Department of Statistics and Data Science, University of Texas at San Antonio
Mismatch Matters: On-Policy Distillation Beyond Token Agreement
Aug 10, 2026Zichao Yu, Chengzhi Yu, Shengze Xu +4Distribution Matching Language Model Distillation
The University of Hong Kong · University of Science and Technology of China · The Chinese University of Hong Kong
ADOPD: Reference-Privileged On-Policy Distillation for MLLM-Based Industrial Anomaly Detection
Aug 10, 2026Jingtai He, Shiyuan Meng, Wenchao Meng +1Industrial Anomaly Detection Multimodal Large Language Models
Zhejiang University
SR-OPSD: Self-Referenced On-Policy Self-Distillation
Aug 10, 2026Zhuo Sun, Entong Li, Yanlong Zhao +9On-Policy Self-Distillation Language Model Distillation
Shanghai University of Finance and Economics · Imperial College London · Independent Researcher +6
Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents
Aug 10, 2026Tianjun Pan, Yuan Li, Hongda Wang +8LLM Agent Skill Learning Self-Distillation
Fudan University · Zhejiang University · Alibaba Group +1
UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation
Aug 10, 2026Xuewan He, Tong Chu, Zihan Cheng +5Synthetic Data Generation Cross-Architecture Knowledge Distillation
University of Electronic Science and Technology of China, Chengdu, China
PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation
Aug 9, 2026Yangyang Feng, Zhuoyan Feng, Junlan ChenOn-Policy Self-Distillation Knowledge Distillation
1The Hong Kong University of Science and Technology (Guangzhou) · 2Sun Yat-sen University
Out-of-Distribution Federated Distillation with Domain-Aware Proxy
Aug 9, 2026Jiahao Xiao, Jiangming LiuOOD Generalization Federated Knowledge Distillation
School of Information Science and Engineering, Yunnan University, China
A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs
Aug 8, 2026Abhinand Balachandran, Praveen PrashantCross-Architecture Knowledge Distillation Feature-Based Knowledge Distillation
Georgia Institute of Technology Atlanta, Georgia, USA
Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
Aug 8, 2026Yongkang Yang, Zhezheng Hao, Hong Zhang +8Adaptive Knowledge Distillation On-Policy Self-Distillation
Tencent, Workbuddy Team · Zhejiang University · Independent Researcher
GeoDistill-Refine: Silhouette-First Geometry Distillation for Annotation-Free Spacecraft Segmentation
Aug 7, 2026Yonglong Zhang, Zongwu Xie, Yang LiuImage Segmentation Pseudo-Labeling
School of Mechatronics Engineering, Harbin Institute of Technology, Harbin, Heilongjiang, China
Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
Aug 7, 2026Taeil Kim, Kangsan Kim, Sung Ju HwangLLM Agent Memory LLM Tool Use
KAIST
InsertFuse: A Unified Framework for Multi-Category Reference-Guided Image Insertion
Aug 6, 2026Guangzhao Li, Qingyan Wei, Huayu Zheng +7Conditional Image Generation Diffusion-Based Image Generation
1Shanghai Jiao Tong University · 2Shanghai Innovation Institute · 4Xi’an Jiaotong University +2
FedTransKD-IDS: Robust Federated Transfer Learning with Knowledge Distillation for Intrusion Detection in IoT
Aug 6, 2026Mohammad Hosssein Gholamrezazadeh, Ahmadreza MontazerolghaemAhmadreza MontazerolghaemIoT Intrusion Detection Cybersecurity
Faculty of Computer Engineering University of Isfahan Isfahan, Iran
The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents
Aug 6, 2026Weiwei Li, Junzhuo Liu, Tong Chu +2GUI Agents Imitation Learning
University of Electronic Science and Technology of China
BioKD: Selective Physiology-to-Video Knowledge Distillation via Reliability Gate for Emotion Recognition
Aug 6, 2026Bojing Hou, Ruohao Li, Yitong Zhu +3Cross-Modal Knowledge Distillation Selective Knowledge Distillation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · New York University, New York, NY, USA
Flow-Map Distillation on Relation Manifolds for Image Restoration
Aug 6, 2026Zihao He, Songhua LiuFlow Matching Flow Map Learning
School of Artificial Intelligence · Shanghai Jiao Tong University · Shanghai City, China
Subliminal Learning is Non-Semantic Distillation
Aug 6, 2026Ethan Hadley, Eren GultepeLLM Auditing Subliminal Learning
Department of Computer Science, Southern Illinois University Edwardsville, Edwardsville, IL, USA.
Spectral Distillation: From Nonlinear Dynamics to Linear State-Space Models
Aug 5, 2026Liane Galanti, Devan Shah, Shlomo Fortgang +1Dynamical Systems Nonlinear System Identification
Princeton University
When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents
Aug 5, 2026Junzhuo Liu, Weiwei Li, Jun Ling +1Language Model Distillation On-Policy Distillation
University of Electronic Science and Technology of China
SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation
Aug 5, 2026Zikun Qu, Min Zhang, Mingze Kong +4Language Model Distillation On-Policy Distillation
The Chinese University of Hong Kong, Shenzhen · East China Normal University · Beihang University +1
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
Aug 4, 2026Bakbergen Ryskulov, Iker García-Ferrero, David Montero +5Online Knowledge Distillation Efficient Language Model Training
Multiverse Computing
Track4Action: Distilling World-Centric 3D Tracker into Vision-Language-Action Policies
Aug 4, 2026Chenyi Wang, Xinkai Wang, Bokai Lin +4Visuomotor Policy Learning Vision-Language-Action Models
1Zhejiang University, China · 3Shanghai Innovation Institute, China · 2Shanghai Jiao Tong University, China +1
When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
Aug 4, 2026Yinuo Jiang, Yongjie Ye, Zhou Tao +4VLM Distillation Language Model Distillation
Zhejiang University · ByteDance · Shanghai Artificial Intelligence Laboratory
Lightweight 3D Object Detection via Mamba-Based Knowledge Distillation
Aug 4, 2026Quoc Cuong Ninh, Huy Xuan Pham, Anh Tung Nguyen +1Lidar-Based 3D Object Detection 3D Object Detection
Viettel AI, Viettel Group, Vietnam · Artificial Intelligence in Robotics Laboratory (AiR Lab), Department of Electrical and Computer Engineering, Aarhus University, 8000 Aarhus C, Denmark · VIST, Viettel Group, Vietnam
Channel-wise Dynamic Knowledge Distillation via Adaptive Sample Generation for Action Recognition
Aug 4, 2026Ping Li, Chenhao Ping, Jie Song +1Video Action Recognition Knowledge Distillation
School of Computer Science, Hangzhou Dianzi University, Hangzhou, China · State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China
Rubrics as Privileged Information for Open-Ended Generation
Aug 3, 2026Deepika Bablani, Ajay Gupta, Wanming ChenOpen-Ended Generation Rubric-Based Evaluation
Apple

Aug 11, 2026J/K move · Enter open · S save

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

Junyong Choi, Cheolhyeon Park, Jaehoon Cho

Hyundai Motor Company · School of Electronics and Avionics Engineering, Korea Aerospace University

Vision Transformer Cross-Architecture Knowledge Distillation Inductive Bias Knowledge Distillation Data-Efficient Learning

Open paper AskPDF arXiv

Vision Transformers demonstrate remarkable global modeling capacity but often underperform in data-scarce regimes. Distilling convolutional inductive biases from a CNN teacher provides an effective remedy while leaving the deployed model unchanged. However, general-purpose feature distillation transfers little in this setting. In CNN-to-CNN distillation, pooling, flattening, and logit-space projections remove the spatial grid that encodes locality and translation equivariance. Unlike a convolutional student, a ViT cannot readily reconstruct this structure on its own. In this paper, we propose iBKD, a distillation framework that preserves the spatial grid throughout the entire transfer process. Its core module, the Inductive Bias Attention Module, aggregates features from all student layers onto the teacher's grid using learned weights. It then enhances structural cues through channel and deformable spatial attention and injects them via convolutional cross-attention operating directly between spatial grids rather than token sets. The module is used only during training, leaving the deployed model as an unmodified ViT with no inference overhead. Across seven Transformer backbones and six data-scarce benchmarks, iBKD consistently outperforms both locality-guidance methods and general knowledge distillation baselines, with its advantage increasing as the amount of training data decreases.