Knowledge Distillation

Follow this topic Ask about this topic

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

Knowledge Distillation for Visual Autoregressive Models
Jun 4, 2026Elia Peruzzo, Aritra Bhowmik, Guillaume Sautiere +2Visual Autoregressive Models Autoregressive Image Generation
Qualcomm AI Research∗ · Qualcomm AI Research · University of Technology Nuremberg
OPRD: On-Policy Representation Distillation
Jun 4, 2026Shenzhi Yang, Guangcheng Zhu, Bowen Song +8Cross-Architecture Knowledge Distillation Language Model Distillation
Zhejiang University · Ant Group
Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation
Jun 4, 2026Maxime Griot, Paul Steven Scotti, Tanishq Mathew AbrahamCoT Distillation Teacher-Student Learning
Université catholique de Louvain · Sophont Inc.
LiAuto-GeoX: Efficient Grounded Driving Transformer
Jun 4, 2026Jiawei Lian, Haoyi Sun, Yang Wu +83D ViTs 3D Reconstruction
Nanjing University of Science and Technology · Li Auto Inc. · Northwestern Polytechnical University +1
ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation
Jun 4, 2026Kanghui Tian, Siyuan Liu, Ziang Yan +3VLM Distillation Multimodal Reasoning
1Shanghai AI Laboratory · 3Nanjing University · 2Fudan University
Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation
Jun 4, 2026Fangbo Tu, Junhua Zhao, Chi Liu +4LLM Quantization Quantization-Aware Distillation
PayPal, Inc.
LoRi: Low-Rank Distillation for Implicit Reasoning
Jun 3, 2026Ryan Solgi, Jiayi Tian, Zheng ZhangCoT Distillation Mathematical Reasoning
University of California-Santa Barbara, USA
Invariant Gradient Alignment for Robust Reasoning Distillation
Jun 3, 2026Zehua Cheng, Wei Dai, Jiahao SunCoT Distillation Domain Generalization
University of Oxford, Oxford, UK
Geometry-Aware Distillation for Prompt Tuning Biomedical Vision-Language Models
Jun 3, 2026Tran Dinh Tien, Zhiqiang ShenVLM Adaptation VLM Distillation
Department of Machine Learning Mohamed bin Zayed University of Artificial Intelligence
DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer
Jun 3, 2026Patomporn Payoungkhamdee, Tinnakit Udsa, Jian Gang Ngui +3Multilingual Language Models Small Language Models
School of Information Science and Technology, VISTEC · AI Singapore · MBZUAI
Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning
Jun 3, 2026Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh AfghahEfficient Multimodal Inference Multimodal ICL
Holcombe Department of Electrical and Computer Engineering, Clemson University, Clemson, United States.
Derivative Informed Learning of Exchange-Correlation Functionals
Jun 2, 2026Eike S. Eberhard, Luca A. Thiede, Abdul Aldossary +5Quantum Chemistry Density Functional Theory
Technical University of Munich, Munich, Germany · Munich Data Science Institute, Munich, Germany · Munich Center for Machine Learning, Munich, Germany +4
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Jun 2, 2026Ali Behrouz, Farnoosh Hashemi, Adel Javanmard +1Continual Learning for LLMs Persistent Memory for Language Models
Google
AAD-1: Asymmetric Adversarial Distillation for One-Step Autoregressive Video Generation
Jun 2, 2026Haobo Li, Yanhong Zeng, Yunhong Lu +6Image-to-Video Generation Autoregressive Video Generation
AutoLab, SAI, SJTU · Ant Group · Department of Automation, Tsinghua University +2
BAHSD: Bridging the Long-tail Gap via Adaptive Distillation in Black-box Sequential Recommendation
Jun 2, 2026Xi Zhou, Famin Wu, Mingming Li +4Adaptive Knowledge Distillation Sequential Recommendation
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Beijing Institute for General Artificial Intelligence, Beijing, China
What Do Students Learn? A Feature-Level Analysis of Dark Knowledge
Jun 2, 2026Seungu Kang, Songkuk KimRepresentation Learning Self-Distillation
Yonsei University, Seoul 03722, Republic of Korea
Pathway-Structured Privileged Distillation for Deployable Computational Pathology
Jun 1, 2026Yongxin Guo, Hao Lu, Onur Koyun +2Computational Pathology Oncology
School of Medicine, Wake Forest University, Winston-Salem, NC, USA.
Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation
Jun 1, 2026Yuying Li, Leqi Zheng, Yongzi Yu +6Selective Knowledge Distillation Language Model Distillation
THU · Meituan · HKUST +2
SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
Jun 1, 2026Hao Li, Jingkun An, Zijun Song +8LLM Safety Alignment On-Policy Distillation
Beihang University · Beijing Institute of Technology · Beijing University of Posts and Telecommunications +4
Cross-Domain Dead Tree Detection via Knowledge Distillation in Aerial Imagery
Jun 1, 2026Anis Ur Rahman, Mete Ahishali, Einari Heinaro +1Remote Sensing Image Understanding Cross-Domain Transfer Learning
CSC – IT Center for Science Ltd., Espoo, Finland · Department of Forest Sciences, University of Helsinki · KOKO Forest Ltd. +1
Pool-Select-Refine for Allocation-Aware Generative Dataset Distillation
Jun 1, 2026Wenmin Li, Shunsuke Sakai, Zhongkai Zhao +1Dataset Distillation Knowledge Distillation
Graduate School of Engineering, University of Fukui, 3-9-1 Bunkyo, Fukui, 910-8507, Fukui, Japan · College of Computer Science and Artificial Intelligence, Southwest Minzu University, No.168, Wenxing Section, Dajian Road, Konggang Development Zone, Shuangliu District, Chengdu, 610225, Sichuan, China
What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression
May 31, 2026Wendao Wu, Fangqing Zhang, Haihan Zhang +1High-Dimensional Linear Regression Weak-to-Strong Generalization
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University, China · School of Mathematical Sciences, Peking University, China
SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation
May 31, 2026Aditya Makineni, Qing TianCross-Architecture Knowledge Distillation Relational Knowledge Distillation
Department of Computer Science University of Alabama at Birmingham Birmingham, US
Rethinking the Role of Temperature in Large Language Model Distillation
May 29, 2026Hoang-Chau Luong, Lingwei ChenLanguage Model Distillation Knowledge Distillation
Golisano College of Computing and Information Sciences Rochester Institute of Technology Rochester, NY, United States
Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
May 29, 2026Yuxuan Jiang, Francis FerraroCoT Distillation Language Model Distillation
University of Maryland, Baltimore County
COLLEAGUE.SKILL: Automated AI Skill Generation via Expert Knowledge Distillation
May 29, 2026Tianyi Zhou, Dongrui Liu, Leitao Yuan +2LLM Agent Skill Learning Persona-Conditioned Generation
Shanghai Artificial Intelligence Laboratory
Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10
May 29, 2026Umut Onur YasarTeacher-Student Learning Feature-Based Knowledge Distillation
Trust-Region Behavior Blending for On-Policy Distillation
May 29, 2026Daniil Plyusov, Alexey Gorbatovski, Alexey Malakhov +4Language Model Distillation Trust-Region Optimization
T-Tech
DEM: A Distilled Explanation Model for Interpretable Anomaly Detection in Physiological Sensor Networks
May 29, 2026Jyotirmoy Singh, Anushka Roy, Shreea Bose +1Interpretable ML Interpretable Anomaly Detection
EVL-ECG: Efficient ECG Interpretation With Multi-Aspect Heterogeneous Knowledge Distillation
May 28, 2026Dang Nguyen Hong, Nhi Ngoc-Yen Nguyen, Huy-Hieu PhamEfficient Neural Network Inference Electrocardiogram Foundation Models
Hanoi University of Science and Technology, Hanoi, Vietnam · VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam · College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam +1

Jun 4, 2026J/K move · Enter open · S save

Knowledge Distillation for Visual Autoregressive Models

Elia Peruzzo, Aritra Bhowmik, Guillaume Sautiere, Yuki M Asano+1

Qualcomm AI Research∗ · Qualcomm AI Research · University of Technology Nuremberg

Visual Autoregressive Models Autoregressive Image Generation Knowledge Distillation

Open paper AskPDF arXiv

Autoregressive (AR) image generation models are highly expressive but computationally intensive, motivating effective model compression. Knowledge distillation (KD) is a natural approach for model compression and has been widely studied in language modeling, yet its behavior in visual AR generation remains underexplored. In this work, we present the first systematic study of distillation strategies for AR image models. Our analysis shows that while standard distillation can yield meaningful gains, recent methods developed for language do not directly transfer to images: long decoding horizons and visual token ambiguity make teacher supervision unreliable especially under student-conditioned contexts. To address this, we propose VarKD, a distillation framework for visual autoregressive models that distills on student samples while selectively applying teacher supervision and reducing token-level ambiguity. Experiments on ImageNet across multiple AR backbones show that VarKD consistently outperforms prior distillation baselines, narrowing the gap to large-scale models.