Knowledge Distillation

Follow this topic Ask about this topic

Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.

Latest papers

829 papers164 papers in the last four weeks, up 95% on the four weeks before. 0.9% of all new papers.

PACD-Net: Pseudo-Augmented Contrastive Distillation for Glycemic Control Estimation from SMBG
May 20, 2026Canyu Lei, David Repaske, Jianxin XieIrregular Time-Series Modeling Contrastive Learning
University of Virginia, School of Data Science, Charlottesville, VA 22903, USA · University of Virginia, Department of Pediatrics, Charlottesville, VA 22903, USA
STAR-IOD: Scale-decoupled Topology Alignment with Pseudo-label Refinement for Remote Sensing Incremental Object Detection
May 20, 2026Yaoteng Zhang, Qing Zhou, Junyu Gao +1Pseudo-Labeling Object Detection
School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China · School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, China · Northwestern Polytechnical University, Xi’an 710072, China
GSA-YOLO: A High-Efficiency Framework via Structured Sparsity and Adaptive Knowledge Distillation for Real-Time X-ray Security Inspection
May 20, 2026Jiahao KongStructured Pruning Real-Time Object Detection
SDU-ANU Joint Science College, Shandong University, Weihai, 264200, China
AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals
May 20, 2026Duy Nguyen, Hanqi Xiao, Archiki Prasad +7On-Policy Self-Distillation Language Model Distillation
UNC Chapel Hill · Capital One · The University of Texas at Austin
Consistently Informative Soft-Label Temperature for Knowledge Distillation
May 19, 2026Hoang-Chau Luong, Nghia Van Vo, Kaiqi Zhao +1Teacher-Student Learning Knowledge Distillation
Rochester Institute of Technology, Rochester, NY, USA · Oakland University, Rochester, MI, USA
Towards Distillation Guarantees under Algorithmic Alignment for Combinatorial Optimization
May 19, 2026Thien Le, Melanie WeberDynamic Programming Combinatorial Optimization
SEAS, Harvard University · Cambridge, MA
Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation
May 19, 2026Yuanpei Zhao, Jie Lin, Chao Zhang +5Pairwise Preference Learning Personalized Image Aesthetics Assessment
1Sichuan University · 2NetEase Fuxi AI Lab · †Work done during an internship at NetEase Fuxi AI Lab. +1
LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models
May 19, 2026Hyunsoo Han, Sangyeop Yeo, Jaejun YooModel Compression Diffusion Model Distillation
Ulsan National Institute of Science and Technology (UNIST)
CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization
May 19, 2026Ahmed Heakl, Abdelrahman M. Shaker, Youssef Mohamed +4Token-Level Credit Assignment RL for Language Model Reasoning
MBZUAI · Linköping University · Australian National University
Cross-Paradigm Knowledge Distillation: A Comprehensive Study of Bidirectional Transfer Between Random Forests and Deep Neural Networks for Big Data Applications
May 19, 2026Mahdi Naser MoghadasiMulti-Teacher Knowledge Distillation Knowledge Distillation
BrightMind AI Research Seattle, Washington, USA
Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation
May 18, 2026Qianhao Yuan, Jie Lou, Xing Yu +4Multimodal Large Language Models On-Policy Distillation
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xiaohongshu Inc.
Distilling Tabular Foundation Models for Structured Health Data
May 18, 2026Aditya Tanna, Nassim Bouarour, Mohamed Bouadi +2Tabular Foundation Models Efficient Neural Network Inference
Lexsi Labs
Pocket Foundation Models: Distilling TFMs into CPU-Ready Gradient-Boosted Trees
May 18, 2026Aditya Tanna, Nassim Bouarour, Mohamed Bouadi +2Tabular Foundation Models Gradient-Boosted Decision Trees
Lexsi Labs
Post-Trained MoE Can Skip Half Experts via Self-Distillation
May 18, 2026Xingtai Lv, Li Sheng, Kaiyan Zhang +12Mixture of Experts Communication-Efficient Mixture-of-Experts
Tsinghua University · Shanghai AI Lab · WeChat AI +1
CommitDistill: A Lightweight Knowledge-Centric Memory Layer for Software Repositories
May 18, 2026Divya Chukkapalli, Thejesh Avula, Aditya Aggarwal +2Agent Memory Software Engineering Agents
Microsoft Corporation
HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents
May 18, 2026Woongyeng Yeo, Yumin Choi, Taekyung Ki +1Credit Assignment in RL LLM Agent Training
KAIST
Balancing Knowledge Distillation for Imbalance Learning with Bilevel Optimization
May 18, 2026Anh B. H. Nguyen, Ba Tho Phan, Viet Cuong TaClass-Imbalanced Learning Long-Tail Learning
VNU University of Engineering and Technology, Hanoi, Vietnam
Sometin Beta Pass Notin (SBPN): Improving Multilingual ASR for Nigerian Languages via Knowledge Distillation
May 18, 2026Sewade OgunLow-Resource Language Processing Low-Resource Speech Recognition
LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs
May 17, 2026Jihwan Kim, Nikhil Parthasarathy, Danfeng Qin +5Efficient VLM Inference Video-Language Models
1Google DeepMind · 2Seoul National University
Collaborative Learning for Semi-Supervised LiDAR Semantic Segmentation
May 16, 2026Bin Yang, Alexandru Paul Condurache3D Semantic Segmentation Semi-Supervised Learning
Bosch Research, Robert Bosch GmbH, Stuttgart, Germany · Institute for Neuro- and Bioinformatics, University of Lübeck, Lübeck, Germany
HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation
May 16, 2026Yihao Liang, Niraj K. JhaCross-Modal Knowledge Distillation Efficient VLM Inference
Princeton University
MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
May 16, 2026Jiarui Liu, Lechen Zhang, Yongjin Yang +5Knowledge Augmentation for Language Models Factual Knowledge in Language Models
Carnegie Mellon University · Jinesis Lab, University of Toronto & Vector Institute · University of Illinois Urbana-Champaign +6
Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation
May 16, 2026Anhao Zhao, Haoran Xin, Yingqi Fan +3Reasoning Distillation On-Policy Distillation
Eastern Institute of Technology, Ningbo · Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Jiao Tong University +1
GeoWorld-VLM: Geometry from World Models for Vision-Language Models
May 15, 2026Renjie Gu, Kaichen Zhou, Yan Luo +1Visual Spatial Reasoning VLM Adaptation
Harvard AI and Robotics Lab · Kempner Institute for the Study of Natural and Artificial Intelligence · Harvard University
Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation
May 15, 2026Jin Shi, Brady Zhang, Yishun LuVLM Distillation Efficient VLA Model Inference
Department of Mechanical Engineering University College London London, United Kingdom · Department of Engineering Science University of Oxford Oxford, United Kingdom
Asking Back: Interaction-Layer Antidistillation Watermarks
May 15, 2026Guang Yang, Amir Ghasemian, Fengchen Liu +3LLM Auditing LLM Watermarking
University of California, Los Angeles · Lawrence Berkeley National Laboratory · Meta
How to Choose Your Teacher for Fine Grained Image Recognition
May 15, 2026Oswin Gosal, Edwin Arkel Rios, Augusto Christian Surya +3Fine-Grained Image Classification Teacher-Student Learning
National Tsing Hua University, Taiwan · National Yang Ming Chiao Tung University, Taiwan
DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation
May 15, 2026Jaehun Jung, Hyunwoo Kim, Brandon Cui +4VLM Distillation VLM Reasoning
NVIDIA Research
DIPA: Distilled Preconditioned Algorithms for Solving Imaging Inverse Problems
May 14, 2026Romario Gualdrón-Hurtado, Roman Jacome, Leon Suarez +1Learning to Optimize Image Reconstruction
Department of Computer Science, Universidad Industrial de Santander, Bucaramanga, 680002, Colombia · Department of Electrical Engineering, Universidad Industrial de Santander, Bucaramanga, 680002, Colombia
Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models
May 14, 2026Senne Deproost, Denis Steckelmacher, Ann NowéReinforcement Learning Vector Quantization
Vrije Universiteit Brussel, Pleinlaan 2, 1050 Brussels, Belgium

May 20, 2026J/K move · Enter open · S save

PACD-Net: Pseudo-Augmented Contrastive Distillation for Glycemic Control Estimation from SMBG

Canyu Lei, David Repaske, Jianxin Xie

University of Virginia, School of Data Science, Charlottesville, VA 22903, USA · University of Virginia, Department of Pediatrics, Charlottesville, VA 22903, USA

Irregular Time-Series Modeling Contrastive Learning Self-Supervised Time Series Representation Learning Knowledge Distillation

Open paper AskPDF arXiv

Effective diabetes management requires continuous monitoring of glycemic levels. Clinically, glycemic control is assessed using metrics such as Time in Range (TIR), Time Below Range (TBR), and Time Above Range (TAR), typically derived from continuous glucose monitoring (CGM). However, many patients rely on self-monitoring of blood glucose (SMBG) due to the high cost and limited accessibility of CGM. Unlike CGM, SMBG provides sparse and irregular measurements, making accurate estimation of these metrics challenging. Conventional supervised learning approaches struggle under such sparsity, leading to poor generalization and unstable performance. To address this, we propose PACD-Net, a self-supervised contrastive knowledge distillation framework for estimating glycemic control from SMBG. Pseudo-SMBG samples with richer temporal coverage are used as teacher signals to guide learning from sparse observations. In addition, multi-view contrastive learning enforces representation consistency across diverse sampling patterns. The model adopts a hybrid Swin Transformer-CNN backbone to capture temporal dependencies in sparse SMBG sequences. Experimental results demonstrate that PACD-Net consistently outperforms existing methods in estimating TAR, TIR, and TBR from real-world SMBG data, achieving improved accuracy as well as enhanced stability and generalization under extremely sparse observation settings. The proposed framework provides a practical tool for clinical SMBG interpretation and offers a generalizable approach for learning from sparse and irregularly sampled sensor data in broader applications.