Feature-Based Knowledge Distillation

Ask about this topic

Overview

Feature-based knowledge distillation trains a compact student to learn from a teacher’s internal representations, adding information beyond the teacher’s final predictions. Common approaches align intermediate activations or attention maps, or transfer relationships between examples, selected tokens, channels, or representation subspaces; some combine these signals with logit matching. The methods are used in settings including image recognition and segmentation, object detection, language and speech models, graph learning, and robotics, where they can reduce inference costs or transfer knowledge across domains and model types. Results depend strongly on student capacity, architecture, and task: extra feature supervision can improve data-scarce or cross-domain learning, but can also overfit or conflict with other objectives, so its benefits require careful, compute-aware comparisons.

Latest papers

CardsList

49 papers18 papers in the last four weeks, up 350% on the four weeks before. 0.1% of all new papers.

  1. Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10

    May 29, 2026Umut Onur YasarTeacher-Student LearningFeature-Based Knowledge Distillation

  2. Generative Diffusion Prior Distillation for Long-Context Knowledge Transfer

    May 12, 2026Nilushika Udayangani, Kishor Nandakishor, Marimuthu PalaniswamiTime Series ClassificationFeature-Based Knowledge Distillation

    Department of Electrical and Electronic Engineering University of Melbourne Parkville, VIC 3052, Australia

  3. Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representations

    May 8, 2026Yuan Zhang, Chenyi Li, Guoqing Ma +7Diffusion Model DistillationDiffusion Sampling

    JD Explore Academy, China · The Hong Kong University of Science and Technology · Tsinghua University +1

  4. The Surprising Effectiveness of Canonical Knowledge Distillation for Semantic Segmentation

    Apr 28, 2026Muhammad Ali, Kevin Alexander Laube, Madan Ravi Ganesh +3Image SegmentationFeature-Based Knowledge Distillation

    University of Freiburg · Bosch Center for Artificial Intelligence · Aleph Alpha Research +1

  5. Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation

    Apr 10, 2026Yutong Zhang, Jiaxin Chen, Honglin Chen +5Memory-Efficient Fine-TuningEfficient Neural Network Inference

    State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China · School of Computer Science and Engineering, Beihang University, China · College of Information Technology, United Arab Emirates University, United Arab Emirates

  6. Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization

    Apr 3, 2026Zihe Liu, Yulong Mao, Jinan Xu +2Language Model DistillationKnowledge Distillation

    Key Laboratory of Big Data & Artificial Intelligence in Transportation · School of Computer Science and Technology, Beijing Jiaotong University

  7. MEDiC: Multi-objective Exploration of Distillation from CLIP

    Mar 30, 2026Konstantinos Georgiou, Maofeng Tang, Hairong QiMasked Image ModelingVLM Distillation

    Min H. Kao Department of Electrical Engineering and Computer Science The University of Tennessee Knoxville, TN 37996

  8. Distilling Genomic Models for Efficient mRNA Representation Learning via Embedding Matching

    Mar 27, 2026Rasched Haidari, Sam Martin, Maxime AllardKnowledge DistillationMolecular Representation Learning

    Helical London, UK

  9. Learnable Instance Attention Filtering for Adaptive Detector Distillation

    Mar 27, 2026Chen Liu, Qizhen Lan, Zhicheng Ding +2Knowledge DistillationFeature-Based Knowledge Distillation

    Dept. of CS, University of Alabama at Birmingham · Dept. of CS, Bowling Green State University

  10. Forecasting with Guidance: Representation-Level Supervision for Time Series Forecasting

    Mar 25, 2026Jiacheng Wang, Liang Fan, Baihua Li +1Time Series ForecastingTime Series Representation Learning

    Xijing University · Loughborough University · Northeastern University

  11. Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement

    Mar 25, 2026Xin Zhang, Jianyang Xu, Hao Peng +5Knowledge DistillationMulti-Teacher Knowledge Distillation

    Hangzhou Dianzi University

  12. Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

    Mar 21, 2026Kai Li, Shiyu ZhaoVision-Based Robot ControlMobile Robotics

    College of Computer Science and Technology at Zhejiang University, Hangzhou, China. · School of Engineering at Westlake University, Hangzhou, China.

  13. Collaborative Adaptive Curriculum for Progressive Knowledge Distillation

    Mar 19, 2026Jing Liu, Zhenchao Ma, Han Yu +5Federated Knowledge DistillationKnowledge Distillation

    Fudan University · The University of British Columbia · Duke Kunshan University +2

  14. CD-FKD: Cross-Domain Feature Knowledge Distillation for Robust Single-Domain Generalization in Object Detection

    Mar 17, 2026Junseok Lee, Sungho Shin, Seongju Lee +1Object DetectionCross-Domain Generalization

    Junseok Lee is with the Advanced Robotics Lab, LG Electronics, Seoul, Republic of Korea. · Sungho Shin is with the Place AI Flatform, Naver, Gyeonggi-do, Republic of Korea · Seongju Lee and Kyoobin Lee are with the Department of AI Convergence, Gwangju Institute of Science and Technology (GIST), Gwangju, Republic of Korea.

  15. DAIT: Distillation from Vision-Language Models to Lightweight Classifiers with Adaptive Intermediate Teacher Transfer

    Mar 16, 2026Zhengxu He, Jun Li, Zhijian WuVLM DistillationFine-Grained Image Classification

    School of Computer and Electronic Information, Nanjing Normal University, Nanjing, China · Medical Artificial Intelligence Lab, Westlake University

  16. SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning

    Mar 9, 2026Kaushik Roy, Giovanni D'urso, Nicholas Lawrance +2Imitation LearningRobot Imitation Learning

    CSIRO Robotics, DATA61, CSIRO, Australia. · Queensland University of Technology (QUT), Brisbane, Australia.

  17. Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation

    Mar 3, 2026Chonghua Lv, Dong Zhao, Shuang Wang +4Domain Generalization for Semantic SegmentationKnowledge Distillation

    School of Artificial Intelligence, Xidian University, China · Department of Information Engineering and Computer Science, University of Trento, Italy · Department of Automation, Tsinghua University, China +1

  18. RestoRect: Degraded Image Restoration via Latent Rectified Flow & Feature Distillation

    Sep 27, 2025Shourya Verma, Mengbo Wang, Nadia Atallah Lanman +1Image RestorationFeature-Based Knowledge Distillation

    Department of Computer Science, Purdue University, USA · Purdue Institute of Cancer Research, USA

  19. Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework

    Mar 2, 2024Junxian Li, Bin Shi, Erfei Cui +2Graph Positional EncodingKnowledge Distillation

    Xi’an Jiaotong University · Arizona State University, Tempe, USA · Tongji University, Shanghai, China