Attention Distillation

Attention distillation is a machine learning technique focused on transferring knowledge from a larger, more complex "teacher" model to a smaller, more efficient "student" model by focusing on the attention mechanisms within the teacher. Current research emphasizes applications across diverse areas, including image classification, object detection, semantic segmentation, and even graph neural networks, often employing vision transformers (ViTs) and convolutional neural networks (CNNs). This approach is significant for model compression, improving performance on resource-constrained devices, enhancing robustness (e.g., against backdoor attacks), and addressing challenges in continual learning and anomaly detection.

Papers

July 29, 2024

MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity
Kanghyun Choi, Hye Yoon Lee, Dain Kwon, SunJong Park, Kyuyeun Kim, Noseong Park, Jinho Lee
Vision Transformer Quantization Operator Low Bit Multi Head Attention Mechanism Data Free Quantization Attention Distillation Vision Transformer Quantization

July 19, 2024

An Attention-based Representation Distillation Baseline for Multi-Label Continual Learning
Martin Menabue, Emanuele Frascaroli, Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Simone Calderara
Continual LEArning Catastrophic Forgetting Continual Learning Multi Label Attention Distillation

June 7, 2024

Attention Fusion Reverse Distillation for Multi-Lighting Image Anomaly Detection
Yiheng Zhang, Yunkang Cao, Tianhang Zhang, Weiming Shen
Anomaly Detection Attention Module Image Anomaly Detection Attention Distillation

May 10, 2024

Attend, Distill, Detect: Attention-aware Entropy Distillation for Anomaly Detection
Sushovan Jena, Vishwas Saini, Ujjwal Shaw, Pavitra Jain, Abhay Singh Raihal, Anoushka Banerjee, Sharad Joshi, Ananth Ganesh, Arnav Bhavsar
Anomaly Detection Unsupervised Anomaly Detection Convolutional Block Attention Multi Class Anomaly Detection Attention Distillation Class Interference

April 18, 2024

Data-free Knowledge Distillation for Fine-grained Visual Categorization
Renrong Shao, Wei Zhang, Jianhua Yin, Jun Wang
Fine Grained Visual Data Free Knowledge Distillation Fine Grained Category Adversarial Distillation Attention Distillation

March 10, 2024

Attention is all you need for boosting graph convolutional neural network
Yinwei Wu
Convolutional Neural Network Knowledge Graph Knowledge Distillation Graph Drawing Human Attention Graph Convolutional Neural Network Attention Distillation

March 8, 2024

Attention-guided Feature Distillation for Semantic Segmentation
Amir M. Mansourian, Arya Jalali, Rozhan Ahmadi, Shohreh Kasaei
Semantic Segmentation Feature Map Convolutional Block Attention Dense Prediction Task Attention Distillation

February 19, 2024

Unveiling the Magic: Investigating Attention Distillation in Retrieval-augmented Generation
Zizhong Li, Haopeng Zhang, Jiawei Zhang
Retrieval Augmented Generation Retrieval Augmented Retrieval Augmented Language Model LangId Magic Spell Attention Distillation

December 14, 2023

Generative Model-based Feature Knowledge Distillation for Action Recognition
Guiqin Wang, Peng Zhao, Yanjiang Shi, Cong Zhao, Shusen Yang
Knowledge Distillation Action Recognition Attention Distillation Latent Distillation

September 26, 2023

ADU-Depth: Attention-based Distillation with Uncertainty Modeling for Depth Estimation
Zizhang Wu, Zhuozheng Li, Zhi-Gang Fan, Yunzhe Wu, Xiaoquan Wang, Rui Tang, Jian Pu
Depth Estimation Monocular Depth Estimation Uncertainty Modeling Attention Distillation Depth Datasets Student Network

June 2, 2023

Group channel pruning and spatial attention distilling for object detection
Yun Chu, Pu Li, Yong Bai, Zhuhua Hu, Yongqing Chen, Jiafeng Lu
Object Detection Model Compression Channel Pruning Dynamic Sparse Training Attention Distillation

April 4, 2023

Label-guided Attention Distillation for Lane Segmentation
Zhikang Liu, Lanyun Zhu
Segmentation Network Lane Detection Attention Distillation Lane Annotation

March 1, 2023

Distilled Reverse Attention Network for Open-world Compositional Zero-Shot Learning
Yun Li, Zhe Liu, Saurav Jha, Sally Cripps, Lina Yao
Knowledge Distillation Attention Distillation Open Set Scenario Optimal Transport Distillation

October 16, 2022

Federated Learning with Privacy-Preserving Ensemble Attention Distillation
Xuan Gong, Liangchen Song, Rishi Vedula, Abhishek Sharma, Meng Zheng, Benjamin Planche, Arun Innanje, Terrence Chen, Junsong Yuan, David Doermann, Ziyan Wu
PRIvacy Leakage Privacy Preserving Federated Learning Privacy Mechanism Attention Distillation

October 3, 2022

Attention Distillation: self-supervised vision transformer students need more guidance
Kai Wang, Fei Yang, Joost van de Weijer
Knowledge Distillation Self Supervised Human Guidance Self Supervised Vision Transformer Pre Trained ViT Attention Distillation

April 21, 2022

Eliminating Backdoor Triggers for Deep Neural Networks Using Attention Relation Graph Distillation
Jun Xia, Ting Wang, Jiepin Ding, Xian Wei, Mingsong Chen
Deep Neural Network Backdoor Attack Backdoor Defense Backdoor Trigger Attention Distillation

February 1, 2022

Continual Attentive Fusion for Incremental Learning in Semantic Segmentation
Guanglei Yang, Enrico Fini, Dan Xu, Paolo Rota, Mingli Ding, Hao Tang, Xavier Alameda-Pineda, Elisa Ricci
Semantic Segmentation Incremental Learning Attention Based Attention Distillation Pixel Wise Dense Prediction

December 7, 2021

ADD: Frequency Attention and Multi-View based Knowledge Distillation to Detect Low-Quality Compressed Deepfake Images
Binh M. Le, Simon S. Woo
Knowledge Distillation Low Quality Deepfake Deepfake Image Attention Distillation Frequency Attention Deep Forgery Detection

Attention Distillation

Papers

MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity

An Attention-based Representation Distillation Baseline for Multi-Label Continual Learning

Attention Fusion Reverse Distillation for Multi-Lighting Image Anomaly Detection

Attend, Distill, Detect: Attention-aware Entropy Distillation for Anomaly Detection

Data-free Knowledge Distillation for Fine-grained Visual Categorization

Attention is all you need for boosting graph convolutional neural network

Attention-guided Feature Distillation for Semantic Segmentation

Unveiling the Magic: Investigating Attention Distillation in Retrieval-augmented Generation

Generative Model-based Feature Knowledge Distillation for Action Recognition

ADU-Depth: Attention-based Distillation with Uncertainty Modeling for Depth Estimation

Group channel pruning and spatial attention distilling for object detection

Label-guided Attention Distillation for Lane Segmentation

Distilled Reverse Attention Network for Open-world Compositional Zero-Shot Learning

Federated Learning with Privacy-Preserving Ensemble Attention Distillation

Attention Distillation: self-supervised vision transformer students need more guidance

Eliminating Backdoor Triggers for Deep Neural Networks Using Attention Relation Graph Distillation

Continual Attentive Fusion for Incremental Learning in Semantic Segmentation

ADD: Frequency Attention and Multi-View based Knowledge Distillation to Detect Low-Quality Compressed Deepfake Images