Adversarial Distillation

Adversarial distillation is a machine learning technique that enhances the robustness and accuracy of smaller "student" models by transferring knowledge from larger, more robust "teacher" models, often trained with adversarial examples. Current research focuses on improving knowledge transfer efficiency through methods like dynamic guidance, feature-level distillation, and adversarial training incorporating dynamic labels or gradient matching. This approach is significant for improving model performance in resource-constrained environments and for bolstering defenses against adversarial attacks, with applications ranging from image classification and natural language processing to medical image analysis and 3D generation.

Papers

October 30, 2024

Multi-student Diffusion Distillation for Better One-step Generators
Yanke Song, Jonathan Lorraine, Weili Nie, Karsten Kreis, James Lucas
Good Better Adversarial Distillation Multi Teacher Distillation Diffusion Distillation

October 19, 2024

Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step
Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, Hai Huang
Good Teacher Cross Over Step Adversarial Loss Score Distillation Adversarial Distillation

September 12, 2024

DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learning
Kangyang Luo, Shuai Wang, Yexuan Fu, Renrong Shao, Xiang Li, Yunshi Lan, Ming Gao, Jinlong Shu
Adversarial Distillation

September 3, 2024

Dynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledge
Hyejin Park, Dongbo Min
Adversarial Input Adversarial Scenario Adversarial Threat Adversarial Distillation

August 23, 2024

Dynamic Label Adversarial Training for Deep Learning Robustness Against Adversarial Attacks
Zhenyu Liu, Haoran Duan, Huizhi Liang, Yang Long, Vaclav Snasel, Guiseppe Nicosia, Rajiv Ranjan, Varun Ojha
Adversarial Attack Adversarial Example Adversarial Training Model Robustness Neural Network Robustness Adversarial Distillation Adversarial Discriminator

April 18, 2024

Data-free Knowledge Distillation for Fine-grained Visual Categorization
Renrong Shao, Wei Zhang, Jianhua Yin, Jun Wang
Fine Grained Visual Data Free Knowledge Distillation Fine Grained Category Adversarial Distillation Attention Distillation

March 11, 2024

PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor
Jaewon Jung, Hongsun Jang, Jaeyong Song, Jinho Lee
Adversarial Example Adversarial Robustness Peer Agent Adversarial Distillation Peer Learning

February 21, 2024

SDXL-Lightning: Progressive Adversarial Diffusion Distillation
Shanchuan Lin, Anran Wang, Xiao Yang
Text to Image Generation Stable Diffusion Stable Diffusion Model Adversarial Distillation Diffusion Distillation Adversarial Diffusion Distillation

December 11, 2023

CAD: Photorealistic 3D Generation via Adversarial Distillation
Ziyu Wan, Despoina Paschalidou, Ian Huang, Hongyu Liu, Bokui Shen, Xiaoyu Xiang, Jing Liao, Leonidas Guibas
Pre Trained Diffusion Model Diffusion Prior Adversarial Distillation 3D Synthesis

December 6, 2023

Indirect Gradient Matching for Adversarial Robust Distillation
Hongsin Lee, Seungju Cho, Changick Kim
Adversarial Training Adversarial Robustness Adversarial Distillation Gradient Matching Adversarial Robustness Distillation Parallel Self Distillation

November 2, 2023

Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models
Andy Zhou, Jindong Wang, Yu-Xiong Wang, Haohan Wang
Knowledge Distillation Data Augmentation Adversarial Example Adversarial Distillation Distribution Robustness

August 15, 2023

SEDA: Self-Ensembling ViT with Defensive Distillation and Adversarial Training for robust Chest X-rays Classification
Raza Imam, Ibrahim Almakky, Salma Alrashdi, Baketah Alrashdi, Mohammad Yaqub
Vision Transformer Adversarial Example Adversarial Training Adversarial Perturbation Chest X Ray Adversarial Distillation Defensive Distillation

June 25, 2023

Feature Adversarial Distillation for Point Cloud Classification
YuXing Lee, Wei Wu
Point Cloud Classification Adversarial Loss Adversarial Distillation LiDAR Distillation

June 13, 2023

DHBE: Data-free Holistic Backdoor Erasing in Deep Neural Networks via Restricted Adversarial Distillation
Zhicong Yan, Shenghong Li, Ruijie Zhao, Yuan Tian, Yuanyuan Zhao
Deep Neural Network Backdoor Attack Backdoor Learning Adversarial Distillation Backdoor Removal

June 7, 2023

Faithful Knowledge Distillation
Tom A. Lamb, Rudy Brunel, Krishnamurthy DJ Dvijotham, M. Pawan Kumar, Philip H. S. Torr, Francisco Eiras
Knowledge Distillation Adversarial Robustness Adversarial Distillation Robust Imitation

May 22, 2023

Lion: Adversarial Distillation of Proprietary Large Language Models
Yuxin Jiang, Chunkit Chan, Mingyang Chen, Wei Wang
Knowledge Transfer Adversarial Distillation Proprietary Large Language Model

November 28, 2022

Lightning Fast Video Anomaly Detection via Adversarial Knowledge Distillation
Nicolae-Catalin Ristea, Florinel-Alin Croitoru, Dana Dascalescu, Radu Tudor Ionescu, Fahad Shahbaz Khan, Mubarak Shah
Anomaly Detection Video Anomaly Anomaly Map Adversarial Distillation Adversarial Discriminator Adversarial Knowledge Distillation

November 2, 2022

LightVessel: Exploring Lightweight Coronary Artery Vessel Segmentation via Similarity Knowledge Distillation
Hao Dang, Yuekai Zhang, Xingqun Qi, Wanting Zhou, Muyi Sun
Knowledge Distillation Adversarial Distillation Coronary Artery Segmentation

June 5, 2022

Vanilla Feature Distillation for Improving the Accuracy-Robustness Trade-Off in Adversarial Training
Guodong Cao, Zhibo Wang, Xiaowei Dong, Zhifei Zhang, Hengchang Guo, Zhan Qin, Kui Ren
Knowledge Distillation Adversarial Example Adversarial Training Adversarial Distillation Accuracy Robustness Adversarial Training Algorithm

March 31, 2022

Adversarial Speaker Distillation for Countermeasure Model on Automatic Speaker Verification
Yen-Lun Liao, Xuanjun Chen, Chung-Che Wang, Jyh-Shing Roger Jang
Knowledge Distillation Adversarial Fine Tuning Adversarial Distillation Adaptive Electronic Counter Counter Measure