Adversarial Robustness Distillation

Adversarial robustness distillation aims to transfer the robustness of a large, adversarially trained "teacher" model to a smaller, more efficient "student" model, mitigating the computational cost and size limitations of direct adversarial training. Current research focuses on improving knowledge transfer techniques, exploring diverse teacher ensembles (including heterogeneous architectures), and addressing issues like fairness and the accuracy-robustness trade-off through methods such as soft label distillation and gradient matching. This approach holds significant promise for deploying robust deep learning models in resource-constrained environments and applications where adversarial attacks pose a critical risk, such as autonomous driving or medical diagnosis.

Papers

May 22, 2024

Adversarial Training via Adaptive Knowledge Amalgamation of an Ensemble of Teachers
Shayan Mohajer Hamidi, Linfeng Ye
Adversarial Attack Adversarial Training Diverse Ensemble Adversarial Sample Good Teacher Robust Deep Adversarial Robustness Distillation Knowledge Amalgamation

February 23, 2024

Distilling Adversarial Robustness Using Heterogeneous Teachers
Jieren Deng, Aaron Palmer, Rigel Mahmood, Ethan Rathbun, Jinbo Bi, Kaleel Mahmood, Derek Aguiar
Adversarial Attack Adversarial Example Adversarial Training Adversarial Robustness Adversarial Robustness Distillation

December 9, 2023

Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation
Shiji Zhao, Ranjie Duan, Xizhe Wang, Xingxing Wei
Adversarial Training Adversarial Robustness Fair Adversarial Soft Label Distillation Adversarial Robustness Distillation

December 6, 2023

Indirect Gradient Matching for Adversarial Robust Distillation
Hongsin Lee, Seungju Cho, Changick Kim
Adversarial Training Adversarial Robustness Adversarial Distillation Gradient Matching Adversarial Robustness Distillation Parallel Self Distillation

June 28, 2023

Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation
Shiji Zhao, Xizhe Wang, Xingxing Wei
Adversarial Attack Adversarial Training Accuracy Robustness Multi Teacher Distillation Adversarial Robustness Distillation

March 21, 2023

Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation
Yuzheng Wang, Zhaoyu Chen, Dingkang Yang, Pinxue Guo, Kaixun Jiang, Wenqiang Zhang, Lizhe Qi
Adversarial Training Adversarial Robustness Robust Model Adversarial Robustness Distillation Thin Air

February 17, 2023

Adversarial Contrastive Distillation with Adaptive Denoising
Yuzheng Wang, Zhaoyu Chen, Dingkang Yang, Yang Liu, Siao Liu, Wenqiang Zhang, Lizhe Qi
Contrastive Distillation General Adversarial Adversarial Robustness Distillation Parameter Aware Robustness

November 1, 2022

ARDIR: Improving Robustness using Knowledge Distillation of Internal Representation
Tomokatsu Takahashi, Masanori Yamada, Yuuki Yamanaka, Tomoya Yamashita
Native Robustness Knowledge Distillation Adversarial Example Adversarial Training Internal Representation Adversarial Robustness Distillation