CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation
Authors: Zhi Li, Haowei Liu, Hongchen Yang, Xiaoxuan Wang, Song Gao, Zhenli He, Wei Zhou, Shaowen Yao
Organizations: Engineering Research Center of Cyberspace and School of Software and AI, Yunnan University, Kunming, China · School of Information Science and Technology, Yunnan Normal University, Kunming, China
Adversarial robustness distillation typically treats a robust teacher as a fixed knowledge source and transfers robustness to a compact student in a one-way manner. This paradigm terminates teacher-student interaction once the student is distilled, thereby discarding potentially useful feedback from the distilled student. However, exploiting such feedback is nontrivial because indiscriminate weak-to-strong transfer may propagate student errors to the teacher. To address this issue, we propose Cyclic Adversarial Robustness Distillation (CARD), a closed-loop framework that turns the distilled student into a feedback source for refining the robust teacher and subsequently transfers the teacher gains back to the same student. CARD first performs strong-to-weak adversarial distillation and then uses reliability-enhanced student feedback to refine the robust teacher. The refinement combines inverse adversarial student feedback with teacher-student discrepancy maximization, after which the refined teacher is used to continue training the same student under the unchanged base distillation objective. We further establish a sample-level sufficient condition for reliable weak-to-strong feedback transfer and derive a distribution-level upper bound on the refined teacher's risk. Experiments on CIFAR-10, CIFAR-100, and TinyImageNet-10 across multiple teacher-student architectures and base distillation methods show that CARD improves adversarial robustness in most evaluated settings while generally preserving clean accuracy. These results show that the distilled student can provide useful feedback to the teacher, enabling robustness transfer to proceed beyond the conventional one-way teacher-to-student paradigm.
Figures & tables
Figure 1: From one-way to closed-loop adversarial robustness distillation. Conventional adversarial distillation treats the robust teacher as a fixed knowledge source and transfers robustness only from teacher to student. CARD closes the teacher-student loop by using the distilled student to refine the robust teacher and then using the refined teacher to continue training the same student. Snowflake and flame symbols denote frozen and trainable networks, respectively.
Figure 2: Overview of CARD. Stage 1 performs strong-to-weak adversarial distillation from a pretrained robust teacher to a compact student. Stage 2 freezes the first-round student and uses inverse adversarial examples and teacher adversarial examples to refine the pretrained teacher. Stage 3 replaces the initial teacher with the refined teacher and continues training from the first-round student checkpoint, transferring the teacher improvement back to the student.
Type
Architecture
Method
CIFAR-10
Clean
FGSM
PGD-20
CW-20
AA
Teacher
WRN-28-10
ST [ 26 ]
84.57
62.17
57.67
54.99
53.77
Student
ResNet-18
PGD-AT [ 28 ]
82.50
57.52
51.53
50.57
48.04
TRADES [ 39 ]
82.17
58.20
52.21
50.48
49.18
ST [ 26 ]
83.43
59.91
53.72
51.66
50.01
ARD [ 13 ]
83.43
59.27
53.04
51.20
49.49
Table 1: Clean and white-box robust accuracy (%) on CIFAR-10 with WRN-28-10 as the teacher and different student architectures. Within each student-architecture group, the best and second-best results in each column are highlighted in bold and underlined, respectively.
Type
Architecture
Method
CIFAR-10
Clean
FGSM
PGD-20
CW-20
AA
Teacher
WRN-34-10
ST [ 26 ]
84.75
62.40
57.83
55.49
54.33
Student
ResNet-18
PGD-AT [ 28 ]
82.50
57.52
51.53
50.57
48.04
TRADES [ 39 ]
82.17
58.20
52.21
50.48
49.18
ST [ 26 ]
83.43
59.91
53.72
51.66
50.01
ARD [ 13 ]
83.29
59.32
53.65
51.63
50.01
Table 2: Clean and white-box robust accuracy (%) on CIFAR-10 with WRN-34-10 as the teacher and different student architectures. Within each student-architecture group, the best and second-best results in each column are highlighted in bold and underlined, respectively.
Type
Architecture
Method
CIFAR-100
Clean
FGSM
PGD-20
CW-20
AA
Teacher
WRN-34-10
ST [ 26 ]
59.40
35.92
33.71
30.30
29.19
Student
ResNet-18
PGD-AT [ 28 ]
57.52
32.12
28.50
27.12
24.68
TRADES [ 39 ]
56.32
29.83
26.95
23.77
22.69
ST [ 26 ]
55.75
33.00
30.92
26.89
26.04
ARD [ 13 ]
58.44
34.92
31.54
28.47
26.42
Table 3: Clean and white-box robust accuracy (%) on CIFAR-100 with WRN-34-10 as the teacher and different student architectures. Within each student-architecture group, the best and second-best results in each column are highlighted in bold and underlined, respectively.
Type
Architecture
Method
TinyImageNet-10
Clean
FGSM
PGD-20
CW-20
AA
Teacher
WRN-34-10
ST [ 26 ]
77.80
62.00
60.80
60.00
60.00
Student
ResNet-18
PGD-AT [ 28 ]
79.60
61.00
57.20
57.20
55.20
TRADES [ 39 ]
77.80
58.80
56.80
55.60
55.40
ST [ 26 ]
76.60
62.20
61.00
59.00
58.20
ARD [ 13 ]
79.60
60.40
58.60
57.00
56.40
Table 4: Clean and white-box robust accuracy (%) on TinyImageNet-10 with WRN-34-10 as the teacher and ResNet-18 as the student. The best and second-best student results in each column are highlighted in bold and underlined, respectively.
Method
ResNet-18
VGG-19
MobileNet-V2
PGD-100
CW-100
PGD-100
CW-100
PGD-100
CW-100
PGD-AT [ 28 ]
33.69
33.41
41.64
42.03
38.36
38.27
TRADES [ 39 ]
35.71
36.49
40.75
41.68
37.55
37.91
ST [ 26 ]
37.67
38.17
42.00
42.68
39.13
39.65
ARD [ 13 ]
39.32
39.31
44.19
44.24
40.73
40.68
RSLAD [ 43 ]
39.18
39.72
43.23
43.86
40.58
40.81
Table 5: Transfer-based black-box accuracy (%) on CIFAR-100 with WRN-34-10 as the teacher and ResNet-18 as the target student. PGD-100 and CW-100 adversarial examples are generated on independently trained robust ResNet-18, VGG-19, and MobileNet-V2 surrogate models. The best and second-best results in each column are shown in bold and underlined, respectively.
Method
ResNet-18
MobileNet-V2
VGG-19
PGD-100
CW-100
PGD-100
CW-100
PGD-100
CW-100
PGD-AT [ 28 ]
56.29
56.31
56.30
56.32
56.49
56.53
TRADES [ 39 ]
54.83
54.96
54.80
54.97
55.12
55.28
ST [ 26 ]
54.43
54.54
54.35
54.31
54.50
54.68
ARD [ 13 ]
56.99
57.20
57.12
57.04
57.38
57.15
RSLAD [ 43 ]
55.16
55.24
55.14
55.16
55.46
55.50
Table 6: Transfer-based black-box accuracy (%) on CIFAR-100 with WRN-34-10 as the teacher and ResNet-18 as the target student. PGD-100 and CW-100 adversarial examples are generated on independently trained natural ResNet-18, MobileNet-V2, and VGG-19 surrogate models. The best and second-best results in each column are shown in bold and underlined, respectively.
Method
ResNet-50
DenseNet-169
WRN-34-10
PGD-100
CW-100
PGD-100
CW-100
PGD-100
CW-100
PGD-AT [ 28 ]
55.99
55.92
56.26
56.26
57.03
57.07
TRADES [ 39 ]
54.75
54.50
54.68
54.89
55.54
55.64
ST [ 26 ]
54.13
54.32
54.48
54.42
54.94
54.99
ARD [ 13 ]
56.85
56.93
57.18
56.96
57.53
57.60
RSLAD [ 43 ]
54.74
54.97
55.04
55.25
55.71
55.75
Table 7: Transfer-based black-box accuracy (%) on CIFAR-100 with WRN-34-10 as the teacher and ResNet-18 as the target student. PGD-100 and CW-100 adversarial examples are generated on independently trained natural ResNet-50, DenseNet-169, and WRN-34-10 surrogate models. The best and second-best results in each column are shown in bold and underlined, respectively.
Variant
W2S
S2W
Clean
CW-20
AA
AdaAD
57.95
27.00
25.55
S2W Only
✓
57.09
27.55
26.15
CARD
✓
✓
58.24
28.11
26.40
Table 8: Component ablation of CARD on CIFAR-100 with WRN-34-10 as the teacher and MobileNet-V2 as the student. W2S and S2W denote weak-to-strong teacher refinement and strong-to-weak gain transfer, respectively.
Method
Clean
PGD-20
CW-20
AA
AdaAD
57.95
30.92
27.00
25.55
Joint
56.93
31.23
28.00
26.30
Separated (Ours)
58.24
31.30
28.11
26.40
Table 9: Comparison of joint and separated optimization on CIFAR-100 with WRN-34-10 as the teacher and MobileNet-V2 as the student. Both variants contain the same W2S teacher-refinement and S2W gain-transfer components.
Refinement Signal
Clean
PGD-20
CW-20
AA
AdaAD
58.91
32.57
29.00
27.80
w/o Refinement
58.48
32.79
29.32
27.92
GT Only
61.85
26.30
26.09
23.64
Teacher Clean + GT
61.75
29.67
28.14
26.49
Student Clean
58.76
32.81
29.22
27.96
Student Inverse
60.98
32.51
29.29
27.73
Table 10: Comparison of teacher-refinement supervision signals on CIFAR-100 with WRN-34-10 as the teacher and ResNet-18 as the student. The reported results correspond to the student obtained after gain transfer from the resulting teacher. GT denotes ground-truth hard-label supervision.
Figure 3: Sensitivity of CARD to the refinement weight λ on CIFAR-10 with WRN-28-10 as the teacher and ResNet-18 as the student. Blue and red curves denote natural and robust accuracy, respectively. Panels (a) and (c) report the refined teacher after the first and second refinement cycles, while panels (b) and (d) report the corresponding students after the subsequent gain-transfer stage.
Figure 4: Grad-CAM visualizations on adversarial CIFAR-100 inputs. The leftmost column shows the corresponding adversarial images, the middle columns show activation maps of different student methods, and the rightmost column shows those of the robust teacher. In the visualized examples, the CARD student exhibits more concentrated activation on object-relevant regions and greater qualitative similarity to the robust teacher than the baseline students.
Figure 5: Normalized confusion matrices on inverse adversarial, clean, and adversarial CIFAR-10 inputs with WRN-28-10 as the teacher and ResNet-18 as the student. The inverse adversarial and adversarial inputs are generated using 10-step iterative optimization. Columns correspond to the three input types, whereas rows show the initial teacher, AdaAD student, refined teacher, and final CARD student. The strong diagonal concentration on inverse adversarial inputs indicates that these inputs provide highly reliable class-consistent predictions. Teacher refinement improves clean and adversarial accuracy, and part of these gains is subsequently transferred to the final student.
Adversarial distillation transfers robustness from high-capacity teachers to compact students. Existing adversarial distillation methods mainly use teacher predictions on clean or adversarial examples to supervise student learning. However, teacher-favorable supervision within the perturbation neighborhood remains underexplored in adversarial distillation. We therefore propose Collaboratively Guided Adversarial Robust Distillation (CGARD), which jointly optimizes distinct student-adversarial and teacher-collaborative examples within the same perturbation neighborhood. The teacher-collaborative example is constrained to incur no greater cross-entropy loss under the teacher than the clean input. CGARD combines collaborative teacher guidance with adversarial teacher supervision to improve robust knowledge transfer. Experiments on CIFAR-10 and CIFAR-100, including white-box evaluation and additional black-box transfer evaluation, demonstrate consistent robustness improvements over strong adversarial distillation baselines.
Zhi Li, Haowei Liu, Hongchen Yang +4
Engineering Research Center of Cyberspace and School of Software and AI, Yunnan University, Kunming, China · School of Information Science and Technology, Yunnan Normal University, Kunming, China
Adversarial Distillation aims to enhance student robustness by guiding the student with a robust teacher's soft labels within the min-max adversarial training framework, yet its success is notoriously inconsistent: a more robust teacher often fails to improve, or even harms, the student's robust generalization. In this paper, we identify a key mechanism of this teacher dependency: the misalignment between the teacher's supervisory confidence and the student's representational limitations on a consistent subset of training data -- the Robustly Unlearnable Set. We present a theoretical framework analyzing the feature learning dynamics of a two-layer neural network, demonstrating that this mismatch creates a dichotomy in distillation outcomes. We prove that when a teacher provides confident supervision on unlearnable samples, it compels the student to memorize spurious noise patterns that eventually overpower the learned robust signal, thereby driving robust overfitting. Conversely, a teacher that exhibits high uncertainty on these samples effectively suppresses noise memorization, allowing the student to rely solely on the learnable signal for robust generalization. We empirically validate our theory across both synthetic simulations and real-image classification datasets, confirming that robust overfitting is driven by the teacher's interaction with unlearnable samples. Finally, we demonstrate that a teacher's predictive entropy on unlearnable samples serves as a strong indicator of student robustness, validating our theoretical framework and offering a principled guideline for robust teacher selection.
Hongsin Lee, Hye Won Chung
School of Electrical Engineering, KAIST, Daejeon, Korea.
Deep neural networks (DNNs) have achieved remarkable success in classical machine learning problems. However, they are known to be vulnerable to adversarial attacks. Countermeasures proposed in the literature, notably Information Bottleneck Distillation (IBD) introduced by Kuang et al., degrade the classification accuracy on clean inputs while improving the robustness to adversarial inputs. In this work, we extend the IBD framework by introducing an extra teacher model (clean teacher) trained with only clean inputs, into the distillation process from a robust teacher model trained by adversarial training. The features of both clean and robust teachers are transferred to the student through a cross-layer attention matrix. Experimental results on the CIFAR-10 and CIFAR-100 datasets show that the proposed method improves classification accuracy on clean samples compared to the original IBD, while maintaining similar accuracy on adversarial samples. Furthermore, our methods are competitive with state-of-the-art approaches, including the recent dual-teacher distillation framework B-MTARD, particularly in terms of the harmonic mean between clean and robust accuracy. We also analyze the impact of different training settings that have different influences on the attention module.
Vincent Ryusuke Takahashi, Yoshinari Takeishi, Jun'ichi Takeuchi +1
Joint Graduate School of Mathematics for Innovation Kyushu University · Faculty of Information Science and Electrical Engineering Kyushu University · Université Savoie Mont Blanc