CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation
Authors: Zhi Li, Haowei Liu, Hongchen Yang, Xiaoxuan Wang, Song Gao, Zhenli He, Wei Zhou, Shaowen Yao
Organizations: Engineering Research Center of Cyberspace and School of Software and AI, Yunnan University, Kunming, China · School of Information Science and Technology, Yunnan Normal University, Kunming, China
Adversarial robustness distillation typically treats a robust teacher as a fixed knowledge source and transfers robustness to a compact student in a one-way manner. This paradigm terminates teacher-student interaction once the student is distilled, thereby discarding potentially useful feedback from the distilled student. However, exploiting such feedback is nontrivial because indiscriminate weak-to-strong transfer may propagate student errors to the teacher. To address this issue, we propose Cyclic Adversarial Robustness Distillation (CARD), a closed-loop framework that turns the distilled student into a feedback source for refining the robust teacher and subsequently transfers the teacher gains back to the same student. CARD first performs strong-to-weak adversarial distillation and then uses reliability-enhanced student feedback to refine the robust teacher. The refinement combines inverse adversarial student feedback with teacher-student discrepancy maximization, after which the refined teacher is used to continue training the same student under the unchanged base distillation objective. We further establish a sample-level sufficient condition for reliable weak-to-strong feedback transfer and derive a distribution-level upper bound on the refined teacher's risk. Experiments on CIFAR-10, CIFAR-100, and TinyImageNet-10 across multiple teacher-student architectures and base distillation methods show that CARD improves adversarial robustness in most evaluated settings while generally preserving clean accuracy. These results show that the distilled student can provide useful feedback to the teacher, enabling robustness transfer to proceed beyond the conventional one-way teacher-to-student paradigm.
Figures & tables
Figure 1: From one-way to closed-loop adversarial robustness distillation. Conventional adversarial distillation treats the robust teacher as a fixed knowledge source and transfers robustness only from teacher to student. CARD closes the teacher-student loop by using the distilled student to refine the robust teacher and then using the refined teacher to continue training the same student. Snowflake and flame symbols denote frozen and trainable networks, respectively.
Figure 2: Overview of CARD. Stage 1 performs strong-to-weak adversarial distillation from a pretrained robust teacher to a compact student. Stage 2 freezes the first-round student and uses inverse adversarial examples and teacher adversarial examples to refine the pretrained teacher. Stage 3 replaces the initial teacher with the refined teacher and continues training from the first-round student checkpoint, transferring the teacher improvement back to the student.
Type
Architecture
Method
CIFAR-10
Clean
FGSM
PGD-20
CW-20
AA
Teacher
WRN-28-10
ST [ 26 ]
84.57
62.17
57.67
54.99
53.77
Student
ResNet-18
PGD-AT [ 28 ]
82.50
57.52
51.53
50.57
48.04
TRADES [ 39 ]
82.17
58.20
52.21
50.48
49.18
ST [ 26 ]
83.43
59.91
53.72
51.66
50.01
ARD [ 13 ]
83.43
59.27
53.04
51.20
49.49
Table 1: Clean and white-box robust accuracy (%) on CIFAR-10 with WRN-28-10 as the teacher and different student architectures. Within each student-architecture group, the best and second-best results in each column are highlighted in bold and underlined, respectively.
Type
Architecture
Method
CIFAR-10
Clean
FGSM
PGD-20
CW-20
AA
Teacher
WRN-34-10
ST [ 26 ]
84.75
62.40
57.83
55.49
54.33
Student
ResNet-18
PGD-AT [ 28 ]
82.50
57.52
51.53
50.57
48.04
TRADES [ 39 ]
82.17
58.20
52.21
50.48
49.18
ST [ 26 ]
83.43
59.91
53.72
51.66
50.01
ARD [ 13 ]
83.29
59.32
53.65
51.63
50.01
Table 2: Clean and white-box robust accuracy (%) on CIFAR-10 with WRN-34-10 as the teacher and different student architectures. Within each student-architecture group, the best and second-best results in each column are highlighted in bold and underlined, respectively.
Type
Architecture
Method
CIFAR-100
Clean
FGSM
PGD-20
CW-20
AA
Teacher
WRN-34-10
ST [ 26 ]
59.40
35.92
33.71
30.30
29.19
Student
ResNet-18
PGD-AT [ 28 ]
57.52
32.12
28.50
27.12
24.68
TRADES [ 39 ]
56.32
29.83
26.95
23.77
22.69
ST [ 26 ]
55.75
33.00
30.92
26.89
26.04
ARD [ 13 ]
58.44
34.92
31.54
28.47
26.42
Table 3: Clean and white-box robust accuracy (%) on CIFAR-100 with WRN-34-10 as the teacher and different student architectures. Within each student-architecture group, the best and second-best results in each column are highlighted in bold and underlined, respectively.
Type
Architecture
Method
TinyImageNet-10
Clean
FGSM
PGD-20
CW-20
AA
Teacher
WRN-34-10
ST [ 26 ]
77.80
62.00
60.80
60.00
60.00
Student
ResNet-18
PGD-AT [ 28 ]
79.60
61.00
57.20
57.20
55.20
TRADES [ 39 ]
77.80
58.80
56.80
55.60
55.40
ST [ 26 ]
76.60
62.20
61.00
59.00
58.20
ARD [ 13 ]
79.60
60.40
58.60
57.00
56.40
Table 4: Clean and white-box robust accuracy (%) on TinyImageNet-10 with WRN-34-10 as the teacher and ResNet-18 as the student. The best and second-best student results in each column are highlighted in bold and underlined, respectively.
Method
ResNet-18
VGG-19
MobileNet-V2
PGD-100
CW-100
PGD-100
CW-100
PGD-100
CW-100
PGD-AT [ 28 ]
33.69
33.41
41.64
42.03
38.36
38.27
TRADES [ 39 ]
35.71
36.49
40.75
41.68
37.55
37.91
ST [ 26 ]
37.67
38.17
42.00
42.68
39.13
39.65
ARD [ 13 ]
39.32
39.31
44.19
44.24
40.73
40.68
RSLAD [ 43 ]
39.18
39.72
43.23
43.86
40.58
40.81
Table 5: Transfer-based black-box accuracy (%) on CIFAR-100 with WRN-34-10 as the teacher and ResNet-18 as the target student. PGD-100 and CW-100 adversarial examples are generated on independently trained robust ResNet-18, VGG-19, and MobileNet-V2 surrogate models. The best and second-best results in each column are shown in bold and underlined, respectively.
Method
ResNet-18
MobileNet-V2
VGG-19
PGD-100
CW-100
PGD-100
CW-100
PGD-100
CW-100
PGD-AT [ 28 ]
56.29
56.31
56.30
56.32
56.49
56.53
TRADES [ 39 ]
54.83
54.96
54.80
54.97
55.12
55.28
ST [ 26 ]
54.43
54.54
54.35
54.31
54.50
54.68
ARD [ 13 ]
56.99
57.20
57.12
57.04
57.38
57.15
RSLAD [ 43 ]
55.16
55.24
55.14
55.16
55.46
55.50
Table 6: Transfer-based black-box accuracy (%) on CIFAR-100 with WRN-34-10 as the teacher and ResNet-18 as the target student. PGD-100 and CW-100 adversarial examples are generated on independently trained natural ResNet-18, MobileNet-V2, and VGG-19 surrogate models. The best and second-best results in each column are shown in bold and underlined, respectively.
Method
ResNet-50
DenseNet-169
WRN-34-10
PGD-100
CW-100
PGD-100
CW-100
PGD-100
CW-100
PGD-AT [ 28 ]
55.99
55.92
56.26
56.26
57.03
57.07
TRADES [ 39 ]
54.75
54.50
54.68
54.89
55.54
55.64
ST [ 26 ]
54.13
54.32
54.48
54.42
54.94
54.99
ARD [ 13 ]
56.85
56.93
57.18
56.96
57.53
57.60
RSLAD [ 43 ]
54.74
54.97
55.04
55.25
55.71
55.75
Table 7: Transfer-based black-box accuracy (%) on CIFAR-100 with WRN-34-10 as the teacher and ResNet-18 as the target student. PGD-100 and CW-100 adversarial examples are generated on independently trained natural ResNet-50, DenseNet-169, and WRN-34-10 surrogate models. The best and second-best results in each column are shown in bold and underlined, respectively.
Variant
W2S
S2W
Clean
CW-20
AA
AdaAD
57.95
27.00
25.55
S2W Only
✓
57.09
27.55
26.15
CARD
✓
✓
58.24
28.11
26.40
Table 8: Component ablation of CARD on CIFAR-100 with WRN-34-10 as the teacher and MobileNet-V2 as the student. W2S and S2W denote weak-to-strong teacher refinement and strong-to-weak gain transfer, respectively.
Method
Clean
PGD-20
CW-20
AA
AdaAD
57.95
30.92
27.00
25.55
Joint
56.93
31.23
28.00
26.30
Separated (Ours)
58.24
31.30
28.11
26.40
Table 9: Comparison of joint and separated optimization on CIFAR-100 with WRN-34-10 as the teacher and MobileNet-V2 as the student. Both variants contain the same W2S teacher-refinement and S2W gain-transfer components.
Refinement Signal
Clean
PGD-20
CW-20
AA
AdaAD
58.91
32.57
29.00
27.80
w/o Refinement
58.48
32.79
29.32
27.92
GT Only
61.85
26.30
26.09
23.64
Teacher Clean + GT
61.75
29.67
28.14
26.49
Student Clean
58.76
32.81
29.22
27.96
Student Inverse
60.98
32.51
29.29
27.73
Table 10: Comparison of teacher-refinement supervision signals on CIFAR-100 with WRN-34-10 as the teacher and ResNet-18 as the student. The reported results correspond to the student obtained after gain transfer from the resulting teacher. GT denotes ground-truth hard-label supervision.
Figure 3: Sensitivity of CARD to the refinement weight λ on CIFAR-10 with WRN-28-10 as the teacher and ResNet-18 as the student. Blue and red curves denote natural and robust accuracy, respectively. Panels (a) and (c) report the refined teacher after the first and second refinement cycles, while panels (b) and (d) report the corresponding students after the subsequent gain-transfer stage.
Figure 4: Grad-CAM visualizations on adversarial CIFAR-100 inputs. The leftmost column shows the corresponding adversarial images, the middle columns show activation maps of different student methods, and the rightmost column shows those of the robust teacher. In the visualized examples, the CARD student exhibits more concentrated activation on object-relevant regions and greater qualitative similarity to the robust teacher than the baseline students.
Figure 5: Normalized confusion matrices on inverse adversarial, clean, and adversarial CIFAR-10 inputs with WRN-28-10 as the teacher and ResNet-18 as the student. The inverse adversarial and adversarial inputs are generated using 10-step iterative optimization. Columns correspond to the three input types, whereas rows show the initial teacher, AdaAD student, refined teacher, and final CARD student. The strong diagonal concentration on inverse adversarial inputs indicates that these inputs provide highly reliable class-consistent predictions. Teacher refinement improves clean and adversarial accuracy, and part of these gains is subsequently transferred to the final student.
Engineering Research Center of Cyberspace and School of Software and AI, Yunnan University, Kunming, China · School of Information Science and Technology, Yunnan Normal University, Kunming, China
Joint Graduate School of Mathematics for Innovation Kyushu University · Faculty of Information Science and Electrical Engineering Kyushu University · Université Savoie Mont Blanc