Organizations: Engineering Research Center of Cyberspace and School of Software and AI, Yunnan University, Kunming, China · School of Information Science and Technology, Yunnan Normal University, Kunming, China
Adversarial distillation transfers robustness from high-capacity teachers to compact students. Existing adversarial distillation methods mainly use teacher predictions on clean or adversarial examples to supervise student learning. However, teacher-favorable supervision within the perturbation neighborhood remains underexplored in adversarial distillation. We therefore propose Collaboratively Guided Adversarial Robust Distillation (CGARD), which jointly optimizes distinct student-adversarial and teacher-collaborative examples within the same perturbation neighborhood. The teacher-collaborative example is constrained to incur no greater cross-entropy loss under the teacher than the clean input. CGARD combines collaborative teacher guidance with adversarial teacher supervision to improve robust knowledge transfer. Experiments on CIFAR-10 and CIFAR-100, including white-box evaluation and additional black-box transfer evaluation, demonstrate consistent robustness improvements over strong adversarial distillation baselines.
Figures & tables
Figure 1: Overview of CGARD. The joint inner search optimizes student-adversarial and teacher-collaborative examples through the discrepancy between teacher and student predictions, while dual alignment uses teacher supervision from both examples.
CIFAR-10
CIFAR-100
Type
Arch.
Method
Clean
PGD
CW ∞
AA
Clean
PGD
CW ∞
AA
Teacher
WRN
ST [ 12 ]
84.75
57.83
55.49
54.33
59.40
33.71
30.30
29.19
Student
RN-18
Natural
95.28
0.01
0.01
0.00
77.72
0.00
0.00
0.00
PGD-AT [ 13 ]
82.50
51.53
50.57
48.04
57.52
28.50
27.12
24.68
TRADES [ 19 ]
82.78
50.83
49.37
47.63
56.32
26.95
23.77
22.69
ST [ 12 ]
83.43
53.72
51.66
50.01
55.75
30.92
26.89
26.04
Table 1: White-box robustness comparison (%) on CIFAR-10 and CIFAR-100 under ℓ∞ attacks with ε=8/255 . Best and second-best student results are shown in bold and underlined, respectively.
Surrogate Model
WRN
VGG-19
MN-V2
Method
PGD
CW ∞
PGD
CW ∞
PGD
CW ∞
PGD-AT [ 13 ]
65.21
65.21
63.84
63.12
63.25
62.41
TRADES [ 19 ]
65.64
65.26
63.89
62.96
63.00
62.26
ST [ 12 ]
66.92
66.54
66.41
65.70
66.10
65.49
ARD [ 6 ]
66.25
66.06
65.57
64.76
64.85
64.19
RSLAD [ 22 ]
66.70
66.66
66.28
65.69
65.82
65.16
Table 2: Black-box transfer robustness (%) on CIFAR-10 with robust WRN, VGG-19, and MN-V2 surrogates and an RN-18 target. Best and second-best results are shown in bold and underlined.
Configuration
Clean
FGSM
PGD
CW ∞
AA
Sequential Search
84.25
60.97
54.92
52.36
51.01
Joint Search (CGARD)
82.96
61.12
56.47
53.69
52.46
Table 3: Comparison of sequential search and constrained joint inner search on CIFAR-10 with WRN as the teacher and RN-18 as the student. Best results are shown in bold.
Lcol
Ladv
Clean
FGSM
PGD
CW ∞
AA
✓
82.64
60.65
56.06
53.21
51.84
✓
83.13
60.71
55.95
53.07
51.83
✓
✓
82.96
61.12
56.47
53.69
52.46
Table 4: Ablation study of the dual-alignment objective on CIFAR-10 with WRN as the teacher and RN-18 as the student. Best results are shown in bold.
Joint Graduate School of Mathematics for Innovation Kyushu University · Faculty of Information Science and Electrical Engineering Kyushu University · Université Savoie Mont Blanc
The Laboratory of Intelligent Collaborative Computing of UESTC, Chengdu, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China · Monash University, Melbourne, Australia