Learning Better Certified Models from Empirically-Robust Teachers
Authors: Alessandro De Palma
Organizations: Department of Statistics, London School of Economics and Political Science, London, United Kingdom · Work parly carried out at Inria, École Normale Supérieure, PSL University, CNRS
Adversarial training attains strong empirical robustness to specific adversarial attacks by training on concrete adversarial perturbations, but it produces neural networks that are not amenable to strong robustness certificates through neural network verification. On the other hand, earlier certified training schemes directly train on bounds from network relaxations to obtain models that are certifiably robust, but display sub-par standard performance. Recent work has shown that state-of-the-art trade-offs between certified robustness and standard performance can be obtained through a family of losses combining adversarial outputs and neural network bounds. Nevertheless, differently from empirical robustness, verifiability still comes at a significant cost in standard performance. In this work, we propose to leverage empirically-robust teachers to improve the performance of certifiably-robust models through knowledge distillation. Using versatile feature-space distillation objectives, we show that distillation from adversarially-trained teachers can significantly improve the performance of state-of-the-art certified training algorithms for ReLU networks on robust computer vision benchmarks.
Figures & tables
Figure 1: Standard, empirical adversarial and certified accuracies (BaB and CROWN/IBP) under ℓ∞ perturbations of networks trained using CC-Dist under varying distillation coefficient β . The legend is reported once for all subfigures in plot 1(b) . Metrics are reported on the CIFAR-10 test set. The β value employed throughout the paper (see \lx@sectionsign 3.2 ) is marked by a dashed vertical line.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Table 10: Effect of distillation beyond CNN-7 students on CIFAR-10 with ϵ=\nicefrac2255 for CC-Dist.
Engineering Research Center of Cyberspace and School of Software and AI, Yunnan University, Kunming, China · School of Information Science and Technology, Yunnan Normal University, Kunming, China