S2O: Enhancing Adversarial Training with Second-Order Statistics of Weights

Gaojie Jin; Xinping Yi; Wei Huang; Sven Schewe; Xiaowei Huang

cs.LGMar 1, 2026

S2O: Enhancing Adversarial Training with Second-Order Statistics of Weights

Authors: Gaojie Jin, Xinping Yi, Wei Huang, Sven Schewe, Xiaowei Huang

Organizations: Department of Computer Science, University of Exeter, Exeter, UK · National Mobile Communications Research Laboratory, Southeast University, Nanjing, China · Department of Computer Science, University of Liverpool, Liverpool, UK

Abstract

Adversarial training has emerged as a highly effective way to improve the robustness of deep neural networks (DNNs). It is typically conceptualized as a min-max optimization problem over model weights and adversarial perturbations, where the weights are optimized using gradient descent methods, such as SGD. In this paper, we propose a novel approach by treating model weights as random variables, which paves the way for enhancing adversarial training through Second-Order Statistics Optimization (S $^2$ O) over model weights. We challenge and relax a prevalent, yet often unrealistic, assumption in prior PAC-Bayesian frameworks: the statistical independence of weights. From this relaxation, we derive an improved PAC-Bayesian robust generalization bound. Our theoretical developments suggest that optimizing the second-order statistics of weights can substantially tighten this bound. We complement this theoretical insight by conducting an extensive set of experiments that demonstrate that S $^2$ O not only enhances the robustness and generalization of neural networks when used in isolation, but also seamlessly augments other state-of-the-art adversarial training techniques. The code is available at https://github.com/Alexkael/S2O.

Figures & tables

Fig. 1: Illustration of the theoretical framework: perturbation bound with consideration of correlated weights. Under this framework, a standard generalization bound is extended to a robust generalization bound with weight correlation matrix, and further to a bound with weight correlation matrix estimated over both clean data and adversarial data.

Fig. 2: Illustration of the optimization framework: Theorem III.2 and Theorem III.5 show the influence of the second-order statistics of weights, over clean and adversarial data, on the robust generalization performance. In Section IV-A , we demonstrate that these statistics can be approximately optimized using the Frobenius norm of the correlation matrix. To simplify this optimization process, we employ the Laplace approximation in Section IV-B and propose our adversarial training methodology, S 2 O, in Section IV-C .

Fig. 3: We sample 10000 9-dimensional correlation matrices and demonstrate (a) $\|\mathbf{R}_{l,\mathbf{x}}\|_{F}^{2}$ , $\|\mathbf{R}_{l,\mathbf{x}^{\prime}}\|_{F}^{2}$ w.r.t $\Lambda^{c}_{l,\max}$ or $\Lambda^{r}_{l,\max}$ ; (b) $\|\mathbf{R}_{l,\mathbf{x}}\|_{F}^{2}$ , $\|\mathbf{R}_{l,\mathbf{x}^{\prime}}\|_{F}^{2}$ w.r.t $\Lambda_{l,\min}^{k_{l}}\Lambda_{l,\max}^{h^{2}-k_{l}}$ ; (c) $\|\mathbf{R}_{l,\mathbf{x}}\|_{F}^{2}$ , $\|\mathbf{R}_{l,\mathbf{x}^{\prime}}\|_{F}^{2}$ w.r.t $\Lambda^{c}_{l,\max}$ or $\Lambda^{r}_{l,\max}$ ; (d) $\|\mathbf{R}_{l,\mathbf{x}}\|_{F}^{2}$ , $\|\mathbf{R}_{l,\mathbf{x}^{\prime}}\|_{F}^{2}$ w.r.t $\Lambda_{l,\min}^{k_{l}}\Lambda_{l,\max}^{h^{2}-k_{l}}$ . Note that the horizontal axis represents both $\|\mathbf{R}_{l,\mathbf{x}}\|_{F}^{2}$ and $\|\mathbf{R}_{l,\mathbf{x}^{\prime}}\|_{F}^{2}$ .

Fig. 4: We train PreAct ResNet18 with AT and AT+S 2 O on CIFAR-10, and show the results of partial weights. The results estimated from adversarial data are presented in the top figures , while the bottom figures show the results estimated from clean data. (a) shows the normalized spectral norm of $\mathbf{R}^{c}$ , ${\mathbf{R}}^{r}$ , and the determinant of $\mathbf{R}$ , with sampling estimation and Laplace approximation respectively. (b) and (c) demonstrate the absolute correlation matrix of partial weights, for AT and AT+S 2 O respectively.

Threat	Method	CIFAR-10			CIFAR-100			SVHN
Model	Method	Clean	PGD	Time	Clean	PGD	Time	Clean	PGD	Time
$\ell_{\infty}$	AT	82.41	52.77	309s	58.02	28.02	307s	93.17	60.91	509s
$\ell_{\infty}$	AT+S 2 O	83.65	55.11	368s	58.45	30.58	371s	93.39	64.83	595s
$\ell_{2}$	AT	88.83	68.83	292s	64.21	42.20	290s	94.02	66.76	477s
$\ell_{2}$	AT+S 2 O	89.57	69.42	364s	65.32	44.07	366s	94.93	76.19	586s

TABLE I: Adversarial training across datasets on PreAct ResNet18 (%).

Dataset	Method	Model	DDPM Data	Clean	AA
CIFAR-10	Lee et al. (2020) [ 35 ]	WRN-34-10	-	92.56	39.70
	Wang et al. (2020) [ 56 ]	WRN-34-10	-	83.51	51.10
	Rice et al. (2020) [ 47 ]	WRN-34-20	-	85.34	53.42
	Zhang et al. (2020) [ 64 ]	WRN-34-10	-	84.52	53.51
	Pang et al. (2021) [ 44 ]	WRN-34-20	-	86.43	54.39
	Jin et al. (2023) [ 28 ]	WRN-34-20	-	85.98	54.20

TABLE III: Experiments with DDPM generated data under

\ell_{\infty}

attack (%).

Radius	Method	Clean	$\ell_{\infty}$	$\ell_{2}$	$\ell_{1}$	Union
$\ell_{\infty}:\frac{2}{255}$ $\ell_{2}:0.5$ $\ell_{1}:12$	Croce et al. (2022) [ 11 ]	87.2	73.3	64.1	55.4	55.4
	Tramer et al. (2019) [ 55 ]	85.6	72.1	63.6	56.4	56.4
	Jiang et al. (2024) [ 25 ]	86.3	73.3	64.9	59.1	59.1
	+ S 2 O	87.1	73.6	65.3	59.9	59.9
$\ell_{\infty}:\frac{8}{255}$ $\ell_{2}:1.5$ $\ell_{1}:12$	Croce et al. (2022) [ 11 ]	83.5	41.0	25.5	52.9	25.5
	Tramer et al. (2019) [ 55 ]	74.6	42.9	35.7	50.3	35.6

TABLE IV: Experiments on CIFAR-10 with multiple

\ell_{p}

attacks (%).

Method	CIFAR-10			CIFAR-100
Method	Clean	AA	Sum	Clean	AA	Sum
AT	86.06	46.26	132.32	59.83	23.94	83.77
Suzuki et al. (2023) [ 51 ]	90.24	50.20	140.44	73.05	24.32	97.37
Waseda et al. (2025) [ 58 ]	90.89	50.77	141.66	72.51	24.18	96.70
+S 2 O	90.45	53.16	143.61	72.36	27.09	99.45

TABLE V: Experiments on CIFAR to measure the trade-off (

\ell_{\infty}

, %).

Network	Method	CIFAR-10		Imagenette
Network	Method	Clean	AA	Clean	AA
ViT-B	AT	85.53	48.33	91.40	64.20
ViT-B	AT+S 2 O	85.66	48.97	91.40	66.40
DeiT-S	AT	84.13	47.26	90.40	60.20
DeiT-S	AT+S 2 O	84.42	48.36	91.00	63.60

TABLE VI: Experiments on adversarially trained ViT, DeiT (

\ell_{\infty}

, %).

Method	CIFAR-10		CIFAR-100		SVHN
Method	FGSM	PGD-20	FGSM	PGD-20	FGSM	PGD-20
AT	64.32	62.63	38.55	37.36	71.77	63.76
AT+S 2 O	65.63	63.87	39.68	38.60	72.20	64.31

TABLE VIII: Experiments on ResNet-18 with transfer attacks (

\ell_{\infty}

, %).

Adversarial Training

Adversarial training improves model reliability by training on examples or conditions designed to expose weaknesses.

See all 177 papers

How Learning Dynamics Drive Adversarially Robust Generalization?
Oct 10, 2024Yuelin Xu, Xiao ZhangAdversarial Training PAC-Bayesian Generalization Bounds
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
SORA: Free Second-Order Attacks in Fast Adversarial Training
May 30, 2026Mazdak Teymourian, Ramtin Moslemi, Farzan Rahmani +1Adversarial Training Neural Network Generalization
Department of Computer Engineering, Sharif University of Technology, Tehran, Iran.
Probabilistic Adversarial Training
Sep 30, 2026Andi Zhang, Xingyu Zhao, Siddartha KhastgirAdversarial Training Adversarial Robustness
WMG, University of Warwick · Wuhan University

Oct 10, 2024J/K move · Enter open · S save

How Learning Dynamics Drive Adversarially Robust Generalization?

Yuelin Xu, Xiao Zhang

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Adversarial Training PAC-Bayesian Generalization Bounds Neural Network Robustness Adversarial Robustness Neural Network Training Dynamics

Open paper AskPDF arXiv

Despite being widely adopted as a canonical framework for learning robust models, adversarial training suffers from robust overfitting. Existing empirical and theoretical explorations fail to provide a satisfactory mechanistic interpretation of the phenomenon. By modeling adversarial training with momentum SGD as a discrete-time dynamical system, we propose a PAC-Bayesian analytical framework that proves time-resolved robust generalization bounds. Specifically, our framework tracks the closed-form evolution of the posterior mean and covariance under both stationary and non-stationary transient regimes, connecting the model's robust generalization performance to learning rate, local loss geometry, and mini-batch stochastic gradients. By estimating the key quantities associated with the bound, we illustrate the underlying mechanism of robust overfitting. Our framework also shows how adversarial weight perturbation reduces robust generalization gaps by suppressing dominant loss-curvature modes, while suggesting that excessive penalization can be sub-optimal for optimization.

Abstract

Adversarial Training

Explore similar work

How Learning Dynamics Drive Adversarially Robust Generalization?

SORA: Free Second-Order Attacks in Fast Adversarial Training

Probabilistic Adversarial Training