ExGrad: Beyond Suppression - Explicit Clean-Gradient Subtraction in Adversarial Training

Zhiyuan Ye; Xiangyu Zhou; Ji Qi; Hao Zhang; Yi Zhou

cs.LGOct 21, 2026

ExGrad: Beyond Suppression - Explicit Clean-Gradient Subtraction in Adversarial Training

Authors: Zhiyuan Ye, Xiangyu Zhou, Ji Qi, Hao Zhang, Yi Zhou

Organizations: University of Science and Technology of China · China Mobile (Suzhou) Software Technology Co., Ltd.

Abstract

Adversarial training derives updates from surrogate losses, but source-loss descent does not directly characterize an update's effect on adversarial margins. We introduce ExGrad (extrapolative gradient recomposition), which constructs clean-input, adversarial-input, and adversarial-weight-perturbed (AWP) gradients. Virtual steps measure their margin responses, and a globally solvable three-variable problem selects coefficients over a bounded affine domain without nonnegativity constraints. Across affine radii on CIFAR-10, negative clean coefficients rapidly predominate and increase in magnitude with the radius. Cross-zero interventions show that setting the clean coefficient to zero is insufficient: mild subtraction improves PGD-20, while stronger subtraction trades clean accuracy for further PGD-20 gains. In CIFAR-10 WRN-28-4 controls, none of the three candidate-removal variants matches full ExGrad. Projection diagnostics show that negative clean coefficients usually coexist with positive net clean-axis projections, although both decrease as the radius grows. Norm-matched controls indicate that update magnitude alone does not explain the moderate-radius gains. Experiments on CIFAR-10, CIFAR-100, and ImageNet-100 show radius-dependent improvements over AWP. On CIFAR-100, ExGrad at $R=0.8$ improves all five reported metrics, including clean and AutoAttack accuracy by 3.84 and 1.75 percentage points. On ImageNet-100, the same radius improves all four robust metrics over both AWP and PGD-AT. These results identify clean-gradient subtraction as a response-revealed update preference whose strength and clean-robust operating point are controlled by the affine radius.

Figures & tables

Figure 1: Overview of ExGrad. (a) A mini-batch update in which three state-dependent gradients are probed from the same nominal parameters; their margin responses determine one affine recomposition, and only the recomposed gradient advances the optimizer. (b) The bounded unit-sum domain intersects the nonnegative simplex and extends beyond it to permit response-supported subtraction.

Dataset	Model	Method	Clean	PGD	APGD-CE	APGD-T	AA
CIFAR-10	WRN-28-4	PGD-AT	83.00 $\pm$ 0.47	53.41 $\pm$ 0.28	52.69 $\pm$ 0.27	49.15 $\pm$ 0.21	49.12 $\pm$ 0.21
		TRADES ( $\beta=3$ )	84.94 $\pm$ 1.84	50.23 $\pm$ 0.39	49.40 $\pm$ 0.41	47.50 $\pm$ 0.33	47.51 $\pm$ 0.33
		TRADES ( $\beta=6$ )	84.21 $\pm$ 0.67	52.33 $\pm$ 0.38	51.65 $\pm$ 0.39	49.54 $\pm$ 0.08	49.53 $\pm$ 0.05
		TRADES ( $\beta=9$ )	82.40 $\pm$ 0.30	53.12 $\pm$ 0.23	52.38 $\pm$ 0.19	49.89 $\pm$ 0.15	49.88 $\pm$ 0.16
		AWP	82.62 $\pm$ 0.34	56.48 $\pm$ 0.22	55.80 $\pm$ 0.27	51.09 $\pm$ 0.09	51.07 $\pm$ 0.11
		ExGrad ( $R=0.8$ )	83.75 $\pm$ 0.21	56.51 $\pm$ 0.17	55.57 $\pm$ 0.19	51.25 $\pm$ 0.31	51.25 $\pm$ 0.30

Table 1: Clean and robust test accuracy (%). Best and second-best results within each dataset–model block are boldfaced and underlined, respectively.

Figure 2: Test-accuracy trajectories on CIFAR-10 with WRN-28-4. The upper and lower panels report PGD-20 and clean accuracy, respectively; ExGrad uses

R=1.0

. All trajectories use a three-epoch moving average.

Figure 3: Clean-coordinate and clean-axis projection dynamics on CIFAR-10 with WRN-28-4. Rows correspond to

R=0.8

1.0

, and

1.2

; the left and right columns show

\beta_{\mathrm{clean}}

and

p_{\mathrm{clean}}

, respectively. Solid curves are within-epoch mini-batch means; shaded regions span the mini-batch minima and maxima.

$\beta_{\mathrm{clean}}$	$\beta_{\mathrm{adv}}$	$\beta_{\mathrm{awp}}$	Clean	PGD
Adap.	Adap.	$0$	83.38 $\pm$ 0.39	53.37 $\pm$ 0.38
Adap.	$0$	Adap.	82.75 $\pm$ 0.37	56.39 $\pm$ 0.16
$0.1$	Adap.	Adap.	84.91 $\pm$ 0.49	55.53 $\pm$ 0.23
$0$	Adap.	Adap.	83.27 $\pm$ 0.33	56.44 $\pm$ 0.10
$-0.1$	Adap.	Adap.	81.81 $\pm$ 0.14	57.12 $\pm$ 0.14
$-0.2$	Adap.	Adap.	81.17 $\pm$ 0.15	57.40 $\pm$ 0.04

Table 2: Ablations on CIFAR-10 with WRN-28-4. All settings use

R=1

. “Adap.” denotes an adaptive coefficient. The fixed triplet averages the coefficients at the best-PGD checkpoint of each seed and is rounded to three decimals.

$R$	Match	Clean	PGD	Norm
0.8	No	83.75 $\pm$ 0.21	56.51 $\pm$ 0.17	0.943
0.8	Yes	83.81 $\pm$ 0.29	56.58 $\pm$ 0.26	1.000
1.0	No	81.19 $\pm$ 0.23	58.95 $\pm$ 0.09	1.083
1.0	Yes	81.13 $\pm$ 0.26	58.65 $\pm$ 0.27	1.000
1.2	No	79.77 $\pm$ 0.08	59.23 $\pm$ 0.27	1.217
1.2	Yes	79.92 $\pm$ 0.46	58.86 $\pm$ 0.12	1.000

Table 3: AWP-norm-matched control on CIFAR-10 with WRN-28-4. Norm is the applied update-to-AWP norm ratio and is unity by construction after matching.

Method	Time/epoch	Rel.	Alloc.	Reserv.
	(s)	time	(MiB)	(MiB)
PGD-AT	44.20	$0.84\times$	1096.90	1248
AWP	52.45	$1.00\times$	1149.26	1258
ExGrad	76.09	$1.45\times$	1173.50	1302

Table 4: Diagnostic-free training cost measured in separate processes on one H100 80GB (CIFAR-10/WRN-28-4, batch 128, FP32). Time averages synchronized epochs 2–5 after warm-up; memory reports PyTorch peaks.

Adversarial Training

Adversarial training improves model reliability by training on examples or conditions designed to expose weaknesses.

See all 184 papers

Augmentation-Guided Adversarial Training via Local Stability-Based Sample Selection
Oct 23, 2026Daniel Agyei Asante, Mark Tetteh, Deborah Asamoah +2Adversarial Training Adversarial Robustness
Ohio University, United States · University of Nebraska Omaha, United States · University of Toronto, Canada +1
Fast Adversarial Attacks with Gradient Prediction
May 14, 2026Kamil Ciosek, Aleksandr V. Petrov, Nicolò Felicioni +1Adversarial Attacks Neural Network Robustness
Spotify
Adversarial Training Without Input Gradients via Low-Rank Householder Expansions
Aug 27, 2026Tiana C. Johnson, Donsub RimAdversarial Training Adversarial Robustness

Oct 23, 2026J/K move · Enter open · S save

Augmentation-Guided Adversarial Training via Local Stability-Based Sample Selection

Daniel Agyei Asante, Mark Tetteh, Deborah Asamoah, Bernard Asante+1

Ohio University, United States · University of Nebraska Omaha, United States · University of Toronto, Canada+1

Adversarial Training Adversarial Robustness

Open paper AskPDF arXiv

Adversarial training remains one of the most effective defenses against adversarial attacks, but its standard PGD-based training is computationally expensive because adversarial examples are generated for every training sample. Selective adversarial training reduces this cost by perturbing only a subset of samples. However, existing selective techniques typically rely on point-wise difficulty signals such as loss, entropy, margin, or gradients. We show that these point-wise criteria do not capture local instability in a sample's neighborhood. Motivated by this, we propose Augmentation-Guided Adversarial Training (AGAT), a selective adversarial training framework that uses augmentation-induced local variation to allocate samples with unstable neighborhoods for adversarial training. AGAT ranks samples using Augmentation-based Local Stability (ALS), a lightweight score that measures the normalized change in model logits between an input and its label-preserving augmented variant. Experiments across different architectures on MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet-200 demonstrate that AGAT achieves comparable or higher robust accuracy while applying adversarial supervision to only 20-30% of the training samples in each minibatch. AGAT also requires less training time than all other adversarial training baselines in the computational cost evaluation.

Abstract

Adversarial Training

Explore similar work

Augmentation-Guided Adversarial Training via Local Stability-Based Sample Selection

Fast Adversarial Attacks with Gradient Prediction

Adversarial Training Without Input Gradients via Low-Rank Householder Expansions