Adversarial Training with Large Step Sizes: Implicit Bias, Optimization, and Sharpness

Yi Feng; Andrea Paudice

cs.LGNov 5, 2026

Adversarial Training with Large Step Sizes: Implicit Bias, Optimization, and Sharpness

Authors: Yi Feng, Andrea Paudice

Organizations: Aarhus University, Denmark

Abstract

Adversarial Training (AT) is an important method for improving model robustness. We study the dynamics of deterministic Gradient Descent (GD) in AT. Most existing works in this direction focus on the small or vanishing step sizes regime, where GD resembles a continuous-time flow. However, such step sizes can make training inefficient and are rarely used in practice. In this paper, we study the AT dynamics of GD with large constant step sizes. We first study GD in adversarial linear logistic regression, focusing on two aspects: where the model parameters converge (implicit bias) and how quickly the losses decrease (optimization). We then turn to neural networks training and study how the sharpness of losses evolve during AT. Our results advance the understanding of the behaviors of GD with large step sizes in AT and highlight three key differences from its behavior in Standard Training (ST): (i) the implicit bias gap between AT and ST, previously established only for small step sizes, persists for arbitrarily large constant step sizes, (ii) different from ST, there is no uniform rate of convergence for the optimization error in AT, and (iii) sharpness can exhibit two-stage dynamics that differ from the Edge-of-Stability behavior observed in ST.

Figures & tables

Figure 1: Illustrations of the main findings of this work. Left : AT and ST have different implicit bias under large constant step sizes. Middle : The ST loss decreases monotonically after a phase transition time $\tau$ , whereas the AT loss increases infinitely often. It can also be observed that the AT loss decreases very slowly. Right : ST and AT can exhibit different sharpness evolutions. In particular, ST exhibits the edge-of-stability phenomenon, in which the sharpness oscillates around $2/\eta$ , while the sharpness of AT decreases to a value much lower than $2/\eta$ .

Figure 2: Sharpness evolution across neural network architectures (MLP, CNN, ResNet, and Vision Transformer) and step sizes under standard training (ST) and $\boldsymbol{\ell_{\infty}}$ -adversarial training on CIFAR-10. Adversarial examples are generated using ( FGSM ). Except for the MLP at step size $0.005$ , sharpness under ST initially increases and then oscillates around $2/\eta$ , consistent with the edge-of-stability phenomenon ( Cohen et al.,, 2021 ) . Under AT, sharpness also initially increases, but subsequently oscillates with an overall decreasing trend. Opaque curves show smoothed trends for clarity, while semi-transparent curves show the raw values.

Figure 3: The experimental settings are identical to those in Figure 2 except that adversarial examples are generated by $\boldsymbol{\ell_{2}}$ attacks, and adversarial radius is adjusted per common practice.

Figure 4: Sharpness evolution under randomly perturbed losses resembles that of standard training.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Sharpness evolution under standard training and $\ell_{\infty}$ -adversarial training on CIFAR-10. Adversarial examples are generated using ( PGD ).

Figure 6: Sharpness evolution under standard training and $\ell_{2}$ -adversarial training on CIFAR-10. Adversarial examples are generated using ( PGD ).

Figure 7: Comparison of the oscillation behaviors between sharpness and losses. It can be seen that when the sharpness starts to oscillate with a decreasing trend, the losses also start to oscillate. The experimental setting is step size $\eta=0.005$ with different architectures on CIFAR-10, and the adversarial attacks are generated by FGSM with $\ell_{\infty}$ geometry (upper) or $\ell_{2}$ geometry (lower).

Figure 8: The experimental setting is same as Figure 7 , besides the adversarial attacks are generated by PGD .

Figure 9: Sharpness evolution of CNN, ResNet, and ViT on the SVHN dataset. The adversarial examples are generated by the FGSM method with $\ell_{\infty}$ geometry. It can be observed that the sharpness also exhibits the two-stage behavior described in Section 5 .

Figure 10: Sharpness evolution of CNN, ResNet, and ViT with different sample sizes.

Adversarial Training

Adversarial training improves model reliability by training on examples or conditions designed to expose weaknesses.

See all 195 papers

Flatland: The Adventures of Gradient Descent with Large Step Sizes
Jun 4, 2026Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus +2Edge of Stability Gradient Descent
LMU Munich · Munich Center for Machine Learning · University of British Columbia +1
When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study
May 26, 2026Jun Yan, Weiquan Huang, Jiankai Zuo +4Adversarial Training Orthogonal Optimization
IT College, Shanghai Ocean University, Shanghai, 201306 · School of Computer Science and Technology, Tongji University, Shanghai, 201804 · SEIE, Suzhou University of Science and Technology, Suzhou, 215009 +2
Homogenization of $\ell_2$ -Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent
Jun 30, 2026Fabrizzio SabelliAdversarial Training High-Dimensional Asymptotics
Université de Montréal

Jun 4, 2026J/K move · Enter open · S save

Flatland: The Adventures of Gradient Descent with Large Step Sizes

Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus, Mark Schmidt+1

LMU Munich · Munich Center for Machine Learning · University of British Columbia+1

Edge of Stability Gradient Descent Adaptive Gradient Methods Gradient Descent Dynamics Neural Network Stability Loss Landscape Sharpness

Open paper AskPDF arXiv

The training of neural networks often entails objective functions that are not globally

L

-smooth. For these functions, it is both theoretically and practically difficult to reply to the question: what is the largest possible step size that ensures the convergence of gradient descent (GD)? We address this longstanding open question in deep learning by providing a unifying definition of "large" step sizes that requires only local Lipschitz (or even Hölder) continuity of the gradient. We design first-order adaptive methods that provably yield large step sizes and show that they operate at the edge of stability (EoS) right from the start of the training. In particular, the loss decreases nonmonotonically and the product between the step size and sharpness, i.e., the largest eigenvalue of the Hessian, stays above the EoS threshold of 2 throughout training. Using our method, we are also able to minimize the sharpness all the way down to its global minimum. Contrary to expectation, we find that encountering globally-flat regions too early in the training may both slow down convergence and jeopardize the generalization ability of the network. Exploiting a self-stabilization argument, we allow GD to enter slightly sharper valleys and turn unsuccessful training runs into very successful ones.

Abstract

Adversarial Training

Explore similar work

Flatland: The Adventures of Gradient Descent with Large Step Sizes

When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study

Homogenization of ℓ2\ell_2ℓ2​-Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent

Homogenization of $\ell_2$ -Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent