Curvature-Scaled Sharpness-Aware Minimization: Convergence Beyond Smoothness

Dimitris Oikonomou; Nicolas Loizou

math.OCOct 21, 2026

Curvature-Scaled Sharpness-Aware Minimization: Convergence Beyond Smoothness

Authors: Dimitris Oikonomou, Nicolas Loizou

Organizations: Mathematical Institute for Data Science (MINDS), Johns Hopkins University · Department of Computer Science, Johns Hopkins University · Department of Applied Mathematics and Statistics, Johns Hopkins University

Abstract

Sharpness-Aware Minimization (SAM) has established itself as a widely used method for training deep neural networks. Despite its popularity, existing convergence guarantees for SAM-type methods require smoothness, a restrictive assumption. Generalized smoothness conditions have recently been proposed in the analysis of gradient methods but remain underexplored for SAM. To analyze the effect of SAM's perturbation under generalized smoothness, we use relative gradient distortion to quantify how the sharpness-aware direction departs from the original gradient. Under $(L_0,L_1)$ -smoothness, we show that this distortion can become unbounded near stationarity for SAM and at large gradients for its unnormalized variant (USAM), with examples where SAM cycles and USAM diverges. We then establish a necessary condition for uniform distortion control among perturbations along the gradient whose nonnegative multipliers depend only on its norm. Motivated by this condition, we propose Curvature-Scaled SAM (CS-SAM), which scales both the perturbation and the descent step by suitable curvature upper bounds, and prove that it uniformly bounds the distortion. Under both $(L_0,L_1)$ - and $(H_0,H_1)$ -smoothness, we establish deterministic $\mathcal{O}(1/T)$ rates for the best squared gradient norm on nonconvex objectives, last-iterate $\mathcal{O}(1/T)$ rates for the function gap on convex objectives with a minimizer, and linear convergence under the Polyak-Lojasiewicz condition. With zero perturbation, our guarantees recover the rates for gradient descent with $(L_0,L_1)$ - or $(H_0,H_1)$ -based step-sizes. Minibatch experiments on computer vision tasks demonstrate competitive performance for both variants, with the $(H_0,H_1)$ variant achieving the highest mean best-test accuracy in all four settings against tuned SAM and USAM with constant and cosine annealing schedules.

Figures & tables

Method	Perturbation $d(x)$	Curvature model	$D$ uniformly bounded	Nonconvex	Convex	PL
SAM ( Foret et al., 2021 ) rates: ( Si and Yun, 2023 ) , ( Oikonomou and Loizou, 2025 )	$r\,\nabla f(x)/\left\lVert\nabla f(x)\right\rVert$	$L$	✗	$\mathcal{O}(1/T)+L^{2}r^{2}$	no last-iterate gap rate	linear to a neighborhood
USAM ( Andriushchenko and Flammarion, 2022 )	$\rho\,\nabla f(x)$	$L$	✓	$\mathcal{O}(1/T)$	$\mathcal{O}(1/T)$	linear
Unified SAM ( Oikonomou and Loizou, 2025 )	$\rho[(1-\lambda)+\lambda/\left\lVert\nabla f(x)\right\rVert]\cdot\nabla f(x)$	$L$	✓ if $\lambda=0$ ; ✗ otherwise	$\mathcal{O}(T^{-1/2})$ (in expectation)	not analyzed	$\lambda=0$ : linear $\lambda>0$ : linear to a neighborhood
Clipped perturbation ( Adly and Le, 2026 )	$\rho_{k}\nabla f(x)/\left\lVert\nabla f(x)\right\rVert^{\alpha}$ , $\rho_{k}=\min\{\rho,\frac{\theta}{L}\left\lVert\nabla f(x)\right\rVert^{\alpha}\}$	$L$	✓	$\mathcal{O}(1/T)$	not analyzed	not analyzed
Inexact-gradient SAM ( Khanh et al., 2024 )	normalized and unnormalized variants	$L$	normalized: ✗ USAM: ✓	$\left\lVert\nabla f(x_{t})\right\rVert\to 0$ (no generic rate)	$f(x_{T})\to f_{\star}$ (no rate)	USAM: linear under KL exponent $1/2$
Ours, CS-SAM with $M_{L}$	$\rho\,\nabla f(x)/(L_{0}+L_{1}\left\lVert\nabla f(x)\right\rVert)$	$(L_{0},L_{1})$	✓	$\mathcal{O}(1/T)$	$\mathcal{O}(1/T)$	linear

Table 1: Comparison with the closest prior work.

D(x,d)=\left\lVert\nabla f(x+d)-\nabla f(x)\right\rVert/\left\lVert\nabla f(x)\right\rVert

is the relative gradient distortion of the perturbation

d

( Section 2.1 ). Curvature models:

L

means

\left\lVert\nabla^{2}f(x)\right\rVert\leq L

;

(L_{0},L_{1})

means

\left\lVert\nabla^{2}f(x)\right\rVert\leq L_{0}+L_{1}\left\lVert\nabla f(x)\right\rVert

;

(H_{0},H_{1})

means

\left\lVert\nabla^{2}f(x)\right\rVert\leq H_{0}+H_{1}(f(x)-f_{\star})

, with

f_{\star}=\inf_{x}f(x)

. ✓:

D

is bounded by a constant; ✗: the bound degenerates. Rates are on the best squared gradient norm over

T

iterations (nonconvex), on the last-iterate gap

f(x_{T})-f_{\star}

for convex objectives with a minimizer (convex), and on the function gap under PL.

Curvature scale $M(x)$	Assumption	$\boldsymbol{\kappa}$	Perturbation step $\rho/M(x_{t})$	Descent step $\gamma/M(y_{t})$	CS-SAM at $\rho=0$ ( CS-GD )
$L$ (constant)	( 2 )	$0$	$\rho/L$	$\gamma/L$	gradient descent with step $\gamma/L$
$M_{L}(x)=L_{0}+L_{1}\left\lVert\nabla f(x)\right\rVert$	( 3 )	$L_{1}$	$\dfrac{\rho}{L_{0}+L_{1}\left\lVert\nabla f(x_{t})\right\rVert}$	$\dfrac{\gamma}{L_{0}+L_{1}\left\lVert\nabla f(y_{t})\right\rVert}$	the gradient method of Vankov et al. (2025) , Gorbunov et al. (2025)
$M_{H}(x)=H_{0}+H_{1}(f(x)-f_{\star})$	( 4 )	$\sqrt{H_{1}}$	$\dfrac{\rho}{H_{0}+H_{1}(f(x_{t})-f_{\star})}$	$\dfrac{\gamma}{H_{0}+H_{1}(f(y_{t})-f_{\star})}$	a loss-dependent step similar to Alimisis et al. (2026)

Table 2: The curvature scales associated with ( 2 ), ( 3 ) and ( 4 ) as instances of CS-SAM . For each scale we list the assumption under which it is a curvature scale, the admissibility constant

\kappa

of Proposition 2.4 , the perturbation and descent steps of CS-SAM , and the curvature-scaled gradient method obtained at

\rho=0

( CS-GD , Section 2.2 ).

Panel of Figure 1 — Figure 1: Verification of Section 2 on the objectives of Proposition 2.1 . (a) Iterates of SAM on $f(x)=x^{2}/2$ against the predicted two-cycle (dashed), with CS-SAM . (b) $|x_{t}|$ of USAM on $f(x)=x^{4}/4$ against the number of gradient evaluations, with CS-SAM for both scales. (c)–(d) Relative gradient distortion ( 5 ) against the gradient norm and the bound $e^{\rho}-1$ of Theorem 2.5 .

Method	ResNet-20 / CIFAR-10	ResNet-32 / CIFAR-10	ResNet-20 / CIFAR-100	ResNet-32 / CIFAR-100
SAM , constant lr	$90.60_{\pm 0.20}$	$91.24_{\pm 0.08}$	$64.66_{\pm 0.61}$	$65.55_{\pm 0.62}$
SAM , cosine	$90.49_{\pm 0.16}$	$91.16_{\pm 0.23}$	$65.32_{\pm 0.21}$	$66.07_{\pm 0.69}$
USAM , constant lr	$90.15_{\pm 0.30}$	$90.91_{\pm 0.11}$	$64.73_{\pm 0.12}$	$65.61_{\pm 0.40}$
USAM , cosine	$90.13_{\pm 0.41}$	$90.84_{\pm 0.19}$	$65.33_{\pm 0.13}$	$66.77_{\pm 0.34}$
CS-SAM with $M_{L}$	$90.87_{\pm 0.30}$	$91.72_{\pm 0.33}$	$65.18_{\pm 0.42}$	$66.80_{\pm 0.59}$
CS-SAM with $M_{H}$	$\mathbf{91.01_{\pm 0.06}}$	$\mathbf{91.73_{\pm 0.22}}$	$\mathbf{65.68_{\pm 0.34}}$	$\mathbf{67.06_{\pm 0.44}}$

Table 3: Best test accuracy (%, maximum over 200 epochs), mean

\pm

standard deviation over 3 seeds, for ResNet-20/32 trained on CIFAR-10/100. Bold marks the best mean in each column.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Algorithm 1 Curvature-Scaled SAM ( CS-SAM ).

Method	Parameters	ResNet-20 / CIFAR-10	ResNet-32 / CIFAR-10	ResNet-20 / CIFAR-100	ResNet-32 / CIFAR-100
SAM , constant lr	$(\eta,r)$	$(0.3,\,0.2)$	$(0.3,\,0.2)$	$(0.1,\,0.2)$	$(0.1,\,0.2)$
SAM , cosine	$(\eta,r)$	$(1,\,0.1)$	$(0.3,\,0.2)$	$(0.3,\,0.2)$	$(0.3,\,0.2)$
USAM , constant lr	$(\eta,\rho)$	$(0.3,\,0.3)$	$(0.3,\,1)$	$(0.3,\,1)$	$(0.3,\,0.3)$
USAM , cosine	$(\eta,\rho)$	$(0.3,\,0.3)$	$(0.3,\,0.3)$	$(0.3,\,0.3)$	$(0.3,\,0.3)$
CS-SAM with $M_{L}$	$(\rho,\gamma,L_{0})$	$(0.8,\,0.4,\,0.02)$	$(0.8,\,0.4,\,0.02)$	$(0.4,\,0.2,\,0.02)$	$(0.4,\,0.4,\,0.02)$
CS-SAM with $M_{H}$	$(\rho,\gamma,H_{1})$	$(3,\,3,\,10)$	$(3,\,1,\,1)$	$(3,\,3,\,3)$	$(3,\,3,\,3)$

Appendix

Table 4: Configurations selected on seed 3107 by the best test accuracy over 200 epochs, for every method and setting of Table 3 . For SAM and USAM ,

\eta

is the constant learning rate or the initial learning rate of the cosine annealing schedule ( 34 ),

r

is the radius of SAM , and

\rho

is the multiplier of USAM . For CS-SAM ,

\rho

and

\gamma

are the perturbation and step multipliers,

L_{0}

is the constant term of the

M_{L}

scale

L_{0}+L_{1}\left\lVert\nabla f_{i}(x)\right\rVert

with

L_{1}=1

, and

H_{1}

is the loss coefficient of the

M_{H}

scale

H_{0}+H_{1}[f_{i}(x)-\ell_{i}^{\star}]_{+}

with

H_{0}=1

and

\ell_{i}^{\star}=0

Sharpness-Aware Minimization

Sharpness-Aware Minimization (SAM) trains models toward locally flat loss regions to improve generalization and robustness.

See all 40 papers

Oct 14, 2026

One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM

Poor generalisation in deep neural networks is frequently associated with convergence to sharp minima. Sharpness-Aware Minimisation (SAM) improves generalisation of deep neural networks by minimising a combination of the training loss and the local sharpness of the loss landscape, biasing convergence towards flat minima. However, SAM relies on a fixed perturbation radius

ρ

that ignores local curvature, leading to inefficient behaviour: the perturbation may be too small to escape poor basins defined by sharp local minima, or too large to allow convergence to solutions with good loss-sharpness tradeoff. We reinterpret

ρ

as the intensity of the perturbation signal and propose Curvature-Aware Adaptive Perturbation for SAM (CAP-SAM), a geometry-aware method that modulates the perturbation radius based on a sharpness ratio, enabling a more effective search for robust minima on the loss landscape. We prove that CAP-SAM preserves the same convergence rate as standard SAM in non-convex stochastic settings. Extensive experiments indicate that our approach consistently outperforms SAM and its most prominent variants across diverse network architectures on CIFAR-10, CIFAR-100, and ImageNet.

Alessia Rossi, Felipe Campelo, Gabriel L. Oliveira

School of Engineering Mathematics and Technology, University of Bristol, Bristol, Lovelace Building, Ada, Bristol, BS8 1TW, UK

Curvature-Aware Optimization Sharpness-Aware Minimization Neural Network Optimization Neural Network Generalization

Ask PDF

Mar 9, 2026

Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation

Sharpness-Aware Minimization (SAM) enhances generalization by minimizing the maximum training loss within a predefined neighborhood around the parameters. However, its practical implementation approximates this as gradient ascent(s) followed by applying the gradient at the ascent point to update the current parameters. This practice can be justified as approximately optimizing the objective by neglecting the (full) derivative of the ascent point with respect to the current parameters. Nevertheless, a direct and intuitive understanding of why using the gradient at the ascent point to update the current parameters works superiorly is still lacking. Our work bridges this gap by proposing a novel and intuitive interpretation. We show that the gradient at the single-step ascent point, \uline{when applied to the current parameters}, provides a better approximation of the direction from the current parameters toward the maximum within the local neighborhood than the local gradient. This improved approximation thereby enables a more direct escape from the maximum within the local neighborhood. Nevertheless, our analysis further reveals two issues. First, the approximation by the gradient at the single-step ascent point is often inaccurate. Second, the approximation quality may degrade as the number of ascent steps increases. To address these limitations, we propose in this paper eXplicit Sharpness-Aware Minimization (XSAM). It tackles the first by explicitly estimating the direction of the maximum during training, while addressing the second by crafting a search space that effectively leverages the gradient information at the multi-step ascent point. XSAM features a unified formulation that applies to both single-step and multi-step settings and only incurs negligible computational overhead. Extensive experiments demonstrate the consistent superiority of XSAM against existing counterparts.

Jianlong Chen, Zhiming Zhou

Key Laboratory of Interdisciplinary Research of Computation and Economics Shanghai University of Finance and Economics

Sharpness-Aware Minimization Deep Learning Optimization Loss Landscape Sharpness Min-Max Optimization

Ask PDF

May 11, 2026

Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization

Sharpness-Aware Minimization (SAM) improves generalization by minimizing the worst-case loss within a fixed parameter-space radius neighborhood. SAM and its variants mainly rely on a first-order linearized surrogate, while flat minima are inherently a second-order (curvature) notion.We revisit this mismatch and propose Loss-Equated SAM (LE-SAM), which inverts the traditional SAM mechanism that fixed perturbation radius with a fixed loss-space budget,effectively removing gradient-norm-dominated learning signals and shifting optimization toward curvature-dominated terms. Extensive experiments across diverse benchmarks and tasks demonstrate the strong generalization ability of LESAM that consistently outperforms SAM and even its variants, achieving the state-of-the-art performance.

Jinping Wang, Qinhan Liu, Zhiwu Xie +1

Wenzhou-Kean University · International Frontier Interdisciplinary Research Institute, Wenzhou-Kean University

Neural Network Generalization Sharpness-Aware Minimization Deep Learning Optimization Nonconvex Optimization Flat Minima Loss Landscape Sharpness

Ask PDF

Abstract

Sharpness-Aware Minimization

Explore similar work

One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM

Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation

Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization