Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation

Jianlong Chen; Zhiming Zhou

cs.LGMar 9, 2026

Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation

Authors: Jianlong Chen, Zhiming Zhou

Organizations: Key Laboratory of Interdisciplinary Research of Computation and Economics Shanghai University of Finance and Economics

Abstract

Sharpness-Aware Minimization (SAM) enhances generalization by minimizing the maximum training loss within a predefined neighborhood around the parameters. However, its practical implementation approximates this as gradient ascent(s) followed by applying the gradient at the ascent point to update the current parameters. This practice can be justified as approximately optimizing the objective by neglecting the (full) derivative of the ascent point with respect to the current parameters. Nevertheless, a direct and intuitive understanding of why using the gradient at the ascent point to update the current parameters works superiorly is still lacking. Our work bridges this gap by proposing a novel and intuitive interpretation. We show that the gradient at the single-step ascent point, \uline{when applied to the current parameters}, provides a better approximation of the direction from the current parameters toward the maximum within the local neighborhood than the local gradient. This improved approximation thereby enables a more direct escape from the maximum within the local neighborhood. Nevertheless, our analysis further reveals two issues. First, the approximation by the gradient at the single-step ascent point is often inaccurate. Second, the approximation quality may degrade as the number of ascent steps increases. To address these limitations, we propose in this paper eXplicit Sharpness-Aware Minimization (XSAM). It tackles the first by explicitly estimating the direction of the maximum during training, while addressing the second by crafting a search space that effectively leverages the gradient information at the multi-step ascent point. XSAM features a unified formulation that applies to both single-step and multi-step settings and only incurs negligible computational overhead. Extensive experiments demonstrate the consistent superiority of XSAM against existing counterparts.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Visualization of the local loss surface of single-step SAM 2 2 2 Data is collected at the first iteration of the 150th epoch in training ResNet-18 on CIFAR-100. on the hyperplane spanned by the gradient $g_{0}$ at the current parameter $\vartheta_{0}$ and the gradient $g_{1}$ at the single-step ascent point $\vartheta_{1}$ . $\vartheta_{0}$ is set as the origin, the $Y$ -axis is defined along the direction of $g_{0}$ , and the $X$ -axis is aligned with the component of $g_{1}$ perpendicular to $g_{0}$ . The visualized arrows of gradients are set to have length $\rho$ . We see that $\bm{g_{1}\!@\vartheta_{0}}$ (i.e., $\bm{g_{1}}$ applied to $\bm{\vartheta_{0}}$ ) points clearly closer to the direction from $\bm{\vartheta_{0}}$ toward the maximum within the local neighborhood than $\bm{g_{0}}$ . The targeted direction is roughly from the origin to the upper-right corner in the figure. The loss along $g_{1}\!@\vartheta_{0}$ (i.e., $L(\vartheta_{0}+\rho_{m}\cdot{g_{1}}/{\|g_{1}\|})$ ) is higher than that along $g_{0}$ (i.e., $L(\vartheta_{0}+\rho_{m}\cdot{g_{0}}/{\|g_{0}\|})$ ), for sufficiently large $\rho_{m}$ . (b) A simulation of multi-step SAM on a 2D test function. The approximation quality by the SAM gradient may get worse as the number of ascent steps increases . $g_{2}\!@\vartheta_{0}$ inferiorly identifies the direction from $\vartheta_{0}$ toward the maximum within the neighborhood (the upper-left high-loss region in yellow) than $g_{1}\!@\vartheta_{0}$ .

Panel of Figure 3 — Figure 3: (a) Training trajectory comparisons on 2D test function. (b)-(c) Test accuracy comparisons of ResNet-18 trained on CIFAR-100 in single-step and multi-step ( $k=3$ ) settings with varying $\rho$ .

Figure 4: XSAM robustness to the $\alpha^{*}$ update frequency.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Visualization of loss surface during training: VGG-11 trained on CIFAR-100.

Panel of Figure 6 — Figure 6: Visualization of loss surface during training: ResNet-18 trained on CIFAR-100.

Panel of Figure 7 — Figure 7: Visualization of loss surface during training: ViT-Ti trained on CIFAR-100.

Panel of Figure 8 — Figure 8: Visualization of loss surface during training: ResNet-18 trained on CIFAR-10.

Panel of Figure 9 — Figure 9: Visualization of loss surface during training: DenseNet-121 trained on CIFAR-10.

Panel of Figure 10 — Figure 10: (a) Visualization of the local loss surface of single-step SAM. The visualization procedure follows the same steps as in Figure 1(a) . Data is collected at the first iteration of the 100th epoch in training ResNet-18 on CIFAR-100. We see that $\bm{g_{1}\!@\vartheta_{0}}$ (i.e., $\bm{g_{1}}$ applied to $\bm{\vartheta_{0}}$ ) points clearly closer to the direction from $\bm{\vartheta_{0}}$ toward the maximum within the local neighborhood than $\bm{g_{0}}$ . he targeted direction is roughly from the origin to the upper-right corner in the figure. The loss along $g_{1}\!@\vartheta_{0}$ (i.e., $L(\vartheta_{0}+\rho_{m}\cdot{g_{1}}/{\|g_{1}\|})$ ) is higher than that along $g_{0}$ (i.e., $L(\vartheta_{0}+\rho_{m}\cdot{g_{0}}/{\|g_{0}\|})$ ), for sufficiently large $\rho_{m}$ . (b) A simulation of multi-step SAM on a test function. The gradient at the multi-step ascent point, when applied to the current parameters, may be an inferior approximation of the direction toward the maximum .

Panel of Figure 11 — Figure 11: Visualization of $L(\vartheta_{0}+\rho_{m}g_{1}/||g_{1}||)-L(\vartheta_{0}+\rho_{m}g_{0}/||g_{0}||)$ during training.

Panel of Figure 12 — Figure 12: Visualization of $L(\vartheta_{0}+\rho_{m}g_{1}/||g_{1}||)-L(\vartheta_{0}+\rho_{m}g_{0}/||g_{0}||)$ during training.

CIFAR-10

CIFAR-100

Tiny-ImageNet

VGG-11

SGD

SAM

WSAM

XSAM

SGD

SAM

WSAM

XSAM

SGD

SAM

WSAM

XSAM

Epoch

200

Batch size

125

Initial learning rate

0.05

Momentum

0.9

Weight decay

1\times 10^{-3}

1\times 10^{-3}

1\times 10^{-3}

Appendix

Table 5: Hyperparameter details for Results in Table 2 .

	$\rho$ =0.10			$\rho$ =0.20			$\rho$ =0.30
	SAM	WSAM	XSAM	SAM	WSAM	XSAM	SAM	WSAM	XSAM
Epoch	200			200			200
Batch size	125			125			125
Initial learning rate	0.05			0.05			0.05
Momentum	0.9			0.9			0.9
Weight decay	$1\times 10^{-3}$			$1\times 10^{-3}$			$1\times 10^{-3}$

Appendix

Table 6: Hyperparameter details for Results in Figure 3(b) . Note that, in this experiment,

\alpha

for WSAM adopts the average value of the dynamic

\alpha^{*}

in the corresponding XSAM. We see from the results that such WSAM already clearly outperforms SAM.

	$\rho$ =0.04			$\rho$ =0.08			$\rho$ =0.12
	SAM	WSAM	XSAM	SAM	WSAM	XSAM	SAM	WSAM	XSAM
Epoch	200			200			200
Batch size	125			125			125
Initial learning rate	0.05			0.05			0.05
Momentum	0.9			0.9			0.9
Weight decay	$1\times 10^{-3}$			$1\times 10^{-3}$			$1\times 10^{-3}$

Appendix

Table 8: Hyperparameter details for Results in Figure 3(c) . Note that, in this experiment,

\alpha

for WSAM adopts the average value of the dynamic

\alpha^{*}

in the corresponding XSAM. We see from the results that such WSAM already clearly outperforms SAM.

	SAM	ASAM	XSAM	XSAM+ASAM
Test Accuracy	$80.93\pm 0.11$	$81.11\pm 0.06$	$81.24\pm 0.07$	81.68 $\pm$ 0.11

Appendix

Table 10: Test accuracy of SAM variants and their combinations with XSAM.

Method	ResNet-18	DenseNet-121
SAM	80.93 $\pm$ 0.11	83.81 $\pm$ 0.02
ASAM	81.11 $\pm$ 0.06	83.99 $\pm$ 0.25
VaSSO	80.84 $\pm$ 0.15	83.78 $\pm$ 0.25
WSAM	80.95 $\pm$ 0.19	83.91 $\pm$ 0.15
XSAM	81.24 $\pm$ 0.07	83.96 $\pm$ 0.10
XSAM + ASAM	81.68 $\pm$ 0.11	84.06 $\pm$ 0.21

Appendix

Table 11: Comparison on CIFAR-100 with ResNet-18 and DenseNet-121. All baseline methods are carefully tuned for optimal performance. XSAM uses the same

\rho

as SAM, as in the paper.

Method	$\rho=0.04$	$\rho=0.08$	$\rho=0.12$
SAM	80.79 ±0.41	80.75 ±0.27	79.72 ±0.33
LSAM	81.00 ±0.21	81.20 ±0.24	81.16 ±0.04
LSAM+	80.56 ±0.20	80.77 ±0.04	80.21 ±0.27
MSAM	81.04 ±0.06	81.12 ±0.17	80.93 ±0.11
MSAM+	80.72 ±0.16	81.16 ±0.05	81.16 ±0.05
XSAM	81.23 ±0.06	81.36 ±0.08	81.29 ±0.06

Appendix

Table 12: Results on CIFAR-100 using ResNet-18 in multi-step (

k=3

) setting.

Method	$k=1$	$k=2$	$k=4$
LSAM	83.81 $\pm$ 0.02	83.82 $\pm$ 0.28	83.40 $\pm$ 0.17
MSAM	83.81 $\pm$ 0.02	83.67 $\pm$ 0.23	83.74 $\pm$ 0.18
XSAM	83.96 $\pm$ 0.10	84.05 $\pm$ 0.04	84.02 $\pm$ 0.31

Appendix

Table 13: Results on DenseNet-121 with CIFAR-100 with different

k

\rho=\rho^{*}/k

with

\rho^{*}

for single-step.

Method	$k=1$	$k=2$	$k=4$
LSAM	96.59 $\pm$ 0.06	96.66 $\pm$ 0.03	96.72 $\pm$ 0.07
MSAM	96.59 $\pm$ 0.06	96.78 $\pm$ 0.05	96.80 $\pm$ 0.07
XSAM	96.74 $\pm$ 0.04	96.81 $\pm$ 0.06	96.81 $\pm$ 0.11

Appendix

Table 14: Results on ResNet-18 with CIFAR-10 with different

k

\rho=\rho^{*}/k

with

\rho^{*}

for single-step.

Table 15: Performance on CIFAR-10-C/CIFAR-100-C with ResNet-18 and DenseNet-121.

Panel of Figure 13 — Figure 13: More visualizations of the dynamic estimations of $\alpha$ .

Panel of Figure 14 — Figure 14: (a) Robustness analysis of XSAM with respect to $\rho_{m}$ . (b)Training statistics of XSAM. (c) The norms of $g_{i}$ during training.

	SGD	SAM	XSAM
$\lambda_{1}$	78.79	36.15	33.92
$\lambda_{1}/\lambda_{5}$	2.26	1.89	1.59

Appendix

Table 16: Hessian spectrum of ResNet-18 using SGD, SAM, and XSAM on CIFAR-10.

Panel of Figure 15 — Figure 15: Loss landscape visualizations of ResNet-18 on CIFAR-10 with SGD, SAM, and XSAM.

Panel of Figure 16 — Figure 16: Visualization of the average sharpness of the loss landscape at the convergence point.

Panel of Figure 17 — Figure 17: Comparison of various gradient scale strategies.

Sharpness-Aware Minimization

Sharpness-Aware Minimization (SAM) trains models toward locally flat loss regions to improve generalization and robustness.

See all 36 papers

EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbations
Aug 15, 2026Tanapat Ratchatorn, Masayuki TanakaLoss Landscape Sharpness Sharpness-Aware Minimization
Institute of Science Tokyo, Meguro, Tokyo, Japan
Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization
May 11, 2026Jinping Wang, Qinhan Liu, Zhiwu Xie +1Neural Network Generalization Sharpness-Aware Minimization
Wenzhou-Kean University · International Frontier Interdisciplinary Research Institute, Wenzhou-Kean University
Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate
Sep 16, 2024Hinata Harada, Hideaki IidukaSharpness-Aware Minimization Nonconvex Optimization
Meiji University, Japan

Aug 15, 2026J/K move · Enter open · S save

EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbations

Tanapat Ratchatorn, Masayuki Tanaka

Institute of Science Tokyo, Meguro, Tokyo, Japan

Loss Landscape Sharpness Sharpness-Aware Minimization

Open paper AskPDF arXiv

Recent progress in optimization research has highlighted the sharpness of the loss landscape as a key factor in narrowing the generalization gap. Motivated by this insight, Sharpness-Aware Minimization (SAM) was proposed as a training strategy that enhances generalization. Despite the promising performance, SAM suffers from its twice computational cost due to its core algorithm requiring an extra gradient computation during the perturbation step. To overcome this limitation, we introduce Exponential Moving Average Sharpness-Aware Minimization (EMASAM), a computationally efficient variant of SAM. EMASAM does not require the loss gradient in the perturbation step. Instead, EMASAM defines the perturbation direction based on the discrepancy between the main model and the EMA shadow model. This perturbation travels away from the stable average position toward the less stable area, acting as a softer yet cheaper alternative to SAM's worst-case scenario perturbation. Moreover, since EMASAM's perturbation does not rely on noisy mini-batch gradients, it mitigates the gradient-induced instability inherent in SAM. Hence, EMASAM eliminates the need for an extra backpropagation while also preserving the generalization ability of the SAM-style training. Several experiments have been performed and confirm the efficiency and robustness of our method.

Abstract

Sharpness-Aware Minimization

Explore similar work

EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbations

Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization

Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate