Bilateral Sharpness-Aware Minimization for Flatter Minima

Jiaxin Deng; Junbiao Pang; Baochang Zhang; Qingming Huang

cs.CVSep 20, 2024

Bilateral Sharpness-Aware Minimization for Flatter Minima

Authors: Jiaxin Deng, Junbiao Pang, Baochang Zhang, Qingming Huang

Organizations: Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China · School of Artificial Intelligence, BeiHang University, Beijing, 100191, China · University of Chinese Academy of Sciences, Chinese Academy of Sciences (CAS), Beijing 100049, China · Institute of Computing Technology, CAS, Beijing 100190, China

Abstract

Sharpness-Aware Minimization (SAM) enhances generalization by reducing a Max-Sharpness (MaxS). Despite the practical success, we empirically found that the MAxS behind SAM's generalization enhancements face the "Flatness Indicator Problem" (FIP), where SAM only considers the flatness in the direction of gradient ascent, resulting in a next minimization region that is not sufficiently flat. A better Flatness Indicator (FI) would bring a better generalization of neural networks. Because SAM is a greedy search method in nature. In this paper, we propose to utilize the difference between the training loss and the minimum loss over the neighborhood surrounding the current weight, which we denote as Min-Sharpness (MinS). By merging MaxS and MinS, we created a better FI that indicates a flatter direction during the optimization. Specially, we combine this FI with SAM into the proposed Bilateral SAM (BSAM) which finds a more flatter minimum than that of SAM. The theoretical analysis proves that BSAM converges to local minima. Extensive experiments demonstrate that BSAM offers superior generalization performance and robustness compared to vanilla SAM across various tasks, i.e., classification, transfer learning, human pose estimation, and network quantization.

Figures & tables

Panel of Figure 1 — Fig. 1: Illustration of the notations of the MaxS, MinS and BilS.

Panel of Figure 2 — Fig. 2: The occurrence of gradient conflict under different $\rho^{min}$ and the variation of $\rho^{min}$ with learning rate in BSAM.

Panel of Figure 3 — Fig. 3: The cosine similarity between the gradients at the point $\mathbf{w}$ and the point $\mathbf{w}+{\hat{\bm{\varepsilon}}}^{\min}$ in training stage under the different settings of ${\rho}^{min}$ .

Fig. 4: The loss curves and accuracy ones of SAM and BSAM when WideResNet-28-10 was used.

$\rho^{min}$	CIFAR-10	CIFAR-100
0.01	96.75 $\pm$ 0.09	81.56 $\pm$ 0.05
0.025	96.78 $\pm$ 0.02	81.29 $\pm$ 0.11
0.05	96.82 $\pm$ 0.12	81.65 $\pm$ 0.04
0.1	96.66 $\pm$ 0.13	81.48 $\pm$ 0.18
0.25	96.18 $\pm$ 0.09	81.58 $\pm$ 0.25

TABLE I: Test accuracy (%) with different

\rho^{min}

on CIFAR-10 and CIFAR-100.

ResNet-18	CIFAR-10	CIFAR-100
SGD	96.18 $\pm$ 0.09	79.89 $\pm$ 0.38
SAM	96.74 $\pm$ 0.05	81.08 $\pm$ 0.27
ASAM	96.63 $\pm$ 0.15	81.68 $\pm$ 0.12
FisherSAM	96.72 $\pm$ 0.03	80.99 $\pm$ 0.13
F-SAM	96.75 $\pm$ 0.09	81.29 $\pm$ 0.12
SGD-SALR	/	81.73 $\pm$ 0.10

TABLE II: Test accuracy (%) comparison of various networks on CIFAR-10 and CIFAR-100.

Panel of Figure 7 — Fig. 5: The distribution of top-50 eigenvalues of Hessian on the test set of CIFAR-100 with SGD, SAM and BSAM.

Methods	Accuracy	Training Time
SGD	76.00	63.5
SAM	76.70	100.2
SSAM-F	77.31	/
SSAM-D	77.25	/
BSAM	77.36	148.1

TABLE III: Test accuracy (%) and training time (hour) of ResNet-50 with SGD, SAM and BSAM on the ImageNet.

Datasets	Methods	Noise rate
Datasets	Methods	20%	40%	60%	80%
CIFAR-10	SGD	85.94	70.25	48.50	28.91
	SAM	90.44	79.71	67.22	76.94
	BSAM	93.34	85.58	76.37	78.27
CIFAR-100	SGD	65.42	48.91	31.46	12.32
	SAM	69.10	55.06	35.59	9.82

TABLE IV: Results of ResNet-18 on CIFAR-10 and CIFAR-100 with different rate of label noises.

Panel of Figure 10 — Fig. 6: Test accuracy of optimizing ResNet-18 on CIFAR-100 using SGD, SAM and BSAM when $\rho_{max}$ is 0.05 and 0.1.

Methods	Head	Shoulder	Elbow	Wrist	Hip	Knee	Ankle	Mean
SimCC+Adam	96.828	95.771	89.654	84.272	88.679	85.151	81.436	89.318
SimCC+SAM	96.965	95.703	89.637	84.222	88.402	85.513	81.862	89.388
SimCC+BSAM	96.794	95.584	89.995	85.317	89.008	85.392	81.082	89.552

TABLE V: Results of training SimCC on MP

\mathrm{II}

with Adam, SAM and BSAM.

Efficientnet_b0	SGD	SAM	BSAM
CIFAR-10	97.32 $\pm$ 0.07	97.48 $\pm$ 0.07	97.52 $\pm$ 0.04
CIFAR-100	87.05 $\pm$ 0.11	87.27 $\pm$ 0.21	87.40 $\pm$ 0.21
Flowers102	77.11 $\pm$ 0.36	78.04 $\pm$ 0.37	79.02 $\pm$ 1.10
Standford Cars	74.47 $\pm$ 0.55	75.75 $\pm$ 0.41	77.54 $\pm$ 0.62
OxfordIIITPet	87.89 $\pm$ 0.43	88.30 $\pm$ 0.16	89.42 $\pm$ 0.25
ResNet-50	SGD	SAM	BSAM

TABLE VI: Test accuracy (%) results of fine-tuning on different datasets.

Methods	PSPNet	DeepLabv3+
SGD	78.70 $\pm$ 0.28	77.12 $\pm$ 0.28
SAM	79.10 $\pm$ 0.14	77.25 $\pm$ 0.16
BSAM	79.20 $\pm$ 0.14	77.96 $\pm$ 0.39

TABLE VII: Mean IoU (%) of semantic segmentation with SGD, SAM and BSAM on the VOC 2012.

Methods	ResNet-18	MobileNetV1
Full prec.	88.72	85.81
SGD	88.86 $\pm$ 0.18	84.04 $\pm$ 0.13
SAM	89.75 $\pm$ 0.21	84.72 $\pm$ 0.11
BSAM	89.84 $\pm$ 0.11	84.76 $\pm$ 0.10

TABLE VIII: Test accuracy (%) results of QAT with SGD, SAM and BSAM on the Cifar-10.

Sharpness-Aware Minimization

Sharpness-Aware Minimization (SAM) trains models toward locally flat loss regions to improve generalization and robustness.

See all 40 papers

Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation
Mar 9, 2026Jianlong Chen, Zhiming ZhouSharpness-Aware Minimization Deep Learning Optimization
Key Laboratory of Interdisciplinary Research of Computation and Economics Shanghai University of Finance and Economics
Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization
May 11, 2026Jinping Wang, Qinhan Liu, Zhiwu Xie +1Neural Network Generalization Sharpness-Aware Minimization
Wenzhou-Kean University · International Frontier Interdisciplinary Research Institute, Wenzhou-Kean University
On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds
Aug 4, 2026Jiaxin Deng, Junbiao PangSharpness-Aware Minimization Flat Minima
Beijing University of Technology

Mar 9, 2026J/K move · Enter open · S save

Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation

Jianlong Chen, Zhiming Zhou

Key Laboratory of Interdisciplinary Research of Computation and Economics Shanghai University of Finance and Economics

Sharpness-Aware Minimization Deep Learning Optimization Loss Landscape Sharpness Min-Max Optimization

Open paper AskPDF arXiv

Sharpness-Aware Minimization (SAM) enhances generalization by minimizing the maximum training loss within a predefined neighborhood around the parameters. However, its practical implementation approximates this as gradient ascent(s) followed by applying the gradient at the ascent point to update the current parameters. This practice can be justified as approximately optimizing the objective by neglecting the (full) derivative of the ascent point with respect to the current parameters. Nevertheless, a direct and intuitive understanding of why using the gradient at the ascent point to update the current parameters works superiorly is still lacking. Our work bridges this gap by proposing a novel and intuitive interpretation. We show that the gradient at the single-step ascent point, \uline{when applied to the current parameters}, provides a better approximation of the direction from the current parameters toward the maximum within the local neighborhood than the local gradient. This improved approximation thereby enables a more direct escape from the maximum within the local neighborhood. Nevertheless, our analysis further reveals two issues. First, the approximation by the gradient at the single-step ascent point is often inaccurate. Second, the approximation quality may degrade as the number of ascent steps increases. To address these limitations, we propose in this paper eXplicit Sharpness-Aware Minimization (XSAM). It tackles the first by explicitly estimating the direction of the maximum during training, while addressing the second by crafting a search space that effectively leverages the gradient information at the multi-step ascent point. XSAM features a unified formulation that applies to both single-step and multi-step settings and only incurs negligible computational overhead. Extensive experiments demonstrate the consistent superiority of XSAM against existing counterparts.

Abstract

Sharpness-Aware Minimization

Explore similar work

Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation

Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization

On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds