ZO-SAM: Zero-Order Sharpness-Aware Minimization for Efficient Sparse Training

Jie Ji; Gen Li; Kaiyuan Deng; Fatemeh Afghah; Xiaolong Ma

cs.LGMar 13, 2026

ZO-SAM: Zero-Order Sharpness-Aware Minimization for Efficient Sparse Training

Authors: Jie Ji, Gen Li, Kaiyuan Deng, Fatemeh Afghah, Xiaolong Ma

Organizations: Clemson University · University of Arizona

Abstract

Deep learning models, despite their impressive achievements, suffer from high computational costs and memory requirements, limiting their usability in resource-constrained environments. Sparse neural networks significantly alleviate these constraints by dramatically reducing parameter count and computational overhead. However, existing sparse training methods often experience chaotic and noisy gradient signals, severely hindering convergence and generalization performance, particularly at high sparsity levels. To tackle this critical challenge, we propose Zero-Order Sharpness-Aware Minimization (ZO-SAM), a novel optimization framework that strategically integrates zero-order optimization within the SAM approach. Unlike traditional SAM, ZO-SAM requires only a single backpropagation step during perturbation, selectively utilizing zero-order gradient estimations. This innovative approach reduces the backpropagation computational cost by half compared to conventional SAM, significantly lowering gradient variance and effectively eliminating associated computational overhead. By harnessing SAM's capacity for identifying flat minima, ZO-SAM stabilizes the training process and accelerates convergence. These efficiency gains are particularly important in sparse training scenarios, where computational cost is the primary bottleneck that limits the practicality of SAM. Moreover, models trained with ZO-SAM exhibit improved robustness under distribution shift, further broadening its practicality in real-world deployments.

Figures & tables

Figure 1: (a) Comparison of Gradient Variance during sparse training at sparsity 90% and 95% between SGD and ZO-SAM; (b) Loss landscape of dense training, sparse training at sparsity 90% and 95%.

Figure 2: Illustration of the optimization mechanism of ZO-SAM.

Datasets	CIFAR-10			CIFAR-100
Pruning ratio	90%	95%	98%	90%	95%	98%
ResNet-32	94.58 (Dense)			74.89 (Dense)
LT [ 14 ]	92.31	91.06	88.78	68.99	65.02	57.37
LT+ ZO-SAM (ours)	92.69 $\pm$ 0.05 (0.38 $\uparrow$ )	91.67 $\pm$ 0.10 (0.61 $\uparrow$ )	89.46 $\pm$ 0.15 (0.68 $\uparrow$ )	69.44 $\pm$ 0.06 (0.45 $\uparrow$ )	65.59 $\pm$ 0.13 (0.57 $\uparrow$ )	57.87 $\pm$ 0.14 (0.50 $\uparrow$ )
SNIP [ 22 ]	92.59 $\pm$ 0.10	91.01 $\pm$ 0.21	87.51 $\pm$ 0.31	68.89 $\pm$ 0.45	65.02 $\pm$ 0.69	57.37 $\pm$ 1.43
SNIP+ ZO-SAM (ours)	93.38 $\pm$ 0.12 (0.79 $\uparrow$ )	91.69 $\pm$ 0.17 (0.68 $\uparrow$ )	88.22 $\pm$ 0.21 (0.71 $\uparrow$ )	69.54 $\pm$ 0.23 (0.65 $\uparrow$ )	65.90 $\pm$ 0.49 (0.88 $\uparrow$ )	58.29 $\pm$ 0.47 (0.92 $\uparrow$ )

Table 1: Test accuracy (%) of pruned ResNet-32 on CIFAR-10/100.

Figure 3: Overview of perturbation step in ZO-SAM.

Figure 4: Loss surface improvement at various sparsity levels.

Model	DeiT-Small	DeiT-Tiny
Dense	79.90	72.20
Pruning ratio	70%	50%
RigL [ 11 ]	77.99	70.02
RigL+ZO-SAM	79.16 (1.17 $\uparrow$ )	70.79 (0.77 $\uparrow$ )
MEST [ 42 ]	77.15	69.69
MEST+ZO-SAM	78.04 (0.89 $\uparrow$ )	70.41 (0.72 $\uparrow$ )

Table 2: Test accuracy (%) of pruned DeiT on ImageNet-1K

Figure 5: Convergence Speed Comparison at sparsity level 90%.

Figure 6: Convergence Speed Comparison at sparsity level 95%.

CIFAR-10, MEST, Acc = $90\%$	SGD	ESAM	LookSAM(k=5)	GSAM	ZO-SAM
sp=0.9, Epoch#	104	75	79	84	70
sp=0.95, Epoch#	131	92	94	113	88

Table 3: Epochs to reach 90% accuracy on CIFAR-10 (lower is better).

Method	Sparsity	ResNet-32			WRN-28-10
Method	Sparsity	CIFAR-10	CIFAR-100	Image/sec	CIFAR-10	CIFAR-100	Image/sec
SGD	-	94.58	74.89	5673.95	96.41	81.63	752.30
SAM	90%	93.77	72.64	2704.84 (47.67%)	95.37	80.14	354.94 (47.18%)
ESAM [ 9 ]	90%	93.67	72.49	2297.23 (40.48%)	95.31	80.23	305.76 (40.64%)
LS(k=5) [ 25 ]	90%	93.42	72.01	3980.62 (70.14%)	94.89	79.30	542.63 (72.13%)
LS(k=10) [ 25 ]	90%	93.03	71.62	4272.22 (75.29%)	94.17	78.84	593.02 (78.83%)

Table 4: Test accuracy (%) and computation efficiency (image/sec) of ResNet-32 and WideResNet-28-10 under 90% sparsity levels on CIFAR-10 and CIFAR-100. Comparison includes SGD, original SAM, efficient SAM variants, and our proposed methods.

Figure 7: Comparison of feature maps for different layers between Original DST and ZO-SAM under sparsity 90%.

Methods	CIFAR-10 (%)	CIFAR-10-C (%)	$\Delta\downarrow$
SNIP [ 22 ]	92.59	59.60	32.99
SNIP + ESAM [ 9 ]	93.40 (0.81 $\uparrow$ )	61.19 (1.59 $\uparrow$ )	32.21
SNIP + LS(k=5) [ 25 ]	93.31 (0.72 $\uparrow$ )	60.41 (0.81 $\uparrow$ )	32.90
SNIP + LS(k=10) [ 25 ]	92.97 (0.38 $\uparrow$ )	60.08 (0.48 $\uparrow$ )	32.89
SNIP + GSAM [ 48 ]	93.71 (1.12 $\uparrow$ )	61.65 (2.05 $\uparrow$ )	32.1
SNIP + ZO-SAM	93.38 (0.79 $\uparrow$ )	62.70 (3.10 $\uparrow$ )	30.68

Table 5: CIFAR-10 vs. CIFAR-10-C under 90% sparsity with SNIP.

Datasets	CIFAR-10			CIFAR-100
Pruning ratio	90%	95%	98%	90%	95%	98%
ResNet-50	95.23 (Dense)			78.42 (Dense)
SNIP [ 22 ]	92.65 $\pm$ 0.12	90.86 $\pm$ 0.17	87.02 $\pm$ 0.35	73.14 $\pm$ 0.32	69.25 $\pm$ 0.37	62.49 $\pm$ 0.32
SNIP+ ZO-SAM (ours)	92.98 $\pm$ 0.09 (0.33 $\uparrow$ )	91.47 $\pm$ 0.12 (0.61 $\uparrow$ )	87.92 $\pm$ 0.14 (0.90 $\uparrow$ )	73.64 $\pm$ 0.23 (0.50 $\uparrow$ )	69.97 $\pm$ 0.49 (0.72 $\uparrow$ )	63.79 $\pm$ 0.47 (1.30 $\uparrow$ )
GraSP [ 40 ]	92.47 $\pm$ 0.17	91.32 $\pm$ 0.15	88.77 $\pm$ 0.24	73.28 $\pm$ 0.21	70.29 $\pm$ 0.13	62.12 $\pm$ 0.33
GraSP+ ZO-SAM (ours)	92.89 $\pm$ 0.17 (0.42 $\uparrow$ )	91.85 $\pm$ 0.12 (0.53 $\uparrow$ )	89.47 $\pm$ 0.11 (0.70 $\uparrow$ )	73.79 $\pm$ 0.16 (0.51 $\uparrow$ )	70.98 $\pm$ 0.24 (0.69 $\uparrow$ )	63.27 $\pm$ 0.20 (1.15 $\uparrow$ )

6.Table: Test accuracy (%) of pruned ResNet-50 on CIFAR-10/100.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

\theta_{t}\leftarrow\theta_{t}\odot M_{t}\quad(\text{elementwise product}).

Appendix

Algorithm 1 Sparse Zero-Order SAM (Sparse ZO-SAM)

Sharpness-Aware Minimization

Sharpness-Aware Minimization (SAM) trains models toward locally flat loss regions to improve generalization and robustness.

See all 39 papers

EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbations
Aug 15, 2026Tanapat Ratchatorn, Masayuki TanakaLoss Landscape Sharpness Sharpness-Aware Minimization
Institute of Science Tokyo, Meguro, Tokyo, Japan
Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning
Mar 14, 2025Junhyuk Jo, Jihyun Lim, Vincent-Daniel Yun +1Deep Learning Optimization Sharpness-Aware Minimization
Inha University
Sample-Adaptive Zeroth-Order Post-Training
Oct 8, 2026Haiyuan Wan, Yanyi Li, Weicheng Lin +1Zeroth-Order Optimization LLM Fine-Tuning
School of Physics, Peking University · School of Intelligence Science and Technology, Peking University

Aug 15, 2026J/K move · Enter open · S save

EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbations

Tanapat Ratchatorn, Masayuki Tanaka

Institute of Science Tokyo, Meguro, Tokyo, Japan

Loss Landscape Sharpness Sharpness-Aware Minimization

Open paper AskPDF arXiv

Recent progress in optimization research has highlighted the sharpness of the loss landscape as a key factor in narrowing the generalization gap. Motivated by this insight, Sharpness-Aware Minimization (SAM) was proposed as a training strategy that enhances generalization. Despite the promising performance, SAM suffers from its twice computational cost due to its core algorithm requiring an extra gradient computation during the perturbation step. To overcome this limitation, we introduce Exponential Moving Average Sharpness-Aware Minimization (EMASAM), a computationally efficient variant of SAM. EMASAM does not require the loss gradient in the perturbation step. Instead, EMASAM defines the perturbation direction based on the discrepancy between the main model and the EMA shadow model. This perturbation travels away from the stable average position toward the less stable area, acting as a softer yet cheaper alternative to SAM's worst-case scenario perturbation. Moreover, since EMASAM's perturbation does not rely on noisy mini-batch gradients, it mitigates the gradient-induced instability inherent in SAM. Hence, EMASAM eliminates the need for an extra backpropagation while also preserving the generalization ability of the SAM-style training. Several experiments have been performed and confirm the efficiency and robustness of our method.

Abstract

Sharpness-Aware Minimization

Explore similar work

EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbations

Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning

Sample-Adaptive Zeroth-Order Post-Training