Sparsity-Adaptive Sharpness-Aware Minimization

Shiryu Ueno; Yoshikazu Hayashi; Kunihito Kato

cs.CVSep 13, 2026

Sparsity-Adaptive Sharpness-Aware Minimization

Authors: Shiryu Ueno, Yoshikazu Hayashi, Kunihito Kato

Organizations: Faculty of Engineering Gifu University Gifu, Japan

Abstract

Deploying deep neural networks in real-world settings requires models that are both compact and robust to common corruptions. However, at deployment-relevant high sparsity, standard pruning pipelines often degrade corruption robustness, and existing sharpness-aware training/pruning approaches provide limited robustness gains. We address this issue by introducing Sparsity-Adaptive Sharpness-Aware Minimization (SA-SAM), which derives a sparsity-dependent SAM/ASAM perturbation radius by keeping the mean absolute perturbation (an $\ell_1$ -based proxy) approximately invariant as sparsity increases. As a simple complementary option, we evaluate Magnitude-Weighted Hessian (MWH), derived from a second-order removal-path analysis, yielding an importance proportional to $\mathrm{Diag}(F)_i\,|w_i|$ , where $\mathrm{Diag}(F)$ is the diagonal empirical Fisher used as a curvature proxy in our implementation. Across CIFAR-10-C, CIFAR-100-C, and ImageNet-100-C, our approach achieved stronger corruption robustness than the considered pruning baselines at 80--90% sparsity, while preserving clean accuracy. We additionally quantify the robustness--throughput trade-off by reporting measured inference throughput under sparse execution at deployment-relevant sparsity levels.

Figures & tables

Fig. 1: Throughput vs. robustness on CIFAR-10-C. The x-axis shows throughput (samples/sec) and the y-axis shows mean Top-1 accuracy (%) on CIFAR-10-C averaged over all corruption types and severities. Throughput is averaged over 10 runs (full passes over the test set). Our method achieves higher corruption robustness than HAP and AdaSAP at comparable throughput (see legend). Note that at a fixed global sparsity, measured throughput can vary across methods due to differences in layer-wise nonzero distributions induced by different pruning criteria.

Method@ $s$	CIFAR-10	CIFAR-10-C	CIFAR-100	CIFAR-100-C	Throughput
[email protected]	96.12 $\pm$ 0.03	78.19 $\pm$ 0.46	79.93 $\pm$ 0.05	53.37 $\pm$ 0.44	200 $\pm$ 0
[email protected]	93.86 $\pm$ 0.10	75.82 $\pm$ 0.54	76.50 $\pm$ 0.30	48.61 $\pm$ 0.41	446 $\pm$ 11
[email protected]	95.29 $\pm$ 0.12	75.08 $\pm$ 0.80	78.46 $\pm$ 0.21	48.97 $\pm$ 0.66	316 $\pm$ 6
[email protected]	93.59 $\pm$ 0.16	72.38 $\pm$ 0.96	72.72 $\pm$ 0.13	45.64 $\pm$ 0.71	689 $\pm$ 12
[email protected]	96.00 $\pm$ 0.07	77.81 $\pm$ 0.53	79.49 $\pm$ 0.05	52.63 $\pm$ 0.53	535 $\pm$ 7
[email protected]	93.64 $\pm$ 0.03	75.50 $\pm$ 0.53	75.19 $\pm$ 0.37	46.32 $\pm$ 0.57	1310 $\pm$ 90

TABLE I: CIFAR-10 / CIFAR-100. Top-1 (%) on the clean test sets and on the corruption sets (mean over 19 corruption types

\times

5 severities). Values are mean

\pm

std over 3 seeds. Dense denotes the baseline with sparsity

s{=}0.0

. The rightmost column reports inference throughput (samples/sec). Ours uses SA-SAM during pruning retraining and MWH for mask updates.

Method@ $s$	ImageNet-100	ImageNet-100-C	Throughput
[email protected]	82.91 $\pm$ 0.30	44.58 $\pm$ 0.69	173 $\pm$ 3
[email protected]	81.81 $\pm$ 0.42	40.86 $\pm$ 1.18	365 $\pm$ 3
[email protected]	81.27 $\pm$ 0.53	42.13 $\pm$ 0.53	381 $\pm$ 14
[email protected]	77.51 $\pm$ 0.57	35.74 $\pm$ 0.82	426 $\pm$ 17
[email protected]	83.14 $\pm$ 0.06	44.45 $\pm$ 0.50	387 $\pm$ 15
[email protected]	80.93 $\pm$ 0.56	39.59 $\pm$ 1.47	420 $\pm$ 17

TABLE II: ImageNet-100. Top-1 (%) on the clean test set and the corruption set.

Fig. 2: Layer-wise remaining parameter ratios at a fixed global sparsity ( ResNet-18 on CIFAR-10 at $s=0.9$ ).

Fig. 3: Fourier heat maps on CIFAR-10 for (a) Dense, (b) OBD, (c) HAP, (d) AdaSAP, and (e) Ours. Methods (b)-(e) used sparsity $s=0.9$ . Our method shows smaller degradation in the low-frequency region compared to other sparse baselines.

Method	Low↓	High↓	Average↓
OBD	9.45	3.80	5.02
HAP	10.45	3.52	5.01
AdaSAP	12.46	15.29	14.68
Ours	1.94	0.13	0.52

TABLE III: Frequency-wise performance degradation for the sparse models (

s=0.9

). Each value is the degradation rate defined in Eq. ( 17 ), lower is better.

Fig. 4: Loss landscapes on CIFAR-10 for (a) Dense, (b) OBD, (c) HAP, (d) AdaSAP, and (e) Ours. Methods (b)-(e) use sparsity $s=0.9$ . Each surface is obtained by applying perturbations along two random directions in weight space and plotting the resulting error rate.

Fig. 5: ESDs ( $\mathbf{\lambda}$ =100) on CIFAR-10 for (a) Dense, (b) OBD, (c) HAP, (d) AdaSAP, and (e) Ours. Methods (b)-(e) use sparsity $s=0.9$ . The x-axis shows eigenvalues and the y-axis shows density. The $\ell_{2}$ energy of the top-100 eigenvalues, $\bigl(\sum_{i=1}^{100}\lambda_{i}^{2}\bigr)^{1/2}$ , is reported in the top-right corner of each plot.

		ImageNet-100		ImageNet-100-C
Sparsity	Method	ASAM	SA-SAM	ASAM	SA-SAM
0.00	Dense	82.91 $\pm$ 0.30	82.91 $\pm$ 0.30	44.58 $\pm$ 0.69	44.58 $\pm$ 0.69
0.80	OBD	$81.81\pm 0.42$	$81.37\pm 0.31$ ( $\downarrow 0.44$ )	$40.86\pm 1.18$	$40.34\pm 1.13$ ( $\downarrow 0.52$ )
	HAP	$81.27\pm 0.53$	$82.08\pm 0.46$ ( $\uparrow 0.81$ )	$42.13\pm 0.53$	$42.32\pm 0.62$ ( $\uparrow 0.19$ )
	AdaSAP	$77.51\pm 0.57$	$77.82\pm 0.17$ ( $\uparrow 0.31$ )	$35.74\pm 0.82$	$36.29\pm 0.73$ ( $\uparrow 0.55$ )
	MWH (Ours)	$82.85\pm 0.42$	83.14 $\pm$ 0.06 ( $\uparrow 0.29$ )	$44.04\pm 0.57$	44.45 $\pm$ 0.50 ( $\uparrow 0.41$ )

TABLE IV: Comparison of ASAM with SA-SAM for ImageNet-100. Top-1 (%) on the clean test set and ImageNet-100-C (averaged over 15 corruption types

\times

5 severities). Throughput is omitted because all settings use the same target sparsity and unstructured masks, and throughput differences across optimizers are negligible in this ablation.

Fig. 6: Perturbation magnitude ratio relative to the dense model across different pruning methods. The gray dashed line represents the dense model.

Sparse Neural Networks

Sparse neural networks use a subset of weights, connections, activations, or experts to reduce model complexity and computation.

See all 63 papers

One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM
Oct 14, 2026Alessia Rossi, Felipe Campelo, Gabriel L. OliveiraCurvature-Aware Optimization Sharpness-Aware Minimization
School of Engineering Mathematics and Technology, University of Bristol, Bristol, Lovelace Building, Ada, Bristol, BS8 1TW, UK
Pruning Deep Neural Networks via the Marchenko--Pastur Distribution
May 23, 2026Leonid Berlyand, Theo Bourdais, Houman Owhadi +1Structured Pruning Sparse Neural Networks
Department of Mathematics Pennsylvania State University University Park, PA 16802, USA · Department of Computing and Mathematical Sciences California Institute of Technology Pasadena, CA 91125, USA
Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm
Jul 28, 2026Wenzhi Zhong, Edward Milsom, Michael MurraySharpness-Aware Minimization Muon Optimizer
Department of Mathematical Sciences University of Bath · Department of Computer Science University of Bath

Oct 14, 2026J/K move · Enter open · S save

One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM

Alessia Rossi, Felipe Campelo, Gabriel L. Oliveira

School of Engineering Mathematics and Technology, University of Bristol, Bristol, Lovelace Building, Ada, Bristol, BS8 1TW, UK

Curvature-Aware Optimization Sharpness-Aware Minimization Neural Network Optimization Neural Network Generalization

Open paper AskPDF arXiv

Poor generalisation in deep neural networks is frequently associated with convergence to sharp minima. Sharpness-Aware Minimisation (SAM) improves generalisation of deep neural networks by minimising a combination of the training loss and the local sharpness of the loss landscape, biasing convergence towards flat minima. However, SAM relies on a fixed perturbation radius

ρ

that ignores local curvature, leading to inefficient behaviour: the perturbation may be too small to escape poor basins defined by sharp local minima, or too large to allow convergence to solutions with good loss-sharpness tradeoff. We reinterpret

ρ

as the intensity of the perturbation signal and propose Curvature-Aware Adaptive Perturbation for SAM (CAP-SAM), a geometry-aware method that modulates the perturbation radius based on a sharpness ratio, enabling a more effective search for robust minima on the loss landscape. We prove that CAP-SAM preserves the same convergence rate as standard SAM in non-convex stochastic settings. Extensive experiments indicate that our approach consistently outperforms SAM and its most prominent variants across diverse network architectures on CIFAR-10, CIFAR-100, and ImageNet.

Abstract

Sparse Neural Networks

Explore similar work

One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm