cs.LGMar 9, 2026

Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization

Authors: Chaewon Moon, Dongkuk Si, Chulhee Yun

Organizations: Kim Jaechul Graduate School of AI, KAIST · Mobilint, Inc.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study the implicit bias of Sharpness-Aware Minimization (SAM) when training LL-layer linear diagonal networks on linearly separable binary classification. For linear models (L=1L=1), both ℓ∞\ell_\infty- and ℓ2\ell_2-SAM recover the ℓ2\ell_2 max-margin classifier, matching gradient descent (GD). However, for depth L=2L = 2, the behavior changes drastically -- even on a single-example dataset. For ℓ∞\ell_\infty-SAM, the limit direction depends critically on initialization and can converge to 0\mathbf{0} or to any standard basis vector, in stark contrast to GD, whose limit aligns with the basis vector of the dominant data coordinate. For ℓ2\ell_2-SAM, we show that although its limit direction matches the ℓ1\ell_1 max-margin solution as in the case of GD, its finite-time dynamics exhibit a phenomenon we call "sequential feature amplification", in which the predictor initially relies on minor coordinates and gradually shifts to larger ones as training proceeds or initialization increases. Our theoretical analysis attributes this phenomenon to ℓ2\ell_2-SAM's gradient normalization factor applied in its perturbation, which amplifies minor coordinates early and allows major ones to dominate later, giving a concrete example where infinite-time implicit-bias analyses are insufficient. Synthetic and real-data experiments corroborate our findings.

Figures & tables

Figure 1: Trajectories of the predictor β(t)∈R2{\bm{\beta}}(t)\in\mathbb{R}^{2} from identical initial conditions under discrete GD, ℓ∞\ell_{\infty} -SAM and ℓ2\ell_{2} -SAM on {(μ,+1)}\{({\bm{\mu}},+1)\} with μ=(1,2){\bm{\mu}}=(1,2) . We used η=0.3\eta=0.3 and ρ=1\rho=1 for SAM.
Figure 2: Trajectories β(t)\beta(t) from identical initializations under GF and ℓ∞\ell_{\infty} -SAM flow with d=2d=2 and μ=(1,2){\bm{\mu}}=(1,2) . For SAM, ρ=1\rho=1 .
Figure 3: Rescaled ℓ2\ell_{2} -SAM flow on Dμ\mathcal{D}_{\bm{\mu}} with μ=(4,5,6,7,8)∈R5{\bm{\mu}}=(4,5,6,7,8)\in\mathbb{R}^{5} and ρ=1\rho=1 .
Figure 4: Loss curves of GD (left) and ℓ2\ell_{2} -SAM (right) on a 2-layer diagonal network in Regime 2 ( α=0.35\alpha=0.35 , μ=(1,2,3,4,5,6)\mu=(1,2,3,4,5,6) , ρ=0.1\rho=0.1 ). Colored regions mark the coordinate with highest growth.
Figure 5: Grad-CAM comparison of GD and ℓ2\ell_{2} -SAM on a CNN trained on MNIST. GD focuses on dominant digit pixels, whereas ℓ2\ell_{2} -SAM highlights minor background regions.
Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Trajectories of the predictor β(t)∈R2{\bm{\beta}}(t)\in\mathbb{R}^{2} from identical initial conditions under GF, ℓ∞\ell_{\infty} -SAM flow and ℓ2\ell_{2} -SAM flow on {(μ,+1)}\{({\bm{\mu}},+1)\} with μ=(1,2){\bm{\mu}}=(1,2) . For SAM, ρ=1\rho=1 .
Figure 7: Dominant index j†:=arg⁡max⁡jβj(t)j^{\dagger}:=\arg\max_{j}\beta_{j}(t) for GF and ℓ∞\ell_{\infty} -SAM flow over (t,α)(t,\alpha) on Dμ\mathcal{D}_{\bm{\mu}} with μ=(4,5,6,7,8)∈R5{\bm{\mu}}=(4,5,6,7,8)\in\mathbb{R}^{5} .
Figure 8: Dominant index for ℓ2\ell_{2} -SAM flow with μ=(1,2,…,12){\bm{\mu}}=(1,2,\dots,12) . The black line indicates the interesting trajectory.
Figure 9: βj(t)\beta_{j}(t) and normalized βj(t)\beta_{j}(t) trajectory of the original ℓ∞\ell_{\infty} -SAM flow.
Figure 10: βj(t)\beta_{j}(t) and normalized βj(t)\beta_{j}(t) trajectory of the rescaled ℓ∞\ell_{\infty} -SAM flow.
Figure 11: Trajectories β(t){\bm{\beta}}(t) under GF, ℓ∞\ell_{\infty} -SAM flow (black line), GD, and discrete ℓ∞\ell_{\infty} -SAM updates (blue dots) for d=2d=2 and μ=(1,2){\bm{\mu}}=(1,2) . For SAM, we set ρ=1\rho=1 . For GD and discrete ℓ∞\ell_{\infty} -SAM, we use step size η=0.1\eta=0.1 .
Figure 12: A randomly generated linearly separable dataset used in our multi-point experiments. We sample two Gaussian clusters centered at ±μ=±(1,2)\pm{\bm{\mu}}=\pm(1,2) with isotropic noise ( ε∼N(0,0.52I2){\bm{\varepsilon}}\sim\mathcal{N}(0,0.5^{2}{\bm{I}}_{2}) ) and assign labels +1+1 and −1-1 accordingly. This dataset is used to evaluate whether the one-point phenomena from Theorems 3.2 and 3.5 persist in the multi-point regime.
Figure 13: Trajectories β(t){\bm{\beta}}(t) under GF, ℓ∞\ell_{\infty} -SAM flow (black line), GD, and discrete ℓ∞\ell_{\infty} -SAM updates (blue dots) for d=2d=2 on random multi-point dataset in Figure 12 . For SAM, we set ρ=1\rho=1 . For GD and discrete ℓ∞\ell_{\infty} -SAM, we use step size η=0.1\eta=0.1 .
Figure 14: Numerical evaluation of LBj(α)\mathrm{LB}_{j}(\alpha) for various choices of μ{\bm{\mu}} .
Figure 15: Dominant index j†j^{\dagger} over α,t\alpha,t and logistic loss on Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(4,5,6,7,8){\bm{\mu}}=(4,5,6,7,8) .
Figure 16: Dominant index j†j^{\dagger} over α,t\alpha,t and logistic loss on Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(4,5,6,7,8){\bm{\mu}}=(4,5,6,7,8) and ρ=1\rho=1 .
Figure 17: Dominant index j†j^{\dagger} over α,t\alpha,t and logistic loss on Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(1,2,3,4,5){\bm{\mu}}=(1,2,3,4,5) and ρ=1\rho=1 .
Figure 18: Dominant index j†j^{\dagger} over α,t\alpha,t and logistic loss on Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(1,3,5,7,9){\bm{\mu}}=(1,3,5,7,9) and ρ=1\rho=1 .
Figure 19: Dominant index j†j^{\dagger} over α,t\alpha,t and logistic loss on Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(1,2,4,8,16){\bm{\mu}}=(1,2,4,8,16) and ρ=1\rho=1 .
Figure 20: First two dimensions of Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(4,5,6,7,8){\bm{\mu}}=(4,5,6,7,8) and the dominant index j†j^{\dagger} over α,t\alpha,t under GF and discrete GD updates.
Figure 21: First two dimensions of Dμ{\mathcal{D}}_{\bm{\mu}} and the dominant index j†j^{\dagger} over α,t\alpha,t under the rescaled ℓ2\ell_{2} -SAM flow and discrete ℓ2\ell_{2} -SAM updates.
Figure 22: Dominant index j†j^{\dagger} over α,t\alpha,t under the GF on Dμ{\mathcal{D}}_{\bm{\mu}} with μ= (4,5,6,7,8){\bm{\mu}}=~(4,5,6,7,8) .
Figure 23: Dominant index j†j^{\dagger} over α,t\alpha,t under the rescaled ℓ2\ell_{2} -SAM flow on Dμ{\mathcal{D}}_{\bm{\mu}} with μ= (4,5,6,7,8){\bm{\mu}}=~(4,5,6,7,8) and ρ=1\rho=1 .
Figure 24: Training loss curves of GD (top) and SAM (bottom) on the 2-layer diagonal network in the intermediate initialization regime ( α=0.06\alpha=0.06 ). The colored areas correspond to regimes where each feature is mostly amplified. Compared to GD, SAM exhibits an early plateau loss curve: in this phase, SAM primarily amplifies minor coordinates, leading to slow loss decrease. Once SAM shifts its focus to major coordinates, the loss drops rapidly. GD does not display this minor-to-major feature focusing behavior, thereby showing a more steadily decreasing loss without such a plateau.
Figure 25: Sequential feature amplification under random initialization in a two-layer diagonal network. Rows correspond to different random seeds (Seed 0 and Seed 1), and columns correspond to different plot types (left: normalized coordinate trajectories, right: balancedness).
Figure 26: Alignment of the effective weight β(t)\beta(t) for GD (left) and SAM (right) across initialization scales.
Figure 27: Dominant band for GD (top) and SAM (bottom) across gaussian initialization with different scales. Each point shows the dominant band (the band that model mostly focuses on) at the end of training; SAM systematically shifts from minor (low-frequency) to major(high-frequency) bands as α\alpha increases, whereas GD remains insensitive to α\alpha .
Figure 28: Grad-CAM comparison between GD and ℓ2\ell_{2} -SAM on MNIST (labels 0–3).
Figure 29: Grad-CAM comparison between GD and ℓ∞\ell_{\infty} -SAM on MNIST (labels 0–1).
Figure 30: Average number of pixels with Grad-CAM activation exceeding 0.5 as a function of the initialization scale α\alpha , comparing GD and ℓ2\ell_{2} -SAM across different MNIST subsets.
Figure 31: Average number of pixels with Grad-CAM activation exceeding 0.5 as a function of the initialization scale α\alpha , comparing GD and ℓ∞\ell_{\infty} -SAM on 1k MNIST images with labels 0 and 1.
Figure 32: Grad-CAM comparison between GD and ℓ2\ell_{2} -SAM on SVHN (1k images, labels 0–1) with dark digits and light backgrounds.
Figure 33: Grad-CAM comparison between GD and ℓ2\ell_{2} -SAM on SVHN (1k images, labels 0–1) with light digits and dark backgrounds.
Figure 34: Average number of activated pixels (Grad-CAM > 0.5) as a function of the initialization scale α\alpha , comparing GD and ℓ2\ell_{2} -SAM.
Figure 35: Average number of activated pixels (Grad-CAM > 0.5) as a function of the initialization scale α\alpha , comparing GD and ℓ2\ell_{2} -SAM.
More in this topic

Sharpness-Aware Minimization

Sharpness-Aware Minimization (SAM) trains models toward locally flat loss regions to improve generalization and robustness.

See all 36 papers

Explore similar work

CardsList
Aug 4, 2026

On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds

Sharpness-Aware Minimization (SAM) improves generalization by seeking parameters whose loss is robust to local adversarial perturbations, but the quantitative mechanism underlying its implicit bias toward flat minima remains unclear. In particular, the perturbation radius ρρ is typically treated as an isolated tuning parameter, despite defining the neighborhood in which SAM measures sharpness. We analyze mini-batch SAM near an interpolating minimum through linear stability. Under local linearization and gradient-noise alignment assumptions, we prove that every linearly stable minimum satisfies λmax⁡≤bΓ/(2ρη2)3λ_{\max}\leq\sqrt[3]{bΓ/(2ρη^2)}, where λmax⁡λ_{\max} is the largest Hessian eigenvalue, bb is the batch size, ηη is the learning rate, and ΓΓ bounds the gradient norm. The bound quantitatively characterizes SAM's implicit flatness bias: holding the other quantities fixed, a smaller batch size, a larger learning rate, or a larger radius restricts linearly stable SAM to flatter minima. It also exposes a necessary trade-off: ρρ should be large enough to promote flatness, yet remain local enough to preserve the approximation and stable training. We validate this prediction in a controlled study of 900 models on CIFAR-100 with ResNet-18 and VGG-19, where increasing ρρ is consistently associated with a smaller largest Hessian eigenvalue across batch-size and learning-rate settings. Finally, we instantiate the analysis in Taylor-Locality Controlled SAM (TLC-SAM), which adjusts ρρ using the observed Taylor-approximation error and further reduces the top Hessian eigenvalue relative to fixed-radius SAM. Our results provide quantitative hyperparameter bounds and a stability--locality perspective for analyzing and designing SAM variants.

Jiaxin Deng, Junbiao Pang

Beijing University of Technology

Sharpness-Aware MinimizationFlat MinimaHyperparameter Optimization
AskPDF
Sep 16, 2024

Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate

The sharpness-aware minimization (SAM) algorithm and its variants, including gap guided SAM (GSAM), have been successful at improving the generalization capability of deep neural network models by finding flat local minima of the empirical loss in training. Meanwhile, it has been shown theoretically and practically that increasing the batch size or decaying the learning rate avoids sharp local minima of the empirical loss. In this paper, we consider the GSAM algorithm with increasing batch sizes or decaying learning rates, such as cosine annealing or linear learning rate, and theoretically show its convergence. Moreover, we numerically compare SAM (GSAM) with and without an increasing batch size and conclude that using an increasing batch size { achieves a lower worst-case ℓ∞\ell_\infty adaptive sharpness} than compared with using a constant batch size and learning rate.

Hinata Harada, Hideaki Iiduka

Meiji University, Japan

Sharpness-Aware MinimizationNonconvex OptimizationStochastic Optimization ConvergenceFlat Minima
AskPDF
Mar 9, 2026

Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation

Sharpness-Aware Minimization (SAM) enhances generalization by minimizing the maximum training loss within a predefined neighborhood around the parameters. However, its practical implementation approximates this as gradient ascent(s) followed by applying the gradient at the ascent point to update the current parameters. This practice can be justified as approximately optimizing the objective by neglecting the (full) derivative of the ascent point with respect to the current parameters. Nevertheless, a direct and intuitive understanding of why using the gradient at the ascent point to update the current parameters works superiorly is still lacking. Our work bridges this gap by proposing a novel and intuitive interpretation. We show that the gradient at the single-step ascent point, \uline{when applied to the current parameters}, provides a better approximation of the direction from the current parameters toward the maximum within the local neighborhood than the local gradient. This improved approximation thereby enables a more direct escape from the maximum within the local neighborhood. Nevertheless, our analysis further reveals two issues. First, the approximation by the gradient at the single-step ascent point is often inaccurate. Second, the approximation quality may degrade as the number of ascent steps increases. To address these limitations, we propose in this paper eXplicit Sharpness-Aware Minimization (XSAM). It tackles the first by explicitly estimating the direction of the maximum during training, while addressing the second by crafting a search space that effectively leverages the gradient information at the multi-step ascent point. XSAM features a unified formulation that applies to both single-step and multi-step settings and only incurs negligible computational overhead. Extensive experiments demonstrate the consistent superiority of XSAM against existing counterparts.

Jianlong Chen, Zhiming Zhou

Key Laboratory of Interdisciplinary Research of Computation and Economics Shanghai University of Finance and Economics

Sharpness-Aware MinimizationDeep Learning OptimizationLoss Landscape SharpnessMin-Max Optimization
AskPDF