cs.LGMar 9, 2026

Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization

Authors: Chaewon Moon, Dongkuk Si, Chulhee Yun

Organizations: Kim Jaechul Graduate School of AI, KAIST · Mobilint, Inc.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study the implicit bias of Sharpness-Aware Minimization (SAM) when training LL-layer linear diagonal networks on linearly separable binary classification. For linear models (L=1L=1), both ℓ∞\ell_\infty- and ℓ2\ell_2-SAM recover the ℓ2\ell_2 max-margin classifier, matching gradient descent (GD). However, for depth L=2L = 2, the behavior changes drastically -- even on a single-example dataset. For ℓ∞\ell_\infty-SAM, the limit direction depends critically on initialization and can converge to 0\mathbf{0} or to any standard basis vector, in stark contrast to GD, whose limit aligns with the basis vector of the dominant data coordinate. For ℓ2\ell_2-SAM, we show that although its limit direction matches the ℓ1\ell_1 max-margin solution as in the case of GD, its finite-time dynamics exhibit a phenomenon we call "sequential feature amplification", in which the predictor initially relies on minor coordinates and gradually shifts to larger ones as training proceeds or initialization increases. Our theoretical analysis attributes this phenomenon to ℓ2\ell_2-SAM's gradient normalization factor applied in its perturbation, which amplifies minor coordinates early and allows major ones to dominate later, giving a concrete example where infinite-time implicit-bias analyses are insufficient. Synthetic and real-data experiments corroborate our findings.

Figures & tables

Figure 1: Trajectories of the predictor β(t)∈R2{\bm{\beta}}(t)\in\mathbb{R}^{2} from identical initial conditions under discrete GD, ℓ∞\ell_{\infty} -SAM and ℓ2\ell_{2} -SAM on {(μ,+1)}\{({\bm{\mu}},+1)\} with μ=(1,2){\bm{\mu}}=(1,2) . We used η=0.3\eta=0.3 and ρ=1\rho=1 for SAM.
Figure 2: Trajectories β(t)\beta(t) from identical initializations under GF and ℓ∞\ell_{\infty} -SAM flow with d=2d=2 and μ=(1,2){\bm{\mu}}=(1,2) . For SAM, ρ=1\rho=1 .
Figure 3: Rescaled ℓ2\ell_{2} -SAM flow on Dμ\mathcal{D}_{\bm{\mu}} with μ=(4,5,6,7,8)∈R5{\bm{\mu}}=(4,5,6,7,8)\in\mathbb{R}^{5} and ρ=1\rho=1 .
Figure 4: Loss curves of GD (left) and ℓ2\ell_{2} -SAM (right) on a 2-layer diagonal network in Regime 2 ( α=0.35\alpha=0.35 , μ=(1,2,3,4,5,6)\mu=(1,2,3,4,5,6) , ρ=0.1\rho=0.1 ). Colored regions mark the coordinate with highest growth.
Figure 5: Grad-CAM comparison of GD and ℓ2\ell_{2} -SAM on a CNN trained on MNIST. GD focuses on dominant digit pixels, whereas ℓ2\ell_{2} -SAM highlights minor background regions.
Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Trajectories of the predictor β(t)∈R2{\bm{\beta}}(t)\in\mathbb{R}^{2} from identical initial conditions under GF, ℓ∞\ell_{\infty} -SAM flow and ℓ2\ell_{2} -SAM flow on {(μ,+1)}\{({\bm{\mu}},+1)\} with μ=(1,2){\bm{\mu}}=(1,2) . For SAM, ρ=1\rho=1 .
Figure 7: Dominant index j†:=arg⁡max⁡jβj(t)j^{\dagger}:=\arg\max_{j}\beta_{j}(t) for GF and ℓ∞\ell_{\infty} -SAM flow over (t,α)(t,\alpha) on Dμ\mathcal{D}_{\bm{\mu}} with μ=(4,5,6,7,8)∈R5{\bm{\mu}}=(4,5,6,7,8)\in\mathbb{R}^{5} .
Figure 8: Dominant index for ℓ2\ell_{2} -SAM flow with μ=(1,2,…,12){\bm{\mu}}=(1,2,\dots,12) . The black line indicates the interesting trajectory.
Figure 9: βj(t)\beta_{j}(t) and normalized βj(t)\beta_{j}(t) trajectory of the original ℓ∞\ell_{\infty} -SAM flow.
Figure 10: βj(t)\beta_{j}(t) and normalized βj(t)\beta_{j}(t) trajectory of the rescaled ℓ∞\ell_{\infty} -SAM flow.
Figure 11: Trajectories β(t){\bm{\beta}}(t) under GF, ℓ∞\ell_{\infty} -SAM flow (black line), GD, and discrete ℓ∞\ell_{\infty} -SAM updates (blue dots) for d=2d=2 and μ=(1,2){\bm{\mu}}=(1,2) . For SAM, we set ρ=1\rho=1 . For GD and discrete ℓ∞\ell_{\infty} -SAM, we use step size η=0.1\eta=0.1 .
Figure 12: A randomly generated linearly separable dataset used in our multi-point experiments. We sample two Gaussian clusters centered at ±μ=±(1,2)\pm{\bm{\mu}}=\pm(1,2) with isotropic noise ( ε∼N(0,0.52I2){\bm{\varepsilon}}\sim\mathcal{N}(0,0.5^{2}{\bm{I}}_{2}) ) and assign labels +1+1 and −1-1 accordingly. This dataset is used to evaluate whether the one-point phenomena from Theorems 3.2 and 3.5 persist in the multi-point regime.
Figure 13: Trajectories β(t){\bm{\beta}}(t) under GF, ℓ∞\ell_{\infty} -SAM flow (black line), GD, and discrete ℓ∞\ell_{\infty} -SAM updates (blue dots) for d=2d=2 on random multi-point dataset in Figure 12 . For SAM, we set ρ=1\rho=1 . For GD and discrete ℓ∞\ell_{\infty} -SAM, we use step size η=0.1\eta=0.1 .
Figure 14: Numerical evaluation of LBj(α)\mathrm{LB}_{j}(\alpha) for various choices of μ{\bm{\mu}} .
Figure 15: Dominant index j†j^{\dagger} over α,t\alpha,t and logistic loss on Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(4,5,6,7,8){\bm{\mu}}=(4,5,6,7,8) .
Figure 16: Dominant index j†j^{\dagger} over α,t\alpha,t and logistic loss on Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(4,5,6,7,8){\bm{\mu}}=(4,5,6,7,8) and ρ=1\rho=1 .
Figure 17: Dominant index j†j^{\dagger} over α,t\alpha,t and logistic loss on Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(1,2,3,4,5){\bm{\mu}}=(1,2,3,4,5) and ρ=1\rho=1 .
Figure 18: Dominant index j†j^{\dagger} over α,t\alpha,t and logistic loss on Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(1,3,5,7,9){\bm{\mu}}=(1,3,5,7,9) and ρ=1\rho=1 .
Figure 19: Dominant index j†j^{\dagger} over α,t\alpha,t and logistic loss on Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(1,2,4,8,16){\bm{\mu}}=(1,2,4,8,16) and ρ=1\rho=1 .
Figure 20: First two dimensions of Dμ{\mathcal{D}}_{\bm{\mu}} with μ=(4,5,6,7,8){\bm{\mu}}=(4,5,6,7,8) and the dominant index j†j^{\dagger} over α,t\alpha,t under GF and discrete GD updates.
Figure 21: First two dimensions of Dμ{\mathcal{D}}_{\bm{\mu}} and the dominant index j†j^{\dagger} over α,t\alpha,t under the rescaled ℓ2\ell_{2} -SAM flow and discrete ℓ2\ell_{2} -SAM updates.
Figure 22: Dominant index j†j^{\dagger} over α,t\alpha,t under the GF on Dμ{\mathcal{D}}_{\bm{\mu}} with μ= (4,5,6,7,8){\bm{\mu}}=~(4,5,6,7,8) .
Figure 23: Dominant index j†j^{\dagger} over α,t\alpha,t under the rescaled ℓ2\ell_{2} -SAM flow on Dμ{\mathcal{D}}_{\bm{\mu}} with μ= (4,5,6,7,8){\bm{\mu}}=~(4,5,6,7,8) and ρ=1\rho=1 .
Figure 24: Training loss curves of GD (top) and SAM (bottom) on the 2-layer diagonal network in the intermediate initialization regime ( α=0.06\alpha=0.06 ). The colored areas correspond to regimes where each feature is mostly amplified. Compared to GD, SAM exhibits an early plateau loss curve: in this phase, SAM primarily amplifies minor coordinates, leading to slow loss decrease. Once SAM shifts its focus to major coordinates, the loss drops rapidly. GD does not display this minor-to-major feature focusing behavior, thereby showing a more steadily decreasing loss without such a plateau.
Figure 25: Sequential feature amplification under random initialization in a two-layer diagonal network. Rows correspond to different random seeds (Seed 0 and Seed 1), and columns correspond to different plot types (left: normalized coordinate trajectories, right: balancedness).
Figure 26: Alignment of the effective weight β(t)\beta(t) for GD (left) and SAM (right) across initialization scales.
Figure 27: Dominant band for GD (top) and SAM (bottom) across gaussian initialization with different scales. Each point shows the dominant band (the band that model mostly focuses on) at the end of training; SAM systematically shifts from minor (low-frequency) to major(high-frequency) bands as α\alpha increases, whereas GD remains insensitive to α\alpha .
Figure 28: Grad-CAM comparison between GD and ℓ2\ell_{2} -SAM on MNIST (labels 0–3).
Figure 29: Grad-CAM comparison between GD and ℓ∞\ell_{\infty} -SAM on MNIST (labels 0–1).
Figure 30: Average number of pixels with Grad-CAM activation exceeding 0.5 as a function of the initialization scale α\alpha , comparing GD and ℓ2\ell_{2} -SAM across different MNIST subsets.
Figure 31: Average number of pixels with Grad-CAM activation exceeding 0.5 as a function of the initialization scale α\alpha , comparing GD and ℓ∞\ell_{\infty} -SAM on 1k MNIST images with labels 0 and 1.
Figure 32: Grad-CAM comparison between GD and ℓ2\ell_{2} -SAM on SVHN (1k images, labels 0–1) with dark digits and light backgrounds.
Figure 33: Grad-CAM comparison between GD and ℓ2\ell_{2} -SAM on SVHN (1k images, labels 0–1) with light digits and dark backgrounds.
Figure 34: Average number of activated pixels (Grad-CAM > 0.5) as a function of the initialization scale α\alpha , comparing GD and ℓ2\ell_{2} -SAM.
Figure 35: Average number of activated pixels (Grad-CAM > 0.5) as a function of the initialization scale α\alpha , comparing GD and ℓ2\ell_{2} -SAM.
More in this topic

Sharpness-Aware Minimization

Sharpness-Aware Minimization (SAM) trains models toward locally flat loss regions to improve generalization and robustness.

See all 36 papers

Explore similar work

CardsList

  1. On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds

    Aug 4, 2026Jiaxin Deng, Junbiao PangSharpness-Aware MinimizationFlat Minima

    Beijing University of Technology

  2. Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate

    Sep 16, 2024Hinata Harada, Hideaki IidukaSharpness-Aware MinimizationNonconvex Optimization

    Meiji University, Japan

  3. Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation

    Mar 9, 2026Jianlong Chen, Zhiming ZhouSharpness-Aware MinimizationDeep Learning Optimization

    Key Laboratory of Interdisciplinary Research of Computation and Economics Shanghai University of Finance and Economics