stat.MLNov 12, 2026

When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective

Authors: Zilin Wang, Binghui Li, Jia-Nan Wang, Lean Wang, Jinbo Wang, Lei Wu

Organizations: Peking University · AI for Science Institute, Beijing

Abstract

We study how the sign operator interacts with gradient noise to shape the learning dynamics and scaling behavior of sign-based optimizers. In power-law kernel regression with diagonal features, we establish a functional scaling law (FSL) for SignSGD under general learning rate schedules. The FSL describes the learning dynamics through two processes: signal learning and noise forgetting. The key mechanism is noise-induced preconditioning: gradient noise smooths the sign operator, inducing an effective preconditioned SGD. Noise-curvature alignment enables this preconditioning to transform the effective Hessian from H\mathbf{H} to H1/2\mathbf{H}^{1/2}. The resulting dynamics exhibit faster signal learning but slower noise forgetting than SGD. This tradeoff yields a regime-dependent comparison with SGD: (i) SignSGD achieves improved data efficiency in the hard-task regime; (ii) both optimizers achieve the same data efficiency in the easy-task regime, but SignSGD maintains this efficiency at larger batch sizes, requiring fewer optimization steps for a fixed data budget. Power-law kernel regression experiments support the proposed mechanism and predicted scaling behavior. LLM pre-training experiments further suggest that the SignSGD FSL better captures the batch-size dependence of loss dynamics for sign-based optimizers than the original SGD FSL.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?

    Mar 2, 2026Jihwan Kim, Dogyoon Song, Chulhee YunSign-Based OptimizationRandom Feature Methods

  2. When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1\ell_1-norm Lower Bounds

    May 7, 2026Hongyi Tao, Dingzhi Yu, Lijun ZhangSign-Based OptimizationOptimization Convergence Analysis