When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective
Organizations: Peking University · AI for Science Institute, Beijing
Abstract
We study how the sign operator interacts with gradient noise to shape the learning dynamics and scaling behavior of sign-based optimizers. In power-law kernel regression with diagonal features, we establish a functional scaling law (FSL) for SignSGD under general learning rate schedules. The FSL describes the learning dynamics through two processes: signal learning and noise forgetting. The key mechanism is noise-induced preconditioning: gradient noise smooths the sign operator, inducing an effective preconditioned SGD. Noise-curvature alignment enables this preconditioning to transform the effective Hessian from to . The resulting dynamics exhibit faster signal learning but slower noise forgetting than SGD. This tradeoff yields a regime-dependent comparison with SGD: (i) SignSGD achieves improved data efficiency in the hard-task regime; (ii) both optimizers achieve the same data efficiency in the easy-task regime, but SignSGD maintains this efficiency at larger batch sizes, requiring fewer optimization steps for a fixed data budget. Power-law kernel regression experiments support the proposed mechanism and predicted scaling behavior. LLM pre-training experiments further suggest that the SignSGD FSL better captures the batch-size dependence of loss dynamics for sign-based optimizers than the original SGD FSL.
Figures & tables
| Optimizer | Intrinsic time | Noise intensity | ||
|---|---|---|---|---|
| SGD ( Li et al., 2025 ) | ||||
| SignSGD ( ours ) |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Sign-Based Optimization
Sign-based optimization uses gradient directions, often with momentum or spectral updates, to train models with lower memory and communication costs.
See all 33 papers