Overview

Sign-based optimization replaces gradient magnitudes with coordinate-wise signs, as in signSGD and Lion, to reduce optimizer memory or communication. Methods add momentum, smooth or stochastic sign transformations, error feedback, adaptive step sizes, and matrix-aware spectral directions such as Muon. Theory studies convergence under different noise, smoothness, and data-geometry assumptions, while experiments test vision models, distributed training, and large language model pretraining. The central tension is that signs can be efficient and robust to some forms of imbalance or sparse noise, but discarding magnitude can hurt stability or convergence when parameters have heterogeneous scales or data is anisotropic.

Latest papers

CardsList

33 papers3 papers in the last four weeks, down 57% on the four weeks before. 0.0% of all new papers.

  1. Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalance

    Nov 30, 2025Robin Yadav, Shuo Xie, Tianhao Wang +1Sign-Based OptimizationClass-Imbalanced Learning

  2. Better Convergence Guarantees for Sign-Based Momentum Methods

    Jul 16, 2025Wei Jiang, Dingzhi Yu, Sifan Yang +3Sign-Based OptimizationMomentum Methods

  3. Convergence Rate Analysis of LION

    Nov 12, 2024Yiming Dong, Huan Li, Zhouchen LinSign-Based OptimizationDeep Learning Optimization