Overview

Sign-based optimization replaces gradient magnitudes with coordinate-wise signs, as in signSGD and Lion, to reduce optimizer memory or communication. Methods add momentum, smooth or stochastic sign transformations, error feedback, adaptive step sizes, and matrix-aware spectral directions such as Muon. Theory studies convergence under different noise, smoothness, and data-geometry assumptions, while experiments test vision models, distributed training, and large language model pretraining. The central tension is that signs can be efficient and robust to some forms of imbalance or sparse noise, but discarding magnitude can hurt stability or convergence when parameters have heterogeneous scales or data is anisotropic.

Latest papers

CardsList

33 papers3 papers in the last four weeks, down 57% on the four weeks before. 0.0% of all new papers.

  1. When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective

    Nov 12, 2026Zilin Wang, Binghui Li, Jia-Nan Wang +3Sign-Based OptimizationStochastic Gradient Descent

  2. ZEMA: confidence-calibrated signed descent with only one state

    Nov 3, 2026Ali Dadsetan, Domenic Rosati, Frank RudziczAdaptive Gradient MethodsSign-Based Optimization

  3. Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training

    Oct 12, 2026Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5Spectral OptimizationDeep Learning Optimization

  4. Why Adaptive Optimizers Underestimate Rare Tokens

    Sep 29, 2026Sangsidhya KarAdaptive Gradient MethodsSign-Based Optimization

  5. The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

    Sep 28, 2026Yihe Zhou, Tongtian Zhu, Yingxiao Huo +4Stochastic OptimizationSign-Based Optimization

  6. LionMuon: Alternating Spectral and Sign Descent for Efficient Training

    Sep 28, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Stochastic OptimizationLanguage Model Pretraining

  7. Revisiting Distributed Sign-Based Variance Reduction

    Sep 16, 2026Wei Jiang, Zechao Li, Lijun ZhangSign-Based OptimizationCommunication-Efficient Distributed Training

  8. Signed p-adic Residual Encodings of Finite-Domain All-Different Systems with a Sudoku Case Study

    Sep 13, 2026Greg BakerConstraint SatisfactionSign-Based Optimization

  9. Beyond the Matrix Sign: Quadratic Spectral Descent

    Sep 7, 2026Qiaozhe Zhang, Jun Sun, Yingzhuang LiuSign-Based OptimizationMuon Optimizer

  10. Using Non-Lipschitz Signum-based Functions for Distributed Optimization and Machine Learning: Trade-off Between Con-vergence Rate and Optimality Gap

    Aug 2, 2026Mohammadreza Doostmohammadian, Amir Ahmad Ghods, Alireza Aghasi +2Sign-Based OptimizationDistributed Optimization

  11. Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data

    Jul 7, 2026Xiangwu Wang, Chengwei Cao, Yicheng Song +2Sign-Based OptimizationImplicit Regularization

  12. Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling

    May 29, 2026Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov +2Stochastic OptimizationSign-Based Optimization

  13. LionMuon: Alternating Spectral and Sign Descent for Efficient Training

    May 19, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Stochastic OptimizationSign-Based Optimization

  14. Phases of Muon: When Muon Eclipses SignSGD

    May 10, 2026Elliot Paquette, Noah Marshall, Lucas Benigni +3Sign-Based OptimizationMuon Optimizer

  15. Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimates

    May 10, 2026Anik Kumar Paul, Nibedita Roy, Nagesh Talagani +3Stochastic OptimizationSign-Based Optimization

  16. When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1\ell_1-norm Lower Bounds

    May 7, 2026Hongyi Tao, Dingzhi Yu, Lijun ZhangSign-Based OptimizationOptimization Convergence Analysis

  17. SignMuon: Communication-Efficient Distributed Muon Optimization

    May 4, 2026Neel Mishra, Jichao Jiang, Kushagara Trivedi +3Sign-Based OptimizationCommunication-Efficient Distributed Training

  18. CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization

    Apr 16, 2026Feihu Huang, Guanyi Zhang, Songcan ChenStochastic OptimizationSign-Based Optimization

  19. SAGE: Sign-Adaptive Gradient for Memory-Efficient LLM Optimization

    Apr 9, 2026Wooin Lee, Hyun-Tae KimLLM Input OptimizationMemory-Efficient Optimization

  20. Prioritizing Gradient Sign Over Modulus: An Importance-Aware Framework for Wireless Federated Learning

    Mar 11, 2026Yiyang Yue, Jiacheng Yao, Wei Xu +3Federated LearningCommunication-Efficient Distributed Training

  21. Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?

    Mar 2, 2026Jihwan Kim, Dogyoon Song, Chulhee YunSign-Based OptimizationRandom Feature Methods

  22. The Power of Decaying Steps: Enhancing Attack Stability and Transferability for Sign-based Optimizers

    Feb 22, 2026Wei Tao, Yang Dai, Jincai Huang +1Adversarial AttacksAdversarial Examples

  23. Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent

    Feb 3, 2026Hiroki Naganuma, Shagun Gupta, Youssef Briki +4Stochastic OptimizationSign-Based Optimization