Sign-Based Optimization

Follow this topic Ask about this topic

Sign-based optimization replaces gradient magnitudes with coordinate-wise signs, as in signSGD and Lion, to reduce optimizer memory or communication. Methods add momentum, smooth or stochastic sign transformations, error feedback, adaptive step sizes, and matrix-aware spectral directions such as Muon. Theory studies convergence under different noise, smoothness, and data-geometry assumptions, while experiments test vision models, distributed training, and large language model pretraining. The central tension is that signs can be efficient and robust to some forms of imbalance or sparse noise, but discarding magnitude can hurt stability or convergence when parameters have heterogeneous scales or data is anisotropic.

Latest papers

30 papers7 papers in the last four weeks, up 133% on the four weeks before. 0.0% of all new papers.

Low-Bit Distributed Constrained Optimization with Application to Optimal Scheduling and Allocation
Oct 17, 2026Mohammadreza Doostmohammadian, Sergio PequitoDistributed Optimization Constrained Optimization
Adding Heavy-Ball Momentum to Distributed Constrained Optimization for Energy Resource Management Subject to Time-Delays and Quantization
Oct 17, 2026Mohammadreza Doostmohammadian, Sergio PequitoDistributed Optimization Constrained Optimization
Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training
Oct 12, 2026Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5Spectral Optimization Deep Learning Optimization
Mila · Université de Montréal · Stanford University +4
Phase-HDC: Replacing Optimizer History with Gradient Thresholds in Discrete Phase Learning
Oct 7, 2026Ahmed NebliMemory-Efficient Training Sign-Based Optimization
Mathalyse Research
Why Adaptive Optimizers Underestimate Rare Tokens
Sep 29, 2026Sangsidhya KarAdaptive Gradient Methods Sign-Based Optimization
Department of Statistics Presidency University, Kolkata
The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics
Sep 28, 2026Yihe Zhou, Tongtian Zhu, Yingxiao Huo +4Stochastic Optimization Sign-Based Optimization
Department of Computer Science, The University of Manchester, Manchester, UK · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
Sep 28, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Stochastic Optimization Language Model Pretraining
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Applied Artificial Intelligence Institute +1
Revisiting Distributed Sign-Based Variance Reduction
Sep 16, 2026Wei Jiang, Zechao Li, Lijun ZhangSign-Based Optimization Communication-Efficient Distributed Training
School of Computer Science and Engineering, Nanjing University of Science and Technology, China · National Key Laboratory for Novel Software Technology,Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Signed p-adic Residual Encodings of Finite-Domain All-Different Systems with a Sudoku Case Study
Sep 13, 2026Greg BakerConstraint Satisfaction Sign-Based Optimization
School of Computing, Australian National University Canberra, Australia
Beyond the Matrix Sign: Quadratic Spectral Descent
Sep 7, 2026Qiaozhe Zhang, Jun Sun, Yingzhuang LiuSign-Based Optimization Muon Optimizer
School of Electronic Information and Communications Huazhong University of Science and Technology
Using Non-Lipschitz Signum-based Functions for Distributed Optimization and Machine Learning: Trade-off Between Con-vergence Rate and Optimality Gap
Aug 2, 2026Mohammadreza Doostmohammadian, Amir Ahmad Ghods, Alireza Aghasi +2Sign-Based Optimization Distributed Optimization
Faculty of Mechanical Engineering, Semnan University, Semnan, Iran · Department of Electrical Engineering and Computer Science, Oregon State University, Oregon, USA · Institute of Computational Mathematics and Information Technologies, Kazan Federal University, Russia +1
Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback
Jul 31, 2026Maria Smirnova, Alexey KravatskiySign-Based Optimization Communication-Efficient Distributed Training
MIRIAI Moscow, Russia
Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data
Jul 7, 2026Xiangwu Wang, Chengwei Cao, Yicheng Song +2Sign-Based Optimization Implicit Regularization
1The University of Hong Kong · University of California, San Diego · 3Beijing University of Aeronautics and Astronautics
Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
May 29, 2026Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov +2Stochastic Optimization Sign-Based Optimization
HSE University, Moscow, Russia · BRAIn Lab, Moscow, Russia · Yandex Research, Moscow, Russia +3
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
May 19, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Stochastic Optimization Sign-Based Optimization
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Applied Artificial Intelligence Institute +1
Phases of Muon: When Muon Eclipses SignSGD
May 10, 2026Elliot Paquette, Noah Marshall, Lucas Benigni +3Sign-Based Optimization Muon Optimizer
Mathematics and Statistics Department, McGill University · Mathematics and Statistics Department, Université de Montréal · Google DeepMind
Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimates
May 10, 2026Anik Kumar Paul, Nibedita Roy, Nagesh Talagani +3Stochastic Optimization Sign-Based Optimization
Computer Science and Automation, Indian Institute of Science, Bengaluru · Edwardson School of Industrial Engineering, Purdue University · Department of Computer Science, IMT Atlantique
When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$ -norm Lower Bounds
May 7, 2026Hongyi Tao, Dingzhi Yu, Lijun ZhangSign-Based Optimization Optimization Convergence Analysis
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
SignMuon: Communication-Efficient Distributed Muon Optimization
May 4, 2026Neel Mishra, Jichao Jiang, Kushagara Trivedi +3Sign-Based Optimization Communication-Efficient Distributed Training
Microsoft · IIIT Hyderabad
Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
Apr 28, 2026Haoran Chen, Wentao WangSign-Based Optimization Stochastic Optimization Convergence
´Ecole polytechnique Palaiseau, France
StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
Apr 16, 2026Dingzhi Yu, Rui Pan, Yuxing Liu +1Stochastic Optimization Sign-Based Optimization
University of Illinois Urbana-Champaign
CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization
Apr 16, 2026Feihu Huang, Guanyi Zhang, Songcan ChenStochastic Optimization Sign-Based Optimization
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China
SAGE: Sign-Adaptive Gradient for Memory-Efficient LLM Optimization
Apr 9, 2026Wooin Lee, Hyun-Tae KimLLM Input Optimization Memory-Efficient Optimization
London, United Kingdom · Seoul, Republic of Korea
Prioritizing Gradient Sign Over Modulus: An Importance-Aware Framework for Wireless Federated Learning
Mar 11, 2026Yiyang Yue, Jiacheng Yao, Wei Xu +3Federated Learning Communication-Efficient Distributed Training
National Mobile Communications Research Laboratory (NCRL), Southeast University, Nanjing 210096, China · Purple Mountain Laboratories, Nanjing 211111, China · Zhejiang Lab, Hangzhou 311121, China +3
Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?
Mar 2, 2026Jihwan Kim, Dogyoon Song, Chulhee YunSign-Based Optimization Random Feature Methods
Seoul National University & KAIST InnoCORE LLM · University of California, Davis · KAIST
The Power of Decaying Steps: Enhancing Attack Stability and Transferability for Sign-based Optimizers
Feb 22, 2026Wei Tao, Yang Dai, Jincai Huang +1Adversarial Attacks Adversarial Examples
Laboratory for Big Data and Decision, National University of Defense and Technology, China · Academy of Military Science, China · Hefei Institute of Technology, China
Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent
Feb 3, 2026Hiroki Naganuma, Shagun Gupta, Youssef Briki +4Stochastic Optimization Sign-Based Optimization
Mila, Montreal, Canada · Université de Montréal, Montreal, Canada · Meta Platforms, Menlo Park, California, USA
Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalance
Nov 30, 2025Robin Yadav, Shuo Xie, Tianhao Wang +1Sign-Based Optimization Class-Imbalanced Learning
Toyota Technological Institute at Chicago · University of British Columbia · University of California, San Diego
Better Convergence Guarantees for Sign-Based Momentum Methods
Jul 16, 2025Wei Jiang, Dingzhi Yu, Sifan Yang +3Sign-Based Optimization Momentum Methods
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China
Convergence Rate Analysis of LION
Nov 12, 2024Yiming Dong, Huan Li, Zhouchen LinSign-Based Optimization Deep Learning Optimization
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, Beijing, 100871, China · Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University, Tianjin, 300071, China · Pazhou Laboratory (Huangpu), Guangzhou, 510555, China

Oct 17, 2026J/K move · Enter open · S save

Low-Bit Distributed Constrained Optimization with Application to Optimal Scheduling and Allocation

Mohammadreza Doostmohammadian, Sergio Pequito

Distributed Optimization Constrained Optimization Sign-Based Optimization

Open paper AskPDF arXiv

This paper proposes a distributed resource allocation algorithm in which each agent communicates only the sign of its local cost gradient, a single bit per directed link per iteration. The algorithm is based on Laplacian-gradient dynamics and preserves the equality (demand) constraint at every iterate, guaranteeing anytime feasibility irrespective of termination time. First, we provide a complete Lyapunov-based convergence analysis for the fixed-step-size regime: the iterates are shown to enter, in at most O(1/\varepsilon^2) iterations, a neighborhood of the optimizer whose radius scales as O(τ) in the decision variable and O(τ^2) in the cost residual, where

τ

is the step-size. Crucially, the step-size bound is a time-invariant network constant (depending only on the graph Laplacian spectrum and the cost curvature bounds), so no adaptive tuning is required. Second, we characterize the invariant set under B-connected, weight-balanced digraphs, strictly weaker than the all-time connectivity and doubly-stochastic weights assumed in most prior work, and show that any odd, sign-preserving nonlinearity yields the same invariant-set and descent structure, with the single-bit protocol being the extreme communication-minimizing instance of a broader nonlinear class. Third, we address local box constraints via smooth penalty relaxation and quantify explicitly how the penalty weight interacts with the step-size bound and the optimality gap. Simulations on economic dispatch (1000 generators) and CPU scheduling (500 servers) demonstrate that the single-bit protocol achieves 5-60 times reduction in total communication cost relative to multi-bit baselines, at the price of a tunable and explicitly bounded optimality gap. A diminishing step-size schedule is shown empirically to eliminate this gap; its formal convergence analysis is identified as an open problem.