Sign-Based Optimization

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 21

All topics

The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics
Sep 28, 2026Yihe Zhou, Tongtian Zhu, Yingxiao Huo +4Stochastic Optimization Sign-Based Optimization
Department of Computer Science, The University of Manchester, Manchester, UK · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
Sep 28, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Stochastic Optimization Language Model Pretraining
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Applied Artificial Intelligence Institute +1
Revisiting Distributed Sign-Based Variance Reduction
Sep 16, 2026Wei Jiang, Zechao Li, Lijun ZhangSign-Based Optimization Communication-Efficient Distributed Training
School of Computer Science and Engineering, Nanjing University of Science and Technology, China · National Key Laboratory for Novel Software Technology,Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Beyond the Matrix Sign: Quadratic Spectral Descent
Sep 7, 2026Qiaozhe Zhang, Jun Sun, Yingzhuang LiuSign-Based Optimization Muon Optimizer
School of Electronic Information and Communications Huazhong University of Science and Technology
Using Non-Lipschitz Signum-based Functions for Distributed Optimization and Machine Learning: Trade-off Between Con-vergence Rate and Optimality Gap
Aug 2, 2026Mohammadreza Doostmohammadian, Amir Ahmad Ghods, Alireza Aghasi +2Sign-Based Optimization Distributed Optimization
Faculty of Mechanical Engineering, Semnan University, Semnan, Iran · Department of Electrical Engineering and Computer Science, Oregon State University, Oregon, USA · Institute of Computational Mathematics and Information Technologies, Kazan Federal University, Russia +1
Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback
Jul 31, 2026Maria Smirnova, Alexey KravatskiySign-Based Optimization Communication-Efficient Distributed Training
MIRIAI Moscow, Russia
Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data
Jul 7, 2026Xiangwu Wang, Chengwei Cao, Yicheng Song +2Sign-Based Optimization Implicit Regularization
1The University of Hong Kong · University of California, San Diego · 3Beijing University of Aeronautics and Astronautics
Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
May 29, 2026Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov +2Stochastic Optimization Sign-Based Optimization
HSE University, Moscow, Russia · BRAIn Lab, Moscow, Russia · Yandex Research, Moscow, Russia +3
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
May 19, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Stochastic Optimization Sign-Based Optimization
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Applied Artificial Intelligence Institute +1
Phases of Muon: When Muon Eclipses SignSGD
May 10, 2026Elliot Paquette, Noah Marshall, Lucas Benigni +3Sign-Based Optimization Muon Optimizer
Mathematics and Statistics Department, McGill University · Mathematics and Statistics Department, Université de Montréal · Google DeepMind
Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimates
May 10, 2026Anik Kumar Paul, Nibedita Roy, Nagesh Talagani +3Stochastic Optimization Sign-Based Optimization
Computer Science and Automation, Indian Institute of Science, Bengaluru · Edwardson School of Industrial Engineering, Purdue University · Department of Computer Science, IMT Atlantique
When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$ -norm Lower Bounds
May 7, 2026Hongyi Tao, Dingzhi Yu, Lijun ZhangSign-Based Optimization Optimization Convergence Analysis
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
SignMuon: Communication-Efficient Distributed Muon Optimization
May 4, 2026Neel Mishra, Kushagara Trivedi, Pawan KumarSign-Based Optimization Communication-Efficient Distributed Training
1Microsoft · 2IIIT Hyderabad
Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
Apr 28, 2026Haoran Chen, Wentao WangSign-Based Optimization Stochastic Optimization Convergence
´Ecole polytechnique Palaiseau, France
StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
Apr 16, 2026Dingzhi Yu, Rui Pan, Yuxing Liu +1Stochastic Optimization Sign-Based Optimization
University of Illinois Urbana-Champaign
CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization
Apr 16, 2026Feihu Huang, Guanyi Zhang, Songcan ChenStochastic Optimization Sign-Based Optimization
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China
Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?
Mar 2, 2026Jihwan Kim, Dogyoon Song, Chulhee YunSign-Based Optimization Linear Regression
Seoul National University & KAIST InnoCORE LLM · University of California, Davis · KAIST
Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent
Feb 3, 2026Hiroki Naganuma, Shagun Gupta, Youssef Briki +4Stochastic Optimization Sign-Based Optimization
Mila, Montreal, Canada · Université de Montréal, Montreal, Canada · Meta Platforms, Menlo Park, California, USA
Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalance
Nov 30, 2025Robin Yadav, Shuo Xie, Tianhao Wang +1Sign-Based Optimization Class-Imbalanced Learning
Toyota Technological Institute at Chicago · University of British Columbia · University of California, San Diego
Better Convergence Guarantees for Sign-Based Momentum Methods
Jul 16, 2025Wei Jiang, Dingzhi Yu, Sifan Yang +3Sign-Based Optimization Momentum Methods
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China
Convergence Rate Analysis of LION
Nov 12, 2024Yiming Dong, Huan Li, Zhouchen LinSign-Based Optimization Deep Learning Optimization
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, Beijing, 100871, China · Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University, Tianjin, 300071, China · Pazhou Laboratory (Huangpu), Guangzhou, 510555, China

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

Yihe Zhou, Tongtian Zhu, Yingxiao Huo, Satya Prakash Dash+3

Department of Computer Science, The University of Manchester, Manchester, UK · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

Stochastic Optimization Sign-Based Optimization Adaptive Gradient Methods Neural Network Training Dynamics

Open paper AskPDF arXiv

Adam is the default optimizer for training modern deep neural networks, yet its adaptive behavior remains poorly understood due to the complex interaction between its first- and second-moment exponential moving averages (EMAs). We study Adam in the tied-

β

regime, where the two EMA decay rates are equal, and show that its adaptive dynamics can be expressed through a transformed ratio with approximately scale-stable behavior. Empirically, this transformed ratio exhibits a stable, heavy-tailed distribution across tasks, model scales, and training stages, in contrast to the variability of raw moment magnitudes. This empirical stability has both practical and conceptual consequences. First, we derive a recurrence for the transformed ratio, yielding a reparameterization of Adam that replaces the second moment with a compressible state. Leveraging its stable distribution, we show that a fixed 4-bit codebook is sufficient in our experiments to store this state without auxiliary scaling, achieving performance competitive with full-precision Adam. Second, the transformed ratio view clarifies Adam's connection to sign-based methods: Adam reduces to sign-based momentum modulated by the transformed ratio, and replacing it with a constant recovers Signum as a limiting case. This perspective further provides a simple rule for transferring learning rates between the two methods. Together, these results suggest that tied-

β

Adam admits a simple and approximately stable ratio structure underlying its adaptive behavior and demonstrate its utility for both analysis and efficient implementation.

Momentum

Related topics

Latest papers 21