Better Convergence Guarantees for Sign-Based Momentum Methods

Wei Jiang; Dingzhi Yu; Sifan Yang; Wenhao Yang; Zechao Li; Lijun Zhang

math.OCJul 16, 2025

Better Convergence Guarantees for Sign-Based Momentum Methods

Authors: Wei Jiang, Dingzhi Yu, Sifan Yang, Wenhao Yang, Zechao Li, Lijun Zhang

Organizations: National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China

Abstract

This paper presents an improved analysis for sign-based methods with momentum updates. Traditional sign-based methods obtain a convergence rate of $\mathcal{O}(T^{-1/4})$ under the separable smoothness assumption, but they typically require large batch sizes or assume unimodal symmetric stochastic noise. To address these limitations, we demonstrate that signSGD with momentum can achieve the same convergence rate using constant batch sizes without additional assumptions. We also establish a convergence rate under the $l_2$ -smoothness condition, improving upon the result of prior work by a factor of $\mathcal{O}(d^{1/2})$ , where $d$ is the problem dimension. Furthermore, we explore sign-based methods in distributed settings and show that the proposed methods yield convergence rates of $\mathcal{O}\left( d^{1/2}T^{-1/2} + dn^{-1/2} \right)$ and $\mathcal{O}\left(d^{1/4}T^{-1/4}\right)$ , which outperform the previous results of $\mathcal{O}\left( dT^{-1/4} + dn^{-1/2} \right)$ and $\mathcal{O}\left( d^{3/8}T^{-1/8} \right)$ , respectively. Numerical experiments also validate the effectiveness of the proposed methods.

Figures & tables

Figure 1: Results for CIFAR-10 dataset in the centralized environment.

Figure 2: Results for CIFAR-100 dataset in the distributed environment.

Method	SGDM	signSGD	EF-signSGD	AdamW	Signum
GPT-2	2.509±.005	2.236±.004	2.267±.003	2.183±.001	2.176±.002
Qwen3	1.677±.006	1.610±.003	1.609±.005	1.592±.002	1.592±.001

Table 1: Training losses of finetuning GPT-2 and Qwen3 on the Alpaca dataset.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
weight decay	$0.1$
batch size	$256$
max sequence length	$512$
gradient accumulation steps	$1$
epochs	$1.0$
learning rate schedule	cosine decay

Appendix

Table 2: Hyperparameter configurations.

Method	SGDM	signSGD	EF-signSGD	AdamW	Signum
lr	1e-1	1e-4	1e-0	5e-4	1e-4
$\beta_{1}$	0.9	–	–	0.9	0.75
$\beta_{2}$	–	–	–	0.95	–

Appendix

Table 3: Optimal hyperparameters for fine-tuning GPT-2 on Alpaca.

Method	SGDM	signSGD	EF-signSGD	AdamW	Signum
lr	5e-2	1e-5	5e-1	5e-5	1e-5
$\beta_{1}$	0.9	–	–	0.9	0.75
$\beta_{2}$	–	–	–	0.95	–

Appendix

Table 4: Optimal hyperparameters for fine-tuning Qwen3-0.6B on Alpaca.

Figure 3: Sensitivity results across different learning rates.

Figure 4: Sensitivity results across different momentum coefficients.

Figure 5: Stochastic gradient trajectory on the CIFAR-100 dataset.

Figure 6: Stochastic gradient trajectory on the Alpaca dataset.

Explore similar work

Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
Apr 28, 2026Haoran Chen, Wentao WangStochastic Gradient Descent Signsgd
´Ecole polytechnique Palaiseau, France
When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$ -norm Lower Bounds
May 7, 2026Hongyi Tao, Dingzhi Yu, Lijun ZhangStochastic Gradient Descent Signsgd
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
Apr 16, 2026Dingzhi Yu, Rui Pan, Yuxing Liu +1Stochastic Gradient Descent Stochastic Convex Optimization
University of Illinois Urbana-Champaign

Apr 28, 2026, cs.LGJ/K move · Enter open · S save

Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy

Haoran Chen, Wentao Wang

´Ecole polytechnique Palaiseau, France

Stochastic Gradient Descent Signsgd Convergence Unimodal Metrics

Open paper AskPDF arXiv

SignSGD compresses each stochastic gradient coordinate to a single bit, offering substantial memory and communication savings, but its 1-bit quantization removes magnitude information and is known to leave a generalization gap relative to well-tuned SGD. We revisit SignSGD from a 1-bit quantization and dithering perspective and contribute three improvements. First, we derive a small-batch convergence rate for SignSGD under unimodal symmetric gradient noise using a signal-to-noise weighted stationarity measure, removing the large-batch assumption of prior analyses. Second, we inject annealed Gaussian noise before the sign operator, which acts as a classical dithering mechanism and probabilistically restores magnitude information lost to hard thresholding. Third, we adapt the SWATS strategy to sign-based updates with a projection-based learning-rate calibration that smoothly transitions from SignSGD to SGD. Single-worker experiments on ResNet-18 isolate optimizer effects from communication aspects: pre-sign dithering surpasses Adam on CIFAR-100, and the calibrated switch reaches 92.18% test accuracy on CIFAR-10, outperforming both pure SGD 91.38% and pure SignSGD with momentum 90.82%.

Abstract

Explore similar work

Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy

When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1\ell_1ℓ1​-norm Lower Bounds

StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models

When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$ -norm Lower Bounds