Arithmetic Simplicity in Stochastic Gradient Methods

Bin Fu; Pengfei Gu; Jose Nunez; Fabian Vazquez

cs.LGSep 26, 2026

Arithmetic Simplicity in Stochastic Gradient Methods

Authors: Bin Fu, Pengfei Gu, Jose Nunez, Fabian Vazquez

Organizations: Department of Computer Science University of Texas Rio Grande Valley Edinburg, TX 78539, USA

Abstract

A gradient descent method is arithmetically simple if the operations are limited to $+,-, \times$ , and division $x/2^t$ with integer $t$ . An arthmetically simple gradient method is easy to implement in chip design. We show how to transform AdaGrad, Adam, and AdamW into arithmetically simple. AdamW is based on the recursion $x_{t+1}=(1-λη)x_t-\frac{η}{s}m_t$ and Adam is the special case of AdamW with $λ=0$ . We transform them into a static case with $s=S(T)$ , where $T$ is the number of iterations, and $S(T)$ is a fixed function. The convergence analysis is given for a static Adam, which is also arithmetically simple.

Figures & tables

Step	Algorithm AdamW	Algorithm Generalized-AdamW
Input	$G(\xi,x)$ , $\beta,\theta,\eta,\lambda,x_{0},T$	$G(\xi,x)$ , $S(\cdot),\beta,\theta,\eta,\lambda,x_{0},T$
	Let $x_{1}=x_{0}$	Let $x_{1}=x_{0}$
	Let $m_{0}=0_{m}$ (zero vector in $\mathbb{R}^{m}$ )	Let $m_{0}=0_{m}$ (zero vector in $\mathbb{R}^{m}$ )
	Let $v_{0}=0$	Let $v_{0}=0$
	Let $j=1$	Let $j=1$
	Repeat	Repeat

Table 1: Side-by-side comparison of AdamW and Generalized-AdamW.

Figure 1: Final loss after 5,000 optimization iterations on each of the 20 independently generated synthetic linear-regression datasets. Arithmetically-Simple-AdamW obtained the lower final loss on 19 of the 20 datasets.

Figure 2: Mean optimizer-step execution time over 20 paired timing trials, with 5,000 updates per optimizer in each trial. Error bars show the standard deviation of the trial-average step times. Arithmetically-Simple-AdamW reduced optimizer-step time by approximately 9.23% relative to the baseline implementation.

Adaptive Gradient Methods

Adaptive Gradient Methods adjust updates using gradient history, curvature, or noise estimates to improve optimization across diverse problems.

See all 157 papers

Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
Nov 10, 2025Steffen Dereich, Thang Do, Arnulf Jentzen +1Stochastic Optimization Stochastic Optimization Convergence
Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany · School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China +1
Revisiting AdaGrad in Stochastic Convex Optimization: Last Iterates, High Probability, and Lower Bounds
Sep 26, 2026Weiming Ou, Xiao WangConvergence Analysis Adaptive Gradient Methods
Shanghai University of Finance and Economics · MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics
Adaptivity via a Parallel Architecture for Stochastic Gradient Methods
Jul 31, 2026Bin FuStochastic Optimization Stochastic Optimization Convergence
University of Texas Rio Grande Valley

Nov 10, 2025J/K move · Enter open · S save

Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer

Steffen Dereich, Thang Do, Arnulf Jentzen, Philippe von Wurstemberger

Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany · School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China+1

Stochastic Optimization Stochastic Optimization Convergence Adaptive Gradient Methods

Open paper AskPDF arXiv

Beside the standard stochastic gradient descent (SGD) method, the Adam optimizer due to Kingma & Ba (2014) is currently probably the best-known optimization method for the training of deep neural networks in artificial intelligence (AI) systems. Despite the popularity and the success of Adam it remains an open research problem to provide a rigorous convergence analysis for Adam even for the class of strongly convex SOPs. In one of the main results of this work we establish convergence rates for Adam in terms of the number of gradient steps (convergence rate \nicefrac{1}{2} w.r.t. the size of the learning rate), the size of the mini-batches (convergence rate 1 w.r.t. the size of the mini-batches), and the size of the second moment parameter of Adam (convergence rate 1 w.r.t. the distance of the second moment parameter to 1) for the class of strongly convex SOPs. In a further main result of this work, which we refer to as Adam symmetry theorem, we illustrate the optimality of the established convergence rates by proving for a special class of simple quadratic strongly convex SOPs that Adam converges as the number of gradient steps increases to infinity to the solution of the SOP (the unique minimizer of the strongly convex objective function) if and only if the random variables in the SOP (the data in the SOP) are symmetrically distributed. In particular, in the standard case where the random variables in the SOP are not symmetrically distributed we disprove that Adam converges to the minimizer of the SOP as the number of Adam steps increases to infinity. We also complement the conclusions of our convergence analysis and the Adam symmetry theorem by several numerical simulations that indicate the sharpness of the established convergence rates and that illustrate the practical appearance of the phenomena revealed in the Adam symmetry theorem.

Abstract

Adaptive Gradient Methods

Explore similar work

Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer

Revisiting AdaGrad in Stochastic Convex Optimization: Last Iterates, High Probability, and Lower Bounds

Adaptivity via a Parallel Architecture for Stochastic Gradient Methods