Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules

Doğay Altınel

cs.LGSep 22, 2025

Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules

Authors: Doğay Altınel

Organizations: Istanbul Medeniyet University

Abstract

Deep learning optimizers are optimization algorithms that enable deep neural networks to learn. The effectiveness of learning is highly dependent on the optimizer employed in the training process. Alongside the rapid advancement of deep learning, a wide range of optimizers with different approaches have been developed. This study aims to provide a review of various optimizers that have been proposed and received attention in the literature. From Stochastic gradient descent to the most recent ones such as Momentum, AdamW, Sophia, and Muon in chronological order, optimizers are examined individually, and their distinctive features are highlighted in the study. The update rule of each optimizer is presented in detail, with an explanation of the associated concepts and variables. The techniques applied by these optimizers, their contributions to the optimization process, and their default hyperparameter settings are also discussed. In addition, insights are offered into the open challenges encountered in the optimization of deep learning models. Thus, a comprehensive resource is provided both for understanding the current state of optimizers and for identifying potential areas of future development.

Figures & tables

Optimizer	Year Introduced	Related References
SGD	1951	[ 24 , 25 , 26 , 27 , 28 ]
Momentum	1964	[ 29 , 30 ]
NAG	1983	[ 31 , 32 ]
Adagrad	2011	[ 33 ]
RMSprop	2012	[ 34 ]
Adadelta	2012	[ 35 ]

TABLE I: Chronological list of deep learning optimizers with related references

Fig. 1: Illustrative trajectories of SGD, Momentum, and NAG optimizers on the Beale function, starting from $(1,1)$ toward the global minimum at $(3,0.5)$

Fig. 2: Illustrative trajectories of Adagrad, Adadelta, and RMSprop optimizers on the Beale function

Fig. 3: Illustrative trajectories of Adam, Adamax, and Nadam optimizers on the Beale function

Fig. 4: Illustrative Nadam trajectories for different values of the first decay rate, $\beta_{1}=0.90,0.96,0.975,0.990$

Fig. 5: Illustrative trajectories of Adam, AdamW, and AMSgrad optimizers, showing a setting where Adam does not reach the target while its variants do

Fig. 6: Illustrative trajectories of Adafactor, LAMB, Radam, and Adabelief optimizers on the Beale function

Fig. 7: Illustrative trajectories of Lookahead, SAM, ASAM, and Ranger21 optimizers on the Beale function

Fig. 8: Illustrative trajectories of Adan, Lion, Sophia, Muon, NorMuon, and ScheduleFree + optimizers on the Beale function

Optimizer Group	Main Mechanism	Visual Behavior in This Section	Key Insight
SGD, Momentum, NAG	Gradient descent, accumulated gradients, lookahead	Momentum-based methods reduce the number of steps but may introduce oscillation	Acceleration and damping must be balanced
Adagrad, Adadelta, RMSprop	Adaptive scaling by squared gradients	Cumulative scaling is conservative, while decayed scaling produces shorter paths	The memory of past gradients controls step shrinkage
Adam, Adamax, Nadam	First and second moment adaptation	Nesterov-style correction changes the path and can reduce steps in this setting	Adaptive momentum variants differ in direction and scaling
Nadam hyperparameter variants	First moment decay tuning	Higher $\beta_{1}$ accelerates but increases overshooting	Hyperparameters affect both speed and stability
Adam, AdamW, AMSgrad	Decoupled weight decay and long-term second moment control	Variants recover convergence where Adam stalls under the selected settings	Stabilizing modifications can change failure modes
Adafactor, LAMB, Radam, Adabelief	Memory factorization, layerwise scaling, variance rectification, belief correction	Paths differ in directness and conservativeness	Similar Adam-family optimizers can behave visibly differently

TABLE II: Summary of the illustrative optimizer behaviors observed on the Beale-function visualizations

Fig. 9: Small-scale transformer experiment for adaptive moment baselines. The dashed curves with small markers show the mean validation loss over three random seeds under the matched 3,000-step training budget. The inset magnifies the early 50–400 update-step region where several curves are closely spaced.

Optimizer	Final Valid. Loss	Best Valid. Loss	Best Step	Tokens/s
Adam	2.581 $\pm$ 0.013	2.480 $\pm$ 0.004	533	387397
AdamW	1.437 $\pm$ 0.005	1.430 $\pm$ 0.004	2933	439791
Nadam	2.590 $\pm$ 0.002	2.498 $\pm$ 0.006	633	435499
Radam	2.608 $\pm$ 0.014	2.484 $\pm$ 0.003	967	426815
Adabelief	1.431 $\pm$ 0.005	1.423 $\pm$ 0.004	2933	334580

TABLE III: Adaptive moment baseline transformer results over three random seeds

Fig. 10: Small-scale transformer experiment for contemporary optimizers. The dashed curves with small markers show the mean validation loss over three random seeds under the matched training budget.

Optimizer	Final Valid. Loss	Best Valid. Loss	Best Step	Tokens/s
Adan	1.480 $\pm$ 0.003	1.476 $\pm$ 0.006	2933	269599
Lion	1.407 $\pm$ 0.004	1.402 $\pm$ 0.005	2967	352776
Sophia	1.662 $\pm$ 0.010	1.661 $\pm$ 0.009	3000	315436
Muon	1.816 $\pm$ 0.005	1.815 $\pm$ 0.005	3000	205421
NorMuon	1.819 $\pm$ 0.005	1.819 $\pm$ 0.005	3000	181964
ScheduleFree +	1.644 $\pm$ 0.008	1.642 $\pm$ 0.007	2967	289990

TABLE IV: Contemporary optimizer transformer results over three random seeds

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 205 papers

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations
Apr 14, 2026Tong Zhang, Jiangning Zhang, Zhucun Xue +9Deep Learning Optimization Adaptive Gradient Methods
Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University +1
Blog: Survey of Optimizers
Aug 28, 2026Ruoran XuNeural Network Optimization Memory-Efficient Optimization
PILOT: Policy-Informed Learned Optimization for Adaptive Deep Network Training
May 23, 2026Sattam Altuuaim, Lama Ayash, Muhammad Mubashar +1Deep Learning Optimization Learning to Optimize
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · University of Strathclyde, Glasgow, Scotland

Apr 14, 2026J/K move · Enter open · S save

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

Tong Zhang, Jiangning Zhang, Zhucun Xue, Juntao Jiang+8

Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University+1

Deep Learning Optimization Adaptive Gradient Methods Stochastic Gradient Descent Second-Order Optimization Zeroth-Order Optimization

Open paper AskPDF arXiv

Balancing convergence speed, generalization capability, and computational efficiency remains a core challenge in deep learning optimization. First-order gradient descent methods, epitomized by stochastic gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines. However, large-scale model training, stringent differential privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency. To mitigate these bottlenecks, researchers explore second-order optimization techniques to surpass first-order performance ceilings, while zeroth-order methods reemerge to alleviate memory constraints inherent to large-scale training. Despite this proliferation of methodologies, the field lacks a cohesive framework that unifies underlying principles and delineates application scenarios for these disparate approaches. In this work, we retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstream optimizers across diverse model architectures and training scenarios. We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research. By synthesizing theoretical insights with extensive empirical evidence, we provide actionable guidance for designing next-generation highly efficient, robust, and trustworthy optimization methods. The code is available at https://github.com/APRIL-AIGC/Awesome-Optimizer.

Abstract

Deep Learning Optimization

Explore similar work

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

Blog: Survey of Optimizers

PILOT: Policy-Informed Learned Optimization for Adaptive Deep Network Training