Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

Tong Zhang; Jiangning Zhang; Zhucun Xue; Juntao Jiang; Yicheng Xu; Chengming Xu; Teng Hu; Xingyu Xie

cs.LGApr 14, 2026

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

Authors: Tong Zhang, Jiangning Zhang, Zhucun Xue, Juntao Jiang, Yicheng Xu, Chengming Xu, Teng Hu, Xingyu Xie,

Organizations: Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University · National University of Singapore

Abstract

Balancing convergence speed, generalization capability, and computational efficiency remains a core challenge in deep learning optimization. First-order gradient descent methods, epitomized by stochastic gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines. However, large-scale model training, stringent differential privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency. To mitigate these bottlenecks, researchers explore second-order optimization techniques to surpass first-order performance ceilings, while zeroth-order methods reemerge to alleviate memory constraints inherent to large-scale training. Despite this proliferation of methodologies, the field lacks a cohesive framework that unifies underlying principles and delineates application scenarios for these disparate approaches. In this work, we retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstream optimizers across diverse model architectures and training scenarios. We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research. By synthesizing theoretical insights with extensive empirical evidence, we provide actionable guidance for designing next-generation highly efficient, robust, and trustworthy optimization methods. The code is available at https://github.com/APRIL-AIGC/Awesome-Optimizer.

Figures & tables

Symbol	Description
$\theta_{t}$	Model parameters (weights and biases) evaluated at iteration $t$ .
$\eta$	Learning rate, determining the step size of parameter updates .
$\nabla J(\theta_{t})$	Exact gradient of the objective function with respect to parameters at step $t$ .
$g_{t}$	Stochastic gradient (mini-batch gradient) of the objective function at step $t$ .
$m_{t}$	Biased first moment estimate (exponential moving average of past gradients) .
$v_{t}$	Biased second raw moment estimate (exponential moving average of past squared gradients) .

Table 1: Nomenclature of key variables and hyperparameters in gradient-based optimization. This table summarizes the unified mathematical notations consistently used throughout the paper.

Optimizer	Venue	Type	Limitation	Highlight
First-Order Methods ( Sec. 3.2 )
SGDM [ 208 ]	Nature’1986	Accelerating Convergence Rate	Complex hyperparameter tuning.	First proposed to introduce a momentum mechanism to accelerate convergence.
Adam [ 123 ]	ICLR’15	Strategies for Adaptive Step-Size Estimation	High memory usage, worse generalization.	First proposed a method combining adaptive step-size estimation with momentum.
AdaBelief [ 344 ]	NeurIPS’20	Strategies for Adaptive Step-Size Estimation	$\epsilon$ requires task-specific tuning.	Compared to adaptive methods like Adam, it achieves better generalization.
Shampoo [ 83 ]	ICML’18	Approximating Curvature Information	Computationally expensive.	First proposed to exploit tensor structures to maintain preconditioners
4-bit Shampoo [ 281 ]	NeurIPS’24	Approximating Curvature Information	Computationally expensive.	Compared to standard Shampoo, it significantly reduces memory usage by quantizing optimizer states to 4-bit.

Table 2: Taxonomy and Comparison of Representative Optimization Methods. This table presents a systematic evaluation of various optimizers categorized into FO, SO, and ZO paradigms, shows their limitations and highlights.

Procedure	DeiT	A2	A3
Dataset	IN-1K	IN-1K	IN-1K
Epochs	300	300	100
Batch size	1024	1024	1024
Optimizer	AdamW	AdamW	AdamW
Learning rate	$1\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$
Optimizer Momentum	0.9, 0.999	0.9, 0.999	0.9, 0.999

Table 3: Ingredients and hyper-parameters used for image classification training settings. Taking ImageNet-1K as the template setups, the settings of A2/A3 [ 257 ] take ResNet-50 [ 87 ] as the examples, the DeiT [ 257 ] setting takes ViT-S [ 49 ] as the example. Gray regions should be modified for each optimizer.

Figure 8: Comparison of Top-1 accuracy and learning rate sensitivity analysis of various optimizers. Each subfigure demonstrates the robustness and sensitivity of individual optimizers across a range of scaled learning rates (from 0.1× to 10× the base learning rate). Figures demonstrate that convergence stability and final Top-1 accuracy heavily depend on precise hyperparameter tuning.

Optimizer	0.1 $\times$ LR	0.2 $\times$ LR	1.0 $\times$ LR	5.0 $\times$ LR	10.0 $\times$ LR	Best LR
SGD [ 205 ]	48.056	55.918	64.360	60.590	59.262	0.1
RMSprop [ 256 ]	54.254	43.416	20.512	6.1140	-	1e-4
Adam [ 123 ]	66.234	70.772	72.240	-	-	1e-3
AdaBelief [ 344 ]	29.248	40.210	58.562	66.528	68.642	1e-2
Adafactor [ 223 ]	66.294	70.834	74.630	73.344	-	1e-3
RAdam [ 148 ]	66.220	70.826	71.190	-	-	1e-3

Table 4: Comparison of various optimizers’ hyperparameter robustness and learning rate settings.

Optimizer	ViT-S	ResNet-50	Llama-60m
Optimizer	Acc (%)	Acc (%)	Base LR	5.0 $\times$ LR	0.2 $\times$ LR
SGD [ 205 ]	64.360	75.926	-	-	-
RMSprop [ 256 ]	54.254	71.536	522.57	269.27	1178.34
Adam [ 123 ]	72.240	74.480	614.93	357.95	964.46
AdaBelief [ 344 ]	68.642	74.232	43.27	41.89	72.43
Adafactor [ 223 ]	74.630	76.808	13.78	16.55	16.48

Table 5: Results on ViT-S [ 49 ] , ResNet-50 [ 87 ] , and Llama-60m [ 153 ] models across various optimizers. We compare different learning rate multipliers on Llama-60m [ 153 ] and show that Muon [ 117 ] consistently achieves strong performance across all architectures.

Figure 9: Optimizers’ performance on Llama-60m. Subfigures show validation loss curves of various optimizers across three learning rate settings. The robustness to learning rate variations differs significantly among methods, with some optimizers (e.g., Muon [ 117 ] , MARS [ 309 ] ) maintaining stable convergence across all scales while others (e.g., SGD [ 205 ] , SGDP [ 89 ] ) rapidly diverge. Missing data indicates gradient explosion (loss becomes NaN) at the initial step or the point of discontinuation.

Figure 11: Top-1 accuracy and trajectory correlation of popular optimizers. The main line plots display the Top-1 accuracy performance of 23 optimizers on ViT-S [ 49 ] and ResNet-50 [ 87 ] models over 100 and 300 epochs. The inset heatmaps present the Pearson correlation matrices calculated via first-order differencing of the validation metrics, revealing the algorithmic categorization and trajectory divergence across different architectures and training horizons.

Model	Optimizer	100 Epoch Acc (%)	300 Epoch Acc (%)	Improvement ( $\Delta$ )
ViT-S	SGD [ 205 ]	64.360	73.774	+ 9.414
	RMSprop [ 256 ]	54.254	57.090	+ 2.836
	Adam [ 123 ]	72.240	77.274	+ 5.034
	AdaBelief [ 344 ]	68.642	74.640	+ 5.998
	Adafactor [ 223 ]	74.630	80.108	+ 5.478
	RAdam [ 148 ]	71.190	77.032	+ 5.842

Table 6: Results on ViT-S [ 49 ] and ResNet-50 [ 87 ] models across various optimizers, where we compare the 100 epoch accuracy and 300 epoch accuracy and show that ViT-S [ 49 ] achieves larger performance improvements from extended training than ResNet-50 [ 87 ] .

Optimizer	Convergence	Accuracy	Memory	Generalization	Hyperparameter
Optimizer	Speed		Efficiency		Robustness
SGD [ 205 ]	★	★★	★★★★★	★★	★
RMSprop [ 256 ]	★	★	★★★★	★★	★★
Adam [ 123 ]	★★★	★★★	★★★	★★	★★★
AdaBelief [ 344 ]	★★★★	★★	★★★	★★	★★
Adafactor [ 223 ]	★★	★★★	★★★★	★★★	★★★

Table 7: Comparison of various optimizers across five key dimensions. We evaluate convergence speed, accuracy, memory efficiency, generalization, and hyperparameter robustness, and show that Muon achieves the best overall performance.

Benchmark Task	Hardware Type	Hyperparameter Search (5 LR Scales $\times$ 23 Optimizers)	Final Evaluation (Best LR $\times$ 23 Optimizers)	Total GPU Hours
ResNet-50 [ 87 ] (ImageNet-1K [ 103 ] )	NVIDIA A100	-	$\sim$ 288 Hours (100/300 epochs)	$\sim$ 288 Hours
ViT-S [ 49 ] (ImageNet-1K [ 103 ] )	NVIDIA A100	$\sim$ 544 Hours (100 epochs)	$\sim$ 213 Hours (300 epochs)	$\sim$ 757 Hours
LlaMA-60M [ 153 ] (WikiText-103 [ 172 ] )	NVIDIA A100	$\sim$ 28 Hours (Chinchilla scaling)	-	$\sim$ 28 Hours
Total Aggregated Cost	Equivalent to $\sim$ 45 Days on a single NVIDIA A100 GPU			$\sim$ 1073 Hours

Table 8: Comprehensive estimation of computational cost (GPU Hours). The table details the estimated single NVIDIA A100 GPU hours required for the entire empirical benchmark. The workload accounts for 23 optimizers, including rigorous grid searches across 5 learning rate scales and extended 300-epoch training horizons. The massive total computational footprint (

\sim

1073 GPU Hours) underscores the rigorousness of our evaluation and highlights the fundamental computational barrier in modern optimizer benchmarking.

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 204 papers

Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered
May 15, 2026Sijia Liu, Yicheng Lang, Soumyadeep Pal +6Deep Learning Optimization Zeroth-Order Optimization
1OPTML Lab, Michigan State University, USA · 2Lawrence Livermore National Laboratory, USA
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
Jul 4, 2026Siyuan Li, Jiabao Pan, Yumou Liu +9LMO Methods Deep Learning Optimization
Shanghai Artificial Intelligence Laboratory · Westlake University · Shanghai University +4
Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
May 9, 2026Aditya RanganathDeep Learning Optimization LLM Input Optimization
Center for Applied Scientific Computing Lawrence Livermore National Laboratory Livermore, CA, USA

May 15, 2026J/K move · Enter open · S save

Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

Sijia Liu, Yicheng Lang, Soumyadeep Pal, Changsheng Wang+5

1OPTML Lab, Michigan State University, USA · 2Lawrence Livermore National Laboratory, USA

Deep Learning Optimization Zeroth-Order Optimization Derivative-Free Optimization

Open paper AskPDF arXiv

Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered. We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance-query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization.

Abstract

Deep Learning Optimization

Explore similar work

Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers