cs.LGApr 14, 2026

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

Authors: Tong Zhang, Jiangning Zhang, Zhucun Xue, Juntao Jiang, Yicheng Xu, Chengming Xu, Teng Hu, Xingyu Xie, 

Organizations: Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University · National University of Singapore

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Balancing convergence speed, generalization capability, and computational efficiency remains a core challenge in deep learning optimization. First-order gradient descent methods, epitomized by stochastic gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines. However, large-scale model training, stringent differential privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency. To mitigate these bottlenecks, researchers explore second-order optimization techniques to surpass first-order performance ceilings, while zeroth-order methods reemerge to alleviate memory constraints inherent to large-scale training. Despite this proliferation of methodologies, the field lacks a cohesive framework that unifies underlying principles and delineates application scenarios for these disparate approaches. In this work, we retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstream optimizers across diverse model architectures and training scenarios. We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research. By synthesizing theoretical insights with extensive empirical evidence, we provide actionable guidance for designing next-generation highly efficient, robust, and trustworthy optimization methods. The code is available at https://github.com/APRIL-AIGC/Awesome-Optimizer.

Figures & tables

SymbolDescription
θt\theta_{t}Model parameters (weights and biases) evaluated at iteration tt .
η\etaLearning rate, determining the step size of parameter updates .
∇J(θt)\nabla J(\theta_{t})Exact gradient of the objective function with respect to parameters at step tt .
gtg_{t}Stochastic gradient (mini-batch gradient) of the objective function at step tt .
mtm_{t}Biased first moment estimate (exponential moving average of past gradients) .
vtv_{t}Biased second raw moment estimate (exponential moving average of past squared gradients) .
Table 1: Nomenclature of key variables and hyperparameters in gradient-based optimization. This table summarizes the unified mathematical notations consistently used throughout the paper.
OptimizerVenueTypeLimitationHighlight
First-Order Methods ( Sec. 3.2 )
SGDM [ 208 ]Nature’1986Accelerating Convergence RateComplex hyperparameter tuning.First proposed to introduce a momentum mechanism to accelerate convergence.
Adam [ 123 ]ICLR’15Strategies for Adaptive Step-Size EstimationHigh memory usage, worse generalization.First proposed a method combining adaptive step-size estimation with momentum.
AdaBelief [ 344 ]NeurIPS’20Strategies for Adaptive Step-Size Estimationϵ\epsilon requires task-specific tuning.Compared to adaptive methods like Adam, it achieves better generalization.
Shampoo [ 83 ]ICML’18Approximating Curvature InformationComputationally expensive.First proposed to exploit tensor structures to maintain preconditioners
4-bit Shampoo [ 281 ]NeurIPS’24Approximating Curvature InformationComputationally expensive.Compared to standard Shampoo, it significantly reduces memory usage by quantizing optimizer states to 4-bit.
Table 2: Taxonomy and Comparison of Representative Optimization Methods. This table presents a systematic evaluation of various optimizers categorized into FO, SO, and ZO paradigms, shows their limitations and highlights.
ProcedureDeiTA2A3
DatasetIN-1KIN-1KIN-1K
Epochs300300100
Batch size102410241024
OptimizerAdamWAdamWAdamW
Learning rate1×10−31\times 10^{-3}1×10−31\times 10^{-3}1×10−31\times 10^{-3}
Optimizer Momentum0.9, 0.9990.9, 0.9990.9, 0.999
Table 3: Ingredients and hyper-parameters used for image classification training settings. Taking ImageNet-1K as the template setups, the settings of A2/A3 [ 257 ] take ResNet-50 [ 87 ] as the examples, the DeiT [ 257 ] setting takes ViT-S [ 49 ] as the example. Gray regions should be modified for each optimizer.
Figure 8: Comparison of Top-1 accuracy and learning rate sensitivity analysis of various optimizers. Each subfigure demonstrates the robustness and sensitivity of individual optimizers across a range of scaled learning rates (from 0.1× to 10× the base learning rate). Figures demonstrate that convergence stability and final Top-1 accuracy heavily depend on precise hyperparameter tuning.
Optimizer0.1 ×\times LR0.2 ×\times LR1.0 ×\times LR5.0 ×\times LR10.0 ×\times LRBest LR
SGD [ 205 ]48.05655.91864.36060.59059.2620.1
RMSprop [ 256 ]54.25443.41620.5126.1140-1e-4
Adam [ 123 ]66.23470.77272.240--1e-3
AdaBelief [ 344 ]29.24840.21058.56266.52868.6421e-2
Adafactor [ 223 ]66.29470.83474.63073.344-1e-3
RAdam [ 148 ]66.22070.82671.190--1e-3
Table 4: Comparison of various optimizers’ hyperparameter robustness and learning rate settings.
OptimizerViT-SResNet-50Llama-60m
Acc (%)Acc (%)Base LR5.0 ×\times LR0.2 ×\times LR
SGD [ 205 ]64.36075.926---
RMSprop [ 256 ]54.25471.536522.57269.271178.34
Adam [ 123 ]72.24074.480614.93357.95964.46
AdaBelief [ 344 ]68.64274.23243.2741.8972.43
Adafactor [ 223 ]74.63076.80813.7816.5516.48
Table 5: Results on ViT-S [ 49 ] , ResNet-50 [ 87 ] , and Llama-60m [ 153 ] models across various optimizers. We compare different learning rate multipliers on Llama-60m [ 153 ] and show that Muon [ 117 ] consistently achieves strong performance across all architectures.
Figure 9: Optimizers’ performance on Llama-60m. Subfigures show validation loss curves of various optimizers across three learning rate settings. The robustness to learning rate variations differs significantly among methods, with some optimizers (e.g., Muon [ 117 ] , MARS [ 309 ] ) maintaining stable convergence across all scales while others (e.g., SGD [ 205 ] , SGDP [ 89 ] ) rapidly diverge. Missing data indicates gradient explosion (loss becomes NaN) at the initial step or the point of discontinuation.
Figure 11: Top-1 accuracy and trajectory correlation of popular optimizers. The main line plots display the Top-1 accuracy performance of 23 optimizers on ViT-S [ 49 ] and ResNet-50 [ 87 ] models over 100 and 300 epochs. The inset heatmaps present the Pearson correlation matrices calculated via first-order differencing of the validation metrics, revealing the algorithmic categorization and trajectory divergence across different architectures and training horizons.
ModelOptimizer100 Epoch Acc (%)300 Epoch Acc (%)Improvement ( Δ\Delta )
ViT-SSGD [ 205 ]64.36073.774+ 9.414
RMSprop [ 256 ]54.25457.090+ 2.836
Adam [ 123 ]72.24077.274+ 5.034
AdaBelief [ 344 ]68.64274.640+ 5.998
Adafactor [ 223 ]74.63080.108+ 5.478
RAdam [ 148 ]71.19077.032+ 5.842
Table 6: Results on ViT-S [ 49 ] and ResNet-50 [ 87 ] models across various optimizers, where we compare the 100 epoch accuracy and 300 epoch accuracy and show that ViT-S [ 49 ] achieves larger performance improvements from extended training than ResNet-50 [ 87 ] .
OptimizerConvergenceAccuracyMemoryGeneralizationHyperparameter
Speed Efficiency Robustness
SGD [ 205 ]★★★★★★★★★★★
RMSprop [ 256 ]★★★★★★★★★★
Adam [ 123 ]★★★★★★★★★★★★★★
AdaBelief [ 344 ]★★★★★★★★★★★★★
Adafactor [ 223 ]★★★★★★★★★★★★★★★
Table 7: Comparison of various optimizers across five key dimensions. We evaluate convergence speed, accuracy, memory efficiency, generalization, and hyperparameter robustness, and show that Muon achieves the best overall performance.
Benchmark TaskHardware TypeHyperparameter Search (5 LR Scales ×\times 23 Optimizers)Final Evaluation (Best LR ×\times 23 Optimizers)Total GPU Hours
ResNet-50 [ 87 ] (ImageNet-1K [ 103 ] )NVIDIA A100-∼\sim 288 Hours (100/300 epochs)∼\sim 288 Hours
ViT-S [ 49 ] (ImageNet-1K [ 103 ] )NVIDIA A100∼\sim 544 Hours (100 epochs)∼\sim 213 Hours (300 epochs)∼\sim 757 Hours
LlaMA-60M [ 153 ] (WikiText-103 [ 172 ] )NVIDIA A100∼\sim 28 Hours (Chinchilla scaling)-∼\sim 28 Hours
Total Aggregated CostEquivalent to ∼\sim 45 Days on a single NVIDIA A100 GPU∼\sim 1073 Hours
Table 8: Comprehensive estimation of computational cost (GPU Hours). The table details the estimated single NVIDIA A100 GPU hours required for the entire empirical benchmark. The workload accounts for 23 optimizers, including rigorous grid searches across 5 learning rate scales and extended 300-epoch training horizons. The massive total computational footprint ( ∼\sim 1073 GPU Hours) underscores the rigorousness of our evaluation and highlights the fundamental computational barrier in modern optimizer benchmarking.
More in this topic

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 204 papers

Explore similar work

CardsList
May 15, 2026

Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered. We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance-query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization.

Sijia Liu, Yicheng Lang, Soumyadeep Pal +6

1OPTML Lab, Michigan State University, USA · 2Lawrence Livermore National Laboratory, USA

Deep Learning OptimizationZeroth-Order OptimizationDerivative-Free Optimization
AskPDF
Jul 4, 2026

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented. We therefore present OmniOpt, a unified survey and benchmark cookbook of optimizers for the research community. OmniOpt rests on four coupled components. First, we treat every optimizer update as a structured transformation through a five-stage meta-pipeline, and show that most methods engage only one or two of these stages. Second, we use norm-constrained linear minimization oracles (LMOs) to unify different optimizers. Third, these two views ground a dual-dimension taxonomy, one dimension assigning each method to a mechanism family and the other recording the measurable training objectives it aims to improve. Fourth, and at the core of this paper, we instantiate the full taxonomy in a unified cross-domain benchmark spanning representative optimizers, model scales, and training regimes from language model pretraining to image classification, systematically analyzing each method family across multiple effect objectives and laying out their trade-offs. OmniOpt thus supplies the research community with an operational coordinate system for selecting optimizers under explicit mechanism and objective assumptions, and charts a direction for the future development of the optimizer community.

Siyuan Li, Jiabao Pan, Yumou Liu +9

Shanghai Artificial Intelligence Laboratory · Westlake University · Shanghai University +4

LMO MethodsDeep Learning OptimizationNeural Network Optimization
AskPDF
May 9, 2026

Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers

Training large language models requires optimization algorithms that are not only statistically effective, but also computationally and memory efficient at extreme scale. Although Adam remains the dominant optimizer for large-scale language-model pretraining and fine-tuning, recent work has revisited nearly every component of the optimization stack: adaptive moment estimation, decoupled weight decay, memory footprint, curvature approximation, sign-based updates, large-batch stability, low-rank gradient structure, and matrix-wise orthogonalized updates. This survey reviews optimizer design for large language models through a systems-and-optimization lens. We organize the literature into classical first-order optimizers, adaptive optimizers, memory-efficient variants, second-order and curvature-aware methods, sign-based and discovered optimizers, low-rank and projection-based methods, and matrix-based optimizers such as Muon. We also discuss benchmarking methodology, including hyperparameter fairness, scale dependence, wall-clock efficiency, token efficiency, memory overhead, and downstream evaluation. We argue that optimizer research for LLMs is entering a new phase: moving from single-algorithm speedup claims toward rigorous, scale-aware comparisons that jointly evaluate convergence, stability, memory, and implementation complexity.

Aditya Ranganath

Center for Applied Scientific Computing Lawrence Livermore National Laboratory Livermore, CA, USA

Deep Learning OptimizationLLM Input OptimizationMemory-Efficient OptimizationMatrix OptimizationLLM Training
AskPDF