cs.LGMar 7, 2026

Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers

Authors: Tao Shi, Liangming Chen, Long Jin, Mengchu Zhou

Organizations: School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · Helen and John C. Hartmann Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In the training of neural networks, adaptive moment estimation (Adam) typically converges fast but exhibits suboptimal generalization performance. A widely accepted explanation for its defect in generalization is that it often tends to converge to sharp minima. To enhance its ability to find flat minima, we propose its new variant named inverse Adam (InvAdam). The key improvement of InvAdam lies in its parameter update mechanism, which is opposite to that of Adam. Specifically, it computes element-wise multiplication of the first-order and second-order moments, while Adam computes the element-wise division of these two moments. This modification aims to increase the step size of the parameter update when the elements in the second-order moments are large and vice versa, which helps the parameter escape sharp minima and stay at flat ones. However, InvAdam's update mechanism may face challenges in convergence. To address this challenge, we propose dual Adam (DualAdam), which integrates the update mechanisms of both Adam and InvAdam, ensuring convergence while enhancing generalization performance. Additionally, we introduce the diffusion theory to mathematically demonstrate InvAdam's ability to escape sharp minima. Extensive experiments are conducted on image classification tasks and large language model (LLM) fine-tuning. The results validate that DualAdam outperforms Adam and its state-of-the-art variants in terms of generalization performance. The code is publicly available at https://github.com/LongJin-lab/DualAdam.

Figures & tables

Fig. 1: Mian idea of DualAdam. (a) The relationship between sharp minimum ϕ\phi and flat one ψ\psi . χ\chi is the saddle point between ϕ\phi and ψ\psi ; (b) The update mechanisms of Adam and InvAdam. m^\boldsymbol{\hat{m}} and v^\boldsymbol{\hat{v}} are the bias-corrected first- and second-order moments, respectively. Subscript ii represents the ii -th element in the vector. Δθi\Delta\boldsymbol{\theta}_{i} is the ii -th element in parameter update in one iteration; and (c) The update mechanism of DualAdam.
Fig. 2: Optimization trajectories of InvAdam and Adam on 2-parameter loss landscapes. The red stars represent the start points, and the black circles represent the end points.
Fig. 3: Test accuracies over epochs on CIFAR-100.
ModelOptimizerCIFAR-10CIFAR-100
Test Accuracy (%)Training TimeTest Accuracy (%)Training Time
ResNet-18Adam [ 10 ]93.66±0.1793.66_{\pm 0.17}36min40s72.56±0.0872.56_{\pm 0.08}36min37s
AdamW [ 17 ]94.09±0.0494.09_{\pm 0.04}36min43s72.81±0.1072.81_{\pm 0.10}37min10s
SWATS [ 21 ]92.96±1.0492.96_{\pm 1.04}43min27s71.40±1.5071.40_{\pm 1.50}44min04s
NAdam [ 19 ]94.01±0.0294.01_{\pm 0.02}37min12s72.06±0.4372.06_{\pm 0.43}37min35s
RAdam [ 18 ]94.11±0.0194.11_{\pm 0.01}37min11s74.79±0.0674.79_{\pm 0.06}36min50s
TABLE I: Top-1 test accuracies (mean±std) and training times on CIFAR-10 and CIFAR-100
ModelOptimizerTest accuracy (%)Traning Time
ResNet-18Adam [ 10 ]50.98±0.4250.98_{\pm 0.42}537min24s
AdamW [ 17 ]51.20±0.2851.20_{\pm 0.28}536min18s
SWATS [ 21 ]49.55±0.4549.55_{\pm 0.45}601min18s
NAdam [ 19 ]51.58±0.0851.58_{\pm 0.08}538min13s
RAdam [ 18 ]55.04±0.3255.04_{\pm 0.32}537min06s
Adan [ 20 ]51.97±0.2151.97_{\pm 0.21}540min18s
TABLE II: Test accuracies (mean±std) on Tiny ImageNet
ModelOptimizerTest Accuracy (%)
ResNet-18Adam [ 10 ]69.5569.55
AdamW [ 17 ]71.2971.29
DualAdam (ours)72.61\mathbf{72.61}
ResNet-50Adam [ 10 ]70.1070.10
AdamW [ 17 ]71.2371.23
DualAdam (ours)73.32\mathbf{73.32}
TABLE III: Test accuracies on ImageNet-1k
Fig. 4: Comparisons of training loss, validation perplexity, and generalization gap of DualAdam and AdamW on the fine-tuning of OpenPangu-1B.
Fig. 5: Comparisons of top Hessian matrix’s eigenvalues, Hessian matrix’s traces, and Hessian matrix’s eigenvalue densities of loss landscapes on the CIFAR-100 dataset using ResNet18.
Fig. 6: Visualization of loss landscapes of Adam and DualAdam. The loss is defined as L(θ⋆+ζl)L(\theta^{\star}+\zeta l) , where ζ\zeta is a scalar that scales the random vector ll drawn from a Gaussian distribution to perturb the optimal parameter θ⋆\theta^{\star} .
Switching RateTraning LossValidation Accuracy (%)
04.58±0.014.58_{\pm 0.01}2.0±0.042.0_{\pm 0.04}
1×10−51\times 10^{-5}5.4±0.35×10−45.4_{\pm 0.35}\times 10^{-4}69.04±0.0869.04_{\pm 0.08}
2×10−52\times 10^{-5}5.5±0.77×10−45.5_{\pm 0.77}\times 10^{-4}70.30±0.2670.30_{\pm 0.26}
3×10−53\times 10^{-5}4.8±0.07×10−44.8_{\pm 0.07}\times 10^{-4}71.04±0.0871.04_{\pm 0.08}
4×10−54\times 10^{-5}5.1±0.35×10−45.1_{\pm 0.35}\times 10^{-4}71.49±0.2171.49_{\pm 0.21}
5×10−55\times 10^{-5}4.8±0.14×10−44.8_{\pm 0.14}\times 10^{-4}71.52±0.2871.52_{\pm 0.28}
TABLE IV: Training losses (mean±std) and validation accuracies (mean±std) of DualAdam on CIFAR-100 using ResNet-18 with different switching rates
Linear SwitchingSwitching RateTest Accuracy (%)
5×10−55\times 10^{-5}75.30±0.3475.30_{\pm 0.34}
8×10−58\times 10^{-5}75.29±0.2175.29_{\pm 0.21}
1×10−41\times 10^{-4}75.13±0.1675.13_{\pm 0.16}
Exponential SwitchingExponential BaseTest Accuracy (%)
0.871.45±0.4771.45_{\pm 0.47}
0.972.82±0.0972.82_{\pm 0.09}
TABLE V: Test accuracies (mean±std) of DualAdam on CIFAR-100 using ResNet-18 with different switching mechanisms
Figure 12
Figure 13
Figure 14
Figure 15
More in this topic

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 331 papers

Explore similar work

CardsList
Jul 7, 2026

Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning

Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data. Building on Sharpness-Aware Minimization (SAM), for seeking flat minima associated with improved generalization, we propose the Extragradient-Inspired Sharpness-Aware Minimization (EISAM), a novel optimizer that enhances generalization via the extragradient technique. EISAM uses a two-step update process: a prediction step investigating the geometry of the loss landscape and a perturbation step that refines updates with a base optimizer. This approach achieves better generalization performance than SAM. Crucially, EISAM reduces sensitivity to the perturbation radius, enhancing robustness, and simplifying the tuning across diverse settings. Extensive experiments on benchmark datasets demonstrate that EISAM consistently outperforms SGD, Adaptive Moment Estimation (Adam), and SAM in test accuracy and training efficiency across various architectures. Theoretical analysis further confirms that EISAM tightens the generalization bound by steering parameters toward flatter minima with reduced curvature. Accompanied by a thorough hyperparameter analysis, EISAM offers practical tuning guidance, establishing it as a robust, scalable, and broadly applicable optimization solution that advances both the theory and practice in deep learning.

Yao Fu, Chunxia Zhang, Junmin Liu +3

School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, 710049, China · SGIT AI Lab, State Grid Corporation of China, Xi’an, 710054, China · School of Software, Xi’an Jiaotong University, Xi’an, 710049, China +2

Neural Network GeneralizationSharpness-Aware MinimizationFlat MinimaExtragradient Methods
AskPDF
Mar 3, 2026

HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization

Adam and AdamW are a class of default optimizers for training deep learning models in machine learning. These adaptive algorithms converge faster but generalize worse compared to SGD. In fact, their proved generalization error O(1N)O(\frac{1}{\sqrt{N}}) also is larger than O(1N)O(\frac{1}{N}) of SGD, where NN denotes training sample size. Recently, although some variants of Adam have been proposed to improve its generalization, their improved generalizations are still unexplored in theory. To fill this gap, in the paper, we restudy generalization of Adam and AdamW via algorithmic stability, and first prove that Adam and AdamW without square-root (i.e., Adam(W)-srf) have a generalization error O(ρ^−2TN)O(\frac{\hatρ^{-2T}}{N}), where TT denotes iteration number and ρ^>0\hatρ>0 denotes the smallest element of second-order momentum plus a small positive number. To improve generalization, we propose a class of efficient clever Adam (i.e., HomeAdam(W)) algorithms via sometimes returning momentum-based SGD. Moreover, we prove that our HomeAdam(W) have a smaller generalization error O(1N)O(\frac{1}{N}) than O(ρ^−2TN)O(\frac{\hatρ^{-2T}}{N}) of Adam(W)-srf, since ρ^\hatρ is generally very small. In particular, it is also smaller than the existing O(1N)O(\frac{1}{\sqrt{N}}) of Adam(W). Meanwhile, we prove our HomeAdam(W) have a faster convergence rate of O(1T1/4)O(\frac{1}{T^{1/4}}) than O(ρ˘−1T1/4)O(\frac{\breveρ^{-1}}{T^{1/4}}) of the Adam(W)-srf, where ρ˘≤ρ^\breveρ\leq\hatρ also is very small. Extensive numerical experiments demonstrate efficiency of our HomeAdam(W) algorithms.

Feihu Huang, Guanyi Zhang, Songcan Chen

College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China

Adaptive Gradient MethodsNeural Network GeneralizationStochastic Optimization Convergence
AskPDF
May 4, 2026

Anon: Extrapolating Adaptivity Beyond SGD and Adam

Adaptive optimizers such as Adam and non-adaptive methods like SGD exhibit distinct generalization capabilities across different architectures. Prior tunable optimizers attempt to bridge this gap by strictly interpolating between SGD and Adam, effectively confining adaptivity within the 0-to-1 bound. However, this restricted interpolation is fundamentally insufficient: we reveal that optimal adaptivity often requires extrapolation, such as negative adaptivity for classical CNNs and adaptivity of at least one (γ≥1γ\geq 1) for Transformers. Extrapolating adaptivity theoretically violates the strict non-decreasing pre-conditioner assumption, often leading to divergence in existing methods. To break this barrier, we propose Anon, an optimizer that achieves fully continuous adaptivity extrapolation across the entire real-number spectrum. To guarantee provable stability in these out-of-bound regimes, we introduce Incremental Delay Update (IDU), a novel mechanism that bypasses hard max-tracking strategies. We theoretically establish Anon's convergence in both convex and non-convex settings. Empirically, by exploring previously unreachable adaptivity landscapes, Anon demonstrates highly competitive and scalable performance among state-of-the-art element-wise optimizers on representative image classification, diffusion, and large language modeling tasks.

Yiheng Zhang, Kaiyan Zhao, Shaowu Wu +5

Wuhan University · University of Macau · University of Calgary

Nonconvex Stochastic OptimizationAdaptive Gradient Methods
AskPDF