cs.LGMar 7, 2026

Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers

Authors: Tao Shi, Liangming Chen, Long Jin, Mengchu Zhou

Organizations: School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · Helen and John C. Hartmann Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In the training of neural networks, adaptive moment estimation (Adam) typically converges fast but exhibits suboptimal generalization performance. A widely accepted explanation for its defect in generalization is that it often tends to converge to sharp minima. To enhance its ability to find flat minima, we propose its new variant named inverse Adam (InvAdam). The key improvement of InvAdam lies in its parameter update mechanism, which is opposite to that of Adam. Specifically, it computes element-wise multiplication of the first-order and second-order moments, while Adam computes the element-wise division of these two moments. This modification aims to increase the step size of the parameter update when the elements in the second-order moments are large and vice versa, which helps the parameter escape sharp minima and stay at flat ones. However, InvAdam's update mechanism may face challenges in convergence. To address this challenge, we propose dual Adam (DualAdam), which integrates the update mechanisms of both Adam and InvAdam, ensuring convergence while enhancing generalization performance. Additionally, we introduce the diffusion theory to mathematically demonstrate InvAdam's ability to escape sharp minima. Extensive experiments are conducted on image classification tasks and large language model (LLM) fine-tuning. The results validate that DualAdam outperforms Adam and its state-of-the-art variants in terms of generalization performance. The code is publicly available at https://github.com/LongJin-lab/DualAdam.

Figures & tables

Fig. 1: Mian idea of DualAdam. (a) The relationship between sharp minimum ϕ\phi and flat one ψ\psi . χ\chi is the saddle point between ϕ\phi and ψ\psi ; (b) The update mechanisms of Adam and InvAdam. m^\boldsymbol{\hat{m}} and v^\boldsymbol{\hat{v}} are the bias-corrected first- and second-order moments, respectively. Subscript ii represents the ii -th element in the vector. Δθi\Delta\boldsymbol{\theta}_{i} is the ii -th element in parameter update in one iteration; and (c) The update mechanism of DualAdam.
Fig. 2: Optimization trajectories of InvAdam and Adam on 2-parameter loss landscapes. The red stars represent the start points, and the black circles represent the end points.
Fig. 3: Test accuracies over epochs on CIFAR-100.
ModelOptimizerCIFAR-10CIFAR-100
Test Accuracy (%)Training TimeTest Accuracy (%)Training Time
ResNet-18Adam [ 10 ]93.66±0.1793.66_{\pm 0.17}36min40s72.56±0.0872.56_{\pm 0.08}36min37s
AdamW [ 17 ]94.09±0.0494.09_{\pm 0.04}36min43s72.81±0.1072.81_{\pm 0.10}37min10s
SWATS [ 21 ]92.96±1.0492.96_{\pm 1.04}43min27s71.40±1.5071.40_{\pm 1.50}44min04s
NAdam [ 19 ]94.01±0.0294.01_{\pm 0.02}37min12s72.06±0.4372.06_{\pm 0.43}37min35s
RAdam [ 18 ]94.11±0.0194.11_{\pm 0.01}37min11s74.79±0.0674.79_{\pm 0.06}36min50s
TABLE I: Top-1 test accuracies (mean±std) and training times on CIFAR-10 and CIFAR-100
ModelOptimizerTest accuracy (%)Traning Time
ResNet-18Adam [ 10 ]50.98±0.4250.98_{\pm 0.42}537min24s
AdamW [ 17 ]51.20±0.2851.20_{\pm 0.28}536min18s
SWATS [ 21 ]49.55±0.4549.55_{\pm 0.45}601min18s
NAdam [ 19 ]51.58±0.0851.58_{\pm 0.08}538min13s
RAdam [ 18 ]55.04±0.3255.04_{\pm 0.32}537min06s
Adan [ 20 ]51.97±0.2151.97_{\pm 0.21}540min18s
TABLE II: Test accuracies (mean±std) on Tiny ImageNet
ModelOptimizerTest Accuracy (%)
ResNet-18Adam [ 10 ]69.5569.55
AdamW [ 17 ]71.2971.29
DualAdam (ours)72.61\mathbf{72.61}
ResNet-50Adam [ 10 ]70.1070.10
AdamW [ 17 ]71.2371.23
DualAdam (ours)73.32\mathbf{73.32}
TABLE III: Test accuracies on ImageNet-1k
Fig. 4: Comparisons of training loss, validation perplexity, and generalization gap of DualAdam and AdamW on the fine-tuning of OpenPangu-1B.
Fig. 5: Comparisons of top Hessian matrix’s eigenvalues, Hessian matrix’s traces, and Hessian matrix’s eigenvalue densities of loss landscapes on the CIFAR-100 dataset using ResNet18.
Fig. 6: Visualization of loss landscapes of Adam and DualAdam. The loss is defined as L(θ⋆+ζl)L(\theta^{\star}+\zeta l) , where ζ\zeta is a scalar that scales the random vector ll drawn from a Gaussian distribution to perturb the optimal parameter θ⋆\theta^{\star} .
Switching RateTraning LossValidation Accuracy (%)
04.58±0.014.58_{\pm 0.01}2.0±0.042.0_{\pm 0.04}
1×10−51\times 10^{-5}5.4±0.35×10−45.4_{\pm 0.35}\times 10^{-4}69.04±0.0869.04_{\pm 0.08}
2×10−52\times 10^{-5}5.5±0.77×10−45.5_{\pm 0.77}\times 10^{-4}70.30±0.2670.30_{\pm 0.26}
3×10−53\times 10^{-5}4.8±0.07×10−44.8_{\pm 0.07}\times 10^{-4}71.04±0.0871.04_{\pm 0.08}
4×10−54\times 10^{-5}5.1±0.35×10−45.1_{\pm 0.35}\times 10^{-4}71.49±0.2171.49_{\pm 0.21}
5×10−55\times 10^{-5}4.8±0.14×10−44.8_{\pm 0.14}\times 10^{-4}71.52±0.2871.52_{\pm 0.28}
TABLE IV: Training losses (mean±std) and validation accuracies (mean±std) of DualAdam on CIFAR-100 using ResNet-18 with different switching rates
Linear SwitchingSwitching RateTest Accuracy (%)
5×10−55\times 10^{-5}75.30±0.3475.30_{\pm 0.34}
8×10−58\times 10^{-5}75.29±0.2175.29_{\pm 0.21}
1×10−41\times 10^{-4}75.13±0.1675.13_{\pm 0.16}
Exponential SwitchingExponential BaseTest Accuracy (%)
0.871.45±0.4771.45_{\pm 0.47}
0.972.82±0.0972.82_{\pm 0.09}
TABLE V: Test accuracies (mean±std) of DualAdam on CIFAR-100 using ResNet-18 with different switching mechanisms
Figure 12
Figure 13
Figure 14
Figure 15
More in this topic

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 332 papers

Explore similar work

CardsList

  1. Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning

    Jul 7, 2026Yao Fu, Chunxia Zhang, Junmin Liu +3Neural Network GeneralizationSharpness-Aware Minimization

    School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, 710049, China · SGIT AI Lab, State Grid Corporation of China, Xi’an, 710054, China · School of Software, Xi’an Jiaotong University, Xi’an, 710049, China +2

  2. HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization

    Mar 3, 2026Feihu Huang, Guanyi Zhang, Songcan ChenAdaptive Gradient MethodsNeural Network Generalization

    College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China

  3. Anon: Extrapolating Adaptivity Beyond SGD and Adam

    May 4, 2026Yiheng Zhang, Kaiyan Zhao, Shaowu Wu +5Nonconvex Stochastic OptimizationAdaptive Gradient Methods

    Wuhan University · University of Macau · University of Calgary