Why $β_1 = β_2$ Is Dynamically Special in Adam

Alberto Fernández-Hernández; Cristian Pérez-Corral; Jose I. Mestre; Manuel F. Dolz; Enrique S. Quintana-Ortí

cs.LGJan 29, 2026

Why $β_1 = β_2$ Is Dynamically Special in Adam

Authors: Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí

Organizations: Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

Abstract

Adam has been at the core of large-scale training for almost a decade, yet the role of its two momentum parameters remains poorly understood. Recent work shows that tying $β_{1}=β_{2}$ can preserve Adam's strong performance despite collapsing two memory scales into one, raising a basic question: what becomes dynamically special when the memories are tied? We identify a concrete mechanism. In the continuous-time limit, each normalized-update coordinate decomposes into a sign component, an explicit magnitude-lag term proportional to the difference between the two memory times, and additional transition, curvature, and nonlinear ratio terms. This lag channel vanishes exactly when $β_{1}=β_{2}$ , making the diagonal the unique regime in which this mismatch-induced response is structurally absent. A full-history discrete decomposition on real training gradients recovers this change in composition: tied updates are sign-dominated, whereas the lag term becomes substantial off the diagonal and leaves a comparatively small residual. Across six vision and language tasks, tied configurations also typically exhibit smoother update-norm trajectories. Overall, our results identify memory-scale mismatch as a concrete source of magnitude sensitivity in Adam and provide a mechanistic account of why tied momentum is dynamically distinctive.

Explore similar work

May 11, 2026cs.LG

Refresh-Scaling the Memory of Balanced Adam

Recent evidence suggests that Adam performs robustly when its momentum parameters are tied,

β_1=β_2

, reducing the optimizer to a single remaining parameter. However, how this parameter should be set remains poorly understood. We argue that, in balanced Adam,

β

should not be treated as a dimensionless constant: it defines a statistical memory horizon

H_β=(1-β)^{-1}

. In terms of the effective learning horizon

T_{\mathrm{ES}}

, estimated from the validation trajectory, we study the refresh count

R_β=(1-β)T_{\mathrm{ES}}

, which measures how many times Adam renews its internal statistics during the useful phase of training. Across 11 vision and language experiments, we find that choosing

β

so that

R_β\approx1000

selects different

β

values depending on the training scale, yet improves robustness over the best fixed-beta baseline. Compared with the strongest fixed choice

β=0.944

, the refresh rule improves worst-case robustness, reducing the maximum relative gap in validation loss by 33.4%, while bringing all 11 runs within 1% of their validation oracle. These results suggest that the remaining hyperparameter of balanced Adam is more naturally viewed as a memory-scale variable than as a fixed constant. This provides a simple budget-aware perspective on optimizer scaling and opens a path toward treating Adam's momentum as part of the learning dynamics rather than as a static default.

Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2

Adam Balanced Learning Response Magnitude

Ask PDF

Sep 16, 2026cs.LG

Beyond Quadratic Loss: The Stability Phase Diagram of Adam

Loss spikes are recurrent instabilities in neural-network training and can arise from multiple mechanisms. For Adam in particular, macroscopic loss spikes have been linked to optimizer dynamics, yet how its two momentum timescales govern them remains unclear. We investigate this dependence by mapping training dynamics across the

(β_1,β_2)

plane. Across a range of model--task settings, an approximately linear boundary,

1-β_2=C(1-β_1)

, separates spiky from non-spiky dynamics, whereas a one-dimensional quadratic loss produces approximately cubic slope. A one-dimensional superquadratic loss

L(x)\propto|x|^n

recovers the near-linear scaling and links the boundary coefficient to the effective loss exponent

n

. We further show that confident cross-entropy losses develop a core--wall landscape comprising a narrow quadratic core followed by a steep wall, which produces effective superquadratic behavior at the scale of an optimizer update. Together, these results connect Adam loss spikes to both the mismatch between momentum timescales and finite-scale superquadratic loss geometry beyond the Hessian.

Gaoxiang Tang, Huanran Chen, Ziming Liu

Adam Loss Landscape Loss Function Phase Transitions Temporal Scales

Ask PDF

Jan 30, 2026cs.LG

Adaptive Momentum and Nonlinear Damping for Neural Network Training

Momentum Stochastic Gradient Descent (mSGD) relies on a fixed momentum coefficient shared across all parameters, failing to account for the heterogeneous structure of modern loss landscapes. In this work, we adopt a continuous-time formulation to introduce individual, adaptive momentum coefficients regulated by the kinetic energy of each model parameter. This mechanism automatically adjusts to evolving training dynamics to maintain stability without sacrificing convergence speed. We demonstrate that this adaptive friction is inextricably linked to cubic damping, a suppression mechanism from structural dynamics. We additionally introduce two optimization schemes by augmenting the continuous dynamics of mSGD and Adam with a cubic damping term. Empirically, our methods demonstrate robustness and match or outperform Adam on training ViT, BERT, and GPT2 tasks where mSGD typically struggles. We further provide theoretical results establishing the exponential convergence of the proposed schemes.

Aikaterini Karoni, Rajit Rajpal, Benedict Leimkuhler +1

Adam

Ask PDF