The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

Yihe Zhou; Tongtian Zhu; Yingxiao Huo; Satya Prakash Dash; Can Wang; Samuel Kaski; Mingfei Sun

cs.LGSep 28, 2026

The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

Authors: Yihe Zhou, Tongtian Zhu, Yingxiao Huo, Satya Prakash Dash, Can Wang, Samuel Kaski, Mingfei Sun

Organizations: Department of Computer Science, The University of Manchester, Manchester, UK · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

Abstract

Adam is the default optimizer for training modern deep neural networks, yet its adaptive behavior remains poorly understood due to the complex interaction between its first- and second-moment exponential moving averages (EMAs). We study Adam in the tied- $β$ regime, where the two EMA decay rates are equal, and show that its adaptive dynamics can be expressed through a transformed ratio with approximately scale-stable behavior. Empirically, this transformed ratio exhibits a stable, heavy-tailed distribution across tasks, model scales, and training stages, in contrast to the variability of raw moment magnitudes. This empirical stability has both practical and conceptual consequences. First, we derive a recurrence for the transformed ratio, yielding a reparameterization of Adam that replaces the second moment with a compressible state. Leveraging its stable distribution, we show that a fixed 4-bit codebook is sufficient in our experiments to store this state without auxiliary scaling, achieving performance competitive with full-precision Adam. Second, the transformed ratio view clarifies Adam's connection to sign-based methods: Adam reduces to sign-based momentum modulated by the transformed ratio, and replacing it with a constant recovers Signum as a limiting case. This perspective further provides a simple rule for transferring learning rates between the two methods. Together, these results suggest that tied- $β$ Adam admits a simple and approximately stable ratio structure underlying its adaptive behavior and demonstrate its utility for both analysis and efficient implementation.

Figures & tables

$c$	$\beta=0.90$	$\beta=0.92$	$\beta=0.95$	$\beta=0.97$
0	3.36	3.31	3.25	3.21
0.1	3.33	3.28	3.22	3.18
0.2	3.24	3.19	3.13	3.09

Table 1 :

y_{\mathrm{const}}^{\mathrm{eff}}

under

Y_{c}=2/(Z+c)^{2}

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$\beta$	Adam	TR-Adam (FP32- $m_{t}$ , 4-bit $y_{t}$ )	TR-Adam (FP8- $m_{t}$ , 4-bit $y_{t}$ )	$\Delta$ FP32	$\Delta$ FP8
Llama-style 20M	0.90	3.694	3.706	3.710	+0.012	+0.016
Llama-style 20M	0.92	3.676	3.689	3.688	+0.013	+0.012
Llama-style 20M	0.95	3.672	3.672	3.670	-0.000	-0.002
Llama-style 20M	0.97	3.692	3.677	3.675	-0.014	-0.016
Pythia-style 160M	0.90	3.373	3.378	3.387	+0.004	+0.013
Pythia-style 160M	0.92	3.355	3.365	3.369	+0.010	+0.014

Appendix

Table 2: Pre-training validation loss averaged over the final three validation points. For Llama-style 20M, the learning rate is selected from the full learning-rate sweep; for Pythia-style 160M, it is selected from

\{1,3,5,8,10\}\times 10^{-3}

. Lower is better. Here

\Delta=\mathrm{TR\text{-}Adam}-\mathrm{Adam}

Model	$\beta$	Adam	TR-Adam (FP32- $m_{t}$ , 4-bit $y_{t}$ )	TR-Adam (FP8- $m_{t}$ , 4-bit $y_{t}$ )	$\Delta$ FP32	$\Delta$ FP8
Pythia-1B	0.90	1.299	1.297	1.297	-0.002	-0.002
Pythia-1B	0.92	1.302	1.295	1.295	-0.007	-0.006
Pythia-1B	0.95	1.309	1.299	1.299	-0.011	-0.011
Llama-3.2-1B	0.90	1.075	1.072	1.072	-0.002	-0.002
Llama-3.2-1B	0.92	1.076	1.071	1.071	-0.005	-0.005
Llama-3.2-1B	0.95	1.082	1.073	1.073	-0.009	-0.009

Appendix

Table 3: SFT validation loss averaged over the final three validation points. Lower is better. Here

\Delta=\mathrm{TR\text{-}Adam}-\mathrm{Adam}

$\beta$	Adam	TR-Adam (FP32- $m_{t}$ , 4-bit $y_{t}$ )	TR-Adam (FP8- $m_{t}$ , 4-bit $y_{t}$ )	$\Delta$ FP32	$\Delta$ FP8
0.90	0.788	0.825	0.813	+0.036	+0.025
0.92	0.800	0.824	0.819	+0.024	+0.019
0.95	0.808	0.817	0.811	+0.009	+0.003

Appendix

Table 4: ReMax validation reward averaged over the final three validation points. Higher is better. Here

\Delta=\mathrm{TR\text{-}Adam}-\mathrm{Adam}

Dataset	FineWeb 10B / 10B / 100B
Models	Llama-style 20M / Pythia-style 160M / Pythia-1B
Training budget	1.5B / 4B / 20B tokens
Sequence length	2048
Batch size	256
Warmup	512 / 1024 / 1024 steps
LR schedule	constant after warmup

Appendix

Table 5 : Pre-training experimental setup.

Dataset	UltraChat
Models	Pythia-1B, Llama-3.2-1B
Training budget	2000 steps
Maximum sequence length	2048
Batch size	128
Learning rate	$1\times 10^{-5}$
LR schedule	constant after warmup

Appendix

Table 6 : SFT experimental setup.

Dataset	UltraFeedback
Actor model	Llama-3.2-1B
Reward model	Skywork-Reward-V2-Llama-3.2-3B
Algorithm	ReMax Li et al. [2024]
Training budget	600 steps
Batch size	64 prompts
Maximum prompt length	768

Appendix

Table 7 : RLHF/ReMax experimental setup.

Explore similar work

Jan 29, 2026cs.LG

Why $β_1 = β_2$ Is Dynamically Special in Adam

Adam has been at the core of large-scale training for almost a decade, yet the role of its two momentum parameters remains poorly understood. Recent work shows that tying

β_{1}=β_{2}

can preserve Adam's strong performance despite collapsing two memory scales into one, raising a basic question: what becomes dynamically special when the memories are tied? We identify a concrete mechanism. In the continuous-time limit, each normalized-update coordinate decomposes into a sign component, an explicit magnitude-lag term proportional to the difference between the two memory times, and additional transition, curvature, and nonlinear ratio terms. This lag channel vanishes exactly when

β_{1}=β_{2}

, making the diagonal the unique regime in which this mismatch-induced response is structurally absent. A full-history discrete decomposition on real training gradients recovers this change in composition: tied updates are sign-dominated, whereas the lag term becomes substantial off the diagonal and leaves a comparatively small residual. Across six vision and language tasks, tied configurations also typically exhibit smoother update-norm trajectories. Overall, our results identify memory-scale mismatch as a concrete source of magnitude sensitivity in Adam and provide a mechanistic account of why tied momentum is dynamically distinctive.

Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2

Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

Adam Momentum

Ask PDF

May 11, 2026cs.LG

Refresh-Scaling the Memory of Balanced Adam

Recent evidence suggests that Adam performs robustly when its momentum parameters are tied,

β_1=β_2

, reducing the optimizer to a single remaining parameter. However, how this parameter should be set remains poorly understood. We argue that, in balanced Adam,

β

should not be treated as a dimensionless constant: it defines a statistical memory horizon

H_β=(1-β)^{-1}

. In terms of the effective learning horizon

T_{\mathrm{ES}}

, estimated from the validation trajectory, we study the refresh count

R_β=(1-β)T_{\mathrm{ES}}

, which measures how many times Adam renews its internal statistics during the useful phase of training. Across 11 vision and language experiments, we find that choosing

β

so that

R_β\approx1000

selects different

β

values depending on the training scale, yet improves robustness over the best fixed-beta baseline. Compared with the strongest fixed choice

β=0.944

, the refresh rule improves worst-case robustness, reducing the maximum relative gap in validation loss by 33.4%, while bringing all 11 runs within 1% of their validation oracle. These results suggest that the remaining hyperparameter of balanced Adam is more naturally viewed as a memory-scale variable than as a fixed constant. This provides a simple budget-aware perspective on optimizer scaling and opens a path toward treating Adam's momentum as part of the learning dynamics rather than as a static default.

Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2

Universitat Polit`ecnica de Val`encia, Val`encia, Spain · Universitat Jaume I, Castell´o de la Plana, Spain

Adam Momentum

Ask PDF

Date pendingstat.ML

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-preconditioned mean-gradient operator, and high-probability projected stationarity guarantees under general

L

-smooth objectives. In the tracking regime, we derive finite-time expected and high-probability bounds that decompose sharply into four components: initialization, objective drift, a first-moment tracking error governed by

\beta_1

, and a preconditioner perturbation governed by

\beta_2

. We characterize the burn-in time required for the transient terms to decay to the asymptotic tracking bound under constant and step-decay schedules. We also prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. Across both analyses, our bounds reveal a noise--drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can yield favorable upper guarantees, whereas in drift-dominated regimes, stale first-moment information and preconditioner perturbations can enlarge Adam's tracking guarantee, potentially allowing vanilla SGD to attain a smaller tracking error. Our explicit

(\beta_1,\beta_2,\epsilon)

-dependent bounds identify mechanisms through which adaptive step-sizing can help or hurt under nonstationarity and provide theoretical explanations consistent with Adam's empirical instability and stabilization under distribution shift.

Sharan Sahu, Abir Sarkar, Cameron J. Hogan +1

Department of Statistics and Data Science Cornell University

Adam Stationarity Stochastic Gradient Descent Spectral Preconditioning Gradient Staleness Monotonicity+1

Ask PDF

Abstract

Explore similar work

Why β1=β2β_1 = β_2β1​=β2​ Is Dynamically Special in Adam

Refresh-Scaling the Memory of Balanced Adam

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

Why $β_1 = β_2$ Is Dynamically Special in Adam