The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

Yihe Zhou; Tongtian Zhu; Yingxiao Huo; Satya Prakash Dash; Can Wang; Samuel Kaski; Mingfei Sun

cs.LGSep 28, 2026

The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

Authors: Yihe Zhou, Tongtian Zhu, Yingxiao Huo, Satya Prakash Dash, Can Wang, Samuel Kaski, Mingfei Sun

Organizations: Department of Computer Science, The University of Manchester, Manchester, UK · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

Abstract

Adam is the default optimizer for training modern deep neural networks, yet its adaptive behavior remains poorly understood due to the complex interaction between its first- and second-moment exponential moving averages (EMAs). We study Adam in the tied- $β$ regime, where the two EMA decay rates are equal, and show that its adaptive dynamics can be expressed through a transformed ratio with approximately scale-stable behavior. Empirically, this transformed ratio exhibits a stable, heavy-tailed distribution across tasks, model scales, and training stages, in contrast to the variability of raw moment magnitudes. This empirical stability has both practical and conceptual consequences. First, we derive a recurrence for the transformed ratio, yielding a reparameterization of Adam that replaces the second moment with a compressible state. Leveraging its stable distribution, we show that a fixed 4-bit codebook is sufficient in our experiments to store this state without auxiliary scaling, achieving performance competitive with full-precision Adam. Second, the transformed ratio view clarifies Adam's connection to sign-based methods: Adam reduces to sign-based momentum modulated by the transformed ratio, and replacing it with a constant recovers Signum as a limiting case. This perspective further provides a simple rule for transferring learning rates between the two methods. Together, these results suggest that tied- $β$ Adam admits a simple and approximately stable ratio structure underlying its adaptive behavior and demonstrate its utility for both analysis and efficient implementation.

Figures & tables

$c$	$\beta=0.90$	$\beta=0.92$	$\beta=0.95$	$\beta=0.97$
0	3.36	3.31	3.25	3.21
0.1	3.33	3.28	3.22	3.18
0.2	3.24	3.19	3.13	3.09

Table 1 :

y_{\mathrm{const}}^{\mathrm{eff}}

under

Y_{c}=2/(Z+c)^{2}

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$\beta$	Adam	TR-Adam (FP32- $m_{t}$ , 4-bit $y_{t}$ )	TR-Adam (FP8- $m_{t}$ , 4-bit $y_{t}$ )	$\Delta$ FP32	$\Delta$ FP8
Llama-style 20M	0.90	3.694	3.706	3.710	+0.012	+0.016
Llama-style 20M	0.92	3.676	3.689	3.688	+0.013	+0.012
Llama-style 20M	0.95	3.672	3.672	3.670	-0.000	-0.002
Llama-style 20M	0.97	3.692	3.677	3.675	-0.014	-0.016
Pythia-style 160M	0.90	3.373	3.378	3.387	+0.004	+0.013
Pythia-style 160M	0.92	3.355	3.365	3.369	+0.010	+0.014

Appendix

Table 2: Pre-training validation loss averaged over the final three validation points. For Llama-style 20M, the learning rate is selected from the full learning-rate sweep; for Pythia-style 160M, it is selected from

\{1,3,5,8,10\}\times 10^{-3}

. Lower is better. Here

\Delta=\mathrm{TR\text{-}Adam}-\mathrm{Adam}

Model	$\beta$	Adam	TR-Adam (FP32- $m_{t}$ , 4-bit $y_{t}$ )	TR-Adam (FP8- $m_{t}$ , 4-bit $y_{t}$ )	$\Delta$ FP32	$\Delta$ FP8
Pythia-1B	0.90	1.299	1.297	1.297	-0.002	-0.002
Pythia-1B	0.92	1.302	1.295	1.295	-0.007	-0.006
Pythia-1B	0.95	1.309	1.299	1.299	-0.011	-0.011
Llama-3.2-1B	0.90	1.075	1.072	1.072	-0.002	-0.002
Llama-3.2-1B	0.92	1.076	1.071	1.071	-0.005	-0.005
Llama-3.2-1B	0.95	1.082	1.073	1.073	-0.009	-0.009

Appendix

Table 3: SFT validation loss averaged over the final three validation points. Lower is better. Here

\Delta=\mathrm{TR\text{-}Adam}-\mathrm{Adam}

$\beta$	Adam	TR-Adam (FP32- $m_{t}$ , 4-bit $y_{t}$ )	TR-Adam (FP8- $m_{t}$ , 4-bit $y_{t}$ )	$\Delta$ FP32	$\Delta$ FP8
0.90	0.788	0.825	0.813	+0.036	+0.025
0.92	0.800	0.824	0.819	+0.024	+0.019
0.95	0.808	0.817	0.811	+0.009	+0.003

Appendix

Table 4: ReMax validation reward averaged over the final three validation points. Higher is better. Here

\Delta=\mathrm{TR\text{-}Adam}-\mathrm{Adam}

Dataset	FineWeb 10B / 10B / 100B
Models	Llama-style 20M / Pythia-style 160M / Pythia-1B
Training budget	1.5B / 4B / 20B tokens
Sequence length	2048
Batch size	256
Warmup	512 / 1024 / 1024 steps
LR schedule	constant after warmup

Appendix

Table 5 : Pre-training experimental setup.

Dataset	UltraChat
Models	Pythia-1B, Llama-3.2-1B
Training budget	2000 steps
Maximum sequence length	2048
Batch size	128
Learning rate	$1\times 10^{-5}$
LR schedule	constant after warmup

Appendix

Table 6 : SFT experimental setup.

Dataset	UltraFeedback
Actor model	Llama-3.2-1B
Reward model	Skywork-Reward-V2-Llama-3.2-3B
Algorithm	ReMax Li et al. [2024]
Training budget	600 steps
Batch size	64 prompts
Maximum prompt length	768

Appendix

Table 7 : RLHF/ReMax experimental setup.

Explore similar work

Why $β_1 = β_2$ Is Dynamically Special in Adam
Jan 29, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Adam Momentum
Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain
Refresh-Scaling the Memory of Balanced Adam
May 11, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Adam Momentum
Universitat Polit`ecnica de Val`encia, Val`encia, Spain · Universitat Jaume I, Castell´o de la Plana, Spain
Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization
Date pendingSharan Sahu, Abir Sarkar, Cameron J. Hogan +1Adam Stationarity
Department of Statistics and Data Science Cornell University

Jan 29, 2026, cs.LGJ/K move · Enter open · S save

Why $β_1 = β_2$ Is Dynamically Special in Adam

Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz+1

Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

Adam Momentum

Open paper AskPDF arXiv

Adam has been at the core of large-scale training for almost a decade, yet the role of its two momentum parameters remains poorly understood. Recent work shows that tying

β_{1}=β_{2}

can preserve Adam's strong performance despite collapsing two memory scales into one, raising a basic question: what becomes dynamically special when the memories are tied? We identify a concrete mechanism. In the continuous-time limit, each normalized-update coordinate decomposes into a sign component, an explicit magnitude-lag term proportional to the difference between the two memory times, and additional transition, curvature, and nonlinear ratio terms. This lag channel vanishes exactly when

β_{1}=β_{2}

, making the diagonal the unique regime in which this mismatch-induced response is structurally absent. A full-history discrete decomposition on real training gradients recovers this change in composition: tied updates are sign-dominated, whereas the lag term becomes substantial off the diagonal and leaves a comparatively small residual. Across six vision and language tasks, tied configurations also typically exhibit smoother update-norm trajectories. Overall, our results identify memory-scale mismatch as a concrete source of magnitude sensitivity in Adam and provide a mechanistic account of why tied momentum is dynamically distinctive.

Abstract

Explore similar work

Why β1=β2β_1 = β_2β1​=β2​ Is Dynamically Special in Adam

Refresh-Scaling the Memory of Balanced Adam

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

Why $β_1 = β_2$ Is Dynamically Special in Adam