Replaying the Same Gradients at a Different $\varepsilon$: A Registered Test of the Fixed-Trajectory Account of Adam's $\varepsilon$ for One Rare Row

Lngel Kyo

cs.LGOct 28, 2026

Replaying the Same Gradients at a Different $\varepsilon$ : A Registered Test of the Fixed-Trajectory Account of Adam's $\varepsilon$ for One Rare Row

Authors: Lngel Kyo

Organizations: A2O Labs

Abstract

Adam divides every update by the square root of a running average of squared gradients plus a small constant, epsilon. For an output-embedding row whose token is rare, which we call the probe row, raising epsilon can be read as shrinking that row's steps by arithmetic alone. We test that reading directly. While a training run proceeds, we also divide the probe row's update by a different epsilon, using the run's own moment estimates; this replay is exact arithmetic on the gradients the run actually saw, so it holds the run's trajectory fixed. A rule for when the replay counts as wrong was written down before the runs. We train a small causal transformer in a standard configuration. When epsilon rises by four orders of magnitude, the replay counts as wrong on each of the three seeds registered for the test: the probe row keeps 8.9--11.4 times more of its displacement than predicted with the main, sampled-negative head, and 5.0--7.7 times more with a full-softmax head scored as a second instrument. At three orders of magnitude, tested with the main head only, it does not, by narrow margins; a further two seeds, registered after this result had been committed and scored on their own, follow the same pattern. In separate real runs, raising epsilon on the probe row alone is enough to reproduce the row's change and raising it on every other parameter alone also moves the row part of the way --- readings about the runs that do not rescue the replay; scaling the row's learning rate instead, with the denominator untouched, is mispredicted by the replay in the same direction. This is one embedding row of one model family on one corpus; we make no claim about rates, about rows in general (other rows were read only as a panel on one trajectory) or about how the effect changes with model size.

Figures & tables

Figure 1: Kill ratio

\rho_{\mathrm{cf}}/\rho_{\mathrm{meas}}

of every pair, per seed, on a log scale. Left, the BCE head: filled, the four-decade pairs; open, the three-decade pairs. Right, the softmax head, four-decade pairs only. Shaded: the interval

[0.25,4]

fixed before the runs; below it, the replay predicts more suppression than the runs show by more than the rule tolerates. Seeds 3 and 4, right of the dotted rule, were registered after the result for seeds 0–2 had been committed and are scored on their own; the two groups stand side by side and are never pooled.

Figure 2: Per-row step size, per seed; populations side by side, never pooled. Left: the interpolation coordinate of the row-only run (

r_{\mathrm{row}}

, filled) and of the rest-only run (

r_{\mathrm{rest}}

, open) for the sampled-negative head on seeds 0–2, the same head on seeds 3–4 (a later registration, read on its own) and the softmax head on seeds 0–2 (read on its own); dashed, the registered thresholds

0.25

and

0.75

r

places a run between the two endpoints; it is not a share. Right: the realised fraction

s_{\mathrm{lr}}

of the learning-rate arm for the sampled-negative head on seeds 0–2 and 3–4; shaded, the near-exact band

[0.9,1.1]

, in which or above which the prediction would have lost.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

head	seed	pair	$c$ (nats)	kill ratio	$e^{F}$	$s$	crossing step (low / high run)	cell word
main	0	four-decade	$4.47$	$0.088$	$11.36$	$0.457$	9,000/ 8,000	FIRES
main	0	three-decade	$2.65$	$0.269$	$3.72$	$0.504$	9,000/ 7,000	NOT-FIRING
main	1	four-decade	$4.01$	$0.112$	$8.90$	$0.455$	8,000/ 8,000	FIRES
main	1	three-decade	$2.29$	$0.310$	$3.23$	$0.488$	8,000/ 7,000	NOT-FIRING
main	2	four-decade	$3.99$	$0.112$	$8.90$	$0.452$	8,000/ 8,000	FIRES
main	2	three-decade	$2.28$	$0.313$	$3.19$	$0.491$	8,000/ 7,000	NOT-FIRING

Appendix

Table 1: The kill test, every cell. Rows: Q-STDCFG-X1 , Q-STDCFG-X1-SOFTMAX , Q-STDCFG-X1EXT .

Test (row)	branches	predicted
Per-row $\varepsilon$ , $r$ (Q-STDCFG-X8, -X8EXT, -X8SM)	A-DENOMINATOR $r\geq 0.75$ ; B-TRAJECTORY $r\leq 0.25$ ; C-MIXED between;	none
	OUT-OF-RANGE $r<0$ or $r>1$ ; UNDEFINED denominator not positive
Per-row lr, $s_{\mathrm{lr}}$ (Q-STDCFG-X9, -X9EXT)	LR-SAME-SIDE non-negative, $<0.9$ ; LR-NEUTRAL $[0.9,1.1]$ ;	LR-SAME-SIDE
	LR-OPPOSITE $>1.1$ ; OUT-OF-RANGE negative
Shadow, $c_{\mathrm{sh}}$ (Q-SHADOW-X3P, -X3PX9)	LOCAL $\geq 0.75$ ; GLOBAL $\leq 0.25$ ; MIXED between;	none
	OUT-OF-RANGE $<0$ or $>1$ ; UNDEFINED $\|F\|<\ln 1.1$

Appendix

Table 2: Registered outcome sets of the decomposition and the shadow (thresholds as registered; the per-run, per-seed and per-unit states that are never a branch, such as VOID, MISSING and NOT-COVERED, are omitted).

Population (row)	seed	$D_{\mathrm{lo}}-D_{\mathrm{hi}}$	$r_{\mathrm{row}}$	word	$r_{\mathrm{rest}}$	word	$r_{\mathrm{row}}+r_{\mathrm{rest}}$
sampled-negative (Q-STDCFG-X8)	0	2.5947	0.8479	A-DENOMINATOR	0.7249	C-MIXED	1.5728
	1	2.3654	0.8391	A-DENOMINATOR	0.7037	C-MIXED	1.5428
	2	2.2836	0.8533	A-DENOMINATOR	0.6938	C-MIXED	1.5471
sampled-negative, read on its own (Q-STDCFG-X8EXT)	3	2.1510	0.8386	A-DENOMINATOR	0.6728	C-MIXED	1.5114
	4	2.3288	0.8483	A-DENOMINATOR	0.6934	C-MIXED	1.5416
softmax, read on its own (Q-STDCFG-X8SM)	0	2.7932	0.8733	A-DENOMINATOR	0.5083	C-MIXED	1.3817

Appendix

Table 3: Per-row

\varepsilon

, per seed, at the readers’ printed precision. The last column is printed, not a criterion.

Population (row)	seed	$c_{s}$	$\rho_{\mathrm{meas}}$	$s_{\mathrm{lr}}$	word	$s_{\mathrm{row}}$ (beside)
sampled-negative (Q-STDCFG-X9)	0	0.01141	0.1345	0.4484	LR-SAME-SIDE	0.2994
	1	0.01812	0.2103	0.3888	LR-SAME-SIDE	0.3033
	2	0.01854	0.2104	0.3908	LR-SAME-SIDE	0.3126
sampled-negative, read on its own (Q-STDCFG-X9EXT)	3	0.02056	0.2387	0.3688	LR-SAME-SIDE	0.2991
	4	0.01661	0.1955	0.3983	LR-SAME-SIDE	0.3059

Appendix

Table 4: Per-row learning rate, per seed.

s_{\mathrm{row}}

is the row-only

\varepsilon

arm’s fraction against the same

c_{s}

, printed beside

s_{\mathrm{lr}}

and never combined with it.

Cell	seed 0		seed 1		seed 2
learning-rate arm, forward	0.9894	LOCAL	1.0036	OUT-OF-RANGE	1.0021	OUT-OF-RANGE
learning-rate arm, reverse	0.9939	LOCAL	1.0086	OUT-OF-RANGE	1.0040	OUT-OF-RANGE
sampled-negative, three-decade, forward	1.4131	OUT-OF-RANGE	1.3431	OUT-OF-RANGE	1.3258	OUT-OF-RANGE
sampled-negative, three-decade, reverse	1.4138	OUT-OF-RANGE	1.2737	OUT-OF-RANGE	1.2687	OUT-OF-RANGE
sampled-negative, four-decade, forward	1.2966	OUT-OF-RANGE	1.2924	OUT-OF-RANGE	1.2598	OUT-OF-RANGE
sampled-negative, four-decade, reverse	1.3937	OUT-OF-RANGE	1.3461	OUT-OF-RANGE	1.3382	OUT-OF-RANGE

Appendix

Table 5: The shadow, per cell: coordinate

c_{\mathrm{sh}}

and word as the readers printed them. Top: the learning-rate arm (Q-SHADOW-X3PX9). Bottom: the pairs of Section 3.1 , every parameter’s

\varepsilon

changed (Q-SHADOW-X3P).

Direction	seed 0		seed 1		seed 2
forward (row-only run)	1.0053	FULL	1.0110	FULL	1.0046	FULL
reverse (rest-only run)	0.9998	FULL	0.9999	FULL	0.9997	FULL

Appendix

Table 6: The same-world comparison, sampled-negative head, seeds 0–2:

c^{\prime}

and word as printed. Descriptive (A1.5): the inputs were printed before the comparison was registered; not a test of its prediction (Q-SHADOW-X3PROW).

Adaptive Gradient Methods

Adaptive Gradient Methods adjust updates using gradient history, curvature, or noise estimates to improve optimization across diverse problems.

See all 161 papers

Apr 24, 2026

Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repair

Many continual-learning methods modify gradients upstream (e.g., projection, penalty rescaling, replay mixing) while treating Adam as a neutral backend. We show this composition has a hidden failure mode. In a high-overlap, non-adaptive 8-domain continual LM, all shared-routing projection baselines collapse close to vanilla forgetting (12.5--12.8 vs. 13.2). A 0.5% replay buffer is the strongest shared alternative but still reaches 11.6, while fixed-strength decoupling falls below vanilla at 14.1. Only adaptive decoupled routing remains stable at 9.4, improving over vanilla by 3.8 units. On a 16-domain stream, its gain over the strongest shared-routing projection baseline grows to 4.5--4.8 units. The failure is largely invisible on clean benchmarks. We explain this effect through Adam's second-moment pathway: in the tested regime, projection induces a 1/(1-alpha) inflation of the old-direction effective learning rate, matching measurements within 8% across eight alpha values. The same conflict appears with penalty methods, replay mixing, and at 7B scale under LoRA. Our fix routes the modified gradient only to the first moment while preserving magnitude-faithful second-moment statistics, with overlap-aware adaptive strength. This simple change is the only tested configuration that consistently avoids collapse across methods, optimizers, and scale.

Yuelin Hu, Zhenbo Yu, Zhengxue Cheng +2

Shanghai Jiao Tong University · Shanghai Maritime University

Continual Learning for LLMs Adaptive Gradient Methods Catastrophic Forgetting

Ask PDF

Oct 6, 2026

Early Memory Selection for Balanced Adam

We propose a method for choosing the shared memory parameter

β_1=β_2=β

in Adam from a short pilot training. The selected

β

remains fixed during the subsequent full training. A local model of Adam's normalized direction balances sampling variability against the delay introduced by averaging past gradients. This balance gives a cubic memory rule, whose two coefficients are estimated from gradient probes at a few pilot checkpoints. The estimator uses the numerator and denominator jointly, preserving their covariance. With a 200-update pilot and sixteen probe gradients at each of four checkpoints, a seed-matched retrospective evaluation on eleven vision and language workloads reduces mean relative validation gap by 40.7% and worst-quarter mean gap by 44.3% against the grid representative of shared

β=0.95

. The mean gap is also 32.3% lower than that of the best constant

β

chosen across all eleven workloads.

Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2

Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

Adaptive Gradient Methods Hyperparameter Optimization

Ask PDF

May 11, 2026

Refresh-Scaling the Memory of Balanced Adam

Recent evidence suggests that Adam performs robustly when its momentum parameters are tied,

β_1=β_2

, reducing the optimizer to a single remaining parameter. However, how this parameter should be set remains poorly understood. We argue that, in balanced Adam,

β

should not be treated as a dimensionless constant: it defines a statistical memory horizon

H_β=(1-β)^{-1}

. In terms of the effective learning horizon

T_{\mathrm{ES}}

, estimated from the validation trajectory, we study the refresh count

R_β=(1-β)T_{\mathrm{ES}}

, which measures how many times Adam renews its internal statistics during the useful phase of training. Across 11 vision and language experiments, we find that choosing

β

so that

R_β\approx1000

selects different

β

values depending on the training scale, yet improves robustness over the best fixed-beta baseline. Compared with the strongest fixed choice

β=0.944

, the refresh rule improves worst-case robustness, reducing the maximum relative gap in validation loss by 33.4%, while bringing all 11 runs within 1% of their validation oracle. These results suggest that the remaining hyperparameter of balanced Adam is more naturally viewed as a memory-scale variable than as a fixed constant. This provides a simple budget-aware perspective on optimizer scaling and opens a path toward treating Adam's momentum as part of the learning dynamics rather than as a static default.

Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2

Universitat Polit`ecnica de Val`encia, Val`encia, Spain · Universitat Jaume I, Castell´o de la Plana, Spain

Deep Learning Optimization Momentum Methods Adaptive Gradient Methods Hyperparameter Optimization

Ask PDF

Abstract

Adaptive Gradient Methods

Explore similar work

Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repair

Early Memory Selection for Balanced Adam

Refresh-Scaling the Memory of Balanced Adam