Replaying the Same Gradients at a Different $\varepsilon$: A Registered Test of the Fixed-Trajectory Account of Adam's $\varepsilon$ for One Rare Row

Lngel Kyo

cs.LGOct 28, 2026

Replaying the Same Gradients at a Different $\varepsilon$ : A Registered Test of the Fixed-Trajectory Account of Adam's $\varepsilon$ for One Rare Row

Authors: Lngel Kyo

Organizations: A2O Labs

Abstract

Adam divides every update by the square root of a running average of squared gradients plus a small constant, epsilon. For an output-embedding row whose token is rare, which we call the probe row, raising epsilon can be read as shrinking that row's steps by arithmetic alone. We test that reading directly. While a training run proceeds, we also divide the probe row's update by a different epsilon, using the run's own moment estimates; this replay is exact arithmetic on the gradients the run actually saw, so it holds the run's trajectory fixed. A rule for when the replay counts as wrong was written down before the runs. We train a small causal transformer in a standard configuration. When epsilon rises by four orders of magnitude, the replay counts as wrong on each of the three seeds registered for the test: the probe row keeps 8.9--11.4 times more of its displacement than predicted with the main, sampled-negative head, and 5.0--7.7 times more with a full-softmax head scored as a second instrument. At three orders of magnitude, tested with the main head only, it does not, by narrow margins; a further two seeds, registered after this result had been committed and scored on their own, follow the same pattern. In separate real runs, raising epsilon on the probe row alone is enough to reproduce the row's change and raising it on every other parameter alone also moves the row part of the way --- readings about the runs that do not rescue the replay; scaling the row's learning rate instead, with the denominator untouched, is mispredicted by the replay in the same direction. This is one embedding row of one model family on one corpus; we make no claim about rates, about rows in general (other rows were read only as a panel on one trajectory) or about how the effect changes with model size.

Figures & tables

Figure 1: Kill ratio

\rho_{\mathrm{cf}}/\rho_{\mathrm{meas}}

of every pair, per seed, on a log scale. Left, the BCE head: filled, the four-decade pairs; open, the three-decade pairs. Right, the softmax head, four-decade pairs only. Shaded: the interval

[0.25,4]

fixed before the runs; below it, the replay predicts more suppression than the runs show by more than the rule tolerates. Seeds 3 and 4, right of the dotted rule, were registered after the result for seeds 0–2 had been committed and are scored on their own; the two groups stand side by side and are never pooled.

Figure 2: Per-row step size, per seed; populations side by side, never pooled. Left: the interpolation coordinate of the row-only run (

r_{\mathrm{row}}

, filled) and of the rest-only run (

r_{\mathrm{rest}}

, open) for the sampled-negative head on seeds 0–2, the same head on seeds 3–4 (a later registration, read on its own) and the softmax head on seeds 0–2 (read on its own); dashed, the registered thresholds

0.25

and

0.75

r

places a run between the two endpoints; it is not a share. Right: the realised fraction

s_{\mathrm{lr}}

of the learning-rate arm for the sampled-negative head on seeds 0–2 and 3–4; shaded, the near-exact band

[0.9,1.1]

, in which or above which the prediction would have lost.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

head	seed	pair	$c$ (nats)	kill ratio	$e^{F}$	$s$	crossing step (low / high run)	cell word
main	0	four-decade	$4.47$	$0.088$	$11.36$	$0.457$	9,000/ 8,000	FIRES
main	0	three-decade	$2.65$	$0.269$	$3.72$	$0.504$	9,000/ 7,000	NOT-FIRING
main	1	four-decade	$4.01$	$0.112$	$8.90$	$0.455$	8,000/ 8,000	FIRES
main	1	three-decade	$2.29$	$0.310$	$3.23$	$0.488$	8,000/ 7,000	NOT-FIRING
main	2	four-decade	$3.99$	$0.112$	$8.90$	$0.452$	8,000/ 8,000	FIRES
main	2	three-decade	$2.28$	$0.313$	$3.19$	$0.491$	8,000/ 7,000	NOT-FIRING

Appendix

Table 1: The kill test, every cell. Rows: Q-STDCFG-X1 , Q-STDCFG-X1-SOFTMAX , Q-STDCFG-X1EXT .

Test (row)	branches	predicted
Per-row $\varepsilon$ , $r$ (Q-STDCFG-X8, -X8EXT, -X8SM)	A-DENOMINATOR $r\geq 0.75$ ; B-TRAJECTORY $r\leq 0.25$ ; C-MIXED between;	none
	OUT-OF-RANGE $r<0$ or $r>1$ ; UNDEFINED denominator not positive
Per-row lr, $s_{\mathrm{lr}}$ (Q-STDCFG-X9, -X9EXT)	LR-SAME-SIDE non-negative, $<0.9$ ; LR-NEUTRAL $[0.9,1.1]$ ;	LR-SAME-SIDE
	LR-OPPOSITE $>1.1$ ; OUT-OF-RANGE negative
Shadow, $c_{\mathrm{sh}}$ (Q-SHADOW-X3P, -X3PX9)	LOCAL $\geq 0.75$ ; GLOBAL $\leq 0.25$ ; MIXED between;	none
	OUT-OF-RANGE $<0$ or $>1$ ; UNDEFINED $\|F\|<\ln 1.1$

Appendix

Table 2: Registered outcome sets of the decomposition and the shadow (thresholds as registered; the per-run, per-seed and per-unit states that are never a branch, such as VOID, MISSING and NOT-COVERED, are omitted).

Population (row)	seed	$D_{\mathrm{lo}}-D_{\mathrm{hi}}$	$r_{\mathrm{row}}$	word	$r_{\mathrm{rest}}$	word	$r_{\mathrm{row}}+r_{\mathrm{rest}}$
sampled-negative (Q-STDCFG-X8)	0	2.5947	0.8479	A-DENOMINATOR	0.7249	C-MIXED	1.5728
	1	2.3654	0.8391	A-DENOMINATOR	0.7037	C-MIXED	1.5428
	2	2.2836	0.8533	A-DENOMINATOR	0.6938	C-MIXED	1.5471
sampled-negative, read on its own (Q-STDCFG-X8EXT)	3	2.1510	0.8386	A-DENOMINATOR	0.6728	C-MIXED	1.5114
	4	2.3288	0.8483	A-DENOMINATOR	0.6934	C-MIXED	1.5416
softmax, read on its own (Q-STDCFG-X8SM)	0	2.7932	0.8733	A-DENOMINATOR	0.5083	C-MIXED	1.3817

Appendix

Table 3: Per-row

\varepsilon

, per seed, at the readers’ printed precision. The last column is printed, not a criterion.

Population (row)	seed	$c_{s}$	$\rho_{\mathrm{meas}}$	$s_{\mathrm{lr}}$	word	$s_{\mathrm{row}}$ (beside)
sampled-negative (Q-STDCFG-X9)	0	0.01141	0.1345	0.4484	LR-SAME-SIDE	0.2994
	1	0.01812	0.2103	0.3888	LR-SAME-SIDE	0.3033
	2	0.01854	0.2104	0.3908	LR-SAME-SIDE	0.3126
sampled-negative, read on its own (Q-STDCFG-X9EXT)	3	0.02056	0.2387	0.3688	LR-SAME-SIDE	0.2991
	4	0.01661	0.1955	0.3983	LR-SAME-SIDE	0.3059

Appendix

Table 4: Per-row learning rate, per seed.

s_{\mathrm{row}}

is the row-only

\varepsilon

arm’s fraction against the same

c_{s}

, printed beside

s_{\mathrm{lr}}

and never combined with it.

Cell	seed 0		seed 1		seed 2
learning-rate arm, forward	0.9894	LOCAL	1.0036	OUT-OF-RANGE	1.0021	OUT-OF-RANGE
learning-rate arm, reverse	0.9939	LOCAL	1.0086	OUT-OF-RANGE	1.0040	OUT-OF-RANGE
sampled-negative, three-decade, forward	1.4131	OUT-OF-RANGE	1.3431	OUT-OF-RANGE	1.3258	OUT-OF-RANGE
sampled-negative, three-decade, reverse	1.4138	OUT-OF-RANGE	1.2737	OUT-OF-RANGE	1.2687	OUT-OF-RANGE
sampled-negative, four-decade, forward	1.2966	OUT-OF-RANGE	1.2924	OUT-OF-RANGE	1.2598	OUT-OF-RANGE
sampled-negative, four-decade, reverse	1.3937	OUT-OF-RANGE	1.3461	OUT-OF-RANGE	1.3382	OUT-OF-RANGE

Appendix

Table 5: The shadow, per cell: coordinate

c_{\mathrm{sh}}

and word as the readers printed them. Top: the learning-rate arm (Q-SHADOW-X3PX9). Bottom: the pairs of Section 3.1 , every parameter’s

\varepsilon

changed (Q-SHADOW-X3P).

Direction	seed 0		seed 1		seed 2
forward (row-only run)	1.0053	FULL	1.0110	FULL	1.0046	FULL
reverse (rest-only run)	0.9998	FULL	0.9999	FULL	0.9997	FULL

Appendix

Table 6: The same-world comparison, sampled-negative head, seeds 0–2:

c^{\prime}

and word as printed. Descriptive (A1.5): the inputs were printed before the comparison was registered; not a test of its prediction (Q-SHADOW-X3PROW).

Adaptive Gradient Methods

Adaptive Gradient Methods adjust updates using gradient history, curvature, or noise estimates to improve optimization across diverse problems.

See all 161 papers

Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repair
Apr 24, 2026Yuelin Hu, Zhenbo Yu, Zhengxue Cheng +2Continual Learning for LLMs Adaptive Gradient Methods
Shanghai Jiao Tong University · Shanghai Maritime University
Early Memory Selection for Balanced Adam
Oct 6, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Adaptive Gradient Methods Hyperparameter Optimization
Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain
Refresh-Scaling the Memory of Balanced Adam
May 11, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Deep Learning Optimization Momentum Methods
Universitat Polit`ecnica de Val`encia, Val`encia, Spain · Universitat Jaume I, Castell´o de la Plana, Spain

Apr 24, 2026J/K move · Enter open · S save

Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repair

Yuelin Hu, Zhenbo Yu, Zhengxue Cheng, Wei Liu+1

Shanghai Jiao Tong University · Shanghai Maritime University

Continual Learning for LLMs Adaptive Gradient Methods Catastrophic Forgetting

Open paper AskPDF arXiv

Many continual-learning methods modify gradients upstream (e.g., projection, penalty rescaling, replay mixing) while treating Adam as a neutral backend. We show this composition has a hidden failure mode. In a high-overlap, non-adaptive 8-domain continual LM, all shared-routing projection baselines collapse close to vanilla forgetting (12.5--12.8 vs. 13.2). A 0.5% replay buffer is the strongest shared alternative but still reaches 11.6, while fixed-strength decoupling falls below vanilla at 14.1. Only adaptive decoupled routing remains stable at 9.4, improving over vanilla by 3.8 units. On a 16-domain stream, its gain over the strongest shared-routing projection baseline grows to 4.5--4.8 units. The failure is largely invisible on clean benchmarks. We explain this effect through Adam's second-moment pathway: in the tested regime, projection induces a 1/(1-alpha) inflation of the old-direction effective learning rate, matching measurements within 8% across eight alpha values. The same conflict appears with penalty methods, replay mixing, and at 7B scale under LoRA. Our fix routes the modified gradient only to the first moment while preserving magnitude-faithful second-moment statistics, with overlap-aware adaptive strength. This simple change is the only tested configuration that consistently avoids collapse across methods, optimizers, and scale.

Abstract

Adaptive Gradient Methods

Explore similar work

Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repair

Early Memory Selection for Balanced Adam

Refresh-Scaling the Memory of Balanced Adam