Physics-Inspired Optimizers beyond the Edge of Stability: Period-Two Orbits and Delayed Instability

Alexander Itin

nlin.CDOct 19, 2026

Physics-Inspired Optimizers beyond the Edge of Stability: Period-Two Orbits and Delayed Instability

Authors: Alexander Itin

Organizations: Independent researcher, Hamburg, Germany

Abstract

Gradient descent with momentum can be viewed as a particle moving with friction in the loss landscape. In deep learning the learning rate is usually large, so training is a discrete dynamical system: in many full-batch experiments the sharpness grows to the optimizer's stability limit, the edge of stability (EoS), and training continues there. Physics-inspired optimizers change this motion; Velocity-Regularized Adam (VRAdam), for example, shrinks the step at high speed. We study kinetic maps, in which the step is the gradient of a kinetic energy of the momentum, and ask what they do beyond the EoS. (i) In a reduced model, a kinetic map creates a symmetric period-two orbit beyond the edge, strictly linearly stable if and only if the map's elasticity (the log-log slope of step length versus momentum) lies between 0 and 1; on it, training sits at an effective edge of stability. VRAdam's damper fails above twice the critical learning rate, where its kinetic energy is not convex; the Legendre dual of VRAdam's own Lagrangian is convex and passes at every learning rate. (ii) Because the sharpness rises gradually, the loss of stability is delayed, as in Neishtadt's theory of slowly swept bifurcations. In a network with a controlled sweep, the dynamics match this theory without fitted parameters, including the memory of where the sweep started and its erasure by noise. (iii) With momentum, reaching the oscillation also requires elasticity at large momentum: maps that bound the step have competing period-four and chaotic attractors already at beta1 = 0.9, while the cubic-Legendre map reached the oscillation from every sampled start up to beta1 = 0.99. Experiments on an exact quadratic problem, a full-batch MLP and ResNet-32 support the theory; on ResNet-18, much of the apparent benefit of the maps is reproduced by matching the effective learning rate.

Figures & tables

Figure 1: (a) Radial speed profiles and (b) the kinetic energies they imply. (c,d) Late iterates of the reduced model without momentum, $y^{\prime}=y-a\phi(y)$ . The VR map has a stable two-cycle only for $2<a<4$ , followed by a symmetry-breaking pitchfork, period doubling and chaos (first positive Lyapunov exponent at $a\approx 5.32$ ); the quartic map has a stable two-cycle for every $a>2$ .

Figure 2: Controlled sweeps through the edge in a network (tanh MLP, Fashion-MNIST, full-batch EMA momentum,

\beta_{1}=0.9

, no map; three seeds). (a) Deviation along the top Hessian direction after a kick

\delta=10^{-6}

R_{\rm in}

, while

R

is ramped at

\rho=0.003

per step (solid: network, seed 0; dashed: linear recurrence ( 10 ) on the measured path; dotted: exit predicted by the slow-sweep relation ( 8 )). (b) Exit point (deviation back at

\delta

) versus entry point: network (markers,

\rho\leq 0.01

), model on an exact linear ramp (thin lines,

\rho\leq 0.003

) and the

\rho\to 0

relation ( 8 ) (thick line). (c) Kick plus gradient noise

\sigma

\rho=0.003

: network (markers) and model median and 10–90% range (lines, bands); the thick line is ( 8 ). (d) Noise only,

R_{\rm in}=0.5

, exit at

|x_{t}|=10^{-4}

, versus sweep rate.

	worst share over $A\in\{3,4,6,10,20,40,100\}$ for $p=1/n=$
$\beta_{1}$	0.67	0.50	0.40	0.33	0.25	0.20	0.14	0.10
0.0	1	1	1	1	1	1	1	1
0.5	1	1	1	1	0.98	0.96	0.96	0.51
0.8	1	1	0.99	0.98	0.97	0.47	0.25	0.15
0.9	1	1	1	1	0.48	0.15	0.11	0.11
0.95	1	1	1	0.99	0.11	0.05	0.03	0.04

Table 1: Legendre power maps in the reduced model: worst share of 300 initial conditions that reach the symmetric orbit, over seven step sizes

A\in[3,100]

p=1/2

is the cubic-Legendre map,

p=1/3

the quartic-Legendre map.

Figure 3: Legendre power maps in the reduced model. (a) Share of initial conditions that reach the symmetric orbit versus the orbit elasticity, all

(p,A)

. (b) Worst share over

A\in[3,100]

versus the asymptotic elasticity

p

Figure 4: Exact quadratic (7,850 parameters, real data). (a–c) Late momentum norm $\|m\|/c$ versus $\mu=\eta\lambda_{\max}/a_{\mathrm{EoS}}$ (markers: last 200 steps of three seeds) against the closed-form orbit (solid: stable, dotted: unstable). (d) Reduced model, $\beta_{1}=0.9$ : share of initial conditions that reach the orbit versus the orbit elasticity, for four families of maps.

Figure 5: Full-batch EMA momentum (

\beta_{1}=0.9

) on a tanh MLP, 1,000 CIFAR-10 images, 10 seeds per setting. Colour: map; filled circles and solid lines: absolute scale

c=0.05

; open triangles and dashed lines: relative scale

c=2\|\nabla_{B}L(\theta_{0})\|

. (a) One seed at

\mu=0.5

, absolute scale: sharpness ratio

R=\eta\lambda_{\max}/a_{\mathrm{EoS}}

(solid) and effective ratio

R_{\mathrm{eff}}

of ( 12 ) (dotted). Plain momentum stays at the edge,

R\approx 1

. With the quartic map the sharpness rises to

1.75

times the edge while

R_{\mathrm{eff}}

stays at one; with the VR map the sharpness rises to more than five times the edge while

R_{\mathrm{eff}}

mostly stays below one. (b) Mode-weighted multiplier

\bar{g}

of ( 12 ) against

R

at every diagnostic with

R\geq 1.1

after step 100, all completed runs. Points on the line

\bar{g}=1/R

satisfy

R_{\mathrm{eff}}=1

: the convex maps lie on it, the VR map lies below it (a smaller step than the edge requires), and the open triangles near

\bar{g}=1

are relative-scale diagnostics at which the map has not yet engaged, so that the effective step is above the edge. (c) Median final loss versus step size; crosses mark settings in which at least half of the runs diverged.

	trained / collapsed / diverged
method	$\mu=0.5$	$\mu=2$	$\mu=4$
momentum (no map)	5/0/0	0/4/1	0/1/4
quartic-Legendre, abs. $c$	5/0/0	5/0/0	5/0/0
cubic-Legendre, abs. $c$	5/0/0	5/0/0	5/0/0
relativistic, abs. $c$	5/0/0	5/0/0	5/0/0
VR map, abs. $c$	5/0/0	5/0/0	5/0/0

Table 2: ResNet-32 (5 seeds): runs that trained (final loss

<10^{-3}

), collapsed (final loss

>0.03

, near the constant-predictor value

0.09

) or diverged; (+1): one run in between.

Figure 6: Full-batch EMA momentum on ResNet-32 (batch normalization without running statistics), 1,000 CIFAR-10 images, 5 seeds; panels as in Fig. 5 . In (b) the multipliers stay near one above the plain edge: the map has not yet engaged.

$\eta$	method	peak $R$	late $R$	min. mult. (steps $\geq 1500$ )	min. mult. (steps $<100$ )	final loss
0.002	Adam	1.29	0.99	1.000	1.00	$3.4{\times}10^{-6}$
0.002	VRAdam	1.08	0.96	1.000	0.59	$2.4{\times}10^{-5}$
0.002	Adam + relativistic map on $\mathrm{rms}(q)$ , $c=0.5$	1.13	1.00	0.994	0.45	$5.4{\times}10^{-6}$
0.002	Adam + relativistic map on $\\|\hat{m}\\|$ , $c=0.05$	1.36	0.99	0.996	0.15	$4.9{\times}10^{-6}$
0.01	Adam	1.08	0.96	1.000	1.00	$1.7{\times}10^{-4}$
0.01	VRAdam	1.26	0.94	1.000	0.30	$3.1{\times}10^{-6}$

Table 3: Full-batch Adam on the ResNet-32 objective, medians over three seeds.

R

is the preconditioned sharpness ratio

\eta\lambda_{\max}(P^{-1}H)/38

; “late” is the median over steps

\geq 1500

configuration	test accuracy (%)	test CE	$\Delta$ acc (pp)	$\Delta$ CE
Muon, tuned	94.02 $\pm$ 0.18	0.351 $\pm$ 0.006	ref.	ref.
Kinetic Muon (quartic EMA map), tuned	94.19 $\pm$ 0.17	0.335 $\pm$ 0.008	$+0.17$ $\pm$ 0.13 ∗	$-0.016$ $\pm$ 0.011 ∗
Muon at Kinetic Muon’s effective lr ( $\times 0.66$ )	94.09 $\pm$ 0.07	0.344 $\pm$ 0.008	$+0.07$ $\pm$ 0.20	$-0.008$ $\pm$ 0.007
Muon at Kinetic Muon’s nominal lr	94.01 $\pm$ 0.13	0.362 $\pm$ 0.009	$0.00$ $\pm$ 0.22	$+0.010$ $\pm$ 0.013
AdamW, tuned	92.36 $\pm$ 0.19	0.338 $\pm$ 0.010	ref.	ref.
AdamW at $3\times$ tuned lr	92.85 $\pm$ 0.18	0.360 $\pm$ 0.007	$+0.50$ $\pm$ 0.23 ∗	$+0.022$ $\pm$ 0.015 ∗

Table 4: ResNet-18 / CIFAR-10, effective-learning-rate controls: mean

\pm

s.d. over 5 seeds, and paired differences to the reference row of each block ( ∗ :

p<0.05

, paired

t

-test).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: EMA momentum, $\beta_{1}=0.9$ . (a) Floquet radius of the symmetric EoS orbit versus $a/a_{\mathrm{EoS}}$ . (b,c) Late iterates.

Figure 8: Block-diagonal quadratic, quartic map at

6\times

the edge step: global versus per-block maps.

Figure 9: Self-stabilization model with

\alpha=1

(edge at

S=2

c=0.3

. (a) Sharpness, (b) loss at

\alpha S_{0}=6

; (c) steps until

S<2.05

Figure 10: Delayed loss of stability, reduced model with the cubic map,

\beta_{1}=0.9

. (a)

R_{\mathrm{eff}}

while

R

is swept linearly through the edge at a slow and a fast rate; the dotted line is

R_{\mathrm{eff}}=R

(map inactive). (b) Peak

R_{\mathrm{eff}}

versus sweep rate: simulation (dots, 64 noise realizations; band: 10–90%) and the linear estimate (lines) for three noise levels; grey band: range over initial amplitudes

10^{-6}

–

1

and the cubic and quartic maps; open markers: largest

R_{\mathrm{eff}}

during the climb of the sharpness in the networks (

\mu=0.5

, all seeds; orange ResNet-32, blue MLP).

Figure 11: Fashion-MNIST MLP: full-batch momentum on a tanh MLP with per-tensor maps (one seed). (a) Final loss and (b) final sharpness versus step size; (c,d) final loss versus

c

8\times

and

16\times

the initial edge step.

method	$\mu=0.5$	$\mu=1$	$\mu=2$	$\mu=4$	$\mu=8$	median [IQR] (runs)
	median final loss (diverged runs)					$R_{\mathrm{eff}}$ above the edge
momentum (no map)	$1.3{\times}10^{-5}$	$5.8{\times}10^{-6}$	$1.6{\times}10^{-6}$ (1)	– (10)	– (10)	– (0)
quartic-Legendre, abs. $c$	$2.4{\times}10^{-4}$	$2.6{\times}10^{-5}$	$1.0{\times}10^{-6}$	$4.7{\times}10^{-9}$	$3.1{\times}10^{-6}$	1.01 [1.00, 1.04] (50)
cubic-Legendre, abs. $c$	$2.2{\times}10^{-4}$	$1.4{\times}10^{-4}$	$4.8{\times}10^{-5}$	$1.8{\times}10^{-5}$	0.20	1.00 [1.00, 1.01] (50)
relativistic, abs. $c$	$2.3{\times}10^{-4}$	$1.3{\times}10^{-5}$	$8.0{\times}10^{-7}$	$1.5{\times}10^{-10}$	$1.9{\times}10^{-7}$	1.01 [1.00, 1.03] (50)
VR map, abs. $c$	$2.6{\times}10^{-3}$	$4.9{\times}10^{-3}$	$3.0{\times}10^{-4}$	$3.4{\times}10^{-7}$	$6.3{\times}10^{-7}$	0.80 [0.61, 0.91] (50)

Appendix

Table 5: MLP on CIFAR-10: median final training loss over 10 seeds (number of diverged runs in brackets) and per-run median of

R_{\mathrm{eff}}

over the diagnostics with

R\geq 1.1

after step 100 (median and IQR over runs; number of runs with such points).

$\beta_{2}$	$\eta$	$R$ (Lanczos)	legacy / Lanczos	$R_{\rm bs}$	$\eta\lambda_{\rm bs}$	$\lambda_{\min}/\lambda_{\max}$
0.95	0.001	0.20	0.96	0.086	3.3	-0.08
0.95	0.002	0.24	0.92	0.106	4.0	-0.07
0.95	0.005	0.23	0.91	0.106	4.0	-0.10
0.95	0.01	0.27	0.96	0.096	3.6	-0.10
0.999	0.001	0.17	0.94	0.076	2.9	-0.09
0.999	0.002	0.20	0.93	0.088	3.3	-0.08

Appendix

Table 6: Minibatch Adam on ResNet-32 (batch 1,000), medians over two seeds of the median over steps

\geq 900

R

\eta\lambda_{\max}/38

of the preconditioned Hessian of the training batch; legacy/Lanczos: ratio of the earlier power-iteration estimate to the Lanczos estimate on the same batch;

R_{\rm bs}

: preconditioned batch sharpness over

38/\eta

Fashion-MNIST CNN

char-level transformer

Method test loss test acc. diverged trials AdamW 0.275

\pm

0.004 90.15

\pm

0.46 0/10 AdamW + clip 0.280

\pm

0.006 90.13

\pm

0.07 0/10 VRAdam 0.269

\pm

0.001 90.40

\pm

0.26 0/10 K-Adam quartic 0.270

\pm

0.004 90.17

\pm

0.53 0/10 K-Adam quartic (EMA

c

) 0.327

\pm

0.001 88.59

\pm

0.08 0/10 K-Adam relativistic 0.267

\pm

0.003 90.53

\pm

0.39 0/10 K-Adam VR map (abl.) 0.268

\pm

0.003 90.38

\pm

0.07 0/10 K-Adam quartic, global (abl.) 0.266

\pm

0.008 90.57

\pm

0.42 0/10 EoS controller (precond.) 0.266

\pm

0.010 90.67

\pm

0.49 0/10 EoS controller (directional) 0.315

\pm

0.007 88.83

\pm

0.14 0/10 Muon 0.261

\pm

0.007 90.95

\pm

0.46 0/10 Muon + quartic map 0.259

\pm

0.008 91.17

\pm

0.54 0/10

Method test loss diverged trials AdamW 1.728

\pm

0.024 0/10 AdamW + clip 1.703

\pm

0.026 0/10 VRAdam 1.708

\pm

0.028 0/10 K-Adam quartic 1.739

\pm

0.027 0/10 K-Adam quartic (EMA

c

) 2.061

\pm

0.001 0/10 K-Adam relativistic 1.719

\pm

0.022 0/10 K-Adam VR map (abl.) 1.722

\pm

0.010 0/10 K-Adam quartic, global (abl.) 1.697

\pm

0.032 0/10 EoS controller (precond.) 1.704

\pm

0.034 0/10 EoS controller (directional) 1.725

\pm

0.039 0/10 Muon 1.592

\pm

0.011 0/10 Muon + quartic map 1.605

\pm

0.013 0/10

Appendix

Table 7: Equal-budget comparison: mean

\pm

s.d. over 3 fresh seeds of the configuration selected from 10 trials.

Figure 12: Validation loss versus peak learning rate (Fashion-MNIST MLP), with the map hyperparameters fixed in advance.

method	test CE	test accuracy (%)
AdamW	0.334 $\pm$ 0.012	92.44 $\pm$ 0.14
AdamW + clip	0.334 $\pm$ 0.004	92.66 $\pm$ 0.08
VRAdam	0.343 $\pm$ 0.005	92.90 $\pm$ 0.23
K-Adam quartic	0.344 $\pm$ 0.009	92.78 $\pm$ 0.22
K-Adam quartic, global	0.367 $\pm$ 0.008	92.85 $\pm$ 0.12
K-Adam relativistic	0.340 $\pm$ 0.006	92.35 $\pm$ 0.11

Appendix

Table 8: ResNet-18 / CIFAR-10, five fresh seeds (mean

\pm

s.d.). † The curvature probes of these two methods also updated the batch-normalization running statistics used at test time (see text); reported for completeness only.

	final validation loss (two seeds) at peak learning rate
method	$0.001$	$0.003$	$0.01$	$0.03$
AdamW	3.77 / 3.78	3.70 / 3.73	6.47 / 6.69	6.68 / 6.71
AdamW + clip 1.0	3.78 / 3.78	3.71 / 3.72	3.79 / 4.07	6.74 / 6.82
VRAdam	3.78 / 3.78	3.70 / 3.71	5.99 / 6.09	6.57 / 6.59
cubic map on $\hat{m}$ , $k=1$	3.87 / 3.88	3.73 / 3.74	3.96 / 4.84	5.24 / 5.86
cubic map on $\hat{m}$ , $k=4$	3.81 / 3.81	3.70 / 3.71	5.05 / 5.71	6.13 / 6.42

Appendix

Table 9: Small GPT (51.5M parameters, 393M tokens of FineWeb-Edu): final validation loss of both seeds (sorted) at each peak learning rate.

Figure 13: Small GPT. (a) Final validation loss versus peak learning rate (lines: mean of two seeds). (b) Training loss at

\eta=0.01

. (c) Step multipliers of VRAdam and the cubic map at

\eta=0.003

(solid) and

0.01

(dotted), seed 0.

regime	method	steps to grok (seeds 0, 1, 2)	final test acc.	largest loss after memorisation (seeds 0, 1, 2)
AdamW, wd 1	AdamW	20.4k, 14.6k, 14.5k	1.00	0.71, 0.21, 0.57
AdamW, wd 1	AdamW + clip 1.0	21.9k, 14.2k, 18.4k	1.00	0.14, 0.04, 0.15
AdamW, wd 1	VRAdam	19.9k, 12.7k, 15.5k	1.00	0.45, 0.05, 0.61
AdamW, wd 1	cubic map on $\hat{m}$ , $k=1$	21.1k, 11.4k, 15.6k	1.00	0.04, 0.23, 0.13
AdamW, wd 1	cubic map on $\hat{m}$ , $k=4$	18.0k, 12.6k, 13.1k	1.00	0.37, 0.02, 0.33
Adam, wd 0	AdamW	–, –, –	0.26	2.62, 5.43, 4.16

Appendix

Table 10: Grokking test (modular addition,

p=113

): steps until test accuracy

\geq 0.95

, final test accuracy (median), and the largest training loss from 1,000 steps after memorization to the end, for each seed.

Figure 14: Grokking test. (a,b) Training loss (solid) and test accuracy (dotted) of AdamW and the cubic map (

k=1

), seed 0, with and without weight decay. (c) Steps to grok versus the largest training loss after memorization, weight decay 1, all methods and seeds.

Symbol	Meaning
$\eta,\ \eta_{t}$	learning rate
$\lambda,\ \lambda_{\max}$	curvature (Hessian eigenvalue; for Adam, of $P^{-1/2}HP^{-1/2}$ ) and its largest value
$b=\beta_{1}$	EMA momentum coefficient; $\kappa=(1-b)/(1+b)$ , $a_{\mathrm{EoS}}=2/\kappa$
$a=\eta\lambda,\ A=\kappa a$	step size of the reduced model; $A=2$ at the edge
$R,\ R_{\mathrm{eff}}$	sharpness ratio $\eta\lambda_{\max}/a_{\mathrm{EoS}}$ ; effective ratio $R\,\bar{g}$ ( Eq. 12 )
$\mu$	value of $R$ at initialization, $\eta=\mu\,a_{\mathrm{EoS}}/\lambda_{0}$ ( Sections 5.2 and 6.1 )

Appendix

Table 11: Notation.

Study	Protocol
Conventions	$\mu=\eta\lambda_{0}/a_{\mathrm{EoS}}$ with $a_{\mathrm{EoS}}=38$ ( $\beta_{1}=0.9$ ); EMA momentum $m\leftarrow\beta_{1}m+(1-\beta_{1})\nabla L$ without bias correction; $q=\hat{m}/(\sqrt{\hat{s}}+\epsilon)$ ; per tensor: one scale $c_{B}$ and one multiplier $g(r_{B})$ per parameter tensor, $r_{B}=\text{argument}/c_{B}$ ; $\omega$ : decoupled weight decay; $\geq$ 2-D: weight matrices, convolution kernels, embeddings; none: verified to be absent.
Exact quadratic ( Section 5.2 )	Data: first 2,000 Fashion-MNIST training images (standardized pixels plus a bias column), one-hot targets; no validation or test set. Model, loss: linear least squares, $W\in\mathbb{R}^{785\times 10}$ (7,850 parameters, one tensor), $L=\tfrac{1}{2N}\\|XW-Y\\|_{F}^{2}$ , $\lambda_{\max}=300.4$ . Optimizer: full-batch EMA momentum, $\beta_{1}=0.9$ , $\omega=0$ ; no map, quartic, cubic, relativistic, VR. LR: constant, $\eta=\mu\cdot 38/\lambda_{\max}$ , 16 values $\mu\in[0.5,12]$ . Batch: full. Length: 6,000 steps. Precision: FP64 (NumPy, integrated in the eigenbasis of $C$ ). Scale: $c=0.05\,\\|\nabla L(W_{0})\\|=0.542$ (initial gradient norm of the seed-0 start, reused for seeds 1, 2); one global block; argument $\\|m\\|$ . Seeds: 0, 1, 2 (initialization $W_{0}\sim\mathcal{N}(0,0.01^{2})$ ); 240 runs. Failure: non-finite loss or excess loss $>10^{12}$ .
Controlled ramps ( Sections 4 and C )	Data: 2,000 Fashion-MNIST training images (fixed random subset, permutation seed 0), one-hot targets; no validation or test set. Model, loss: tanh MLP 784–128–128–10, $L=\tfrac{1}{2N}\sum_{i}\\|f(x_{i})-e_{y_{i}}\\|^{2}$ . Optimizer: full-batch EMA momentum, $\beta_{1}=0.9$ , $\omega=0$ , no map; reference and perturbed copy take the same $\eta_{t}$ . LR: pretraining 1,500 steps at $R=0.5$ ( $\eta$ reset from $\lambda_{\max}$ every 50 steps); ramp $\eta_{t}=(R_{\rm in}+\rho t)\,38/\hat{\lambda}_{t}$ , power iteration warm-started from the end of pretraining (200 iterations), then 4 iterations every 5 steps. Grid: kick $\delta=10^{-6}$ , $\rho\in\{0.003,0.01,0.03,0.1\}$ , $R_{\rm in}\in\{0.1,0.3,0.5,0.7,0.9,0.97\}$ ; $\delta=10^{-4}$ at $\rho=0.001$ , $R_{\rm in}\in\{0.5,0.7,0.9,0.97\}$ ; noise only ( $R_{\rm in}=0.5$ , $\sigma\in\{10^{-9},10^{-7},10^{-5}\}$ , five rates $\rho=0.001$ to $0.1$ , exit at $\|x_{t}\|=10^{-4}$ ); kick $10^{-6}$ plus noise ( $\rho=0.003$ , $R_{\rm in}\leq 0.9$ ); 174 runs. Batch: full. Length: up to $R=2$ ( $R=4$ for $\rho\geq 0.05$ ), stopped 10 steps after $\|x_{t}\|$ exceeds $100\times$ the exit threshold. Precision: FP64 (CPU). Scale: no map. Seeds: 0, 1, 2 (initialization; noise generator seed $1000+{}$ seed). Failure: none defined. Tracking test ( Appendix C ): seed 2, $R_{\rm in}=0.7$ , $\rho=0.001$ , kick $10^{-4}$ or $10^{-6}$ , sharpness re-estimated every fifth step or at every step; 3 runs.
Full-batch CIFAR-10: MLP and ResNet-32 (E1; Sections 6 and D )	Data: balanced 1,000-image subset (first 100 per class of the 45k training split; split by permutation seed 0 of the 50k images), normalized with subset statistics; 2,000 images of the 5k validation split for diagnostics only; no test set. Model: tanh MLP 3072–128–128–10 (411,146 parameters); ResNet-32 ( $3\times 5$ basic blocks, widths 16/32/64; 466,906 parameters), BN without running statistics; loss and gradients summed over 4 fixed blocks of 250 images (BN statistics per block). Loss: MLP $\tfrac{1}{2N}\sum_{i}\\|f(x_{i})-e_{y_{i}}\\|^{2}$ ; ResNet mean MSE, $\tfrac{1}{10N}\sum_{i}\\|f(x_{i})-e_{y_{i}}\\|^{2}$ . Optimizer: full-batch EMA momentum, $\beta_{1}=0.9$ , $\omega=0$ ; no map, quartic, cubic, relativistic, VR. LR: constant, $\eta=\mu\cdot 38/\lambda_{0}$ , $\lambda_{0}$ by Lanczos at initialization for each seed (2 starts; MLP 72–93, ResNet 27–47); MLP $\mu\in\{0.5,1,2,4,8\}$ , ResNet $\mu\in\{0.5,2,4\}$ . Batch: full. Length: 3,000 steps. Precision: FP32, TF32 off, deterministic cuDNN (A100). Scale: per tensor, argument $\\|m_{B}\\|$ ; absolute $c=0.05$ or relative $c_{B}=2\\|\nabla_{B}L(\theta_{0})\\|$ . Seeds: MLP 0–9, ResNet 0–4 (initialization; data fixed); 450 and 135 runs. Failure: loss non-finite or $>10^{3}$ , non-finite gradient, or parameter norm non-finite or $>10^{6}$ .
Full-batch Adam (E2; Section 6.2 )	Data, model, loss, batch, precision, failure: as for ResNet-32 in E1. Optimizer: Adam, $\beta=(0.9,0.999)$ , $\epsilon=10^{-7}$ , bias-corrected, $\omega=0$ ; VRAdam ( $\beta_{3}=1$ , $\alpha_{1}=19$ , global $\\|m\\|^{2}$ of the uncorrected first moment, same $\beta,\epsilon$ ); Adam with a per-tensor relativistic map. LR: constant, $\eta\in\{0.002,0.01\}$ . Length: 3,000 steps. Scale: map on $q$ : argument $\mathrm{rms}(q_{B})$ , $c=0.5$ ; map on $\hat{m}$ : argument $\\|\hat{m}_{B}\\|$ (Euclidean norm), absolute $c=0.05$ ; both multiply $q_{B}$ . Seeds: 0, 1, 2; 24 runs.
Fashion-MNIST MLP ( Appendix D )	Data, model, loss: as for the controlled ramps. Optimizer: full-batch EMA momentum, $\beta_{1}=0.9$ , $\omega=0$ ; per-tensor none, VR, quartic, relativistic and clipping maps; global VR and quartic maps. LR: constant, $\eta=\mu\cdot 38/\lambda_{0}$ , $\lambda_{0}=33.9$ (power iteration at initialization), $\mu\in\{0.5,1,2,4,8\}$ ; scale sweep at $\mu\in\{8,16\}$ . Batch: full. Length: 3,000 steps. Precision: FP32 (CPU). Scale: absolute $c=0.05$ , argument $\\|m_{B}\\|$ per tensor (or $\\|m\\|$ for the global maps); sweep $c\in\{0.005,0.015,0.05,0.15\}$ (per tensor; VR, relativistic, quartic, clipping). Seeds: 0 only; $35+32$ runs. Failure: loss non-finite or $>10^{3}$ .

Appendix

Table 12: Study protocols.

Gradient Descent Dynamics

Gradient Descent Dynamics studies how iterative gradient-based updates shape optimization trajectories, stability, convergence, and the solutions models learn.

See all 153 papers

Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal
Aug 21, 2026Yiman Fong, Heng YangEdge of Stability Adaptive Gradient Methods
School of Engineering and Applied Sciences, Harvard University.
A Journey to the Edge of Stability
Sep 26, 2026Jaerin Lee, Kyoung Mu LeeDeep Learning Optimization Neural Network Training Dynamics
Computer Vision Lab, ASRI Seoul National University Seoul 08826, Korea
The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System
Jul 6, 2026Thomas HofmannEdge of Stability Gradient Descent
Department of Computer Science, ETH Zurich

Aug 21, 2026J/K move · Enter open · S save

Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal

Yiman Fong, Heng Yang

School of Engineering and Applied Sciences, Harvard University.

Edge of Stability Adaptive Gradient Methods Neural Network Training Dynamics

Open paper AskPDF arXiv

The edge-of-stability (EoS) phenomenon of full-batch Adam has been widely observed, yet its underlying dynamical mechanism remains poorly understood. In this paper, we identify Adam's second-moment adaptation as a negative-feedback mechanism that drives the dynamics toward the stability boundary. We characterize this mechanism through the active curvature, namely, the preconditioned curvature along the preconditioned gradient direction, and establish rigorous characterizations in progressively richer settings: rank-one quadratics with momentum, diagonal quadratics, on which the active curvature separates from the sharpness, and general objectives. Importantly, the mechanism predicts gradient reversal of full-batch Adam near the edge: consecutive gradients repeatedly point in nearly opposite directions, as we observe across fully connected networks, ResNets, ViTs, LSTMs, GPT-2 medium, and Adam-family optimizers. Consistent with this picture, averaging iterates suppresses these fast oscillations and produces smoother and lower loss curves. Together, these results provide an important first step towards fully understanding the dynamical behavior of Adam's EoS through active curvature and gradient reversal.

Abstract

Gradient Descent Dynamics

Explore similar work

Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal

A Journey to the Edge of Stability

The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System