Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal

Yiman Fong; Heng Yang

cs.LGAug 21, 2026

Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal

Authors: Yiman Fong, Heng Yang

Organizations: School of Engineering and Applied Sciences, Harvard University.

Abstract

The edge-of-stability (EoS) phenomenon of full-batch Adam has been widely observed, yet its underlying dynamical mechanism remains poorly understood. In this paper, we identify Adam's second-moment adaptation as a negative-feedback mechanism that drives the dynamics toward the stability boundary. We characterize this mechanism through the active curvature, namely, the preconditioned curvature along the preconditioned gradient direction, and establish rigorous characterizations in progressively richer settings: rank-one quadratics with momentum, diagonal quadratics, on which the active curvature separates from the sharpness, and general objectives. Importantly, the mechanism predicts gradient reversal of full-batch Adam near the edge: consecutive gradients repeatedly point in nearly opposite directions, as we observe across fully connected networks, ResNets, ViTs, LSTMs, GPT-2 medium, and Adam-family optimizers. Consistent with this picture, averaging iterates suppresses these fast oscillations and produces smoother and lower loss curves. Together, these results provide an important first step towards fully understanding the dynamical behavior of Adam's EoS through active curvature and gradient reversal.

Figures & tables

Figure 1 : Left: the normalized sharpness (Eq. ( 4 ), orange, top) and the active curvature (Eq. ( 5 ), blue, bottom) along an Adam training trajectory; the stability boundary of the frozen step is normalized to

2

(dashed). Right: the negative-feedback loop of the second moment that holds the active curvature around the threshold (Section 3 ). Experimental details are in Appendix B.2 .

Figure 2 : Sharpness and active curvature along the same run. BatchNorm ResNet, full-batch MSE on

1000

CIFAR-10 images,

\eta=3\cdot 10^{-4}

\beta_{2}=0.999

4000

steps, at

\beta_{1}=0

(left) and

\beta_{1}=0.9

(right). Training details are in Appendix B.3 .

Figure 3 : Active curvature and gradient reversal in six full-batch Adam runs with

\beta_{1}=0.9

\beta_{2}=0.999

. (a–c) The width-

200

fc-tanh MLP of Appendix C.1 . (d, e) The ResNet with BatchNorm and the ViT of Appendix B.3 . (f) GPT-2 medium of Appendix B.4 . Blue:

W_{\mathrm{act}}

; dashed line at

2

. Red shading: the stretches on which consecutive full-batch gradients reverse.

Figure 4 : Period-two orbits at the edge in GPT-2 medium, trained full-batch on

64

fixed sequences with label-smoothed cross-entropy,

\eta=3\cdot 10^{-4}

: (a)

\beta_{1}=0

\beta_{2}=0.95

; (b)

\beta_{1}=0.9

\beta_{2}=0.999

. Blue: excess training loss

L-L^{\star}

over the entropy of the smoothed target. Red shading: the stretches on which the full-batch gradient reverses. Training details are in Appendix B.4 .

Appendix figures & tables45 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5 : The sharpness and the active curvature on a diagonal quadratic. Diagonal quadratic in

n=100

dimensions, Adam with

\beta_{1}=0.9

\beta_{2}=0.999

\eta=10^{-3}

. (a) The sharpness

\lambda_{\max}(P_{t})=\max_{i}{w_{i,t+1}}

(orange) and the active curvature

W_{\mathrm{act}}

(blue). (b) Forty consecutive plateau steps, one color per coordinate

i

: the line is its normalized curvature

w_{i,t+1}

and the dots mark the steps on which it attains the sharpness. Spectrum, initialization and the readings are described below.

Figure 6 : Width-

200

network (

3072{-}200{-}200{-}10

6.6\cdot 10^{5}

parameters), full-batch MSE,

\beta_{1}=0.9

6000

steps, at four learning rates: training loss (purple) above

\lambda_{\max}(P_{t})

(orange) and

W_{\mathrm{act}}

(blue),

31

-step rolling medians, threshold

2

dashed. The curvature axis is cut at

4.5

; at

\eta=10^{-2}

the sharpness rises above it (Table 1 ).

Figure 7 : The network and readings of Figure 6 , MSE,

\eta=10^{-3}

, at

\beta_{1}\in\{0,0.3,0.5,0.9\}

. The

\beta_{1}=0.9

panel is the

\eta=10^{-3}

run of Figure 6 .

Figure 8 : The network and readings of Figure 6 ,

\beta_{1}=0.9

\eta=10^{-3}

, under MSE and under cross-entropy with label smoothing (LS)

0.1

and

0.2

; the label-smoothed panels show the excess loss

L-L^{\star}

over the entropy of the smoothed target. The MSE panel is the

\eta=10^{-3}

run of Figures 6 and 7 .

loss	$\beta_{1}$	$\eta$	$\lambda_{\max}(P_{t})$	$W_{\mathrm{act}}$	train acc
MSE	0	$10^{-3}$	3.35	2.018	0.973
MSE	0.3	$10^{-3}$	2.25	2.013	0.992
MSE	0.5	$10^{-3}$	2.17	2.011	0.996
MSE	0.9	$10^{-3}$	2.01	1.965	0.996
MSE	0.9	$10^{-5}$	2.00	1.885	1.000
MSE	0.9	$10^{-4}$	2.01	1.970	1.000

Appendix

Table 1 : The runs of Figures 6 – 8 : width-

200

MLP on

5000

CIFAR-10 images, full batch,

6000

steps; medians of

\lambda_{\max}(P_{t})

and

W_{\mathrm{act}}

over steps

3000

–

6000

and the training accuracy (median over steps

5800

–

5999

). “CE, LS

0.1

” and “CE, LS

0.2

” are cross-entropy with label smoothing

0.1

and

0.2

Figure 9 : BatchNorm ResNet and ViT (full-batch MSE on

1000

CIFAR-10 images) and LSTM at

\beta_{1}=0

\eta=3\cdot 10^{-4}

\beta_{2}=0.999

4000

steps. Top: training loss (purple). Bottom:

\lambda_{\max}(P_{t})

(orange, every

50

steps) and the

31

-step rolling median of

W_{\mathrm{act}}

(blue), dashed line at

2

Figure 10 : As Figure 9 at

\beta_{1}=0.9

architecture	params	$\beta_{1}$	$\lambda_{\max}(P_{t})$	$W_{\mathrm{act}}$	loss
ResNet-BN	1.22M	0	2.89	2.023	0.00406
ResNet-BN	1.22M	0.9	2.02	1.957	0.000164
ViT	413k	0	2.39	2.021	0.000777
ViT	413k	0.9	1.99	1.955	0.000377
LSTM	6.70M	0	2.46	2.102	0.139
LSTM	6.70M	0.9	1.95	1.466	1.11

Appendix

Table 2 : Second-half medians (steps

2000

–

3999

) of the two curvature readings for the six runs of Figures 9 and 10 ; loss is the training loss over the last

400

steps (MSE for the vision networks, excess cross-entropy over the floor

1.4075

for the LSTM).

Figure 11 : GPT-2 medium, full batch, over the whole of each run: (a)

\beta_{1}=0

\beta_{2}=0.95

8600

steps; (b)

\beta_{1}=0.9

\beta_{2}=0.999

, and (c)

\beta_{1}=0.9

\beta_{2}=0.95

30{,}000

steps. Top: training loss, excess over the label-smoothing floor

1.4076

(purple). Bottom: the exact full-batch sharpness

\lambda_{\max}(P_{t})

(orange; one reading every

100

steps in (a), every

250

in (b, c)) and active curvature

W_{\mathrm{act}}

(blue; every

10

steps in (a), every

50

in (b, c)); faint markers are the individual readings, lines their rolling medians over about

500

steps (

750

for the sharpness in (b, c)); dashed line at

2

; readings outside

[-0.25,4.5]

are drawn as triangles at the edge.

variant	$c$	threshold	$\lambda_{\max}$	$W_{\mathrm{act}}$	locked
Adam (uncorrected)	$(1-\beta_{1})/(1+\beta_{1})$	$38/\eta$	2.01	1.95	100%
Adam, bias-corrected	$c_{\mathrm{Adam}}$	$38/\eta$	2.02	1.94	99%
AdamW	$c_{\mathrm{Adam}}/(1-\eta\lambda/2)$	$38(1-\eta\lambda/2)/\eta$	2.00	1.92	94%
Adam + $L_{2}$	$c_{\mathrm{Adam}}$	$38/\eta$	1.87	1.76	43%
Padam ( $p=1/4$ )	$c_{\mathrm{Adam}}$	$38/\eta$	2.01	1.97	100%
Nadam	$c_{\mathrm{Adam}}(1+2\beta_{1})$	$13.57/\eta$	2.09	2.00	100%

Appendix

Table 3 : Ten variants, with

\beta_{1}=0.9

or momentum

\mu=0.9

where the variant has a momentum term: the normalization constant

c

and the raw threshold of each, and their readings on the width-

200

network (fc-tanh

3072{-}200{-}200{-}10

5000

CIFAR-10 images, MSE,

3000

steps, last quarter): means of the normalized

\lambda_{\max}

and

W_{\mathrm{act}}

, and the fraction of plateau steps with

W_{\mathrm{act}}\in(1.8,2.1)

(“locked”).

Figure 12 : The ten variants of Table 3 on the width-

200

MLP (

5000

CIFAR-10 images, MSE, full batch,

3000

steps), one column per optimizer: training loss (purple) above

\lambda_{\max}(P_{t})

(orange) and

W_{\mathrm{act}}

(blue),

31

-step rolling medians, each normalized by the variant’s own constant

c

so that the threshold (dashed) is

2

for all. AdamW and

L_{2}

use decay

0.1

; Padam and Adagrad run at

\eta=10^{-2}

, the others at

10^{-3}

Figure 13 : Width-

512

MLPs with an exponential learning-rate decay over the shaded window (steps

2000

–

5000

31

-step rolling medians of

\lambda_{\max}(P_{t})

(orange) and

W_{\mathrm{act}}

(blue) normalized by the instantaneous

c\eta_{t}

Figure 14 : Batch-size scan on the width-

200

MLP,

\beta_{1}=0.9

\eta=10^{-3}

, MSE,

30{,}000

steps: minibatch training, in which Adam sees at each step the gradient of a random subset of the fixed training set (

N=5000

images) of size

B

, for

B/N=1

(full batch),

0.2

0.05

and

0.01

(four of the six fractions of Table 4 ). Top: the training loss on all

N

images (purple). Middle: the two readings on the full-set loss of that minibatch trajectory, the sharpness

\lambda_{\max}(P_{t})

of the full-set Hessian (orange, Lanczos every

20

steps) and the active curvature

W_{\mathrm{act}}

from the full-set gradient and Hessian (blue). Bottom: the same two readings on the loss of the step’s own minibatch , its Hessian and its gradient (

\lambda_{\max}

orange;

W_{\mathrm{act}}

blue, every step).

Figure 15 : Batch-size scan on the width-

200

MLP at

\beta_{1}=0

(otherwise as Figure 14 : minibatch training on random subsets of the fixed

N=5000

images,

B/N=1

0.2

0.05

and

0.01

\eta=10^{-3}

, MSE,

30{,}000

steps). Top: training loss on all

N

images (purple). Middle: sharpness and active curvature (blue) of the full-set loss along the minibatch trajectory. Bottom: the same two readings on the loss of the step’s own minibatch.

Figure 16 : Batch-size scan on GPT-2 medium,

\beta_{1}=0

\beta_{2}=0.95

8000

steps: minibatch training in which Adam sees at each step the gradient of

B=32

16

8

of the fixed

64

training sequences (

B/N=\tfrac{1}{2},\tfrac{1}{4},\tfrac{1}{8}

; the full-batch run,

B/N=1

, is Figure 11 (a)). Top: the training loss on all

64

sequences, as excess over the label-smoothing floor

1.4076

(purple). Middle: the readings on the full-set loss of the minibatch trajectory, the exact active curvature

W_{\mathrm{act}}

from the full-set gradient and Hessian-vector product (blue squares, one reading every

100

steps) and the sharpness

\lambda_{\max}(P_{t})

of the full-set Hessian, which was not recorded along the run and is recomputed at the saved checkpoints

x_{2000}

x_{4000}

x_{6000}

and

x_{8000}

(orange circles where it fits the axis, orange triangles at the top edge with the value written below them where it does not). Bottom: the same two readings on the loss of the step’s own minibatch :

W_{\mathrm{act}}

of the batch loss (blue squares, one sample every

100

steps) and its sharpness recomputed at

x_{4000}

x_{6000}

and

x_{8000}

on two random batches of the run’s size each (orange, median of the two). Dashed lines at

2

Figure 17 : Batch-size scan on GPT-2 medium at

\beta_{1}=0.9

\beta_{2}=0.999

(otherwise as Figure 16 : minibatch training on

32

16

8

of the fixed

64

sequences per step,

8000

steps; the full-batch run is Figure 11 (b)). Top: training loss on all

64

sequences, excess over the floor

1.4076

(purple). Middle: full-set readings, exact

W_{\mathrm{act}}

(blue squares, every

100

steps) and the full-set sharpness recomputed at the checkpoints

x_{2000}

–

x_{8000}

(orange circles). Bottom: the readings on the step’s own minibatch loss,

W_{\mathrm{act}}

of the batch loss (blue squares, every

100

steps) and its sharpness at the checkpoints

x_{4000}

–

x_{8000}

(orange circles). Dashed lines at

2

			full-set loss		minibatch loss
model	$\beta_{1}$	$B/N$	$W_{\mathrm{act}}$	$\lambda_{\max}(P_{t})$	$W_{\mathrm{act}}$	$\lambda_{\max}$
GPT-2	0	full	2.14	2.76	2.14	2.76
GPT-2	0	1/2	2.11	15.09	2.59	19.18
GPT-2	0	1/4	3.58	21.12	3.36	10.80
GPT-2	0	1/8	4.22	29.69	6.36	58.58
GPT-2	0.9	full	1.82	1.91	1.82	1.91

Appendix

Table 4 : Batch-fraction scan: late-window medians of the active curvature and of the sharpness. MLP: width

200

N=5000

, steps

20{,}000

–

30{,}000

. GPT-2 medium: fixed

64

sequences, exact full-set

W_{\mathrm{act}}

, steps

4000

–

8000

. The full-batch GPT-2 rows are the runs of Appendix B.4 over the same window; on the subsampled runs the sharpness was not recorded during training and is recomputed at their saved checkpoints

x_{4000}

x_{6000}

and

x_{8000}

. Minibatch-loss columns: the same two readings taken on the loss of a minibatch of the run’s size, with its own gradient and Hessian. On the MLP both are logged along the run (

W_{\mathrm{act}}

every step,

\lambda_{\max}

by Lanczos every

20

steps, on the step’s own batch).

Figure 18 : Width-

200

fc-tanh MLP on CIFAR-10, full-batch MSE,

\beta_{1}=0.9

\beta_{2}=0.999

, at

\eta=10^{-3}

3\cdot 10^{-3}

and

10^{-2}

(first

10{,}000

steps). Top: training loss at the iterate

L(x_{t})

(blue, solid), at the midpoint of consecutive iterates (cyan, dashed) and at the EMA iterate (orange, dotted). Bottom:

\cos(g_{t},g_{t-1})

(red) and

\cos(m_{t},m_{t-1})

(cyan). The run protocol and the numbers are in Appendix C.1 .

Figure 19 : Width-

200

fc-tanh MLP on CIFAR-10, full-batch MSE,

\eta=10^{-3}

, at

\beta_{1}=0

(a) and

\beta_{1}=0.9

(b): loss at the iterate

x_{t}

(blue, solid), at the midpoint of consecutive iterates (cyan, dashed) and at the EMA of the iterate with rate

0.99

(orange, dotted). The orbit height

E_{\mathrm{osc}}=L(x_{t})-L(\mathrm{mid})

settles to a constant, at once at

\beta_{1}=0

and after about

3000

steps at

\beta_{1}=0.9

, while the loss keeps falling (Figure 20 ); details are in Appendix C.1 .

Figure 20 : The orbit height

E_{\mathrm{osc}}=L(x_{t})-L(\tfrac{1}{2}(x_{t}+x_{t-1}))

of the two runs of Figure 19 , as a

501

-step rolling median:

\beta_{1}=0

(red) and

\beta_{1}=0.9

(blue). It is flat from step

1000

(

\beta_{1}=0

) and from about step

3000

(

\beta_{1}=0.9

) on, while the loss at the iterate keeps falling, by a factor of

6

or more.

Figure 21 : Momentum. The small network at the reference setting of Appendix C.2 with

\beta_{1}\in\{0,0.3,0.7,0.9,0.95\}

; the

\beta_{1}=0

and

\beta_{1}=0.9

runs use seeds

2

and

1

, the others seed

0

. Top: training loss at the iterate (blue, solid), at the midpoint of consecutive iterates

\tfrac{1}{2}(x_{t}+x_{t-1})

(cyan, dashed) and at the EMA of the iterate with rate

0.99

(orange, dotted). Bottom:

101

-step rolling medians of

\cos(g_{t},g_{t-1})

(red) and

\cos(m_{t},m_{t-1})

(cyan, dashed); the dashed line is

-1

Figure 22 : Second-moment constants. The network and readings of Figure 21 with

\beta_{2}=0.95

(at

\beta_{1}=0

0.99

and

0.9999

, and with

\varepsilon=10^{-6}

Figure 23 : Learning rate. The network and readings of Figure 21 at

\eta=3\cdot 10^{-4}

and

3\cdot 10^{-3}

Figure 24 : Network and data. The readings of Figure 21 with ReLU (at

\beta_{1}=0.9

and

0

), on CIFAR-100, with

n=1000

training images, and with a single hidden layer of

512

setting	$\beta_{1}$	$\beta_{2}$	$\eta$	$\cos(g_{t},g_{t-1})$	$\cos(g_{t},g_{t-2})$	$\cos(m_{t},m_{t-1})$	reversal steps	loss
seed 2, $\beta_{1}{=}0$	0	0.999	0.001	-0.993	+0.990	-0.993	100%	0.156
$\beta_{1}{=}0.3$	0.3	0.999	0.001	-0.992	+0.982	-0.985	100%	0.102
$\beta_{1}{=}0.7$	0.7	0.999	0.001	-0.982	+0.938	-0.941	100%	0.0742
seed 1	0.9	0.999	0.001	-0.982	+0.943	-0.402	99%	0.0611
$\beta_{1}{=}0.95$	0.95	0.999	0.001	-0.971	+0.908	+0.480	98%	0.0608
$\beta_{2}{=}0.95$ , $\beta_{1}{=}0$	0	0.95	0.001	-0.963	+0.904	-0.963	100%	0.062

Appendix

Table 5 : The sixteen settings of Figures 21 – 24 , grouped as the figures: medians over steps

2000

–

4000

\cos(g_{t},g_{t-1})

\cos(g_{t},g_{t-2})

and

\cos(m_{t},m_{t-1})

, the fraction of steps with

\cos(g_{t},g_{t-1})<-0.5

, and the training loss (median over steps

3800

–

3999

Figure 25 : Reversal on the three runs of Figure 9 (

\beta_{1}=0

). Top: training loss at the iterate (purple, solid), at the midpoint of consecutive iterates (cyan, dashed) and at the EMA of the iterate with rate

0.99

(orange, dotted); MSE for the vision networks, excess cross-entropy over the floor

1.4075

for the LSTM. Bottom:

101

-step rolling medians of

\cos(g_{t},g_{t-1})

(red, solid) and

\cos(m_{t},m_{t-1})

(cyan, dashed); dashed line at

-1

. At

\beta_{1}=0

the momentum is the gradient,

m_{t+1}=g_{t}

, so the two curves coincide.

Figure 26 : As Figure 25 for the three runs of Figure 10 (

\beta_{1}=0.9

architecture	$\beta_{1}$	$\cos(g_{t},g_{t-1})$	$\cos(g_{t},g_{t-2})$	$\cos(m_{t},m_{t-1})$	reversal steps
ResNet-BN	0	-0.998	+0.994	-0.998	100%
ResNet-BN	0.9	-0.966	+0.872	-0.944	100%
ViT	0	-0.996	+0.990	-0.996	100%
ViT	0.9	-0.975	+0.905	-0.966	100%
LSTM	0	-0.986	+0.991	-0.986	79%
LSTM	0.9	-0.498	-0.286	-0.027	50%

Appendix

Table 6 : Second-half medians (steps

2000

–

3999

) of the cosines for the six runs of Figures 9 and 10 , and the fraction of steps with

\cos(g_{t},g_{t-1})<-0.5

$\beta_{1}$	$\beta_{2}$	steps	$\cos(g_{t},g_{t-1})$	$\cos(m_{t},m_{t-1})$	reversal
$0$	$0.95$	$2{,}000$ – $4{,}000$	$-0.97$	$-0.97$	$89\%$
		$4{,}000$ – $8{,}000$	$-0.98$	$-0.98$	$93\%$
		$8{,}000$ – $8{,}600$	$-0.99$	$-0.99$	$94\%$
$0.9$	$0.999$	$2{,}000$ – $8{,}000$	$-0.90$	$0.36$	$61\%$
		$8{,}000$ – $12{,}000$	$-0.97$	$-0.41$	$78\%$
		$12{,}000$ – $20{,}000$	$-0.93$	$-0.30$	$75\%$

Appendix

Table 7 : Gradient reversal on the three full-batch GPT-2 runs of Appendix B.4 : median

\cos(g_{t},g_{t-1})

and

\cos(m_{t},m_{t-1})

, and fraction of steps with

\cos(g_{t},g_{t-1})<-0.5

over every step of the window; at

\beta_{1}=0

the momentum is the gradient.

Figure 27 : Gradient reversal along the three full-batch GPT-2 runs of Figure 11 : (a)

\beta_{1}=0

\beta_{2}=0.95

8600

steps; (b)

\beta_{1}=0.9

\beta_{2}=0.999

, and (c)

\beta_{1}=0.9

\beta_{2}=0.95

30{,}000

steps. Top: training loss, excess over the label-smoothing floor

1.4076

, at the iterate (purple, solid), at the midpoint of consecutive iterates (cyan, dashed) and at the EMA of the iterate with rate

0.99

(orange, dotted). Bottom:

101

-step rolling medians of

\cos(g_{t},g_{t-1})

(red, solid) and

\cos(m_{t},m_{t-1})

(cyan, dashed); the dashed line is

-1

; at

\beta_{1}=0

the two curves coincide. Red shading in both rows: stretches on which

\cos(g_{t},g_{t-1})<-0.8

on the majority of a

51

-step window, as in Figure 4 ; no stretch qualifies in (c).

Figure 28 : Cosine traces of the two full-batch GPT-2 runs of Figure 4 :

101

-step rolling medians of

\cos(g_{t},g_{t-1})

(red) and

\cos(m_{t},m_{t-1})

(cyan); the dashed line is

-1

. (a)

\beta_{1}=0

\beta_{2}=0.95

(b)

\beta_{1}=0.9

\beta_{2}=0.999

Figure 29 : Reversal on the ten configurations of Figure 12 (

\beta_{1}=0.9

\mu=0.9

), one column per optimizer in the order of Table 3 . Top of each block: training loss at the iterate (purple, solid), at the midpoint of consecutive iterates (cyan, dashed) and at the EMA of the iterate with rate

0.99

(orange, dotted). Bottom:

101

-step rolling medians of

\cos(g_{t},g_{t-1})

(red, solid) and

\cos(m_{t},m_{t-1})

(cyan, dashed); dashed line at

-1

. RMSProp and Adagrad have no momentum and show the gradient cosine only.

Figure 30 : As Figure 29 at

\beta_{1}=0

for the six variants that are run again there (Table 8 );

m_{t+1}=g_{t}

, so the two cosine curves coincide.

	$\beta_{1}=0.9$ or $\mu=0.9$						$\beta_{1}=0$
variant	$\cos(g_{t},$ $g_{t-1})$	$\cos(g_{t},$ $g_{t-2})$	$\cos(m_{t},$ $m_{t-1})$	reversal	$W_{\mathrm{act}}$	$\lambda_{\max}$	$\cos(g_{t},$ $g_{t-1})$	$\cos(g_{t},$ $g_{t-2})$	reversal	$W_{\mathrm{act}}$	$\lambda_{\max}$
Adam (uncorrected)	-0.987	+0.947	-0.909	99%	1.97	2.01	-0.998	+0.995	100%	2.05	3.70
Adam, bias-corrected	-0.959	+0.832	-0.905	99%	1.94	2.02	-0.983	+0.982	100%	2.28	2.84
AdamW	-0.972	+0.872	-0.753	94%	1.92	2.00	-0.996	+0.994	100%	2.09	3.69
Adam + $L_{2}$	+0.677	+0.942	-0.128	1%	1.76	1.87	-0.119	+0.844	3%	1.94	2.90
Padam ( $p=1/4$ )	-0.982	+0.930	-0.957	100%	1.97	2.01	-1.000	+0.999	100%	2.00	2.06

Appendix

Table 8 : Reversal of the ten variants of Table 3 , last quarter of the run (steps

2250

–

2999

): medians of

\cos(g_{t},g_{t-1})

\cos(g_{t},g_{t-2})

and

\cos(m_{t},m_{t-1})

, the fraction of steps with

\cos(g_{t},g_{t-1})<-0.5

(“reversal”), and the means of the normalized

W_{\mathrm{act}}

and

\lambda_{\max}

, the statistics of Table 3 .

Figure 31 : Gradient cosine against the batch fraction

B/N

, late-window medians, for (a) the width-

200

MLP and (b) GPT-2 medium, at

\beta_{1}=0

(red circles) and

\beta_{1}=0.9

(blue squares). Filled markers, solid lines: consecutive full-set gradients; hollow markers, dashed lines: consecutive minibatch gradients along the same runs (at

B/N=1

the two coincide). Dashed horizontal line at

-1

model	$\beta_{1}$	$B/N$	$\cos(g_{t},g_{t-1})$	reversal	$\cos(g_{B,t},g_{B,t-1})$	reversal	$\cos(g_{B,t},g_{t})$
			full-set gradient		minibatch gradient
GPT-2	0	full	-0.98	93%	-0.98	93%	+1.00
GPT-2	0	1/2	-0.94	90%	-0.95	89%	+0.95
GPT-2	0	1/4	-0.79	75%	-0.75	70%	+0.90
GPT-2	0	1/8	-0.44	43%	-0.36	30%	+0.62
GPT-2	0.9	full	-0.93	65%	-0.93	65%	+1.00

Appendix

Table 9 : Batch-fraction scan, the reversal read on the full-set gradient and on the minibatch gradient of the step: late-window medians of

\cos(g_{t},g_{t-1})

and of

\cos(g_{B,t},g_{B,t-1})

, the fraction of steps on which each is below

-0.5

(“reversal”), and the alignment

\cos(g_{B,t},g_{t})

of the batch gradient with the full gradient at the same iterate. MLP: width

200

N=5000

, steps

20{,}000

–

30{,}000

, every column logged at every step. GPT-2 medium: fixed

64

sequences, steps

4000

–

8000

Figure 32 : Gradient reversal in the batch-size scan on the width-

200

MLP,

\beta_{1}=0

, the runs of Table 9 (

\eta=10^{-3}

, MSE,

30{,}000

steps; at each step Adam sees a random subset of size

B

of the fixed

N=5000

images,

B/N=1

0.2

0.05

0.01

). Top: the training loss on all

N

images at the iterate (blue, solid), at the midpoint of consecutive iterates (cyan, dashed) and at the EMA of the iterate with rate

0.99

(orange, dotted), and the loss of the step’s own minibatch (grey, thin). Bottom:

101

-step rolling medians of the cosine of consecutive full-set gradients

\cos(g_{t},g_{t-1})

(red), of consecutive momenta

\cos(m_{t},m_{t-1})

(cyan, dashed; at

\beta_{1}=0

the momentum is the gradient,

m_{t+1}=g_{t}

, and the two curves coincide), of consecutive minibatch gradients

\cos(g_{B,t},g_{B,t-1})

(grey) and of the alignment

\cos(g_{B,t},g_{t})

of the batch gradient with the full gradient at the same iterate (purple, dotted). Dashed line at

-1

. In the full-batch panel the minibatch is the full set and only the full-set curves are drawn.

Figure 33 : As Figure 32 at

\beta_{1}=0.9

(the same fractions and readings).

Figure 34 : Gradient reversal in the batch-size scan on GPT-2 medium,

\beta_{1}=0

\beta_{2}=0.95

8000

steps, the runs of Table 9 (at each step Adam sees

B=32

16

8

of the fixed

64

training sequences; the full-batch run is Figure 27 (a)). Top: the training loss on all

64

sequences as excess over the label-smoothing floor

1.4076

, at the iterate (blue, solid) and at the midpoint of consecutive iterates (cyan, dashed), and the loss of the step’s own minibatch (grey, thin);

101

-step rolling medians of

\cos(g_{t},g_{t-1})

on the full set (red), of

\cos(m_{t},m_{t-1})

(cyan, dashed; at

\beta_{1}=0

it coincides with the gradient cosine) and of

\cos(g_{B,t},g_{B,t-1})

on the minibatches (grey). The alignment

\cos(g_{B,t},g_{t})

was not logged along these runs; its checkpoint values are in Table 9 . Dashed line at

-1

Figure 35 : As Figure 34 at

\beta_{1}=0.9

\beta_{2}=0.999

Panel of Figure 36 — Figure 36 : Proof roadmap for $\beta_{1}>0$ (Theorem 3.3 ). Left: below the edge, an adaptive Lyapunov function certifies contraction up to a cutoff $\overline{W}<2$ when $\beta_{2}>\beta_{1}^{2}$ (Proposition F.2 , Corollary F.4 ), while strictly subcritical four-cycles exist for complementary parameter choices when $\varepsilon=0$ (Proposition F.5 ). Right: above the edge, the sign of $x^{\scriptscriptstyle\parallel}_{t}h_{t}$ separates aligned trajectories, which exit every fixed supercritical band in finite time (Lemma F.6 , Corollary F.7 ), from exceptional persistently misaligned trajectories that converge while remaining uniformly supercritical (Proposition F.8 ).

Figure 37 : The four-cycle of Proposition F.5 at

\beta_{1}=0.5

\beta_{2}=0.2

\varepsilon=0

\eta=0.1

: the constructed

\zeta=0.25868

gives an orbit of exact period four (a) whose two sharpness values,

0.7095

and

1.3825

, both stay strictly below

2

(b).

Figure 38 : Zero momentum (

\eta=0.05

\beta_{2}=0.9

w_{\max}=3

). (a) From

w_{0}=1.2

the target

\widehat{w}=1.8

is reached at

\sigma=17

, against the certificate

N_{\rm sub}=19

of Theorem E.1 . (b) From

w_{0}=w_{\max}

the band

w\geq 2.3

is left at

\tau_{\delta}=6

, against

N_{\rm sup}=10

of Theorem E.3 . (c) A longer run crosses

w=2

from both sides and then settles on the exact edge two-cycle of Proposition 3.1 .

Figure 39 : Positive momentum (

\eta=0.9

\beta_{1}=0.9

\beta_{2}=0.99

w_{\max}=2.84

\overline{W}=1.9722

). (a) Starting at

w_{0}=1

, the normalized sharpness leaves the band

w\leq b=1.9

t=268

, as forced by Corollary F.4 . (b) The Lyapunov functional

\Psi_{t}

falls over that stretch, an observed per-step factor

0.905

; the certified factor is

1-\Delta(a,b)

with

a=w_{0}=1

b=1.9

and

\Delta=1.93\cdot 10^{-6}

. (c) An aligned supercritical start leaves

w\geq 2.3

\tau=5

, against the closed-form bound

16

of Corollary F.7 .

Figure 40 : The misaligned orbit of Proposition F.8 at

\beta_{1}=0.5

\beta_{2}=0.9

\eta=0.3

w_{\max}=6

: over

120

steps

\left|x^{\scriptscriptstyle\parallel}_{t}\right|

falls below

10^{-130}

while

w_{t}

rises from

2.6

5.98

, so the iterate converges to the minimizer manifold without ever leaving the supercritical region.

Explore similar work

May 7, 2026cs.LG

A Rod Flow Model for Adam at the Edge of Stability

Cohen et al. (arXiv:2207.14484) observed that adaptive gradient methods such as Adam operate at the edge of stability. While there has been significant work on continuous-time modeling of gradient descent at the edge of stability, extending these models to momentum methods remains underdeveloped. In the gradient descent setting, Regis et al. (arXiv:2602.01480) introduced rod flow, which models consecutive iterates as an extended one-dimensional object -- a "rod." Here we extend rod flow to Adam by working in the joint phase space of parameters and first moment

(w, m)

and treating the second moment

ν

as a smooth auxiliary variable. We also develop rod flows for heavy ball momentum, Nesterov momentum, and scalar and per-component versions of RMSProp, Adam, and NAdam. For all eight optimizers, we empirically evaluate rod flow on representative machine learning architectures, where it tracks the discrete iterates through the edge-of-stability regime significantly more accurately than the corresponding stable flow.

Eric Regis, Sinho Chewi

Yale University

Adam Momentum Gradient Descent Edge

Ask PDF

Date pendingstat.ML

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-preconditioned mean-gradient operator, and high-probability projected stationarity guarantees under general

L

-smooth objectives. In the tracking regime, we derive finite-time expected and high-probability bounds that decompose sharply into four components: initialization, objective drift, a first-moment tracking error governed by

\beta_1

, and a preconditioner perturbation governed by

\beta_2

. We characterize the burn-in time required for the transient terms to decay to the asymptotic tracking bound under constant and step-decay schedules. We also prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. Across both analyses, our bounds reveal a noise--drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can yield favorable upper guarantees, whereas in drift-dominated regimes, stale first-moment information and preconditioner perturbations can enlarge Adam's tracking guarantee, potentially allowing vanilla SGD to attain a smaller tracking error. Our explicit

(\beta_1,\beta_2,\epsilon)

-dependent bounds identify mechanisms through which adaptive step-sizing can help or hurt under nonstationarity and provide theoretical explanations consistent with Adam's empirical instability and stabilization under distribution shift.

Sharan Sahu, Abir Sarkar, Cameron J. Hogan +1

Department of Statistics and Data Science Cornell University

Adam Stationarity Stochastic Gradient Descent Spectral Preconditioning Gradient Staleness Monotonicity+1

Ask PDF

Jul 4, 2026cs.LG

Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam

The local sharpness of the loss, the top Hessian eigenvalue

λ_1

, determines the largest stable gradient step, but measuring it normally requires Lanczos or Hessian-vector iterations. We observe that a single Armijo backtracking line search already carries this information at the cost of a few forward passes: the accepted step

α

brackets the \emph{directional} curvature

q = g^\top H g/\|g\|^2

within the multiplicative band set by the backtracking factor. Across CIFAR-10, Fashion-MNIST and Imagenette,

\logα

tracks

\logλ_1

at Pearson

-0.91

-0.95

, giving a low-cost online Edge-of-Stability reading. Used once at initialisation, this measurement yields a learning-rate cap (a safeguard, not a faster optimiser) that makes Adam robust to a too-large initial learning rate across more than three orders of magnitude (

10^{-3}

3.0

), at about one percent overhead, and it is a no-op when the chosen rate is already safe. One probe is enough: periodic in-training probing adds no robust benefit. The raw-gradient probe exposes the mechanism but needs a safety factor calibrated to the architecture by a one-minute divergence sweep. Probing along Adam's own update direction removes this calibration: a single fixed safety factor

κ= 2

avoids divergence on all nine architectures we test and across the full learning-rate grids of all four benchmarks, and the recipe transfers to AdamW unchanged.

Ashmitha R, Jörg Frochte

Department of Artificial Intelligence and Data Science, Sri Ramakrishna Engineering College, Anna University, Coimbatore 641022, Tamil Nadu, India · Interdisciplinary Institute for Applied AI and Data Science Ruhr (AKIS), Department of Electrical Engineering and Computer Science, Bochum University of Applied Sciences, Am Hochschulcampus 1, 44801 Bochum, Germany

Adam Sharpness-Aware Minimization Batch Hessian

Ask PDF