Gradient descent with momentum can be viewed as a particle moving with friction in the loss landscape. In deep learning the learning rate is usually large, so training is a discrete dynamical system: in many full-batch experiments the sharpness grows to the optimizer's stability limit, the edge of stability (EoS), and training continues there. Physics-inspired optimizers change this motion; Velocity-Regularized Adam (VRAdam), for example, shrinks the step at high speed. We study kinetic maps, in which the step is the gradient of a kinetic energy of the momentum, and ask what they do beyond the EoS. (i) In a reduced model, a kinetic map creates a symmetric period-two orbit beyond the edge, strictly linearly stable if and only if the map's elasticity (the log-log slope of step length versus momentum) lies between 0 and 1; on it, training sits at an effective edge of stability. VRAdam's damper fails above twice the critical learning rate, where its kinetic energy is not convex; the Legendre dual of VRAdam's own Lagrangian is convex and passes at every learning rate. (ii) Because the sharpness rises gradually, the loss of stability is delayed, as in Neishtadt's theory of slowly swept bifurcations. In a network with a controlled sweep, the dynamics match this theory without fitted parameters, including the memory of where the sweep started and its erasure by noise. (iii) With momentum, reaching the oscillation also requires elasticity at large momentum: maps that bound the step have competing period-four and chaotic attractors already at beta1 = 0.9, while the cubic-Legendre map reached the oscillation from every sampled start up to beta1 = 0.99. Experiments on an exact quadratic problem, a full-batch MLP and ResNet-32 support the theory; on ResNet-18, much of the apparent benefit of the maps is reproduced by matching the effective learning rate.
Figures & tables
Figure 1: (a) Radial speed profiles and (b) the kinetic energies they imply. (c,d) Late iterates of the reduced model without momentum, y′=y−aϕ(y) . The VR map has a stable two-cycle only for 2<a<4 , followed by a symmetry-breaking pitchfork, period doubling and chaos (first positive Lyapunov exponent at a≈5.32 ); the quartic map has a stable two-cycle for every a>2 .
Figure 2: Controlled sweeps through the edge in a network (tanh MLP, Fashion-MNIST, full-batch EMA momentum, β1=0.9 , no map; three seeds). (a) Deviation along the top Hessian direction after a kick δ=10−6 at Rin , while R is ramped at ρ=0.003 per step (solid: network, seed 0; dashed: linear recurrence ( 10 ) on the measured path; dotted: exit predicted by the slow-sweep relation ( 8 )). (b) Exit point (deviation back at δ ) versus entry point: network (markers, ρ≤0.01 ), model on an exact linear ramp (thin lines, ρ≤0.003 ) and the ρ→0 relation ( 8 ) (thick line). (c) Kick plus gradient noise σ , ρ=0.003 : network (markers) and model median and 10–90% range (lines, bands); the thick line is ( 8 ). (d) Noise only, Rin=0.5 , exit at ∣xt∣=10−4 , versus sweep rate.
worst share over A∈{3,4,6,10,20,40,100} for p=1/n=
β1
0.67
0.50
0.40
0.33
0.25
0.20
0.14
0.10
0.0
1
1
1
1
1
1
1
1
0.5
1
1
1
1
0.98
0.96
0.96
0.51
0.8
1
1
0.99
0.98
0.97
0.47
0.25
0.15
0.9
1
1
1
1
0.48
0.15
0.11
0.11
0.95
1
1
1
0.99
0.11
0.05
0.03
0.04
Table 1: Legendre power maps in the reduced model: worst share of 300 initial conditions that reach the symmetric orbit, over seven step sizes A∈[3,100] . p=1/2 is the cubic-Legendre map, p=1/3 the quartic-Legendre map.
Figure 3: Legendre power maps in the reduced model. (a) Share of initial conditions that reach the symmetric orbit versus the orbit elasticity, all (p,A) . (b) Worst share over A∈[3,100] versus the asymptotic elasticity p .
Figure 4: Exact quadratic (7,850 parameters, real data). (a–c) Late momentum norm ∥m∥/c versus μ=ηλmax/aEoS (markers: last 200 steps of three seeds) against the closed-form orbit (solid: stable, dotted: unstable). (d) Reduced model, β1=0.9 : share of initial conditions that reach the orbit versus the orbit elasticity, for four families of maps.
Figure 5: Full-batch EMA momentum ( β1=0.9 ) on a tanh MLP, 1,000 CIFAR-10 images, 10 seeds per setting. Colour: map; filled circles and solid lines: absolute scale c=0.05 ; open triangles and dashed lines: relative scale c=2∥∇BL(θ0)∥ . (a) One seed at μ=0.5 , absolute scale: sharpness ratio R=ηλmax/aEoS (solid) and effective ratio Reff of ( 12 ) (dotted). Plain momentum stays at the edge, R≈1 . With the quartic map the sharpness rises to 1.75 times the edge while Reff stays at one; with the VR map the sharpness rises to more than five times the edge while Reff mostly stays below one. (b) Mode-weighted multiplier gˉ of ( 12 ) against R at every diagnostic with R≥1.1 after step 100, all completed runs. Points on the line gˉ=1/R satisfy Reff=1 : the convex maps lie on it, the VR map lies below it (a smaller step than the edge requires), and the open triangles near gˉ=1 are relative-scale diagnostics at which the map has not yet engaged, so that the effective step is above the edge. (c) Median final loss versus step size; crosses mark settings in which at least half of the runs diverged.
trained / collapsed / diverged
method
μ=0.5
μ=2
μ=4
momentum (no map)
5/0/0
0/4/1
0/1/4
quartic-Legendre, abs. c
5/0/0
5/0/0
5/0/0
cubic-Legendre, abs. c
5/0/0
5/0/0
5/0/0
relativistic, abs. c
5/0/0
5/0/0
5/0/0
VR map, abs. c
5/0/0
5/0/0
5/0/0
Table 2: ResNet-32 (5 seeds): runs that trained (final loss <10−3 ), collapsed (final loss >0.03 , near the constant-predictor value 0.09 ) or diverged; (+1): one run in between.
Figure 6: Full-batch EMA momentum on ResNet-32 (batch normalization without running statistics), 1,000 CIFAR-10 images, 5 seeds; panels as in Fig. 5 . In (b) the multipliers stay near one above the plain edge: the map has not yet engaged.
η
method
peak R
late R
min. mult. (steps ≥1500 )
min. mult. (steps <100 )
final loss
0.002
Adam
1.29
0.99
1.000
1.00
3.4×10−6
0.002
VRAdam
1.08
0.96
1.000
0.59
2.4×10−5
0.002
Adam + relativistic map on rms(q) , c=0.5
1.13
1.00
0.994
0.45
5.4×10−6
0.002
Adam + relativistic map on ∥m^∥ , c=0.05
1.36
0.99
0.996
0.15
4.9×10−6
0.01
Adam
1.08
0.96
1.000
1.00
1.7×10−4
0.01
VRAdam
1.26
0.94
1.000
0.30
3.1×10−6
Table 3: Full-batch Adam on the ResNet-32 objective, medians over three seeds. R is the preconditioned sharpness ratio ηλmax(P−1H)/38 ; “late” is the median over steps ≥1500 .
configuration
test accuracy (%)
test CE
Δ acc (pp)
Δ CE
Muon, tuned
94.02 ± 0.18
0.351 ± 0.006
ref.
ref.
Kinetic Muon (quartic EMA map), tuned
94.19 ± 0.17
0.335 ± 0.008
+0.17± 0.13 ∗
−0.016± 0.011 ∗
Muon at Kinetic Muon’s effective lr ( ×0.66 )
94.09 ± 0.07
0.344 ± 0.008
+0.07± 0.20
−0.008± 0.007
Muon at Kinetic Muon’s nominal lr
94.01 ± 0.13
0.362 ± 0.009
0.00± 0.22
+0.010± 0.013
AdamW, tuned
92.36 ± 0.19
0.338 ± 0.010
ref.
ref.
AdamW at 3× tuned lr
92.85 ± 0.18
0.360 ± 0.007
+0.50± 0.23 ∗
+0.022± 0.015 ∗
Table 4: ResNet-18 / CIFAR-10, effective-learning-rate controls: mean ± s.d. over 5 seeds, and paired differences to the reference row of each block ( ∗ : p<0.05 , paired t -test).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: EMA momentum, β1=0.9 . (a) Floquet radius of the symmetric EoS orbit versus a/aEoS . (b,c) Late iterates.
Figure 8: Block-diagonal quadratic, quartic map at 6× the edge step: global versus per-block maps.
Figure 9: Self-stabilization model with α=1 (edge at S=2 ), c=0.3 . (a) Sharpness, (b) loss at αS0=6 ; (c) steps until S<2.05 .
Figure 10: Delayed loss of stability, reduced model with the cubic map, β1=0.9 . (a) Reff while R is swept linearly through the edge at a slow and a fast rate; the dotted line is Reff=R (map inactive). (b) Peak Reff versus sweep rate: simulation (dots, 64 noise realizations; band: 10–90%) and the linear estimate (lines) for three noise levels; grey band: range over initial amplitudes 10−6 – 1 and the cubic and quartic maps; open markers: largest Reff during the climb of the sharpness in the networks ( μ=0.5 , all seeds; orange ResNet-32, blue MLP).
Figure 11: Fashion-MNIST MLP: full-batch momentum on a tanh MLP with per-tensor maps (one seed). (a) Final loss and (b) final sharpness versus step size; (c,d) final loss versus c at 8× and 16× the initial edge step.
median final loss (diverged runs)
Reff above the edge
method
μ=0.5
μ=1
μ=2
μ=4
μ=8
median [IQR] (runs)
momentum (no map)
1.3×10−5
5.8×10−6
1.6×10−6 (1)
– (10)
– (10)
– (0)
quartic-Legendre, abs. c
2.4×10−4
2.6×10−5
1.0×10−6
4.7×10−9
3.1×10−6
1.01 [1.00, 1.04] (50)
cubic-Legendre, abs. c
2.2×10−4
1.4×10−4
4.8×10−5
1.8×10−5
0.20
1.00 [1.00, 1.01] (50)
relativistic, abs. c
2.3×10−4
1.3×10−5
8.0×10−7
1.5×10−10
1.9×10−7
1.01 [1.00, 1.03] (50)
VR map, abs. c
2.6×10−3
4.9×10−3
3.0×10−4
3.4×10−7
6.3×10−7
0.80 [0.61, 0.91] (50)
Appendix
Table 5: MLP on CIFAR-10: median final training loss over 10 seeds (number of diverged runs in brackets) and per-run median of Reff over the diagnostics with R≥1.1 after step 100 (median and IQR over runs; number of runs with such points).
β2
η
R (Lanczos)
legacy / Lanczos
Rbs
ηλbs
λmin/λmax
0.95
0.001
0.20
0.96
0.086
3.3
-0.08
0.95
0.002
0.24
0.92
0.106
4.0
-0.07
0.95
0.005
0.23
0.91
0.106
4.0
-0.10
0.95
0.01
0.27
0.96
0.096
3.6
-0.10
0.999
0.001
0.17
0.94
0.076
2.9
-0.09
0.999
0.002
0.20
0.93
0.088
3.3
-0.08
Appendix
Table 6: Minibatch Adam on ResNet-32 (batch 1,000), medians over two seeds of the median over steps ≥900 . R : ηλmax/38 of the preconditioned Hessian of the training batch; legacy/Lanczos: ratio of the earlier power-iteration estimate to the Lanczos estimate on the same batch; Rbs : preconditioned batch sharpness over 38/η .
Table 7: Equal-budget comparison: mean ± s.d. over 3 fresh seeds of the configuration selected from 10 trials.
Figure 12: Validation loss versus peak learning rate (Fashion-MNIST MLP), with the map hyperparameters fixed in advance.
method
test CE
test accuracy (%)
AdamW
0.334 ± 0.012
92.44 ± 0.14
AdamW + clip
0.334 ± 0.004
92.66 ± 0.08
VRAdam
0.343 ± 0.005
92.90 ± 0.23
K-Adam quartic
0.344 ± 0.009
92.78 ± 0.22
K-Adam quartic, global
0.367 ± 0.008
92.85 ± 0.12
K-Adam relativistic
0.340 ± 0.006
92.35 ± 0.11
Appendix
Table 8: ResNet-18 / CIFAR-10, five fresh seeds (mean ± s.d.). † The curvature probes of these two methods also updated the batch-normalization running statistics used at test time (see text); reported for completeness only.
final validation loss (two seeds) at peak learning rate
method
0.001
0.003
0.01
0.03
AdamW
3.77 / 3.78
3.70 / 3.73
6.47 / 6.69
6.68 / 6.71
AdamW + clip 1.0
3.78 / 3.78
3.71 / 3.72
3.79 / 4.07
6.74 / 6.82
VRAdam
3.78 / 3.78
3.70 / 3.71
5.99 / 6.09
6.57 / 6.59
cubic map on m^ , k=1
3.87 / 3.88
3.73 / 3.74
3.96 / 4.84
5.24 / 5.86
cubic map on m^ , k=4
3.81 / 3.81
3.70 / 3.71
5.05 / 5.71
6.13 / 6.42
Appendix
Table 9: Small GPT (51.5M parameters, 393M tokens of FineWeb-Edu): final validation loss of both seeds (sorted) at each peak learning rate.
Figure 13: Small GPT. (a) Final validation loss versus peak learning rate (lines: mean of two seeds). (b) Training loss at η=0.01 . (c) Step multipliers of VRAdam and the cubic map at η=0.003 (solid) and 0.01 (dotted), seed 0.
regime
method
steps to grok (seeds 0, 1, 2)
final test acc.
largest loss after memorisation (seeds 0, 1, 2)
AdamW, wd 1
AdamW
20.4k, 14.6k, 14.5k
1.00
0.71, 0.21, 0.57
AdamW, wd 1
AdamW + clip 1.0
21.9k, 14.2k, 18.4k
1.00
0.14, 0.04, 0.15
AdamW, wd 1
VRAdam
19.9k, 12.7k, 15.5k
1.00
0.45, 0.05, 0.61
AdamW, wd 1
cubic map on m^ , k=1
21.1k, 11.4k, 15.6k
1.00
0.04, 0.23, 0.13
AdamW, wd 1
cubic map on m^ , k=4
18.0k, 12.6k, 13.1k
1.00
0.37, 0.02, 0.33
Adam, wd 0
AdamW
–, –, –
0.26
2.62, 5.43, 4.16
Appendix
Table 10: Grokking test (modular addition, p=113 ): steps until test accuracy ≥0.95 , final test accuracy (median), and the largest training loss from 1,000 steps after memorization to the end, for each seed.
Figure 14: Grokking test. (a,b) Training loss (solid) and test accuracy (dotted) of AdamW and the cubic map ( k=1 ), seed 0, with and without weight decay. (c) Steps to grok versus the largest training loss after memorization, weight decay 1, all methods and seeds.
Symbol
Meaning
η,ηt
learning rate
λ,λmax
curvature (Hessian eigenvalue; for Adam, of P−1/2HP−1/2 ) and its largest value
b=β1
EMA momentum coefficient; κ=(1−b)/(1+b) , aEoS=2/κ
a=ηλ,A=κa
step size of the reduced model; A=2 at the edge
R,Reff
sharpness ratio ηλmax/aEoS ; effective ratio Rgˉ ( Eq. 12 )
μ
value of R at initialization, η=μaEoS/λ0 ( Sections 5.2 and 6.1 )
Appendix
Table 11: Notation.
Study
Protocol
Conventions
μ=ηλ0/aEoS with aEoS=38 ( β1=0.9 ); EMA momentum m←β1m+(1−β1)∇L without bias correction; q=m^/(s^+ϵ) ; per tensor: one scale cB and one multiplier g(rB) per parameter tensor, rB=argument/cB ; ω : decoupled weight decay; ≥ 2-D: weight matrices, convolution kernels, embeddings; none: verified to be absent.
Exact quadratic ( Section 5.2 )
Data: first 2,000 Fashion-MNIST training images (standardized pixels plus a bias column), one-hot targets; no validation or test set. Model, loss: linear least squares, W∈R785×10 (7,850 parameters, one tensor), L=2N1∥XW−Y∥F2 , λmax=300.4 . Optimizer: full-batch EMA momentum, β1=0.9 , ω=0 ; no map, quartic, cubic, relativistic, VR. LR: constant, η=μ⋅38/λmax , 16 values μ∈[0.5,12] . Batch: full. Length: 6,000 steps. Precision: FP64 (NumPy, integrated in the eigenbasis of C ). Scale: c=0.05∥∇L(W0)∥=0.542 (initial gradient norm of the seed-0 start, reused for seeds 1, 2); one global block; argument ∥m∥ . Seeds: 0, 1, 2 (initialization W0∼N(0,0.012) ); 240 runs. Failure: non-finite loss or excess loss >1012 .
Controlled ramps ( Sections 4 and C )
Data: 2,000 Fashion-MNIST training images (fixed random subset, permutation seed 0), one-hot targets; no validation or test set. Model, loss: tanh MLP 784–128–128–10, L=2N1∑i∥f(xi)−eyi∥2 . Optimizer: full-batch EMA momentum, β1=0.9 , ω=0 , no map; reference and perturbed copy take the same ηt . LR: pretraining 1,500 steps at R=0.5 ( η reset from λmax every 50 steps); ramp ηt=(Rin+ρt)38/λ^t , power iteration warm-started from the end of pretraining (200 iterations), then 4 iterations every 5 steps. Grid: kick δ=10−6 , ρ∈{0.003,0.01,0.03,0.1} , Rin∈{0.1,0.3,0.5,0.7,0.9,0.97} ; δ=10−4 at ρ=0.001 , Rin∈{0.5,0.7,0.9,0.97} ; noise only ( Rin=0.5 , σ∈{10−9,10−7,10−5} , five rates ρ=0.001 to 0.1 , exit at ∣xt∣=10−4 ); kick 10−6 plus noise ( ρ=0.003 , Rin≤0.9 ); 174 runs. Batch: full. Length: up to R=2 ( R=4 for ρ≥0.05 ), stopped 10 steps after ∣xt∣ exceeds 100× the exit threshold. Precision: FP64 (CPU). Scale: no map. Seeds: 0, 1, 2 (initialization; noise generator seed 1000+ seed). Failure: none defined. Tracking test ( Appendix C ): seed 2, Rin=0.7 , ρ=0.001 , kick 10−4 or 10−6 , sharpness re-estimated every fifth step or at every step; 3 runs.
Full-batch CIFAR-10: MLP and ResNet-32 (E1; Sections 6 and D )
Data: balanced 1,000-image subset (first 100 per class of the 45k training split; split by permutation seed 0 of the 50k images), normalized with subset statistics; 2,000 images of the 5k validation split for diagnostics only; no test set. Model: tanh MLP 3072–128–128–10 (411,146 parameters); ResNet-32 ( 3×5 basic blocks, widths 16/32/64; 466,906 parameters), BN without running statistics; loss and gradients summed over 4 fixed blocks of 250 images (BN statistics per block). Loss: MLP 2N1∑i∥f(xi)−eyi∥2 ; ResNet mean MSE, 10N1∑i∥f(xi)−eyi∥2 . Optimizer: full-batch EMA momentum, β1=0.9 , ω=0 ; no map, quartic, cubic, relativistic, VR. LR: constant, η=μ⋅38/λ0 , λ0 by Lanczos at initialization for each seed (2 starts; MLP 72–93, ResNet 27–47); MLP μ∈{0.5,1,2,4,8} , ResNet μ∈{0.5,2,4} . Batch: full. Length: 3,000 steps. Precision: FP32, TF32 off, deterministic cuDNN (A100). Scale: per tensor, argument ∥mB∥ ; absolute c=0.05 or relative cB=2∥∇BL(θ0)∥ . Seeds: MLP 0–9, ResNet 0–4 (initialization; data fixed); 450 and 135 runs. Failure: loss non-finite or >103 , non-finite gradient, or parameter norm non-finite or >106 .
Full-batch Adam (E2; Section 6.2 )
Data, model, loss, batch, precision, failure: as for ResNet-32 in E1. Optimizer: Adam, β=(0.9,0.999) , ϵ=10−7 , bias-corrected, ω=0 ; VRAdam ( β3=1 , α1=19 , global ∥m∥2 of the uncorrected first moment, same β,ϵ ); Adam with a per-tensor relativistic map. LR: constant, η∈{0.002,0.01} . Length: 3,000 steps. Scale: map on q : argument rms(qB) , c=0.5 ; map on m^ : argument ∥m^B∥ (Euclidean norm), absolute c=0.05 ; both multiply qB . Seeds: 0, 1, 2; 24 runs.
Fashion-MNIST MLP ( Appendix D )
Data, model, loss: as for the controlled ramps. Optimizer: full-batch EMA momentum, β1=0.9 , ω=0 ; per-tensor none, VR, quartic, relativistic and clipping maps; global VR and quartic maps. LR: constant, η=μ⋅38/λ0 , λ0=33.9 (power iteration at initialization), μ∈{0.5,1,2,4,8} ; scale sweep at μ∈{8,16} . Batch: full. Length: 3,000 steps. Precision: FP32 (CPU). Scale: absolute c=0.05 , argument ∥mB∥ per tensor (or ∥m∥ for the global maps); sweep c∈{0.005,0.015,0.05,0.15} (per tensor; VR, relativistic, quartic, clipping). Seeds: 0 only; 35+32 runs. Failure: loss non-finite or >103 .
The edge-of-stability (EoS) phenomenon of full-batch Adam has been widely observed, yet its underlying dynamical mechanism remains poorly understood. In this paper, we identify Adam's second-moment adaptation as a negative-feedback mechanism that drives the dynamics toward the stability boundary. We characterize this mechanism through the active curvature, namely, the preconditioned curvature along the preconditioned gradient direction, and establish rigorous characterizations in progressively richer settings: rank-one quadratics with momentum, diagonal quadratics, on which the active curvature separates from the sharpness, and general objectives. Importantly, the mechanism predicts gradient reversal of full-batch Adam near the edge: consecutive gradients repeatedly point in nearly opposite directions, as we observe across fully connected networks, ResNets, ViTs, LSTMs, GPT-2 medium, and Adam-family optimizers. Consistent with this picture, averaging iterates suppresses these fast oscillations and produces smoother and lower loss curves. Together, these results provide an important first step towards fully understanding the dynamical behavior of Adam's EoS through active curvature and gradient reversal.
Yiman Fong, Heng Yang
School of Engineering and Applied Sciences, Harvard University.
It has recently been found that deep learning often occurs at the "edge of stability (EoS)," where the maximum Hessian eigenvalue of the model is stabilized at a value reciprocal to the learning rate. However, what happens before we reach that regime? We fix a deep learning problem and vary first order optimization methods with dense learning rate sweeps. We then track the characterizing quantities of a learning trajectory: the loss, the sharpness, and the alignment between consecutive gradients. To our surprise, if we scale the learning rate by the dc gain of the optimizer, these traces from the sweeps from different optimizers almost perfectly overlap across a large range of learning rates. The dc-normalized optimizers have another role that only becomes apparent in high learning rates: they select when the sharpness value detaches from this universal curve and enters the edge of stability. Upon this discovery, we specify three distinct regimes with respect to the dc-adjusted learning rate: the low-LR regime where the trajectory is nearly insensitive to the optimizer, the high-LR regime, where the optimizer governs the sharpness according to the EoS reciprocal rule, and the in-between mid-LR regime where so-called progressive sharpening originates independently of the optimizer. This distinguishes the role of the optimizer, the learning rate, and the model in shaping the learning progress.
Jaerin Lee, Kyoung Mu Lee
Computer Vision Lab, ASRI Seoul National University Seoul 08826, Korea
Many phenomena of deep learning are dynamical: they concern not only which minima exist, but how gradient descent reaches, avoids, or selects among them. Edge-of-stability behavior, sharpness oscillations, catapult phases, balancing, and movement toward flatter representations are effects of the training map itself, and are poorly captured by the small-step gradient-flow limit. This paper studies fixed-step gradient descent as a discrete dynamical system in a hierarchy of exactly solvable models retaining basic structures of deep learning: depth, factorization, width, data coupling, activation, and stochasticity. The starting point is the balanced scalar reduction of a deep linear chain, giving a quartic loss and a cubic gradient map whose post-edge behavior is explicit. Under the natural large-depth scaling, this dynamics converges to a universal Ricker-type map. The edge of stability is therefore not a breakdown of optimization, but the first bifurcation of the training map. Embedding the scalar dynamics back into factored models turns these regimes into learning phenomena. Finite steps break conservation laws of gradient flow and contract factorization imbalance; residual oscillations move parameters toward flatter, more balanced representations. Wider linear networks produce a ladder of spectral edges, so the optimal learning rate can lie beyond the first edge. Data coupling, nonlinear activations, and stochastic targets preserve the same organizing principle: finite-step oscillations drive alignment, balancing, and representation selection. Thus the learning rate is not merely a numerical stability parameter. It is a structural parameter of the training dynamics, determining its attractors and shaping the representations gradient descent selects.