Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training

Yongzhong Xu

cs.LGMar 30, 2026

Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training

Authors: Yongzhong Xu

Abstract

We develop the spectral edge analysis: phase transitions in neural network training -- grokking, capability gains, loss plateaus -- are controlled by the spectral gap of the rolling-window Gram matrix of parameter updates. In the extreme aspect ratio regime (parameters $P \sim 10^8$ , window $W \sim 10$ ), the classical BBP detection threshold is vacuous; the operative structure is the intra-signal gap separating dominant from subdominant modes at position $k^* = \mathrm{argmax}\, σ_j/σ_{j+1}$ . From three assumptions we derive: (i) gap dynamics governed by a Dyson-type ODE with curvature asymmetry, damping, and gradient driving; (ii) a spectral loss decomposition linking each mode's learning contribution to its Davis--Kahan stability coefficient; (iii) the Gap Maximality Principle, showing that $k^*$ is the unique dynamically privileged position -- its collapse is the only one that disrupts learning, and it sustains itself through an $α$ -feedback loop requiring no assumption on the optimizer. The adiabatic parameter $\mathcal{A} = \|ΔG\|_F / (η\, g^2)$ controls circuit stability: $\mathcal{A} \ll 1$ (plateau), $\mathcal{A} \sim 1$ (phase transition), $\mathcal{A} \gg 1$ (forgetting). Tested across six model families (150K--124M parameters): gap dynamics precede every grokking event (24/24 with weight decay, 1/24 without), the gap position is optimizer-dependent (Muon: $k^*=1$ , AdamW: $k^*=2$ on the same model), and 19/20 quantitative predictions are confirmed. The framework is consistent with the edge of stability, Tensor Programs, Dyson Brownian motion, the Lottery Ticket Hypothesis, and neural scaling laws.

Figures & tables

Figure 1: The spectral edge framework. (A) Singular value spectrum of the Gram matrix

\bm{G}

for TinyStories 51M, showing the intra-signal gap at

k^{*}=2

. All eigenvalues are far above the BBP noise detection threshold (dashed; see Proposition 4.1 ). (B) Ratio profile

\sigma_{k}/\sigma_{k+1}

: the maximum at

k=2

defines the signal rank. (C) Gap ratio

\sigma_{2}/\sigma_{3}

over training for TinyStories 51M, showing the three-phase pattern: rise, plateau, collapse. (D) Gap ratio

\sigma_{3}/\sigma_{4}

tracks validation loss for GPT-2 124M. The spectral edge event (“Shift”) coincides with the loss plateau.

Model	$\sigma_{W}/d_{\mathrm{crit}}$ min	median	max
GPT-2 124M (41 windows)	$22.9\times$	$56.3\times$	$63.1\times$
TinyStories 51M (72 windows)	$8.0\times$	$53.3\times$	$62.5\times$

Table 1: Empirical verification of Proposition 4.1 : every singular value exceeds the BBP detection threshold by at least one order of magnitude.

Figure 2: Spectral edge analysis for TinyStories 51M. Top: Consecutive singular value ratios $\sigma_{k}/\sigma_{k+1}$ over training. The $\sigma_{1}/\sigma_{2}$ gap (red) dominates during the plateau phase, while $\sigma_{2}/\sigma_{3}$ (blue) shows the three-phase rise–plateau–collapse pattern. Middle: Eigenvalue gaps $\sigma_{k}^{2}-\sigma_{k+1}^{2}$ over training, showing the same three-phase structure. Bottom: Gap ratio $\sigma_{2}/\sigma_{3}$ (blue) overlaid with validation loss (red), confirming that the spectral edge tracks learning progress.

Model	Mode $k^{*}$	Frequency	Max ratio $R$
GPT-2 124M	2	51.2% (21/41)	$\bar{R}=1.73$ , peak $2.86$
TinyStories 51M	1	77.8% (56/72)	$\bar{R}=2.17$ , peak $5.90$

Table 2: Distribution of

k^{*}

over training windows. The gap position is stable within each training phase.

Model	$\eta$	$1/(\eta W)$	Expected $K$	Observed $k^{*}$
TinyStories 51M	$10^{-3}$	100	2–3	1
GPT-2 124M	$3\times 10^{-5}$	3333	3–4	2

Table 3: Krylov bound consistency. The observed

k^{*}

is bounded by the predicted number of Hessian outliers.

Model	Zero-lag $\|r\|$	Best lag $\|r\|$
GPT-2 124M	0.656	0.656 (lag 0)
TinyStories 51M	0.359	0.669 (lag $-5$ )

Table 4: Cross-correlation between gap ratio

R(t)

and validation loss.

Region	Position	Mean $\alpha_{j}$
Dominant ( $j<k^{*}$ )	$j=1$	0.818
At gap ( $j=k^{},k^{}{+}1$ )	$j=2,3$	0.234
Subdominant ( $j>k^{*}{+}1$ )	$j=4,\ldots,10$	$\approx 0$

Table 5: Stability coefficient hierarchy for GPT-2 124M (

k^{*}=2

). Mean

\alpha_{j}

across 21 windows where

k^{*}=2

Figure 3: Stability coefficient hierarchy for GPT-2 124M. Heatmap of $\alpha_{j}$ (Definition 16.1 ) across eigenvalue index $j$ (vertical) and training step (horizontal). The dominant mode ( $j=1$ , green) has $\alpha\approx 1$ throughout training. The gap mode ( $j=2$ ) fluctuates between stable (green) and unstable (red) as $k^{*}$ shifts. All subdominant modes ( $j\geq 3$ , red) have $\alpha\approx 0$ . The hierarchy $\alpha_{1}>\alpha_{2}>\cdots$ is visually obvious.

Task	Seed	Grok Step	$R_{\mathrm{grok}}$	$R_{\mathrm{terminal}}$	Groks?
Dyck ( $\omega{=}1$ )	42	600	1.13	3187	✓
Dyck ( $\omega{=}1$ )	137	1400	1.19	2841	✓
Dyck ( $\omega{=}1$ )	2024	1000	1.09	1756	✓
SCAN ( $\omega{=}1$ )	42	3000	1.05	12.4	✓
SCAN ( $\omega{=}1$ )	137	4000	1.11	8.7	✓
SCAN ( $\omega{=}1$ )	2024	2500	1.02	6.3	✓

Table 6: Gap ratio at grokking and terminal training for Dyck and SCAN. Weight decay

\omega=1.0

enables grokking;

\omega=0

does not.

Figure 4: Grokking as a spectral edge event (Dyck-1). Grokking runs (weight decay $\omega=1.0$ , red/solid, 3 seeds) vs. control runs ( $\omega=0$ , blue/dashed, 3 seeds). Top: singular value ratio $\sigma_{1}/\sigma_{2}$ of $W_{Q}$ . With weight decay, the ratio rises dramatically as $\sigma_{2}\to 0$ (gap opens); without weight decay, the ratio stays flat (no gap). Middle: Frobenius norm $\|W_{Q}\|_{F}$ . Weight decay compresses the weights; the control runs retain large norms. Bottom: test accuracy. Grokking (accuracy $\to 1$ ) occurs only in the weight-decay runs, coinciding with the gap opening above. All 6 grokking runs show gap opening; all 6 controls show neither gap opening nor grokking.

Task	Seed	Grok	$g_{23}^{\mathrm{early}}$	$g_{23}^{\mathrm{grok}}$	Decline	$k^{*}_{\mathrm{term}}$
Dyck ( $\omega{=}1$ )	42	600	3.88	0.081	$48.1\times$	1
Dyck ( $\omega{=}1$ )	137	1400	11.34	0.240	$47.2\times$	1
Dyck ( $\omega{=}1$ )	2024	1000	10.11	2.230	$4.5\times$	1
SCAN ( $\omega{=}1$ )	42	3000	27.73	0.555	$49.9\times$	1
SCAN ( $\omega{=}1$ )	137	4000	26.59	0.858	$31.0\times$	1
SCAN ( $\omega{=}1$ )	2024	3000	8.21	0.166	$49.5\times$	1

Table 7: Gram-matrix eigenvalue gap decline for Dyck and SCAN grokking. All grokking runs show

g_{23}

compression (

33

–

43\times

mean);

k^{*}=1

universally.

Op	Seed	Grok	$g_{23}^{\mathrm{early}}$	$g_{23}^{\mathrm{grok}}$	Decline	$k^{*}_{\mathrm{term}}$
add	42	3100	1.32	0.18	$7.5\times$	3
add	137	3000	1.25	0.19	$6.6\times$	1
add	2024	2500	1.34	0.23	$5.8\times$	3
sub	42	3200	1.14	0.15	$7.5\times$	1
sub	137	3300	1.15	0.15	$7.5\times$	1
sub	2024	3600	1.17	0.14	$8.4\times$	1

Table 8: Eigenvalue sub-leading gap decline for modular arithmetic (4 operations, 3 seeds each). The gap

g_{23}

decreases monotonically from early training to grokking, with

k^{*}

shifting to 1 at terminal training in 10/12 runs.

Op	Seed	Grok	$g_{23}^{\mathrm{early}}$	$g_{23}^{\mathrm{grok}}$	Decline	$k^{*}_{\mathrm{term}}$
add	42	3100	14.70	0.63	$23.3\times$	1
add	137	3000	14.74	0.93	$15.8\times$	1
add	2024	2500	14.43	0.13	$110.1\times$	8
sub	42	3300	14.39	0.52	$27.7\times$	1
sub	137	3300	14.55	0.26	$55.4\times$	1
sub	2024	3700	14.34	0.67	$21.5\times$	1

Table 9: Gram-matrix sub-leading gap decline for modular arithmetic (2-layer transformer,

d{=}128

). The gap

g_{23}=\sigma_{2}^{2}-\sigma_{3}^{2}

of the rolling-window Gram matrix declines in 12/12 grokking runs (mean

40.2\times

) and 1/12 controls (the anomalous late-grokking

x^{2}{+}y^{2}

run). Weighted

k^{*}

reaches 1 in 9/12 runs.

Prediction	Formula	GPT-2 124M	TinyStories 51M	Dyck/SCAN	Status
BBP vacuous ( 4.1 )	$\sigma_{W}/d_{\mathrm{crit}}\gg 1$	$23$ – $63\times$	$8$ – $63\times$	—	✓
$k^{*}$ = argmax ( 6.1 )	$\argmax_{j}\sigma_{j}/\sigma_{j+1}$	mode $k^{*}=3$	mode $k^{*}=2$	mode $k^{*}=1$	✓
Krylov bound ( 5.5 )	$k^{*}\leq K$	$3\leq 3$ – $4$	$2\leq 2$ – $3$	—	✓
Gap–loss corr ( 17.1 )	$\|r(R,L_{\mathrm{val}})\|>0$	$\|r\|=0.66$	$\|r\|=0.67$	—	✓
Stability ( 16.3 )	$\alpha_{\mathrm{dom}}>\alpha_{\mathrm{gap}}$	$0.82>0.23$	(N/A, $k^{*}{=}1$ )	—	✓
Gap flow ( 14.2 )	$\rho(g,\dot{g})<0$	$r=+0.08$	—	—	$\sim$

Table 10: Theory–experiment match across all predictions and models.

Figure 5: Dependency graph of the spectral edge framework. Green = no dependence on

[\mathcal{P},H]\approx 0

. Yellow (dashed) = qualitative conclusion survives; only exact coefficients need commutativity. Orange (thick dashed) = both structure and conclusion require

[\mathcal{P},H]\approx 0

. The core diagnostic and learning-theoretic results (left column and NTK branch) are entirely clean. The flow equations are weakly dependent. Only the Krylov bound, collapse/opening times, and

\beta_{2}

theorem have strong dependence.

Phase Transitions in Neural Networks

See all 25 papers

Jun 26, 2026

Spectral phase transitions and trainability in neural network learning dynamics

The emergence of low-dimensional structures in the spectra of neural network weight matrices is a common empirical feature of trained models, but the dynamical origin of this phenomenon during learning remains an open problem. We formulate neural network training as the stochastic evolution of an initially random matrix ensemble, driven by stochastic gradient descent (SGD) updates that reshape the spectral bulk while amplifying signal strength. This induces a Baik-Ben Arous-Péché (BBP) transition during training, where isolated eigenvalues detach from the random bulk distribution, providing a dynamical framework for representation formation in high-dimensional learning dynamics. We demonstrate this in a solvable linear teacher-student model, where spectral evolution is analytically tractable and a phase diagram of trainability governed by the step size (or learning rate) and initial weight variance is obtained, and subsequently extend our formalism beyond the linear regime to nonlinear and stochastic settings. Numerical simulations in realistic settings support this picture, showing robust emergence of spectral alignment during training. Our results suggest that spectral analysis may provide a unified perspective of stochastic learning dynamics, linking trainability, optimisation hyperparameters, spectral phase transitions, and representation learning in neural networks.

Chanju Park, Dario Bocchi, Francesco D'Amico +2

Centre for Quantum Fields and Gravity, Department of Physics, Swansea University, Swansea SA2 8PP, United Kingdom · Physics Department, University of Rome Sapienza, Piazzale Aldo Moro 5, 00185 Rome, Italy · Institute of Nanotechnology, CNR-NANOTEC, Piazzale Aldo Moro 5, 00185 Rome, Italy +1

Random Matrix Theory Neural Network Training Dynamics Stochastic Gradient Descent Phase Transitions in Neural Networks

Ask PDF

May 22, 2026

Non-normal spectral signatures of instability in neural network training dynamics

Training instabilities in deep networks - loss spikes, oscillatory convergence, and gradient pathologies - are empirically prevalent but lack a rigorous operator-theoretic explanation. We show that the linearized update operators for practically used optimizers are generically non-normal: for Adam, non-normality is controlled by the commutator [H, M] between the Hessian and the diagonal adaptive preconditioner, while for SGD with momentum it arises from the augmented state-space structure of the update map. Applying non-normal stability theory to these operators, we derive a conservative pseudospectral precursor bound in which κ(V) serves as an early-warning indicator of transient amplification even when the spectral radius remains below one, and we establish that exceptional points of the update operator appear as the κ(V) -> \infty limiting case of this framework. Numerical experiments on two-layer networks confirm that the spectral radius ρ(J) provides no separation between stable and unstable training phases while κ(V) separates them by approximately one order of magnitude, complementing the classical sharpness criterion with a continuous severity measure of non-normal amplification. These results establish non-Hermitian operator theory as a useful and underexplored framework for neural network optimization stability, offering a diagnostic language and proof-of-concept benchmark for understanding adaptive optimization stability.

Souvik Ghosh

Department of Physics, National Sun Yat-sen University, Kaohsiung 80424, Taiwan

Dynamical Systems Neural Network Optimization Adaptive Gradient Methods Neural Network Training Dynamics Neural Network Stability

Ask PDF

Mar 14, 2026

Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales

Despite hundreds of millions of parameters, transformer training trajectories evolve within only a few coherent directions. We introduce Spectral Edge Dynamics (SED) to quantify this structure: a rolling-window SVD of parameter updates reveals a sharp boundary -- the spectral edge -- between coherent optimization directions and stochastic noise, identified via the maximum consecutive singular value ratio

σ_k / σ_{k+1}

. Across a 51M-parameter TinyStories model (4 seeds) and GPT-2 124M under distribution shift, the spectral edge exhibits a universal three-phase pattern (rise, plateau, collapse). The effective signal rank adapts to task complexity (

k^* = 2

at 51M,

k^* = 3

at 124M), and the directional coupling between spectral geometry and validation loss reverses with window size -- a lag flip reflecting the timescale of trajectory integration. Johnson--Lindenstrauss projection to

d = 10W

dimensions (e.g.,

d = 100

for

W = 10

) preserves the spectral gap within

5.7\%

, making the framework applicable to models of arbitrary scale. In companion work, the same spectral geometry provides early-warning signals of grokking -- predicting generalization

600

1{,}700

steps before it occurs across modular arithmetic, Dyck languages, and the SCAN benchmark.

Yongzhong Xu

Transformer Training Dynamics Neural Network Training Dynamics Gradient Descent Dynamics Spectral Methods

Ask PDF

Abstract

Phase Transitions in Neural Networks

Explore similar work

Spectral phase transitions and trainability in neural network learning dynamics

Non-normal spectral signatures of instability in neural network training dynamics

Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales