Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training

Yongzhong Xu

cs.LGMar 30, 2026

Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training

Authors: Yongzhong Xu

Abstract

We develop the spectral edge analysis: phase transitions in neural network training -- grokking, capability gains, loss plateaus -- are controlled by the spectral gap of the rolling-window Gram matrix of parameter updates. In the extreme aspect ratio regime (parameters $P \sim 10^8$ , window $W \sim 10$ ), the classical BBP detection threshold is vacuous; the operative structure is the intra-signal gap separating dominant from subdominant modes at position $k^* = \mathrm{argmax}\, σ_j/σ_{j+1}$ . From three assumptions we derive: (i) gap dynamics governed by a Dyson-type ODE with curvature asymmetry, damping, and gradient driving; (ii) a spectral loss decomposition linking each mode's learning contribution to its Davis--Kahan stability coefficient; (iii) the Gap Maximality Principle, showing that $k^*$ is the unique dynamically privileged position -- its collapse is the only one that disrupts learning, and it sustains itself through an $α$ -feedback loop requiring no assumption on the optimizer. The adiabatic parameter $\mathcal{A} = \|ΔG\|_F / (η\, g^2)$ controls circuit stability: $\mathcal{A} \ll 1$ (plateau), $\mathcal{A} \sim 1$ (phase transition), $\mathcal{A} \gg 1$ (forgetting). Tested across six model families (150K--124M parameters): gap dynamics precede every grokking event (24/24 with weight decay, 1/24 without), the gap position is optimizer-dependent (Muon: $k^*=1$ , AdamW: $k^*=2$ on the same model), and 19/20 quantitative predictions are confirmed. The framework is consistent with the edge of stability, Tensor Programs, Dyson Brownian motion, the Lottery Ticket Hypothesis, and neural scaling laws.

Figures & tables

Figure 1: The spectral edge framework. (A) Singular value spectrum of the Gram matrix

\bm{G}

for TinyStories 51M, showing the intra-signal gap at

k^{*}=2

. All eigenvalues are far above the BBP noise detection threshold (dashed; see Proposition 4.1 ). (B) Ratio profile

\sigma_{k}/\sigma_{k+1}

: the maximum at

k=2

defines the signal rank. (C) Gap ratio

\sigma_{2}/\sigma_{3}

over training for TinyStories 51M, showing the three-phase pattern: rise, plateau, collapse. (D) Gap ratio

\sigma_{3}/\sigma_{4}

tracks validation loss for GPT-2 124M. The spectral edge event (“Shift”) coincides with the loss plateau.

Model	$\sigma_{W}/d_{\mathrm{crit}}$ min	median	max
GPT-2 124M (41 windows)	$22.9\times$	$56.3\times$	$63.1\times$
TinyStories 51M (72 windows)	$8.0\times$	$53.3\times$	$62.5\times$

Table 1: Empirical verification of Proposition 4.1 : every singular value exceeds the BBP detection threshold by at least one order of magnitude.

Figure 2: Spectral edge analysis for TinyStories 51M. Top: Consecutive singular value ratios $\sigma_{k}/\sigma_{k+1}$ over training. The $\sigma_{1}/\sigma_{2}$ gap (red) dominates during the plateau phase, while $\sigma_{2}/\sigma_{3}$ (blue) shows the three-phase rise–plateau–collapse pattern. Middle: Eigenvalue gaps $\sigma_{k}^{2}-\sigma_{k+1}^{2}$ over training, showing the same three-phase structure. Bottom: Gap ratio $\sigma_{2}/\sigma_{3}$ (blue) overlaid with validation loss (red), confirming that the spectral edge tracks learning progress.

Model	Mode $k^{*}$	Frequency	Max ratio $R$
GPT-2 124M	2	51.2% (21/41)	$\bar{R}=1.73$ , peak $2.86$
TinyStories 51M	1	77.8% (56/72)	$\bar{R}=2.17$ , peak $5.90$

Table 2: Distribution of

k^{*}

over training windows. The gap position is stable within each training phase.

Model	$\eta$	$1/(\eta W)$	Expected $K$	Observed $k^{*}$
TinyStories 51M	$10^{-3}$	100	2–3	1
GPT-2 124M	$3\times 10^{-5}$	3333	3–4	2

Table 3: Krylov bound consistency. The observed

k^{*}

is bounded by the predicted number of Hessian outliers.

Model	Zero-lag $\|r\|$	Best lag $\|r\|$
GPT-2 124M	0.656	0.656 (lag 0)
TinyStories 51M	0.359	0.669 (lag $-5$ )

Table 4: Cross-correlation between gap ratio

R(t)

and validation loss.

Region	Position	Mean $\alpha_{j}$
Dominant ( $j<k^{*}$ )	$j=1$	0.818
At gap ( $j=k^{},k^{}{+}1$ )	$j=2,3$	0.234
Subdominant ( $j>k^{*}{+}1$ )	$j=4,\ldots,10$	$\approx 0$

Table 5: Stability coefficient hierarchy for GPT-2 124M (

k^{*}=2

). Mean

\alpha_{j}

across 21 windows where

k^{*}=2

Figure 3: Stability coefficient hierarchy for GPT-2 124M. Heatmap of $\alpha_{j}$ (Definition 16.1 ) across eigenvalue index $j$ (vertical) and training step (horizontal). The dominant mode ( $j=1$ , green) has $\alpha\approx 1$ throughout training. The gap mode ( $j=2$ ) fluctuates between stable (green) and unstable (red) as $k^{*}$ shifts. All subdominant modes ( $j\geq 3$ , red) have $\alpha\approx 0$ . The hierarchy $\alpha_{1}>\alpha_{2}>\cdots$ is visually obvious.

Task	Seed	Grok Step	$R_{\mathrm{grok}}$	$R_{\mathrm{terminal}}$	Groks?
Dyck ( $\omega{=}1$ )	42	600	1.13	3187	✓
Dyck ( $\omega{=}1$ )	137	1400	1.19	2841	✓
Dyck ( $\omega{=}1$ )	2024	1000	1.09	1756	✓
SCAN ( $\omega{=}1$ )	42	3000	1.05	12.4	✓
SCAN ( $\omega{=}1$ )	137	4000	1.11	8.7	✓
SCAN ( $\omega{=}1$ )	2024	2500	1.02	6.3	✓

Table 6: Gap ratio at grokking and terminal training for Dyck and SCAN. Weight decay

\omega=1.0

enables grokking;

\omega=0

does not.

Figure 4: Grokking as a spectral edge event (Dyck-1). Grokking runs (weight decay $\omega=1.0$ , red/solid, 3 seeds) vs. control runs ( $\omega=0$ , blue/dashed, 3 seeds). Top: singular value ratio $\sigma_{1}/\sigma_{2}$ of $W_{Q}$ . With weight decay, the ratio rises dramatically as $\sigma_{2}\to 0$ (gap opens); without weight decay, the ratio stays flat (no gap). Middle: Frobenius norm $\|W_{Q}\|_{F}$ . Weight decay compresses the weights; the control runs retain large norms. Bottom: test accuracy. Grokking (accuracy $\to 1$ ) occurs only in the weight-decay runs, coinciding with the gap opening above. All 6 grokking runs show gap opening; all 6 controls show neither gap opening nor grokking.

Task	Seed	Grok	$g_{23}^{\mathrm{early}}$	$g_{23}^{\mathrm{grok}}$	Decline	$k^{*}_{\mathrm{term}}$
Dyck ( $\omega{=}1$ )	42	600	3.88	0.081	$48.1\times$	1
Dyck ( $\omega{=}1$ )	137	1400	11.34	0.240	$47.2\times$	1
Dyck ( $\omega{=}1$ )	2024	1000	10.11	2.230	$4.5\times$	1
SCAN ( $\omega{=}1$ )	42	3000	27.73	0.555	$49.9\times$	1
SCAN ( $\omega{=}1$ )	137	4000	26.59	0.858	$31.0\times$	1
SCAN ( $\omega{=}1$ )	2024	3000	8.21	0.166	$49.5\times$	1

Table 7: Gram-matrix eigenvalue gap decline for Dyck and SCAN grokking. All grokking runs show

g_{23}

compression (

33

–

43\times

mean);

k^{*}=1

universally.

Op	Seed	Grok	$g_{23}^{\mathrm{early}}$	$g_{23}^{\mathrm{grok}}$	Decline	$k^{*}_{\mathrm{term}}$
add	42	3100	1.32	0.18	$7.5\times$	3
add	137	3000	1.25	0.19	$6.6\times$	1
add	2024	2500	1.34	0.23	$5.8\times$	3
sub	42	3200	1.14	0.15	$7.5\times$	1
sub	137	3300	1.15	0.15	$7.5\times$	1
sub	2024	3600	1.17	0.14	$8.4\times$	1

Table 8: Eigenvalue sub-leading gap decline for modular arithmetic (4 operations, 3 seeds each). The gap

g_{23}

decreases monotonically from early training to grokking, with

k^{*}

shifting to 1 at terminal training in 10/12 runs.

Op	Seed	Grok	$g_{23}^{\mathrm{early}}$	$g_{23}^{\mathrm{grok}}$	Decline	$k^{*}_{\mathrm{term}}$
add	42	3100	14.70	0.63	$23.3\times$	1
add	137	3000	14.74	0.93	$15.8\times$	1
add	2024	2500	14.43	0.13	$110.1\times$	8
sub	42	3300	14.39	0.52	$27.7\times$	1
sub	137	3300	14.55	0.26	$55.4\times$	1
sub	2024	3700	14.34	0.67	$21.5\times$	1

Table 9: Gram-matrix sub-leading gap decline for modular arithmetic (2-layer transformer,

d{=}128

). The gap

g_{23}=\sigma_{2}^{2}-\sigma_{3}^{2}

of the rolling-window Gram matrix declines in 12/12 grokking runs (mean

40.2\times

) and 1/12 controls (the anomalous late-grokking

x^{2}{+}y^{2}

run). Weighted

k^{*}

reaches 1 in 9/12 runs.

Prediction	Formula	GPT-2 124M	TinyStories 51M	Dyck/SCAN	Status
BBP vacuous ( 4.1 )	$\sigma_{W}/d_{\mathrm{crit}}\gg 1$	$23$ – $63\times$	$8$ – $63\times$	—	✓
$k^{*}$ = argmax ( 6.1 )	$\argmax_{j}\sigma_{j}/\sigma_{j+1}$	mode $k^{*}=3$	mode $k^{*}=2$	mode $k^{*}=1$	✓
Krylov bound ( 5.5 )	$k^{*}\leq K$	$3\leq 3$ – $4$	$2\leq 2$ – $3$	—	✓
Gap–loss corr ( 17.1 )	$\|r(R,L_{\mathrm{val}})\|>0$	$\|r\|=0.66$	$\|r\|=0.67$	—	✓
Stability ( 16.3 )	$\alpha_{\mathrm{dom}}>\alpha_{\mathrm{gap}}$	$0.82>0.23$	(N/A, $k^{*}{=}1$ )	—	✓
Gap flow ( 14.2 )	$\rho(g,\dot{g})<0$	$r=+0.08$	—	—	$\sim$

Table 10: Theory–experiment match across all predictions and models.

Figure 5: Dependency graph of the spectral edge framework. Green = no dependence on

[\mathcal{P},H]\approx 0

. Yellow (dashed) = qualitative conclusion survives; only exact coefficients need commutativity. Orange (thick dashed) = both structure and conclusion require

[\mathcal{P},H]\approx 0

. The core diagnostic and learning-theoretic results (left column and NTK branch) are entirely clean. The flow equations are weakly dependent. Only the Krylov bound, collapse/opening times, and

\beta_{2}

theorem have strong dependence.

Phase Transitions in Neural Networks

See all 25 papers

Spectral phase transitions and trainability in neural network learning dynamics
Jun 26, 2026Chanju Park, Dario Bocchi, Francesco D'Amico +2Random Matrix Theory Neural Network Training Dynamics
Centre for Quantum Fields and Gravity, Department of Physics, Swansea University, Swansea SA2 8PP, United Kingdom · Physics Department, University of Rome Sapienza, Piazzale Aldo Moro 5, 00185 Rome, Italy · Institute of Nanotechnology, CNR-NANOTEC, Piazzale Aldo Moro 5, 00185 Rome, Italy +1
Non-normal spectral signatures of instability in neural network training dynamics
May 22, 2026Souvik GhoshDynamical Systems Neural Network Optimization
Department of Physics, National Sun Yat-sen University, Kaohsiung 80424, Taiwan
Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales
Mar 14, 2026Yongzhong XuTransformer Training Dynamics Neural Network Training Dynamics

Jun 26, 2026J/K move · Enter open · S save

Spectral phase transitions and trainability in neural network learning dynamics

Chanju Park, Dario Bocchi, Francesco D'Amico, Biagio Lucini+1

Centre for Quantum Fields and Gravity, Department of Physics, Swansea University, Swansea SA2 8PP, United Kingdom · Physics Department, University of Rome Sapienza, Piazzale Aldo Moro 5, 00185 Rome, Italy · Institute of Nanotechnology, CNR-NANOTEC, Piazzale Aldo Moro 5, 00185 Rome, Italy+1

Random Matrix Theory Neural Network Training Dynamics Stochastic Gradient Descent Phase Transitions in Neural Networks

Open paper AskPDF arXiv

The emergence of low-dimensional structures in the spectra of neural network weight matrices is a common empirical feature of trained models, but the dynamical origin of this phenomenon during learning remains an open problem. We formulate neural network training as the stochastic evolution of an initially random matrix ensemble, driven by stochastic gradient descent (SGD) updates that reshape the spectral bulk while amplifying signal strength. This induces a Baik-Ben Arous-Péché (BBP) transition during training, where isolated eigenvalues detach from the random bulk distribution, providing a dynamical framework for representation formation in high-dimensional learning dynamics. We demonstrate this in a solvable linear teacher-student model, where spectral evolution is analytically tractable and a phase diagram of trainability governed by the step size (or learning rate) and initial weight variance is obtained, and subsequently extend our formalism beyond the linear regime to nonlinear and stochastic settings. Numerical simulations in realistic settings support this picture, showing robust emergence of spectral alignment during training. Our results suggest that spectral analysis may provide a unified perspective of stochastic learning dynamics, linking trainability, optimisation hyperparameters, spectral phase transitions, and representation learning in neural networks.

Abstract

Phase Transitions in Neural Networks

Explore similar work

Spectral phase transitions and trainability in neural network learning dynamics

Non-normal spectral signatures of instability in neural network training dynamics

Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales