Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin

Truong Xuan Khanh; Truong Quynh Hoa; Luu Duc Trung; Phan Thanh Duc

cs.LGApr 13, 2026

Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin

Authors: Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung, Phan Thanh Duc

Organizations: H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam

Abstract

Grokking - the delayed transition from memorisation to generalisation in neural networks - remains poorly understood. We study this phenomenon through the geometry of learned representations and identify a consistent empirical signature preceding generalisation: collapse of the spectral entropy of the representation covariance matrix. Across modular arithmetic tasks and multiple random seeds, spectral entropy decreases gradually during training and crosses a stable task-specific threshold before test accuracy rises. A representation-mixing intervention that delays this collapse also delays grokking, including under norm-matched controls, indicating that the effect is not explained by parameter norm alone. We further show that the entropy gap predicts the remaining time until grokking with useful out-of-sample accuracy. To probe the structure underlying this transition, we introduce a Fourier-alignment observable for cyclic-group tasks. Entropy collapse is strongly coupled to the emergence of Fourier-aligned representations, suggesting that spectral entropy tracks concentration of the representation into task-structured directions rather than generic compression alone. The same qualitative dynamics appear in non-abelian group composition tasks, while MLP controls show that entropy collapse by itself is insufficient for grokking in the absence of appropriate inductive bias. Taken together, the results support a view of grokking as a representational phase transition with an observable geometric signature. We discuss the scope and limitations of this interpretation, connections to recent feature-learning and spectral-dynamics work, and directions for testing whether similar transitions appear in larger-scale learning systems.

Figures & tables

Figure 1: Entropy collapse precedes grokking. Mean $\pm$ $1.96\,\mathrm{SE}$ over $10$ seeds. (A) Accuracy curves showing the classic grokking delay. (B) Normalised spectral entropy $\tilde{H}(t)$ decreasing monotonically, crossing the critical threshold $\tilde{H}^{*}=0.609$ (dashed line) before test accuracy rises. (C) Parameter norm $\|\theta\|_{2}$ increasing then plateauing. Vertical dashed line marks mean $T_{\mathrm{grok}}=14{,}360$ steps.

Figure 2: Stability of the critical threshold $\tilde{H}^{*}$ . Histogram of $\tilde{H}(T_{\mathrm{grok}})$ across $10$ seeds. Mean $\tilde{H}^{*}=0.609$ (95% CI: $[0.595,0.624]$ ), demonstrating low variance across initialisations.

Figure 3: Training dynamics decompose into three qualitative phases. All $1{,}556$ evaluation points pooled across $10$ baseline seeds. (A) $\tilde{H}(t)$ vs. training step, coloured by phase: Phase I (memorising, train acc $<0.95$ , $5.9\%$ of points); Phase II (entropy collapse, train acc $\geq 0.95$ and $t<T_{\mathrm{grok}}$ , $86.4\%$ of points); Phase III (generalised, $t\geq T_{\mathrm{grok}}$ , $7.6\%$ of points). The dashed horizontal line marks $\tilde{H}^{*}=0.609$ . (B) Phase-plane $(\tilde{H},d\tilde{H}/dt)$ showing the same colour coding: in Phase II the system moves leftward and slightly downward, consistent with $\tilde{H}$ decreasing monotonically; in Phase III the system continues slowly leftward (see Section 13.7 ).

Figure 4: Interventional probe delays grokking. Mean $\pm$ $1.96\,\mathrm{SE}$ over $10$ seeds. (A) Test accuracy: intervention (orange) and norm-controlled (green) conditions generalise later than baseline (blue). (B) Entropy: mixing prevents $\tilde{H}$ from collapsing below $\tilde{H}^{*}$ . (C) Norm: norm-controlled condition matches baseline, confirming that the delay is attributable to entropy, not norm. $\Delta T_{\mathrm{grok}}=+5{,}020$ steps ( $p=0.044$ , $d=0.70$ ).

Condition	Grokked	$\bar{T}_{\mathrm{grok}}$	$\Delta T$	$p$ -value
Baseline	$10/10$	$14{,}360$	—	—
Intervention	$10/10$	$19{,}420$	$+5{,}020$	$0.044$
Norm-control ( $n=30$ )	$28/30$	$22{,}664$	$+8{,}304$	$5\times 10^{-5}$

Table 1: Interventional probe results. Baseline and intervention:

10

seeds each. Norm-control:

28/30

seeds grokked (

n=30

total),

p=97

, train fraction

0.20

Predictor $x$	$R^{2}$	$\|\rho(x,\Delta T)\|$	$\gamma$
$\tilde{H}$ -gap $\|\tilde{H}-\tilde{H}^{*}\|$ (ours)	$\mathbf{0.507}$	$\mathbf{0.735}$	$1.73$
Absolute entropy $\tilde{H}$	$0.379$	$0.733$	$15.0$
Parameter norm $\\|\theta\\|_{t}$	$0.074$	$0.433$	$-1.53$
Inverse norm $1/\\|\theta\\|_{t}$	$0.074$	$0.452$	$1.53$
Train-loss proxy $(1-\mathrm{train\;acc})$	$-0.074$	$0.162$	$0.06$
Entropy velocity $\|d\tilde{H}/dt\|$	$-0.119$	$0.041$	$-0.20$

Table 2: Comprehensive predictor comparison. Power-law fit

\Delta T=C_{1}x^{\gamma}

for seven candidate predictors, across

10

seeds and

1{,}436

pre-grokking evaluation points. The

\tilde{H}

-gap outperforms all alternatives by at least a factor of

6.8\times

R^{2}

, confirming that spectral entropy carries predictive information about grokking proximity that no norm-, loss-, or velocity-based predictor captures. Negative

R^{2}

values indicate predictions worse than the sample mean.

Panel of Figure 5 — Figure 5: Left: Each point is one evaluation step from one seed, coloured by test accuracy. Pearson $\rho(\|\theta\|_{2},\tilde{H})=-0.248$ (95% CI: $[-0.049,0.050]$ ; $p=4\times 10^{-23}$ ), confirming that norm and entropy carry independent information. The green cluster (high test accuracy) occurs at high norm but low entropy, while the red cluster (low test accuracy) spans all norm values. Right: Power-law fit $\Delta T=C_{1}(\tilde{H}-\tilde{H}^{*})^{\gamma}$ to $1{,}428$ data points pooled across $10$ seeds. $R^{2}=0.543$ (95% CI: $[0.513,0.573]$ ), $\gamma=1.65$ .

Figure 6: Leave-one-out predictive validation of the power-law $\Delta T=C_{1}(\tilde{H}-\tilde{H}^{*})^{\gamma}$ . (A) Absolute percentage error in $\hat{T}_{\mathrm{grok}}$ (log scale) vs. training steps remaining until grokking, for each of $10$ LOO held-out seeds (light blue) with median across seeds (dark red). Prediction accuracy improves monotonically as the system approaches the transition $(\tilde{H}-\tilde{H}^{*}\to 0^{+})$ , crossing the $20\%$ error threshold at roughly $15{,}000$ steps before $T_{\mathrm{grok}}$ on average. (B) Distribution of final-step APE (at $t=T_{\mathrm{grok}}-1$ ) across $10$ LOO-held-out seeds: median $1.69\%$ , mean $3.21\%$ . All seeds achieve final-step APE $<20\%$ .

Figure 7: Entropy collapse is consistent across modular arithmetic tasks. Each panel shows mean $\pm 1.96\,\mathrm{SE}$ test accuracy (orange, left axis) and normalised spectral entropy $\tilde{H}$ (blue dashed, right axis) for $5$ seeds. The critical threshold $\tilde{H}^{*}$ (dotted red) is remarkably consistent: $0.605$ (add), $0.589$ (mul), $0.589$ (sub), a range of $2.7\%$ . Vertical dashed line marks mean $T_{\mathrm{grok}}$ .

Figure 8: Entropy collapse in $S_{5}$ permutation composition (non-abelian group). (A) Mean $\pm 1.96\,\mathrm{SE}$ trajectories: test accuracy (orange) and $\tilde{H}$ (blue dashed) for $10$ seeds on the $S_{5}$ composition task. All $10$ seeds grokked; $\tilde{H}$ approaches $\tilde{H}^{*}=0.655$ just before generalisation, with moderate oscillations due to the small training set ( $n_{\mathrm{train}}=1{,}440$ , $12$ per class). (B) $\tilde{H}^{*}$ across group structures: modular arithmetic ( $\mathbb{Z}/p\mathbb{Z}$ , abelian, blue) versus $S_{5}$ composition (non-abelian, orange). The mechanism is qualitatively the same; the threshold shifts upward ( $\tilde{H}^{*}=0.655$ vs. $0.594$ for modular), consistent with the higher output complexity of $S_{5}$ ( $120$ classes vs. $97$ ).

Task	Group	Grokked	$\bar{T}_{\mathrm{grok}}$	$\tilde{H}^{*}$
$(a+b)\bmod 97$	$\mathbb{Z}/97\mathbb{Z}$ (abelian)	$5/5$	$14{,}040$	$0.605$
$(a\times b)\bmod 97$	$\mathbb{Z}/97\mathbb{Z}$ (abelian)	$5/5$	$21{,}560$	$0.589$
$(a-b)\bmod 97$	$\mathbb{Z}/97\mathbb{Z}$ (abelian)	$5/5$	$21{,}560$	$0.589$
$\sigma_{1}\circ\sigma_{2}$	$S_{5}$ (non-abelian)	$10/10$	$4{,}380$	$0.655$

Table 3: Universality results across group structures.

\mathbb{Z}/p\mathbb{Z}

tasks:

p=97

5

seeds.

S_{5}

10

seeds, train_frac

=0.10

. All use

1

-layer Transformer, AdamW,

\lambda=1.0

Figure 9: Three-phase decomposition of grokking dynamics, averaged across $5$ seeds. (A) Full trajectory: $\tilde{H}$ (blue), $A_{\mathrm{Fourier}}$ (orange) and test accuracy (red) vs. training step, with $95\%$ CI bands across seeds. Mean grokking step $T_{\mathrm{grok}}=14{,}260$ (green dashed). (B) Pre-grok window: $A_{\mathrm{Fourier}}$ rises monotonically while $\tilde{H}$ decreases. The two quantities are not independent; their Pearson anti-correlation is consistent across seeds ( $\bar{\rho}=-0.82$ ; range $[-0.86,-0.76]$ ; combined Fisher $p<10^{-168}$ ).

Seed	$T_{\mathrm{grok}}$	$\tilde{H}^{*}$	$A_{\mathrm{Fourier}}(T_{\mathrm{grok}})$	$\rho_{\mathrm{pre}}(\tilde{H},A_{\mathrm{Fourier}})$	$A_{\mathrm{post}}$ mean	$A_{\mathrm{post}}$ std
0	$12{,}300$	$0.607$	$0.752$	$-0.855$ ( $n{=}123$ )	$0.640$	$0.039$
1	$11{,}300$	$0.617$	$0.576$	$-0.826$ ( $n{=}113$ )	$0.581$	$0.033$
2	$12{,}000$	$0.555$	$0.641$	$-0.761$ ( $n{=}120$ )	$0.681$	$0.043$
3	$9{,}200$	$0.654$	$0.544$	$-0.806$ ( $n{=}92$ )	$0.614$	$0.045$
4	$26{,}500$	$0.621$	$0.582$	$-0.833$ ( $n{=}265$ )	$0.603$	$0.037$
Mean	$14{,}260$	$0.611$	$0.619$	$-0.816$	$0.624$	$0.039$

Table 4: Per-seed summary, paper-exact GrokTransformer on

(a+b)\bmod 97

with

f_{\mathrm{train}}=0.20

n=5

seeds.

\rho_{\mathrm{pre}}

is the Pearson correlation between

\tilde{H}

and

A_{\mathrm{Fourier}}

across pre-grokking evaluation points (

n_{\mathrm{pre}}

points per seed).

A_{\mathrm{post}}

statistics are computed over steps

\geq T_{\mathrm{grok}}+2000

Figure 10: Phase portrait in the $(\tilde{H},A_{\mathrm{Fourier}})$ plane across $5$ seeds ( $\bigstar$ $=$ grok step). All seeds trace a structurally identical monotone manifold from $(\tilde{H},A_{\mathrm{Fourier}})\approx(0.80,0.11)$ at initialisation to $(\approx 0.55,\approx 0.65)$ post-grokking, despite a $2.9\times$ range in the number of steps each seed takes to traverse it. The manifold, not wall-clock time, is the invariant of the dynamics.

Figure 11: Entropy-based grokking forecaster (representative seed). (A) Training dynamics: the purple region marks the interval between the first accurate prediction ( $t_{\mathrm{warn}}$ ) and actual grokking, during which the practitioner has advance notice. (B) Online prediction $\hat{T}_{\mathrm{grok}}(t)$ converging to the true value (dashed) within the $\pm 20\%$ band. (C) Prediction error over time, crossing the $20\%$ threshold early.

Figure 12: Prediction accuracy across all $10$ seeds. (D) Final prediction error per seed (sorted); all below $20\%$ . (E) Distribution of advance warning lead times (mean $12{,}370$ steps, min $8{,}800$ steps).

Figure 13: Entropy collapse without grokking in an MLP architecture. Top row (MLP, $p=41$ ): (A) Train accuracy reaches $1.0$ rapidly; test accuracy remains near zero for $80{,}000$ steps despite entropy collapse. (B) $\tilde{H}$ drops well below $\tilde{H}^{*}=0.609$ (red region) — entropy collapses but grokking does not follow. (C) Fourier alignment (max Pearson $|r|$ between top eigenvectors and task-relevant Fourier features) remains near zero throughout (peak $=0.052$ ). Bottom row ( $1$ -layer Transformer): (D) Grokking occurs at step $1{,}600$ after a memorisation plateau. (E) $\tilde{H}$ crosses $\tilde{H}^{*}$ and generalisation follows. Entropy collapse is present in both architectures, but grokking only occurs in the Transformer; the discriminating factor is likely the inductive bias of attention for learning task-structured representations, though we leave a causal characterisation of this difference for future work.

Figure 14: Post-grokking entropy dynamics are consistent with continued refinement, not collapse reversal. (A) Each point is one seed: $\tilde{H}$ at grokking onset ( $\tilde{H}(T_{\mathrm{grok}})$ , $x$ -axis) vs. at end of training ( $\tilde{H}(\mathrm{final})$ , $y$ -axis); all points lie below the identity line, confirming continued decrease. (B) Distribution of post-grok entropy trend slopes $d\tilde{H}/d\mathrm{step}$ : all $10$ seeds are negative (decreasing), with mean $-3.60\times 10^{-5}$ . This is evidence that the representation continues to refine after generalisation, consistent with continued low-rank concentration, and inconsistent with an immediate anti-grokking rebound on the $\leq 50{,}000$ -step horizon.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Architecture	$1$ -layer Transformer
$d_{\mathrm{model}}$	$128$
Attention heads	$4$
Feedforward dim	$512$
Dropout	$0.0$
Optimiser	AdamW

Appendix

Table 5: Full hyperparameter specification.

Figure 15: Probe robustness: $\tilde{H}$ is stable across probe sets. (A) Mean $\tilde{H}$ trajectories computed with a training-set probe (blue) and a test-set probe (orange) are nearly identical throughout training. (B) The absolute difference $|\tilde{H}^{\mathrm{train}}-\tilde{H}^{\mathrm{test}}|$ remains below $0.02$ at all steps (mean $=0.006$ , max $=0.015$ ). (C) Per-seed $\tilde{H}^{*}$ values under the two probes are highly correlated ( $r=0.998$ ) and lie close to the $y=x$ line. These results confirm that the entropy threshold is not an artefact of using a training-set probe.

Signal	Interpretation	Recommended action
$\tilde{H}\gg\tilde{H}^{*}$ , $\Phi\approx 0$	Phase I: norm expanding, representation isotropic	Normal. Continue training.
$\tilde{H}>\tilde{H}^{*}$ , $\Phi>0.01$	Phase II onset: entropy collapsing	Activate predictor. Grokking approaching.
$\|\tilde{H}-\tilde{H}^{*}\|<0.02$	Near critical threshold	Grokking imminent ( $\leq 1{,}000$ steps).
$\tilde{H}<\tilde{H}^{*}$ , test acc. low	Collapse without generalisation	Architecture likely lacks sufficient inductive bias (cf. MLP, § 12 ).
$\tilde{H}$ stagnant $>30{,}000$ steps	No collapse signal	Increase weight decay; reduce learning rate; verify task can grok.

Appendix

Table 6: Diagnostic signals for entropy-based grokking monitoring.

\Phi(t)=\tilde{H}(t-5\tau)-\tilde{H}(t)

denotes the entropy drop over the last

5

evaluation steps (window size

5\tau

Grokking

Grokking is delayed generalization in which neural networks memorize training data before abruptly learning a rule that works on unseen examples.

See all 77 papers

The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization
Mar 5, 2026Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1Grokking Neural Network Generalization
H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam
How to Tame Grokking: Representation Geometry as a Control Signal
Jul 13, 2026Maksim A KazanskiiNeural Network Generalization Spectral Regularization
Independent Researcher
Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent
May 26, 2026Chi-Ning Chou, Oscar Uzdelewicz, Neng-Chun Chiu +2Double Descent Neural Network Generalization
Center for Computational Neuroscience, Flatiron Institute, New York, NY, USA. · Department of Physics, Harvard University, Cambridge, MA, USA. · Kempner Institute, Harvard University, Allston, MA, USA. +1

Mar 5, 2026J/K move · Enter open · S save

The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung, Phan Thanh Duc

H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam

Grokking Neural Network Generalization Gradient Descent Dynamics Neural Network Memorization

Open paper AskPDF arXiv

Grokking -- the sudden generalisation that appears long after a model has perfectly memorised its training data -- has been widely observed but lacks a quantitative theory explaining the length of the delay. We show that grokking is a norm-driven representational phase transition in regularised training dynamics, and establish the Norm-Separation Delay Law:

T_{\mathrm{grok}} - T_{\mathrm{mem}} = Θ(γ_{\mathrm{eff}}^{-1} \log(\|θ_{\mathrm{mem}}\|^2 / \|θ_{\mathrm{post}}\|^2))

, where

γ_{\mathrm{eff}}

is the optimiser's effective contraction rate (

γ_{\mathrm{eff}} = ηλ

for SGD,

γ_{\mathrm{eff}} \ge ηλ

for AdamW). The upper bound follows from a discrete Lyapunov contraction argument; the matching lower bound from dynamical constraints of regularised first-order optimisation. Across 293 training runs spanning modular addition, modular multiplication, and sparse parity, we confirm three falsifiable predictions: inverse scaling with weight decay (

R^2 = 0.97

), inverse scaling with learning rate (

R^2 = 0.92

), and logarithmic dependence on the norm ratio (Pearson

r = 0.91

). A fourth finding reveals that grokking requires an optimiser capable of decoupling memorisation from contraction: SGD fails entirely at the same hyperparameters where AdamW reliably groks. These results reframe grokking not as a mysterious optimisation artefact but as a predictable consequence of norm separation between competing interpolating representations. We further derive a practical three-input algorithm that predicts grokking delay at memorisation time with 34.6% mean absolute error (bootstrap 95% CI [30.0%, 39.4%],

N=60

seeds), enabling principled early stopping.

Abstract

Grokking

Explore similar work

The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

How to Tame Grokking: Representation Geometry as a Control Signal

Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent