The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

Truong Xuan Khanh; Truong Quynh Hoa; Luu Duc Trung; Phan Thanh Duc

cs.AIMar 5, 2026

The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

Authors: Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung, Phan Thanh Duc

Organizations: H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam

Abstract

Grokking -- the sudden generalisation that appears long after a model has perfectly memorised its training data -- has been widely observed but lacks a quantitative theory explaining the length of the delay. We show that grokking is a norm-driven representational phase transition in regularised training dynamics, and establish the Norm-Separation Delay Law: $T_{\mathrm{grok}} - T_{\mathrm{mem}} = Θ(γ_{\mathrm{eff}}^{-1} \log(\|θ_{\mathrm{mem}}\|^2 / \|θ_{\mathrm{post}}\|^2))$ , where $γ_{\mathrm{eff}}$ is the optimiser's effective contraction rate ( $γ_{\mathrm{eff}} = ηλ$ for SGD, $γ_{\mathrm{eff}} \ge ηλ$ for AdamW). The upper bound follows from a discrete Lyapunov contraction argument; the matching lower bound from dynamical constraints of regularised first-order optimisation. Across 293 training runs spanning modular addition, modular multiplication, and sparse parity, we confirm three falsifiable predictions: inverse scaling with weight decay ( $R^2 = 0.97$ ), inverse scaling with learning rate ( $R^2 = 0.92$ ), and logarithmic dependence on the norm ratio (Pearson $r = 0.91$ ). A fourth finding reveals that grokking requires an optimiser capable of decoupling memorisation from contraction: SGD fails entirely at the same hyperparameters where AdamW reliably groks. These results reframe grokking not as a mysterious optimisation artefact but as a predictable consequence of norm separation between competing interpolating representations. We further derive a practical three-input algorithm that predicts grokking delay at memorisation time with 34.6% mean absolute error (bootstrap 95% CI [30.0%, 39.4%], $N=60$ seeds), enabling principled early stopping.

Figures & tables

Figure 1: Conceptual overview of the Norm-Separation Delay Law. After memorisation (

T_{\mathrm{mem}}

), weight decay contracts parameter norms exponentially from the high-norm memorisation region toward the low-norm Fourier manifold. The grokking delay is the time required for this exponential contraction to traverse the norm gap

\log(\|\theta_{\mathrm{mem}}\|^{2}/\|\theta_{\mathrm{post}}\|^{2})

. Generalisation (

T_{\mathrm{grok}}

) occurs once parameters enter the Fourier region and the validation gap becomes detectable.

Symbol	Meaning
$p$	Modulus (problem size)
$\eta$	Learning rate
$\lambda$	Weight decay coefficient
$V_{t}=\left\\|\theta_{t}\right\\|^{2}$	Squared parameter norm at step $t$
$V_{\text{mem}},\,V_{\text{post}}$	Norm at memorisation / after grokking
$V_{\infty}=\eta\sigma^{2}/\lambda$	Asymptotic noise floor

Table 1: Summary of notation.

Seed	$T_{\text{mem}}$	$T_{\text{grok}}$	Delay	$V_{\text{mem}}$	Fitted $\rho$	$T_{\text{esc}}^{\text{th}}$	$R^{2}$
42	800	1800	1000	3891	0.99865	1332	0.9992
43	800	2000	1200	3846	0.99862	1226	0.9993
44	800	1600	800	3916	0.99857	1368	0.9990
45	800	1600	800	3887	0.99847	1223	0.9991
46	800	1800	1000	3944	0.99857	1321	0.9989
47	800	2200	1400	3879	0.99862	1291	0.9987

Table 2: Lyapunov escape validation (

p=97

\eta=10^{-3}

\lambda=1.0

, 10 seeds). The fitted rate

\rho

is the base of the exponential fit

V_{t}=A\rho^{t}+C

;

\gamma_{\text{fit}}=1-\rho

;

T_{\text{esc}}^{\text{th}}=\gamma_{\text{fit}}^{-1}\log(V_{\text{mem}}/V_{\text{final}})

Figure 2: Lyapunov escape validation (real data). (a) Fitted contraction rates

\rho

across 10 seeds; all exceed the weight-decay baseline

1-\eta\lambda=0.999

(green), confirming AdamW amplification (

\gamma_{\text{fit}}\approx 1.41\cdot\eta\lambda

). (b) Distribution of grokking times across 10 seeds (mean

1840\pm 215

steps). (c) Norm separation:

V_{\text{mem}}\approx 3900

V_{\text{post}}\approx 300

across seeds.

$\lambda$	Regime	Grokked	$T_{\text{grok}}$	$V_{\text{mem}}$	$V_{\text{post}}$	$\log\frac{V_{\text{mem}}}{V_{\text{post}}}$	$\tilde{T}$
0.001	I	0/10	—	14808	15060	—	—
0.01	I	0/10	—	14686	11877	—	—
0.1	II	10/10	$12700\pm 1229$	13419	1942	1.94	$625\pm 40$
0.3	II	10/10	$4900\pm 539$	10272	970	2.37	$530\pm 38$
0.5	II	10/10	$3150\pm 320$	6225	568	2.40	$447\pm 59$
1.0	II	10/10	$2100\pm 374$	2648	266	2.28	$434\pm 144$

Table 3: Weight decay sweep (

p=97

\eta=10^{-3}

, 10 seeds per

\lambda

). Regime: I (no grokking), II (reliable), III (over-regularisation).

$p$	$T_{\text{grok}}$	$T_{\text{mem}}$	Delay	$V_{\text{mem}}$	$V_{\text{post}}$	$T_{\text{esc}}^{\text{th}}$
53	$2336\pm 426$	250	2086	5366	188	1679
67	$2079\pm 376$	364	1715	5456	220	1610
89	$1957\pm 221$	629	1328	4653	277	1416
97	$1686\pm 198$	736	950	4297	307	1322
101	$1721\pm 217$	779	942	4112	291	1326
113	$1607\pm 142$	936	671	3645	390	1119

Table 4: Modulus dependence experiment (mean over 7 seeds per

p

T_{\text{esc}}^{\text{th}}=\gamma_{\text{fit}}^{-1}\log(V_{\text{mem}}/V_{\text{post}})

$\eta$	Mean $T_{\text{grok}}$	$T_{\text{grok}}\cdot\eta$	Seeds grokked
$2.0\times 10^{-3}$	$840\pm 80$	$1.68\pm 0.16$	5/5
$1.0\times 10^{-3}$	$1760\pm 150$	$1.76\pm 0.15$	5/5
$5.0\times 10^{-4}$	$5280\pm 2010$	$2.64\pm 1.01$	5/5
$2.0\times 10^{-4}$	$35680\pm 10189$	$7.14\pm 2.04$	5/5
$1.0\times 10^{-4}$	$100350\pm 18500$	$10.04\pm 1.85$	4/5

Table 5: Learning rate sweep (

p=97

\lambda=1.0

, 5 seeds). The rising

T_{\text{grok}}\cdot\eta

product reflects the noise floor effect from Theorem 3.2 .

Figure 3: Scaling laws. (a)

T_{\text{grok}}

1/\lambda

in Regime II (

R^{2}=0.97

). (b)

T_{\text{grok}}

1/\eta

(

R^{2}=0.92

). (c) Delay vs log norm ratio across 7 moduli (

r=0.91

Seed	$T_{\text{grok}}$	$K_{\text{final}}$	$\mathcal{R}_{\text{pre}}$	$\mathcal{R}_{\text{post}}$	$\mathcal{R}_{\text{final}}$
42	1800	21	0.128	0.00271	0.00271
43	2000	21	0.109	0.00228	0.00228
44	1600	17	0.104	0.00101	0.00101
45	1600	11	0.095	0.00236	0.00236
46	1800	11	0.159	0.00394	0.00394
Mean $\pm$ std		$16.2\pm 4.7$	$0.119\pm 0.025$	$0.00246\pm 0.00098$	$0.00246\pm 0.00098$

Table 6: Spectral separation results (

p=97

, 5 seeds).

Figure 4: Cross-task generalization. (a) Modular multiplication: all 20 runs grok across 4 moduli. (b) Exponential fit quality (

R^{2}

) for all 20 multiplication runs; all exceed 0.988. (c) Sparse parity:

V_{\text{final}}>V_{\text{mem}}

in all 15 runs—no norm separation, no grokking.

Optimizer	Convention	Grokked	$T_{\text{grok}}$	$V_{\text{final}}$	Fitted $\rho$	$R^{2}$
SGD	$w=2\lambda$	0/5	—	$3.6\!\times\!10^{-5}$	—	—
SGD	$w=\lambda$	0/5	—	$1.1\!\times\!10^{-4}$	—	—
AdamW	$w=\lambda$	5/5	$1760\pm 150$	$233\pm 35$	$0.99859\pm 6.5\!\times\!10^{-5}$	0.9990

Table 7: SGD vs AdamW ablation (

p=97

\eta=10^{-3}

\lambda=1.0

, 5 seeds). SGD fails to memorise under these hyperparameters, precluding grokking entirely.

Figure 5: AdamW contraction analysis. (a) Fitted contraction rates across 10 seeds; all exceed the weight-decay baseline

1-\eta\lambda

\sim

41%. (b) Grokking success: AdamW groks in 5/5 seeds; SGD fails entirely. (c)

R^{2}

of exponential fits across all seeds (mean

R^{2}=0.9990

$p$	Grokked	$T_{\text{grok}}$	Delay	$V_{\text{mem}}$	$V_{\text{post}}$	$\gamma_{\text{fit}}$	$R^{2}$
53	5/5	$4480\pm 500$	4280	10431	257	0.00180	0.992
67	5/5	$4440\pm 320$	4040	9337	305	0.00170	0.992
89	5/5	$4040\pm 150$	3440	8859	388	0.00158	0.992
97	5/5	$3920\pm 240$	3320	9622	394	0.00156	0.992

Table 8: Modular multiplication: all 20 runs grok. Exponential contraction fits with

R^{2}>0.988

$n$	Grokked	Delay	$V_{\text{mem}}$	$V_{\text{final}}$	$V_{\text{final}}/V_{\text{mem}}$
20	5/5 †	0	141	231	1.64
30	5/5 †	0	112	220	1.97
40	5/5 †	$-120$	116	202	1.74
† Val. acc $>95\%$ achieved, but simultaneously with train acc — no delayed transition.

Table 9: Sparse parity: no grokking observed. Generalisation occurs simultaneously with (or before) memorisation; norm ratio is inverted (

V_{\text{final}}>V_{\text{mem}}

Figure 6: Phase diagram and cross-task universality. (a) Phase diagram in

(\eta,\lambda)

space showing three regimes. Red triangle: SGD fails where AdamW succeeds. (b) Delay vs log norm ratio for both addition (S1, blue circles) and multiplication (S9, coral diamonds); the linear relationship holds across tasks.

Prediction	Theory	Empirical Result
Lyapunov contraction rate	$1-\eta\lambda=0.999$	$0.99859\pm 6.5\!\times\!10^{-5}$ , $R^{2}=0.9990$
$T\propto 1/\lambda$ (Regime II)	slope $=$ const	slope $=1182$ , $R^{2}=0.971$ , CI $[1082,1271]$
Norm-ratio formula	$\gamma_{\text{fit}}^{-1}\log\frac{V_{\text{mem}}}{V_{\text{post}}}$	slope $1.08\pm 0.15$ , Pearson $r=0.91$
$T\propto 1/\eta$	slope $=$ const	slope $10.62\!\times\!10^{3}$ , $R^{2}=0.921$
Joint $\eta\lambda$ universality	$T\cdot\eta\lambda=$ const	mean $T\cdot\eta\lambda=9.58$
Gap $\propto\mathcal{R}(f_{\theta})$ (OLS)	positive slope	hi-res: 15.81, $R^{2}=0.767$ ; lo-res: 13.49, $R^{2}=0.736$

Table 10: Theoretical predictions versus empirical measurements across all experiments.

Contribution	Prior work	This paper
Empirical documentation	( Power et al., 2022 )	—
Circuit analysis	( Nanda et al., 2023 )	—
Qualitative phase transition	Various	—
Quantitative delay formula	—	✓
Tight upper + lower bounds	—	✓
Cross-task validation	—	✓

Table 11: Positioning of this work relative to prior grokking research.

Explore similar work

The Weight Norm Sets the Grokking Timescale: A Causal Delay Law
Jun 11, 2026Truong Xuan Khanh, Doan Hoang Viet, Luu Duc Trung +1Neural Network Generalization Neural Network Training Dynamics
H&K Research Studio / Clevix LLC, Hanoi, Vietnam · Bac A Bank, Hanoi · Banking Academy of Vietnam
First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation
May 13, 2026Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1Neural Network Memorization Neural Network Training Dynamics
H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam
A Stochastic--Geometric Theory of Scaling Laws in Grokking
Jun 29, 2026Róisín Luo, Christian Gagné, Jonas Ngnawé +2Neural Network Generalization Neural Network Optimization
Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval +2

Jun 11, 2026J/K move · Enter open · S save

The Weight Norm Sets the Grokking Timescale: A Causal Delay Law

Truong Xuan Khanh, Doan Hoang Viet, Luu Duc Trung, Phan Thanh Duc

H&K Research Studio / Clevix LLC, Hanoi, Vietnam · Bac A Bank, Hanoi · Banking Academy of Vietnam

Neural Network Generalization Neural Network Training Dynamics Grokking

Open paper AskPDF arXiv

Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data. Whether the weight norm causes this delay is disputed: some studies report a critical norm at the transition, others observe grokking with no fixed norm at all. We settle this by intervening on the norm during training rather than only observing it. Under free training with weight decay, networks grok when the weight norm reaches a value Wc that varies little across seeds and learning rates (CV 1 to 2 percent) and grows with the modular base as a power law. When we instead clamp the norm to a fixed multiple rho of Wc and hold it there, the network still groks, but the delay follows T_grok proportional to exp(alpha rho). One exponent, alpha near 7.5, fits this delay across four moduli (R^2 = 0.996). Over the swept ranges the held norm moves the delay by about 19x and the learning rate by only about 2x, and holding the norm above Wc slows grokking rather than preventing it. A final LayerNorm removes the dependence by decoupling weight scale from the network function; without it the exponential law returns. This pinned-norm delay is the exponential counterpart to the logarithmic delay predicted for a freely contracting norm.

Abstract

Explore similar work

The Weight Norm Sets the Grokking Timescale: A Causal Delay Law

First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation

A Stochastic--Geometric Theory of Scaling Laws in Grokking