Statistical Vignettes on the Grokking Phenomenon

Baye Malick Gning; Elvis Dohmatob

cs.LGOct 13, 2026

Statistical Vignettes on the Grokking Phenomenon

Authors: Baye Malick Gning, Elvis Dohmatob

Organizations: Concordia University · Mila – Quebec AI Institute

Abstract

In this work, we study grokking in a simple, analytically tractable setting whose rich phenomenology allows us to disentangle the effects of initialization, signal strength, weight decay, sample size, and input dimension. These factors can produce superficially similar learning curves through distinct mechanisms, complicating their interpretation. Our analysis distinguishes early fitting through nuisance features, core feature learning, and delayed generalization. We identify sharp conditions under which grokking provably occurs and derive analytic asymptotic expressions for the lengths of the generalization plateaus. Our results also characterize regimes of early generalization and failure to reach a prescribed test accuracy target. We show that the interpolation edge constitutes a distinct dynamical regime, where vanishing spectral gaps separate core learning from generalization, helping clarify the role of proximity to interpolation in existing accounts of grokking. The resulting picture is analogous to fast and slow phases in chemical kinetics and suggests that the lazy-to-rich transition discussed in the literature may reflect a similar separation of dynamical time scales. Our theoretical results are faithfully supported by extensive experiments.

Figures & tables

Figure 1: Grokking near the interpolation edge (

\gamma=1

). Experiments for the simple model ( 2 ) trained without regularization / weight decay (

\lambda=0

) in the optimization objective ( 3 ), on

n=400

samples and

d=395

nuisance features (

d/n\approx 1

) from the data distribution ( 1 ), for different values of the signal strength

\varepsilon

. In all cases, the effective weight decay is

b=0<1/z_{p}

: the core coefficient

\theta=\varepsilon uv

approaches

1

, and the test accuracy target

p=95\%

is attained at a time consistent with the prediction of our theory ( 21 ). Under a very small signal, note that when weight decay is active (

\lambda\in\{10^{-4},10^{-5}\}

b

exceeds the threshold, preventing grokking from occurring. See Appendix C for details and additional experiments.

Panel of Figure 2 — Figure 2: Grokking below interpolation threshold ( $\gamma<1$ ). Experiments use $n=400$ samples and $d=360$ nuisance features ( $\gamma=0.9$ ). The results support Theorem 4 across zero, weak and strong weight decay $\lambda$ . The core coefficient approaches the limit $(1-\beta)_{+}$ predicted by our Theorem 4 , while weight determines whether the test accuracy target $p=0.95$ is attained. Core convergence and target attainment need not coincide. Whenever the target is reached, the observed hitting time agrees with the theoretical prediction (vertical line). See Appendix C for details and more extensive experiments.

Panel of Figure 3 — Figure 3: Fast/slow dynamics of learning. Triple arrows indicate fast dynamics, double arrows indicate slow, and single arrows indicate very slow dynamics. In the fast phase, the dynamics ( 7 ) quickly relaxes onto the plane $\mathcal{M}$ in constant time $T(w)\asymp O(1)$ . Then a slower phase begins where the dynamics relaxes towards the line $\mathcal{L}\subseteq\mathcal{M}$ , corresponding to learning the core feature weight $\theta:=\varepsilon uv$ in the student ( 2 ). This happens on a time scale of order $T(\theta)\asymp 1/\varepsilon\gg 1$ , where $\varepsilon\ll 1$ is the SNR. This phase is responsible for the long test accuracy, long after memorization (perfect training accuracy) has occurred. In the final / slowest phase (which only exists in the case of unbalanced initialization), the ”harmonic dual” variable $\phi:=\varepsilon(u^{2}-v^{2})/2$ relaxes to zero in time of order $T(\phi)\asymp 1/\lambda\gg T(\theta)$ , where $\lambda\ll\varepsilon$ is the weight decay.

Regime	Effective dynamics of core feature	Effective dynamics of nuisance	Conclusion
Under-parameterized, weak regularization ( $\gamma<1$ , $\beta<1-\delta_{p}$ )	Logistic ODE ( 13 ); limiting core trajectory converges to $1-\beta>\delta_{p}$	Enters fixed vicinity of the slow sheet $\mathcal{M}_{n}$ in constant time, then tracks the core feature	Grokking occurs on time scale of order $\ell_{n}\ll 1$ (given in ( 16 )), which diverges with the sample size $n$
Under-parameterized, strong regularization ( $\gamma<1$ , $\beta>1-\delta_{p}$ )	Logistic ODE ( 13 ); limiting core trajectory converges to $(1-\beta)_{+}<\delta_{p}$	Enters fixed vicinity of $\mathcal{M}_{n}$ in constant time, then tracks core	Target test accuracy $p$ is never reached
Interpolation edge, weak regularization ( $\gamma=1$ , $b<1/z_{p}$ )	Logistic ODE ( 13 ); limiting core trajectory converges to $1$	Entry time diverges, but is of order $o(\ell_{n})$ and $O(\rho_{n}^{-2}\log\rho_{n}^{-1})$	Fixed core levels reached on scale $\ell_{n}$ (diverging with $n$ ); grokking occurs on time scale of order $\ell_{n}\log\rho_{n}^{-1}\gg\ell_{n}$
Interpolation edge, strong regularization ( $\gamma=1$ , $b>1/z_{p}$ )	Logistic ODE ( 13 ); limiting core trajectory converges to $1$	Same upper-bounds as above; stationary nuisance norm $\\|w_{n}(t)\\|$ tends to $b$	Fixed core levels reached on scale of order $\ell_{n}$ , but target test accuracy $p$ is never reached

Table 1: Phase diagram for the (non)emergence of grokking (Theorem 4 ). Here, the scales

\rho_{n}=O(1)

and

\ell_{n}\to\infty

are as defined in ( 16 ), while

z_{p}

and

\delta_{p}

are as defined in ( 17 )–( 18 ). The under-parametrized regimes additionally assume

\beta_{n}\to\beta

, while interpolation-edge regimes assume

\varepsilon_{n}\rho_{n}^{-2}\to 0

and

\beta_{n}\sqrt{\rho_{n}^{-1}-1}\to b\in[0,\infty)

; in particular,

\beta=0

in both edge rows. Grokking denotes early attainment of the training target followed by delayed generalization, for the given test target

p

. A precise hitting-time equivalent is provided in ( 19 ); the logarithmic factor at the edge is already included in the displayed time scale. The scale

\ell_{n}

diverges with sample size

n

, accounting for long plateaus. Regularization / weight decay

\lambda_{n}

is classified relative to the target

p

. Core convergence refers to the limiting slow trajectory as slow time tends to infinity; stationary nuisance limits are taken as

n\to\infty

. Limits and time estimates are in probability; non-attainment holds with probability tending to one. Detailed results are provided in

T^{\xi}(\|e_{n}\|)

, defined in ( 14 ). Detailed claims are provided in Appendix A.2 .

Figure 4: Logarithmic scaling of grokking time w.r.t core feature initialization

\theta_{0}

. Experiment with

n=400

samples, nuisance dimension

d=360

, a small signal strength

\varepsilon=10^{-2}

, and no weight decay (

\lambda=0

). Perfect terminal test accuracy (

100\%

) is always attained in this setting. Empirically measured normalized hitting time

T_{p}/\ell_{n}

scales linearly in

\log(1/\theta_{0})

in view of ( 20 ), as predicted by Theorem 4 .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Grokking at hard interpolation $\gamma=1$ . Here we use $n=400$ samples and $d=395$ nuisance features ( $d/n\approx 1$ ). This is an extension of experiments in Figure 1 , using regularized models $\lambda\in\{10^{-5},10^{-4}\}$ . Here, critical threshold is $1/z_{p}=0.60$ and $b>1/z_{p}$ in last two columns. For fixed $n$ and $d$ , $\beta_{n}/\sqrt{\rho_{n}^{-1}-1}\to b$ increases as the regularization increases, making grokking harder to happen.

Panel of Figure 6 — Figure 6: Grokking in the under-parameterized regime ( $\gamma<1$ ) for gradient descent and continuous gradient flow. Both experiments use $n=400$ samples, $d=360$ nuisance features ( $\gamma=d/n=0.9$ ), and initial core feature scale $\theta_{0}=10^{-1}$ . Signal strength is $\varepsilon=10^{-1}$ . (a) Gradient Descent with rescaled time. (b) Continuous Gradient Flow ODE. The close match confirms that the theoretical continuous-time analysis accurately captures discrete-time dynamics.

Figure 7: Phase portrait $\|w\|$ against $\theta$ in under-parametrized model ( $\gamma=0.9$ ) . Experiments use $n=400$ samples, and $d=360$ nuisance features. Results illustrate phase portraits from experiments in Figure 2 . For small SNR $\varepsilon=10^{-2}$ (top row), dynamics exhibit a fast inital relaxation in time scale of $O(1)$ (visible as a near-vertical trajectory at the beginning of training), after which nuisance feature weights $w$ are trapped in slow manifold $\mathcal{M}_{0}$ , directly validating Theorem 4 via Fenichel theory. Notice that moderate SRN $\varepsilon=10^{-1}$ (bottom row) accelerates grokking, yielding a reduced hitting time, in exact agreement with our analytical $T_{p}\simeq H_{p}(\beta)\ell_{n}$ .

Figure 8: Logarithmic scaling of grokking time w.r.t core feature initialization

\theta_{0}

under Gradient Descent. Comparing gradient flow ( left ) and gradient-descent ( right ). The settings are identity to Figure 4 of the main text. GD is run with setup as described in Section C.1 .

Figure 9: Plateau length as a function of initialization scale. For these experiments, the signal strength is set to

\varepsilon=10^{-2}

and the weight decay is turned off (

\lambda=0

). In each case, the straight broken line (theory) represents

y=x+b

, where

x=\log 1/\theta_{0}

and the intercept

b

is an explicit function of the target test accuracy (here

p=0.95

), but not on the initialization

\theta_{0}

, etc. The grokking time

T_{p}

is measured empirically. The scale

\ell_{n}

is as defined in ( 16 ). Left: Under-parametrized regime

\gamma<1

. We use

n=400

and

d=360

as in Figure 4 . The theoretical curve is given by ( 20 ). Right: Near the hard interpolation edge

\gamma=1

. Here, we use

n=2000

and

d=1800

. For the theory curve, we use ( 21 ). As in ( 20 ), notice the perfect match between experiment and our theoretical predictions from Section 4.3 .

Figure 10: Plateau length in under-parametrized regime $\gamma<1$ . Here $n=10000$ samples and $d=8000$ , making $k=n-d$ larger. Signal strength is $\varepsilon=10^{-2}$ and regularization is odd. Under the limit ( 16 ), the measured normalized grokking time is close to the theory as expected.

Grokking

Grokking is delayed generalization in which neural networks memorize training data before abruptly learning a rule that works on unseen examples.

See all 77 papers

A Stochastic--Geometric Theory of Scaling Laws in Grokking
Jun 29, 2026Róisín Luo, Christian Gagné, Jonas Ngnawé +2Neural Network Generalization Neural Network Optimization
Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval +2
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Sep 22, 2026Lenz Pracher, Pascal de Jong, Oskar Lieshaus +2Neural Network Generalization Weight Decay
Arnold-Sommerfeld-Center for Theoretical Physics, Ludwig-Maximilians-Universität München, Munich, Germany · Department of Applied Physics, Standford University, Standford, CA, USA · Department of Mathematics, University of Cambridge, Cambridge, UK
Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent
May 26, 2026Chi-Ning Chou, Oscar Uzdelewicz, Neng-Chun Chiu +2Double Descent Neural Network Generalization
Center for Computational Neuroscience, Flatiron Institute, New York, NY, USA. · Department of Physics, Harvard University, Cambridge, MA, USA. · Kempner Institute, Harvard University, Allston, MA, USA. +1

Jun 29, 2026J/K move · Enter open · S save

A Stochastic--Geometric Theory of Scaling Laws in Grokking

Róisín Luo, Christian Gagné, Jonas Ngnawé, Ihsan Ullah+1

Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval+2

Neural Network Generalization Neural Network Optimization Neural Network Memorization Neural Scaling Laws Grokking

Open paper AskPDF arXiv

Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell--core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model's parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and

\ell_2

regularization coefficient, which are further validated through experiments and shown to recover results from prior literature.

Abstract

Grokking

Explore similar work

A Stochastic--Geometric Theory of Scaling Laws in Grokking

A Spectral Theory of Grokking: Weight Decay induces Feature Learning

Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent