Statistical Vignettes on the Grokking Phenomenon

Baye Malick Gning; Elvis Dohmatob

cs.LGOct 13, 2026

Statistical Vignettes on the Grokking Phenomenon

Authors: Baye Malick Gning, Elvis Dohmatob

Organizations: Concordia University · Mila – Quebec AI Institute

Abstract

In this work, we study grokking in a simple, analytically tractable setting whose rich phenomenology allows us to disentangle the effects of initialization, signal strength, weight decay, sample size, and input dimension. These factors can produce superficially similar learning curves through distinct mechanisms, complicating their interpretation. Our analysis distinguishes early fitting through nuisance features, core feature learning, and delayed generalization. We identify sharp conditions under which grokking provably occurs and derive analytic asymptotic expressions for the lengths of the generalization plateaus. Our results also characterize regimes of early generalization and failure to reach a prescribed test accuracy target. We show that the interpolation edge constitutes a distinct dynamical regime, where vanishing spectral gaps separate core learning from generalization, helping clarify the role of proximity to interpolation in existing accounts of grokking. The resulting picture is analogous to fast and slow phases in chemical kinetics and suggests that the lazy-to-rich transition discussed in the literature may reflect a similar separation of dynamical time scales. Our theoretical results are faithfully supported by extensive experiments.

Figures & tables

Figure 1: Grokking near the interpolation edge (

\gamma=1

). Experiments for the simple model ( 2 ) trained without regularization / weight decay (

\lambda=0

) in the optimization objective ( 3 ), on

n=400

samples and

d=395

nuisance features (

d/n\approx 1

) from the data distribution ( 1 ), for different values of the signal strength

\varepsilon

. In all cases, the effective weight decay is

b=0<1/z_{p}

: the core coefficient

\theta=\varepsilon uv

approaches

1

, and the test accuracy target

p=95\%

is attained at a time consistent with the prediction of our theory ( 21 ). Under a very small signal, note that when weight decay is active (

\lambda\in\{10^{-4},10^{-5}\}

b

exceeds the threshold, preventing grokking from occurring. See Appendix C for details and additional experiments.

Panel of Figure 2 — Figure 2: Grokking below interpolation threshold ( $\gamma<1$ ). Experiments use $n=400$ samples and $d=360$ nuisance features ( $\gamma=0.9$ ). The results support Theorem 4 across zero, weak and strong weight decay $\lambda$ . The core coefficient approaches the limit $(1-\beta)_{+}$ predicted by our Theorem 4 , while weight determines whether the test accuracy target $p=0.95$ is attained. Core convergence and target attainment need not coincide. Whenever the target is reached, the observed hitting time agrees with the theoretical prediction (vertical line). See Appendix C for details and more extensive experiments.

Panel of Figure 3 — Figure 3: Fast/slow dynamics of learning. Triple arrows indicate fast dynamics, double arrows indicate slow, and single arrows indicate very slow dynamics. In the fast phase, the dynamics ( 7 ) quickly relaxes onto the plane $\mathcal{M}$ in constant time $T(w)\asymp O(1)$ . Then a slower phase begins where the dynamics relaxes towards the line $\mathcal{L}\subseteq\mathcal{M}$ , corresponding to learning the core feature weight $\theta:=\varepsilon uv$ in the student ( 2 ). This happens on a time scale of order $T(\theta)\asymp 1/\varepsilon\gg 1$ , where $\varepsilon\ll 1$ is the SNR. This phase is responsible for the long test accuracy, long after memorization (perfect training accuracy) has occurred. In the final / slowest phase (which only exists in the case of unbalanced initialization), the ”harmonic dual” variable $\phi:=\varepsilon(u^{2}-v^{2})/2$ relaxes to zero in time of order $T(\phi)\asymp 1/\lambda\gg T(\theta)$ , where $\lambda\ll\varepsilon$ is the weight decay.

Regime	Effective dynamics of core feature	Effective dynamics of nuisance	Conclusion
Under-parameterized, weak regularization ( $\gamma<1$ , $\beta<1-\delta_{p}$ )	Logistic ODE ( 13 ); limiting core trajectory converges to $1-\beta>\delta_{p}$	Enters fixed vicinity of the slow sheet $\mathcal{M}_{n}$ in constant time, then tracks the core feature	Grokking occurs on time scale of order $\ell_{n}\ll 1$ (given in ( 16 )), which diverges with the sample size $n$
Under-parameterized, strong regularization ( $\gamma<1$ , $\beta>1-\delta_{p}$ )	Logistic ODE ( 13 ); limiting core trajectory converges to $(1-\beta)_{+}<\delta_{p}$	Enters fixed vicinity of $\mathcal{M}_{n}$ in constant time, then tracks core	Target test accuracy $p$ is never reached
Interpolation edge, weak regularization ( $\gamma=1$ , $b<1/z_{p}$ )	Logistic ODE ( 13 ); limiting core trajectory converges to $1$	Entry time diverges, but is of order $o(\ell_{n})$ and $O(\rho_{n}^{-2}\log\rho_{n}^{-1})$	Fixed core levels reached on scale $\ell_{n}$ (diverging with $n$ ); grokking occurs on time scale of order $\ell_{n}\log\rho_{n}^{-1}\gg\ell_{n}$
Interpolation edge, strong regularization ( $\gamma=1$ , $b>1/z_{p}$ )	Logistic ODE ( 13 ); limiting core trajectory converges to $1$	Same upper-bounds as above; stationary nuisance norm $\\|w_{n}(t)\\|$ tends to $b$	Fixed core levels reached on scale of order $\ell_{n}$ , but target test accuracy $p$ is never reached

Table 1: Phase diagram for the (non)emergence of grokking (Theorem 4 ). Here, the scales

\rho_{n}=O(1)

and

\ell_{n}\to\infty

are as defined in ( 16 ), while

z_{p}

and

\delta_{p}

are as defined in ( 17 )–( 18 ). The under-parametrized regimes additionally assume

\beta_{n}\to\beta

, while interpolation-edge regimes assume

\varepsilon_{n}\rho_{n}^{-2}\to 0

and

\beta_{n}\sqrt{\rho_{n}^{-1}-1}\to b\in[0,\infty)

; in particular,

\beta=0

in both edge rows. Grokking denotes early attainment of the training target followed by delayed generalization, for the given test target

p

. A precise hitting-time equivalent is provided in ( 19 ); the logarithmic factor at the edge is already included in the displayed time scale. The scale

\ell_{n}

diverges with sample size

n

, accounting for long plateaus. Regularization / weight decay

\lambda_{n}

is classified relative to the target

p

. Core convergence refers to the limiting slow trajectory as slow time tends to infinity; stationary nuisance limits are taken as

n\to\infty

. Limits and time estimates are in probability; non-attainment holds with probability tending to one. Detailed results are provided in

T^{\xi}(\|e_{n}\|)

, defined in ( 14 ). Detailed claims are provided in Appendix A.2 .

Figure 4: Logarithmic scaling of grokking time w.r.t core feature initialization

\theta_{0}

. Experiment with

n=400

samples, nuisance dimension

d=360

, a small signal strength

\varepsilon=10^{-2}

, and no weight decay (

\lambda=0

). Perfect terminal test accuracy (

100\%

) is always attained in this setting. Empirically measured normalized hitting time

T_{p}/\ell_{n}

scales linearly in

\log(1/\theta_{0})

in view of ( 20 ), as predicted by Theorem 4 .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Grokking at hard interpolation $\gamma=1$ . Here we use $n=400$ samples and $d=395$ nuisance features ( $d/n\approx 1$ ). This is an extension of experiments in Figure 1 , using regularized models $\lambda\in\{10^{-5},10^{-4}\}$ . Here, critical threshold is $1/z_{p}=0.60$ and $b>1/z_{p}$ in last two columns. For fixed $n$ and $d$ , $\beta_{n}/\sqrt{\rho_{n}^{-1}-1}\to b$ increases as the regularization increases, making grokking harder to happen.

Panel of Figure 6 — Figure 6: Grokking in the under-parameterized regime ( $\gamma<1$ ) for gradient descent and continuous gradient flow. Both experiments use $n=400$ samples, $d=360$ nuisance features ( $\gamma=d/n=0.9$ ), and initial core feature scale $\theta_{0}=10^{-1}$ . Signal strength is $\varepsilon=10^{-1}$ . (a) Gradient Descent with rescaled time. (b) Continuous Gradient Flow ODE. The close match confirms that the theoretical continuous-time analysis accurately captures discrete-time dynamics.

Figure 7: Phase portrait $\|w\|$ against $\theta$ in under-parametrized model ( $\gamma=0.9$ ) . Experiments use $n=400$ samples, and $d=360$ nuisance features. Results illustrate phase portraits from experiments in Figure 2 . For small SNR $\varepsilon=10^{-2}$ (top row), dynamics exhibit a fast inital relaxation in time scale of $O(1)$ (visible as a near-vertical trajectory at the beginning of training), after which nuisance feature weights $w$ are trapped in slow manifold $\mathcal{M}_{0}$ , directly validating Theorem 4 via Fenichel theory. Notice that moderate SRN $\varepsilon=10^{-1}$ (bottom row) accelerates grokking, yielding a reduced hitting time, in exact agreement with our analytical $T_{p}\simeq H_{p}(\beta)\ell_{n}$ .

Figure 8: Logarithmic scaling of grokking time w.r.t core feature initialization

\theta_{0}

under Gradient Descent. Comparing gradient flow ( left ) and gradient-descent ( right ). The settings are identity to Figure 4 of the main text. GD is run with setup as described in Section C.1 .

Figure 9: Plateau length as a function of initialization scale. For these experiments, the signal strength is set to

\varepsilon=10^{-2}

and the weight decay is turned off (

\lambda=0

). In each case, the straight broken line (theory) represents

y=x+b

, where

x=\log 1/\theta_{0}

and the intercept

b

is an explicit function of the target test accuracy (here

p=0.95

), but not on the initialization

\theta_{0}

, etc. The grokking time

T_{p}

is measured empirically. The scale

\ell_{n}

is as defined in ( 16 ). Left: Under-parametrized regime

\gamma<1

. We use

n=400

and

d=360

as in Figure 4 . The theoretical curve is given by ( 20 ). Right: Near the hard interpolation edge

\gamma=1

. Here, we use

n=2000

and

d=1800

. For the theory curve, we use ( 21 ). As in ( 20 ), notice the perfect match between experiment and our theoretical predictions from Section 4.3 .

Figure 10: Plateau length in under-parametrized regime $\gamma<1$ . Here $n=10000$ samples and $d=8000$ , making $k=n-d$ larger. Signal strength is $\varepsilon=10^{-2}$ and regularization is odd. Under the limit ( 16 ), the measured normalized grokking time is close to the theory as expected.

Grokking

Grokking is delayed generalization in which neural networks memorize training data before abruptly learning a rule that works on unseen examples.

See all 77 papers

Jun 29, 2026

A Stochastic--Geometric Theory of Scaling Laws in Grokking

Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell--core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model's parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and

\ell_2

regularization coefficient, which are further validated through experiments and shown to recover results from prior literature.

Róisín Luo, Christian Gagné, Jonas Ngnawé +2

Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval +2

Neural Network Generalization Neural Network Optimization Neural Network Memorization Neural Scaling Laws Grokking

Ask PDF

Sep 22, 2026

A Spectral Theory of Grokking: Weight Decay induces Feature Learning

In grokking an early fit to the training data separates from a much later improvement in generalization. During this delay, training can move from a fixed neural tangent kernel (NTK) regime to one in which task-relevant kernel eigendirections continue to evolve. We provide a quantitative theory for how this transition from lazy to rich learning can produce delayed generalization. For homogeneous networks trained with squared loss and

L_2

weight decay, we show that a finite residual remains after memorization, with larger residual fractions in target components associated with smaller NTK eigenvalues. These residuals feed back into the dynamics of the NTK itself, and projecting the resulting dynamics onto task-relevant spectral directions yields a reduced system in which residual-driven kernel growth competes with weight decay. This system predicts that the grokking timescale is controlled by the product of learning rate and weight decay, that feature learning slows logarithmically near a critical decay above which task-aligned NTK structure can no longer support generalization, and that stronger decay can prevent fitting altogether. We test these predictions in modular addition. In a homogeneous MLP, task-aligned Fourier structure continues to emerge in the NTK after training accuracy has saturated, and an 84

\times

90-grid of trained networks across varying learning rate and weight decay recovers the predicted phase geometry and inverse-product scaling of the generalization time with learning rate and weight decay. A one-block Transformer shows similar macroscopic phase structure in a 42

\times

45-grid, as well as the same transition-time scaling despite violating exact homogeneity. Together, these results provide a mechanistic derivation connecting post-fit feature learning to both the onset of generalization and its phase structure in the learning rate and weight decay plane.

Lenz Pracher, Pascal de Jong, Oskar Lieshaus +2

Arnold-Sommerfeld-Center for Theoretical Physics, Ludwig-Maximilians-Universität München, Munich, Germany · Department of Applied Physics, Standford University, Standford, CA, USA · Department of Mathematics, University of Cambridge, Cambridge, UK

Neural Network Generalization Weight Decay Modular Addition Neural Tangent Kernel Grokking

Ask PDF

May 26, 2026

Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent

Training loss and accuracy are the standard signals used to monitor generalization during deep neural network training. Two well-documented phenomena complicate this picture: in grokking, train loss falls rapidly while test performance improves abruptly only after a long delay; in epoch-wise double descent, train loss decreases monotonically while test loss or error rises and falls. Existing accounts are often task-specific, and a task-agnostic analysis framework for diagnosing and explaining these phenomena across realistic tasks and architectures is missing. We address this challenge by analyzing two competing processes that underlie learning dynamics: representation learning in the encoder and readout calibration in the final classifier. Using tools from representational geometry, neural tangent kernels, and linear probing, we show that both processes are active throughout training, with the fluctuations of their relative speed giving rise to seemingly anomalous generalization dynamics. Applying the representation-readout decomposition to grokking across a wide range of tasks and architectures, we find that the readout is train-biased before grokking onset, and representation learning is gradual but not absent, contrary to the lazy-to-rich account. The framework further provides diagnostic signatures distinguishing spurious from genuine generalization: in a previously reported MNIST grokking example and an epoch-wise double descent example, apparent delayed or non-monotone generalization is shown to arise from representation degradation and readout misalignment induced by non-standard training recipes. Together, these results establish the representation-readout decomposition as a top-down framework for understanding learning dynamics and revealing underlying algorithms for interpretability research.

Chi-Ning Chou, Oscar Uzdelewicz, Neng-Chun Chiu +2

Center for Computational Neuroscience, Flatiron Institute, New York, NY, USA. · Department of Physics, Harvard University, Cambridge, MA, USA. · Kempner Institute, Harvard University, Allston, MA, USA. +1

Double Descent Neural Network Generalization Representation Learning Neural Network Training Dynamics Grokking

Ask PDF

Abstract

Grokking

Explore similar work

A Stochastic--Geometric Theory of Scaling Laws in Grokking

A Spectral Theory of Grokking: Weight Decay induces Feature Learning

Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent