Grokking as a Falsifiable Finite-Size Transition

Yuda Bi; Chenyu Zhang; Qiheng Wang; Vince D Calhoun

cs.LGMar 25, 2026

Grokking as a Falsifiable Finite-Size Transition

Authors: Yuda Bi, Chenyu Zhang, Qiheng Wang, Vince D Calhoun

Organizations: Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS Center), Atlanta, Georgia 30303, USA · Beijing Institute of Technology, Beijing 100081, China · Georgia Institute of Technology, Atlanta, Georgia 30332, USA

Abstract

Grokking -- the delayed onset of generalization after early memorization -- is often described with phase-transition language, but that claim has lacked falsifiable finite-size inputs. Here we supply those inputs by treating the group order $p$ of $\mathbb{Z}_p$ as an admissible extensive variable and a held-out spectral head-tail contrast as a representation-level order parameter, then apply a condensed-matter-style diagnostic chain to coarse-grid sweeps and a dense near-critical addition audit. Binder-like crossings reveal a shared finite-size boundary, and susceptibility comparison strongly disfavors a smooth-crossover interpretation ( $Δ\mathrm{AIC}=16.8$ in the near-critical audit). Phase-transition language in grokking can therefore be tested as a quantitative finite-size claim rather than invoked as analogy alone, although the transition order remains unresolved at present.

Figures & tables

Figure 1: Raw spectral order-parameter curves for the canonical addition task. Panel A shows $m_{\mathrm{HTC}}$ versus training fraction $f$ for all $13$ coarse-grid primes; Panel B zooms the transition region around the shared crossing in this first diagnostic step. The ordering structure sharpens monotonically with $p$ .

Figure 2: Binder-based diagnostics for the canonical addition task. Panel A: Binder-like cumulant curves cross near a common fraction. Panel B: all pairwise crossing estimates are shown, with the selected dominant branch highlighted for the drift fit; that branch shows no significant linear drift versus inverse size. Panel C: the coarse-grid Binder minimum is continuity-leaning but method-dependent, motivating the near-critical stress test in Fig. 3 A. Panel D: susceptibility model comparison favors power-law scaling over the minimal saturating null, disfavoring a smooth-crossover interpretation.

Figure 3: Near-critical follow-up for the addition task. (A) Binder-minimum extrapolation on both grids. The coarse grid (open symbols, $p\leq 251$ ) extrapolates toward zero; the near-critical grid (filled symbols, $p$ up to $397$ ) develops a negative trend. (B) Seed-level HTC distributions at representative near-critical points for the two largest primes ( $p=307,f=0.42$ and $p=397,f=0.40$ ). Both distributions remain unimodal and do not provide clean evidence of first-order coexistence despite the negative Binder minimum at the largest size.

Diagnostic	Coarse grid	Near-critical
$f_{\mathrm{c}}$	$0.390$	$0.418$
Crossing spread	$0.057$	$0.019$
Crossing drift	not significant	not significant
$\Delta$ AIC (vs. crossover)	$11.4$	$16.8$
$U_{4,\min}(p\to\infty)$	$0.001\pm 0.157$	$-0.67\pm 0.11$
Seed bimodality	—	absent

Table 1: Summary of FSS diagnostics for canonical addition.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Addition	Subtraction	Multiplication
$f_{\mathrm{c}}$	$0.411$	$0.465$	$0.431$
Original-sweep crossing spread	$0.057$	$0.064$	$0.071$
$\Delta$ AIC (vs. crossover)	$11.4$	$9.2$	$8.7$

Appendix

Table 2: Contextual coarse-grid diagnostics by operation, quoted from the original per-operation sweeps rather than the audited main-text dominant-branch locator.

Figure 4: Order-parameter screening summary for context. Panel A reports time-resolved scans over common observables. Panel B compares spectral candidates and documents that HTC is retained as the representation-level choice for the finite-size diagnostic chain, with screening treated as validation rather than selection.

Figure 5: Robustness of the HTC head–tail cutoff in near-critical and coarse-grid data. Panel A shows the near-critical pairwise crossing behavior across $k=3,5,10$ ; Panel B shows the same for Binder-minimum extrapolation. $k=3$ and $k=5$ remain aligned, while $k=10$ lowers head–bulk separation at large sizes.

Figure 6: Phase diagram in the $(f,\lambda)$ plane at fixed $p=113$ for addition. Panel A shows the majority phase map under the four-label protocol. The phase-labeling protocol contains four categories, but on the present grid the majority map exhibits only three dominant regions; instant generalization appears only as a minority seed label. Panel B shows the fraction of seeds that grok. Panel C shows the mean grokking time where grokking occurs. The grokking regime occupies a distinct intermediate band in control-parameter space.

Explore similar work

Grokking as Dimensional Phase Transition in Neural Networks
Apr 6, 2026Ping WangGrokking Neural Network Training Dynamics
Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China
Dimensional Criticality at Grokking Across MLPs and Transformers
Apr 6, 2026Ping WangNeural Network Training Dynamics Grokking
Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China
An Exploratory Replica-Overlap Probe of the Grokking Transition
Sep 22, 2026A. C. Opus, J. Q. LuStatistical Physics of Learning Neural Network Training Dynamics
Department of Physics, University of Puerto Rico, Mayagüez, PR 00680, USA

Apr 6, 2026J/K move · Enter open · S save

Grokking as Dimensional Phase Transition in Neural Networks

Ping Wang

Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China

Grokking Neural Network Training Dynamics Statistical Physics of Learning Gradient Descent Dynamics

Open paper AskPDF arXiv

Neural network grokking -- the abrupt memorization-to-generalization transition -- challenges our understanding of learning dynamics. Through finite-size scaling of gradient avalanche dynamics across eight model scales, we find that grokking is a dimensional phase transition: effective dimensionality~

D

crosses from sub-diffusive (subcritical,

D < 1

) to super-diffusive (supercritical,

D > 1

) at generalization onset, exhibiting self-organized criticality (SOC). Crucially,

D

reflects gradient field geometry, not network architecture: synthetic i.i.d.\ Gaussian gradients maintain

D \approx 1

regardless of graph topology, while real training exhibits dimensional excess from backpropagation correlations. The grokking-localized

D(t)

crossing -- robust across topologies -- offers new insight into the trainability of overparameterized networks.

Abstract

Explore similar work

Grokking as Dimensional Phase Transition in Neural Networks

Dimensional Criticality at Grokking Across MLPs and Transformers

An Exploratory Replica-Overlap Probe of the Grokking Transition