cs.LGApr 6, 2026

Grokking as Dimensional Phase Transition in Neural Networks

Authors: Ping Wang

Organizations: Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Neural network grokking -- the abrupt memorization-to-generalization transition -- challenges our understanding of learning dynamics. Through finite-size scaling of gradient avalanche dynamics across eight model scales, we find that grokking is a dimensional phase transition: effective dimensionality~DD crosses from sub-diffusive (subcritical, D<1D < 1) to super-diffusive (supercritical, D>1D > 1) at generalization onset, exhibiting self-organized criticality (SOC). Crucially, DD reflects gradient field geometry, not network architecture: synthetic i.i.d.\ Gaussian gradients maintain D≈1D \approx 1 regardless of graph topology, while real training exhibits dimensional excess from backpropagation correlations. The grokking-localized D(t)D(t) crossing -- robust across topologies -- offers new insight into the trainability of overparameterized networks.

Figures & tables

Figure 1: Grokking as transient SOC and dimensional phase transition. (a) Training (blue) and evaluation (purple) accuracies for representative XOR case (h=21, N=85; train and evaluation share the same four patterns), showing a synchronized abrupt transition at epoch 27. Inset: multi-scale analysis across h=20–500 reveals scale-dependent grokking timing spanning epochs 12–134. (b) Time-resolved FSS analysis shows effective dimensionality DD evolves continuously during training. Yellow region: multi-scale grokking window. Orange line: single-scale grokking. Red line: time-averaged D=1.00±0.02D=1.00\pm 0.02 . Inset: FSS fit quality R2>0.98R^{2}>0.98 . (c) Representative example: Weight concentration (Gini coefficient of ∣θ∣|\bm{\theta}| ; teal) exhibits transient peak coinciding with grokking. Multi-seed statistical validation (1000 seeds) described in text.
Figure 2: Finite-size scaling analysis of avalanche dynamics. (a) Complementary cumulative distributions (CCDF) of avalanche sizes across eight model scales ( h=20h=20 – 500500 ), showing heavy-tailed, scale-dependent behavior with systematic cutoff growth. (b) X-only data collapse: plotting P(>s)P(>s) vs s/NDs/N^{D} collapses all scales toward a common curve using a single exponent DD , validating the FSS exponent without additional fitting parameters. (c) FSS of maximum ( smax⁡∼NDs_{\max}\sim N^{D} , left axis) and mean ( ⟨s⟩∼Nγ\langle s\rangle\sim N^{\gamma} , right axis) avalanche sizes, yielding D=1.00±0.02D=1.00\pm 0.02 ( R2=1.00R^{2}=1.00 ) and γ=1.15±0.06\gamma=1.15\pm 0.06 ( R2=0.99R^{2}=0.99 ) across eight scales.
Figure 3: Gradient Geometry Determines Dimensionality. (a) Bootstrap distributions (10,000 resamples) of the FSS exponent DD , where each run is phase-split at its own grokking epoch: pre-grokking real gradients (green, D=0.90±0.02D=0.90\pm 0.02 , sub-diffusive), post-grokking real gradients (red, D=1.20±0.02D=1.20\pm 0.02 , super-diffusive), and synthetic i.i.d. Gaussian gradients (blue, D=0.99±0.01D=0.99\pm 0.01 ). Three non-overlapping peaks confirm statistically distinct scaling regimes. (b) Leave-One-Out FSS analysis: removing any single scale preserves DD , confirming scale invariance across N=81N=81 – 20012001 . Inset: Five network topologies collapse to D≈0.99D\approx 0.99 for synthetic gradients, demonstrating topology invariance.
More in this topic

Grokking

Grokking is delayed generalization in which neural networks memorize training data before abruptly learning a rule that works on unseen examples.

See all 75 papers

Explore similar work

CardsList

  1. Dimensional Criticality at Grokking Across MLPs and Transformers

    Apr 6, 2026Ping WangNeural Network Training DynamicsGrokking

    Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China

  2. Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

    Sep 9, 2026Anish KatariaMultilayer PerceptronsNeural Network Memorization

    Princeton University, Princeton, NJ, USA.

  3. Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networks

    Jun 15, 2026Ibrahim Talha Ersoy, Karoline WiesnerStatistical Physics of LearningNeural Network Generalization

    Complexity Science Group, Institute of Physics and Astronomy, University of Potsdam, Potsdam, Germany