Is Grokking a Loss of Normal Hyperbolicity of the Interpolation Manifold?

Suvinava Basak

cs.LGAug 14, 2026

Is Grokking a Loss of Normal Hyperbolicity of the Interpolation Manifold?

Authors: Suvinava Basak

Organizations: Technische Universität Braunschweig, Data Science, Universitätsplatz 2, 38106 Braunschweig, Germany

Abstract

A recent line of work recasts the post-memorization phase of grokking as constrained optimization: once a network interpolates the training set, weight decay drives a slow drift along the zero-loss manifold toward lower norm. In the language of dynamical systems, this is a fast-slow system in which the interpolation manifold plays the role of a slow manifold. We ask a question that this framing makes natural but the existing literature does not address: is the sharp generalization transition a loss of normal hyperbolicity of that manifold: a fold- or bifurcation-like event in which a normal restoring direction goes flat? Or does the manifold stay uniformly attracting while generalization happens by smooth drift? We propose a simple, optimizer-agnostic diagnostic: the smallest nonzero singular value $σ_{\min}^{+}(\mathbf J)$ of the residual Jacobian, which, for the squared loss, equals the slowest normal restoring rate of the manifold. On a two-layer ReLU network trained to grok modular addition under squared loss, $σ_{\min}^{+}(\mathbf J)$ does not collapse at the transition; it is near zero only before memorization and attains its largest values during the transition. The result holds across five seeds, and the six smallest singular values behave identically; there is no subspace-local collapse either. This is preliminary evidence against the bifurcation hypothesis and in favor of the smooth-contraction picture. We are explicit that a single-setting, gradual-transition experiment under Adam optimizer does not prove the absence of a bifurcation; it constrains where one could hide.

Figures & tables

Figure 1: Test/training accuracy (left axis) and the normal restoring rate $\sigma_{\min}^{+}(\mathbf{J})$ (right axis). The shaded band marks the generalization transition. $\sigma_{\min}^{+}(\mathbf{J})$ does not dip during the transition; it is small only before memorization and largest while the model generalizes. Two-layer ReLU network, squared loss, AdamW, modular addition $p=11$ ; single seed.

Phase	Step range	median $\sigma_{\min}^{+}(\mathbf{J})$	min $\sigma_{\min}^{+}(\mathbf{J})$
Pre-memorization	$500$ – $1800$	$\approx 0.010$	$0.0096$
Memorization plateau	$3000$ – $6000$	$0.128$	$0.045$
Generalization transition	$7000$ – $17000$	$0.220$	$0.202$
Post-transition	$\geq 22000$	$0.182$	$0.177$

Table 1: Normal restoring rate by training phase (single seed). The only regime in which

\sigma_{\min}^{+}(\mathbf{J})

approaches zero is pre-memorization, which is unrelated to grokking; during the transition it is at its maximum. The pre-memorization reading also shows the diagnostic is sensitive enough to register a genuinely near-degenerate Jacobian when one occurs.

Figure 2: (a) The six smallest singular values ( $\sigma_{1}^{+}$ being the smallest) of $\mathbf{J}$ form a tight cluster bounded away from zero across the transition; the smallest dips only pre-memorization. The yellow shaded region identifies the grokking transition. (b) Robustness across five seeds: thin red lines are per-seed $\sigma_{\min}^{+}(\mathbf{J})$ , the bold red curve and band are mean $\pm$ s.d., and the blue curve/band is mean $\pm$ s.d. test accuracy. No seed shows a dip at its transition.

Grokking

Grokking is delayed generalization in which neural networks memorize training data before abruptly learning a rule that works on unseen examples.

See all 76 papers

Sep 9, 2026

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

Neural networks trained past memorization frequently undergo a delayed transition to generalization, a phenomenon known as grokking. Despite theoretical progress on why this transition occurs, the quantitative structure of when it occurs in hyperparameter space remains uncharacterized. We map the memorization-to-generalization boundary across 384 configurations of two-hidden-layer MLPs on modular arithmetic, fitting a power-law scaling relation for generalization onset time:

T_{\mathrm{grok}} \propto H^{-0.27}\, D^{-2.04}\, η^{-0.50}\, λ^{-0.64}

(

R^2 = 0.732

;

0.821

with interactions). The exponent hierarchy reveals that data complexity (

D^{-2.04}

) is the dominant driver of regime transition, not model capacity (

H^{-0.27}

): doubling data accelerates generalization by

{\sim}4\times

, while doubling width yields only

{\sim}1.2\times

. A sharp phase boundary at weight decay

λ\gtrsim 1.0

separates grokking from non-grokking configurations, and weight norm trajectories show monotonic compression during the transition, consistent with implicit regularization selecting low-complexity solutions. These results provide a quantitative foundation for predicting and controlling regime transitions in overparameterized networks.

Anish Kataria

Princeton University, Princeton, NJ, USA.

Neural Network Training Dynamics Neural Scaling Laws Grokking Neural Network Memorization Multilayer Perceptrons Phase Transitions in Neural Networks+1

Ask PDF

Mar 5, 2026

The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

Grokking -- the sudden generalisation that appears long after a model has perfectly memorised its training data -- has been widely observed but lacks a quantitative theory explaining the length of the delay. We show that grokking is a norm-driven representational phase transition in regularised training dynamics, and establish the Norm-Separation Delay Law:

T_{\mathrm{grok}} - T_{\mathrm{mem}} = Θ(γ_{\mathrm{eff}}^{-1} \log(\|θ_{\mathrm{mem}}\|^2 / \|θ_{\mathrm{post}}\|^2))

, where

γ_{\mathrm{eff}}

is the optimiser's effective contraction rate (

γ_{\mathrm{eff}} = ηλ

for SGD,

γ_{\mathrm{eff}} \ge ηλ

for AdamW). The upper bound follows from a discrete Lyapunov contraction argument; the matching lower bound from dynamical constraints of regularised first-order optimisation. Across 293 training runs spanning modular addition, modular multiplication, and sparse parity, we confirm three falsifiable predictions: inverse scaling with weight decay (

R^2 = 0.97

), inverse scaling with learning rate (

R^2 = 0.92

), and logarithmic dependence on the norm ratio (Pearson

r = 0.91

). A fourth finding reveals that grokking requires an optimiser capable of decoupling memorisation from contraction: SGD fails entirely at the same hyperparameters where AdamW reliably groks. These results reframe grokking not as a mysterious optimisation artefact but as a predictable consequence of norm separation between competing interpolating representations. We further derive a practical three-input algorithm that predicts grokking delay at memorisation time with 34.6% mean absolute error (bootstrap 95% CI [30.0%, 39.4%],

N=60

seeds), enabling principled early stopping.

Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1

H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam

Grokking Neural Network Generalization Gradient Descent Dynamics Neural Network Memorization

Ask PDF

Jun 29, 2026

A Stochastic--Geometric Theory of Scaling Laws in Grokking

Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell--core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model's parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and

\ell_2

regularization coefficient, which are further validated through experiments and shown to recover results from prior literature.

Róisín Luo, Christian Gagné, Jonas Ngnawé +2

Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval +2

Neural Network Generalization Neural Network Optimization Neural Network Memorization Neural Scaling Laws Grokking

Ask PDF

Abstract

Grokking

Explore similar work

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

A Stochastic--Geometric Theory of Scaling Laws in Grokking