Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold

Pratyush Acharya; Habish Dhakal

cs.LGMar 16, 2026

Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold

Authors: Pratyush Acharya, Habish Dhakal

Abstract

Standard optimization theories struggle to explain grokking, where generalization occurs long after training convergence. While geometric studies attribute this to slow drift, they often overlook the interaction between the optimizer's noise structure and landscape curvature. This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a Spectral Gating'' mechanism that regulates the transition from memorization to generalization. We find that AdamW operates as a variance-gated stochastic system. Grokking is constrained by a stability condition: the generalizing solution resides in a sharp basin ($λ_{max}^H$) initially inaccessible under low-variance regimes. The delayed'' phase represents the accumulation of gradient variance required to lift the effective stability ceiling, permitting entry into this sharp manifold. Our ablation studies identify three complexity regimes: (1) Capacity Collapse ( $P < 23$ ), where rank-deficiency prevents structural learning; (2) The Variance-Limited Regime ( $P \approx 41$ ), where generalization waits for the spectral gate to open; and (3) Stability Override ( $P > 67$ ), where memorization becomes dimensionally unstable. Furthermore, we challenge the "Flat Minima" hypothesis for algorithmic tasks, showing that isotropic noise injection fails to induce grokking. Generalization requires the anisotropic rectification unique to adaptive optimizers, which directs noise into the tangent space of the solution manifold.

Figures & tables

Figure 1: Stability-Complexity Phase Diagram. We define three regimes based on the time-to-generalization: (1) Capacity Collapse ( $P<23$ ), where rank-deficiency prevents the representation of the solution; (2) Variance-Limited Regime ( $29\leq P\leq 59$ ), where generalization is delayed by spectral gating; and (3) Stability Override ( $P\geq 67$ ), where the high dimensionality of the memorization manifold forces immediate structural learning. The white dashed line marks $\epsilon=\sigma_{noise}$ , representing the intrinsic gradient noise level.

Figure 2: Capacity Collapse ( $P=17$ ). Despite aggressive stability tuning ( $\epsilon\to 10^{-15}$ ), the optimizer fails to locate the generalizing minimum. The dynamics exhibit stochastic tunneling without convergence, indicating that the embedding space lacks the geometric capacity to separate the task’s Fourier modes.

Figure 3: Non-Monotonic Generalization Dynamics. Generalization time peaks at intermediate complexity ( $P=41$ , Red). Hard tasks ( $P=97$ , Teal) generalize immediately (Stability Override), while easy tasks ( $P=23$ , Purple) stagnate. The results reveal a "Complexity Valley" where the competition between the entropic pull of memorization and the spectral gate of generalization is maximized.

Panel of Figure 4 — Figure 4: Radial Stationarity. (a) AdamW generates a constant diffusive "push" (peach line) that counters the restorative force. (b) The restorative drift stabilizes, confirming a fixed radial orbit. (c) Stronger weight decay (peach) forces a lower-norm equilibrium, compressing the search space onto the manifold [ 7 ] .

Panel of Figure 5 — Figure 5: Acceleration via Constraint. (a) Higher weight decay (Peach) significantly accelerates the phase transition. (b) Strong regularization suppresses the peak gradient variance, enforcing a disciplined search in the tangent space.

Figure 6: The Stability Threshold. Class C (Hard): Performance degrades as $\epsilon\to 10^{-1}$ , confirming the need for anisotropic noise (Limit I). Class B (Intermediate): Exhibits a convex "optimal stability" region around $\epsilon=10^{-4}$ . Class A (Easy): Fails regardless of stability settings.

Figure 7: Violation of the Initial Stability Condition. We compare Hessian Sharpness (Blue) to the Initial Stability Limit determined by $\epsilon$ (Red Dotted Line). For the grokking task ( $P=41$ , Center), the generalizing solution resides in a basin where $\lambda_{max}$ is **721x higher** than the initial stability threshold. The delayed phase (0–200k steps) corresponds to the time required for gradient variance $\sqrt{v_{t}}$ to grow sufficiently to lift the stability ceiling (Black Dashed Line) above the basin’s curvature.

Panel of Figure 8 — Figure 8: Failure of Isotropic SGLD. (a) Despite $1\times 10^{6}$ steps and extensive noise tuning ( $\sigma\in[10^{-3},10^{-1}]$ ), SGLD fails to generalize. (b) Isotropic noise successfully maintains the weight norm, proving that radial energy alone is insufficient.

Panel of Figure 9 — Figure 9: Structural Stagnation in SGLD. (a) and (b) Fourier sparsity remains high, indicating no circuit formation. (c) The Hessian trace lacks the characteristic "Ignition" spike of AdamW.

Figure 10: Capacity Collapse. We ablate embedding dimension $V$ against modulus $P$ . The color scale indicates the time required to grok (Log10 Steps). The sharp diagonal boundary reveals that grokking requires a minimum dimensional capacity proportional to task complexity ( $V\propto P$ ). Black crosses indicate configurations that failed to generalize within $10^{6}$ steps.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Embedding Geometry across Complexity Regimes. Learned embeddings projected onto their first two principal components. Left ( $P=17$ ): The Signal Starvation regime exhibits a lack of structural organization; embeddings are scattered, indicating failure to discover the modular circle. Center ( $P=41$ ): The Grokking regime reveals a recognizable but noisy circular geometry, consistent with a variance-limited equilibrium where the optimizer maintains high entropy to navigate the basin. Right ( $P=97$ ): The Stability Override regime forms a clear circular structure immediately, indicating that high-complexity tasks force rapid geometric condensation.

Panel of Figure 12 — Figure 12: Distribution of Update Scalings. Histograms display the effective learning rates applied by AdamW per parameter. Unlike Isotropic SGLD (which implies a Dirac distribution), AdamW generates a **heavy-tailed distribution** spanning multiple orders of magnitude. This indicates that the optimizer actively shapes the noise geometry, suppressing variance in high-curvature directions (left tail) while amplifying it in flat directions (right tail).

Figure 13: Spectral Dynamics for Varying Moduli. Green lines denote Test Accuracy; Blue lines denote Hessian Sharpness. For successful generalization cases ( $P\geq 29$ ), we observe the characteristic "Ignition" event where the variance-driven stability ceiling (Black Dashed) rises to meet the local curvature.

Figure 14: Isotropic Noise Injection. Injecting isotropic noise ( $\sigma=0.01$ , cyan lines) allows the model to escape the lazy basin and reach $\approx 60-80\%$ accuracy, outperforming the noiseless baseline. However, it fails to induce the perfect generalization seen in grokking. Thus, isotropic noise aids optimization but does not replace the geometric steering required for structural convergence.

Grokking

Grokking is delayed generalization in which neural networks memorize training data before abruptly learning a rule that works on unseen examples.

See all 77 papers

A Stochastic--Geometric Theory of Scaling Laws in Grokking
Jun 29, 2026Róisín Luo, Christian Gagné, Jonas Ngnawé +2Neural Network Generalization Neural Network Optimization
Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval +2
Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking
Feb 18, 2026Yongzhong XuGrokking Neural Network Training Dynamics
First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation
May 13, 2026Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1Neural Network Memorization Neural Network Training Dynamics
H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam

Jun 29, 2026J/K move · Enter open · S save

A Stochastic--Geometric Theory of Scaling Laws in Grokking

Róisín Luo, Christian Gagné, Jonas Ngnawé, Ihsan Ullah+1

Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval+2

Neural Network Generalization Neural Network Optimization Neural Network Memorization Neural Scaling Laws Grokking

Open paper AskPDF arXiv

Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell--core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model's parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and

\ell_2

regularization coefficient, which are further validated through experiments and shown to recover results from prior literature.

Abstract

Grokking

Explore similar work

A Stochastic--Geometric Theory of Scaling Laws in Grokking

Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking

First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation