Latent Algorithmic Structure Precedes Grokking: A Mechanistic Study of ReLU MLPs on Modular Arithmetic

Grokking the delayed transition from memorization to generalization in neural networks remains poorly understood, in part because prior empirical studies confound the roles of architecture, optimization, and regularization. We present a controlled study that systematically disentangles these factors on modular addition (mod 97), with matched and carefully tuned training regimes across models. Our central finding is that grokking dynamics are not primarily determined by architecture, but by interactions between optimization stability and regularization. Specifically, we show: (1) depth has a non-monotonic effect, with depth-4 MLPs consistently failing to grok while depth-8 residual networks recover generalization, demonstrating that depth requires architectural stabilization; (2) the apparent gap between Transformers and MLPs largely disappears (1.11$\times$ delay) under matched hyperparameters, indicating that previously reported differences are largely due to optimizer and regularization confounds; (3) activation function effects are regime-dependent, with GELU up to 4.3$\times$ faster than ReLU only when regularization permits memorization; and (4) weight decay is the dominant control parameter, exhibiting a narrow ``Goldilocks'' regime in which grokking occurs, while too little or too much prevents generalization. Across 3--5 seeds per configuration, these results provide a unified empirical account of grokking as an interaction-driven phenomenon. Our findings challenge architecture-centric interpretations and clarify how optimization and regularization jointly govern delayed generalization.

Neural networks trained on modular arithmetic exhibit grokking, a delayed transition from memorisation to generalisation known to depend on model capacity: too little and the network memorises slowly or not at all, too much and it generalises almost immediately. What happens at the extreme of this spectrum, when the architecture's expressible function class collapses to a finite-dimensional algebraic variety? We study two-layer networks with a holomorphic monomial activation sigma(z)=z^k, trained on modular tasks encoded via roots of unity. Here the network output, regardless of hidden width, is confined to a (k+1)-dimensional subspace of characters of (Z_p)^2, an O(k/p^2) slice of the full function space. We give a complete algebraic characterisation of this subspace: a task is representable if and only if its discrete Fourier support lies on the diagonal u+v = k (mod p), which for linear-phase targets reduces to the arithmetic criterion m+n=k. This is not merely a constraint on eventual generalisation but on memorisation itself: because the outputs are algebraically confined, a non-representable target cannot be fit even on the training set, and we prove a positive lower bound on the training loss, independent of width. Across 585 runs the algebraic prediction matches the observed outcome with 99.8% accuracy, with no memorisation regime and no grokking; outcomes split cleanly into instant success and outright failure. This binary behaviour is the limiting case of the capacity-grokking relationship: when the expressible class shrinks to a fixed algebraic object, the question of when a network will grok dissolves into whether it can represent the target at all. A bottleneck ablation connects this extreme to standard networks, tracing a continuous path from representational failure, through memorisation without generalisation, to grokking with a shrinking gap as capacity grows.

Analyses of grokking usually read hidden activations in one basis, most often the Fourier basis modulo $p$. We ask how much the answer to "has the network organised yet?" depends on that choice. The measure FSD$_{\rm gen}$ averages each unit's activation over inputs with the same answer and counts how many units share the same strongest irreducible representation. For modular multiplication we read one answer-averaged profile in three orders: discrete-logarithm order, in which the characters of the multiplicative group become Fourier modes; natural order; and random orders. Across 60 networks at $p\in\{59,97,113\}$, the discrete-log reading reached a fixed threshold before both other readings in 60 of 60 networks (59 of 60 when it had to hold 500 steps). Because untrained networks already score higher in discrete-log order, we also required an equal rise above each reading's untrained level; the discrete-log reading was still first in at least 16 of 20 networks at every prime. At $p=59$ and $p=97$ this happened during the memorisation plateau in 35 of 39 networks; at $p=113$ in 12 of 20. Memorisation preceded grokking in all 179 runs that grokked, by more than any synchronisation signal on average, so a lead over grokking is not by itself evidence of a detected circuit. Synchronisation time is only weakly related to grokking time (pooled $ρ=0.18$). On six finite groups, read with the same measure in every basis, the irreducible-representation reading came before random element orders on all six; with equal-rise thresholds this held in every case on four groups, in 7 of 10 on $\mathbb{Z}_5\times\mathbb{Z}_5$ and in none on $D_{16}$, where about half the profile power lies in one-dimensional representations the measure leaves out. We also correct a conjugate-pair artefact that roughly halves the measure on abelian groups. This version withdraws the causal and predictive claims of v1.