A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization

Shalima Binta Manir; Anamika Paul Rupa

cs.LGMar 26, 2026

A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization

Authors: Shalima Binta Manir, Anamika Paul Rupa

Organizations: Howard University

Abstract

Grokking the delayed transition from memorization to generalization in neural networks remains poorly understood, in part because prior empirical studies confound the roles of architecture, optimization, and regularization. We present a controlled study that systematically disentangles these factors on modular addition (mod 97), with matched and carefully tuned training regimes across models. Our central finding is that grokking dynamics are not primarily determined by architecture, but by interactions between optimization stability and regularization. Specifically, we show: (1) depth has a non-monotonic effect, with depth-4 MLPs consistently failing to grok while depth-8 residual networks recover generalization, demonstrating that depth requires architectural stabilization; (2) the apparent gap between Transformers and MLPs largely disappears (1.11 $\times$ delay) under matched hyperparameters, indicating that previously reported differences are largely due to optimizer and regularization confounds; (3) activation function effects are regime-dependent, with GELU up to 4.3 $\times$ faster than ReLU only when regularization permits memorization; and (4) weight decay is the dominant control parameter, exhibiting a narrow ``Goldilocks'' regime in which grokking occurs, while too little or too much prevents generalization. Across 3--5 seeds per configuration, these results provide a unified empirical account of grokking as an interaction-driven phenomenon. Our findings challenge architecture-centric interpretations and clarify how optimization and regularization jointly govern delayed generalization.

Figures & tables

Hyperparameter	Value
Optimizer	SGD (MLP); AdamW (Transformer)
Learning rate	$10^{-2}$ (SGD); $10^{-3}$ (AdamW)
Momentum	0.9 (SGD only)
Weight decay (default)	$2\times 10^{-3}$ (SGD); $1.0$ (AdamW)
Gradient clip norm	1.0 (depth $\geq$ 8)
Batch size	Full train set

Table 1: Default training hyperparameters.

Figure 1: Canonical grokking on modular addition mod 97 (direct experiment output). Train accuracy (blue) reaches 100% at step $T_{\text{train}}=1{,}000$ , while test accuracy (orange) remains near chance until a sharp transition at step $T_{\text{grok}}=33{,}000$ (grokking delay $=32{,}000$ steps). Test accuracy stabilizes at $\approx$ 99.35% and continues to slowly improve through step 400,000. The dashed line marks the 99% threshold.

Depth	Seed	$T_{\text{train}}$	$T_{\text{grok}}$	$\Delta T$
2	0	2,000	34,000	32,000
2	1	2,000	22,000	20,000
2	2	2,000	28,000	26,000
2	3	2,000	—	DNF
2	4	2,000	212,000	210,000
Depth 2 mean (4 seeds)				$72{,}000\pm 85{,}536$

Table 2: Grokking results by depth.

\Delta T=

grokking delay (steps). DNF

=

did not grok within budget. Mean excludes DNF seeds.

Figure 2: H1: Depth-8 residual MLP grokking delay across 5 seeds. Seeds 3 and 4 (orange) did not grok within the 400,000-step budget (shown as 0); seeds 0–2 grokked with a mean $\Delta T=33{,}333\pm 12{,}858$ steps (dashed line).

Architecture	Seed	$T_{\text{train}}$	$T_{\text{grok}}$	$\Delta T$
Transformer	0	4,000	46,000	42,000
Transformer	1	4,000	36,000	32,000
Transformer	2	4,000	80,000	76,000
Transformer	3	2,000	32,000	30,000
Transformer	4	2,000	76,000	74,000
Transformer mean				$50{,}800\pm 22{,}565$

Table 3: H2 Architecture comparison over 5 seeds (corrected configs). Transformer uses AdamW (

\lambda=1.0

, lr

=10^{-3}

); MLP uses SGD (

\lambda=5\times 10^{-4}

, lr

=3\times 10^{-2}

Figure 3: H2: Transformer vs. MLP grokking comparison over 5 seeds (corrected hyperparameter configs). (a) Per-seed grokking delays $\Delta T=T_{\text{test}}-T_{\text{train}}$ ; dashed lines are per-architecture means. (b) Mean delay $\pm$ std: MLP achieves $45.6\text{k}\pm 5.6\text{k}$ vs. Transformer $50.8\text{k}\pm 22.6\text{k}$ , a 1.11 $\times$ difference with $4.1\times$ higher variance. (c) $T_{\text{train}}$ vs. $T_{\text{test}}$ scatter: both architectures memorize rapidly but diverge in generalization onset.

Sweep	Activation	Grokked (of 5)	Mean $\Delta T$	Std $\Delta T$
A	GELU	0	— (DNF all)	—
A	ReLU	2	266,000	96,167
A	Tanh	3	242,667	134,288
B	GELU	5	45,600	5,550
B	ReLU	5	196,800	89,728
B	Tanh	2	188,000	36,770

Table 4: Grokking results by activation function at two hyperparameter configurations. Sweep A: lr

=10^{-2}

\lambda=2\times 10^{-3}

, width 256. Sweep B: lr

=3\times 10^{-2}

\lambda=5\times 10^{-4}

, width 512. All depth-4 MLPs, SGD, 5 seeds, 400,000 steps. DNF = did not grok within budget. Mean excludes DNF seeds.

Figure 4: H3: Activation function comparison at two hyperparameter configurations. Left (Sweep A) : original H3 config (lr $=10^{-2}$ , $\lambda=2\times 10^{-3}$ , width 256). GELU fails entirely (0/5 seeds); ReLU and Tanh grok in 2/5 and 3/5 seeds with large delays. Right (Sweep B) : H2 MLP baseline config (lr $=3\times 10^{-2}$ , $\lambda=5\times 10^{-4}$ , width 512). GELU dominates with 5/5 seeds at 45.6k steps—4.32 $\times$ faster than ReLU. The reversal between sweeps demonstrates an activation–weight decay interaction: GELU’s advantage only emerges when regularization is light enough to permit memorization.

$\lambda$	Grokked/3	Mean $\Delta T$	Std
$10^{-5}$	0/3	DNF ${}_{\text{te}}$ : memorizes, no grok
$5\times 10^{-5}$	1/3	388,000	—
$10^{-4}$	3/3	220,000	36,056
$5\times 10^{-4}$	3/3	45,333	7,572
$10^{-3}$	3/3	25,333	5,033
$2\times 10^{-3}$	0/3	DNF ${}_{\text{tr}}$ : cannot memorize

Table 5: H4a: MLP weight decay sweep (depth-4 GELU, width 512, SGD, lr

=3\times 10^{-2}

, 3 seeds, 400,000 steps). DNF

{}_{\text{tr}}

= failed to memorize within 60,000 steps. DNF

{}_{\text{te}}

= memorized but did not grok within budget.

$\lambda$	Grokked/3	Mean $\Delta T$	Std
$0.01$	0/3	DNF ${}_{\text{te}}$ : no grok in 400,000 steps
$0.1$	3/3	202,667	141,454
$0.5$	3/3	35,333	17,010
$1.0$	3/3	44,000	12,166
$2.0$	3/3	65,333	42,771
$5.0$	3/3	24,000	10,583

Table 6: H4b: Transformer weight decay sweep (width 512, 4 heads, AdamW, lr

=10^{-3}

, 3 seeds, 400,000 steps). DNF

{}_{\text{te}}

= memorized but did not grok within budget.

Figure 5: H4: Weight decay sweeps for MLP (SGD) and Transformer (AdamW), plus fair comparison at optimal $\lambda$ each (5 seeds). (a) MLP $\lambda$ sweep: optimal at $\lambda=10^{-3}$ ; $\lambda\geq 2\times 10^{-3}$ prevents memorization entirely. (b) Transformer $\lambda$ sweep: optimal at $\lambda=5.0$ , requiring 5,000 $\times$ stronger regularization than the MLP. (c) Fair comparison at optimal $\lambda$ each (5 seeds): MLP achieves $26.8\text{k}\pm 6.4\text{k}$ vs. Transformer $50.8\text{k}\pm 38.7\text{k}$ , a 1.90 $\times$ gap—the definitive architecture comparison superseding H2’s 1.11 $\times$ .

Configuration	Delay (steps)	$\\|W\\|$ at init	$\\|W\\|$ at grokking
Depth-2 GELU (baseline)	32,000	0.372	0.0695
MLP GELU (H2 config)	54,000	0.209	0.0236
MLP GELU (optimal $\lambda$ )	26,000	0.209	0.0225
Transformer ( $\lambda=1.0$ )	58,000	0.126	0.0254
Transformer ( $\lambda=5.0$ )	12,000	0.120	0.0160
MLP ReLU (Sweep-B)	180,000	0.209	0.0219

Table 7: Weight norm at grokking step across six configurations. RMS norm is computed over all parameters. Depth-2 uses width 256; all others use width 512.

Figure 6: Weight norm dynamics and training curves across six configurations. (a) Test accuracy over training; vertical dotted lines mark grokking steps. (b) RMS weight norm over training; dots mark the weight norm at each grokking step. All width-512 models converge to $\|W\|\approx 0.022$ at grokking, regardless of architecture or activation. (c) Weight norm at grokking step: the depth-2 baseline (width 256) is an outlier; the five width-512 models cluster tightly (CV $=14.5\%$ ). (d–f) Individual train/test accuracy curves for the three MLP configurations, showing the characteristic memorization plateau followed by abrupt generalization.

Model	Top-5 frequencies	Top-5 conc.	Grokking delay
MLP (GELU)	6, 12, 21, 37, 46	74.7%	26,000
MLP (ReLU)	6, 7, 11, 16, 21	75.6%	80,000
Transformer	0, 16, 29, 32, 34	98.5%	12,000

Table 8: Fourier concentration of learned embeddings post-grokking. Top-5 concentration = fraction of total Fourier energy in the five highest-energy frequency components.

Figure 7: Fourier analysis of learned embeddings post-grokking. Top row (a–c): Normalised Fourier energy spectrum of the embedding matrix $W_{E}$ for each model. Highlighted bars (colour) show the top-5 frequencies. The Transformer (c) concentrates 98.5% of energy in 5 frequencies; MLPs concentrate $\sim$ 75%. Bottom row (d–f): Each residue’s embedding projected onto the two highest-energy Fourier directions, coloured by residue value. The circular/elliptical structure in all three panels confirms that the learned representations encode modular arithmetic via sinusoidal patterns Nanda et al. (2023) .

Arch.	Seed	$T_{\text{train}}$	$T_{\text{grok}}$	$\Delta T$
MLP	0	2,000	24,000	22,000
MLP	1	2,000	28,000	26,000
MLP	2	2,000	26,000	24,000
MLP	3	2,000	24,000	22,000
MLP	4	2,000	32,000	30,000
MLP mean				$24{,}800\pm 3{,}347$

Table 9: H2-mul: Architecture comparison on

(a\times b)\bmod 97

. Both architectures use optimal

\lambda

from H4 (MLP:

\lambda=10^{-3}

; Transformer:

\lambda=5.0

Activation	Grokked/5	Mean $\Delta T$	Std
GELU	5/5	24,400	3,578
ReLU	5/5	93,200	47,045
Tanh	3/5	102,667	19,732

Table 10: H3-mul: Activation comparison on

(a\times b)\bmod 97

(depth-4 MLP, width 512, SGD,

\lambda=10^{-3}

, 5 seeds).

Figure 8: Modular multiplication $(a\times b)\bmod 97$ results. (a) Per-seed grokking delays for MLP vs. Transformer; ratio = 1.42 $\times$ (vs. 1.90 $\times$ on addition). (b) Activation comparison: GELU dominates at 3.82 $\times$ faster than ReLU; Tanh grokks in 3/5 seeds. (c) Direct comparison of addition vs. multiplication for each condition. Both architectures and GELU grokk faster on multiplication; the ReLU gap is preserved.

Explore similar work

Sep 9, 2026

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

Neural networks trained past memorization frequently undergo a delayed transition to generalization, a phenomenon known as grokking. Despite theoretical progress on why this transition occurs, the quantitative structure of when it occurs in hyperparameter space remains uncharacterized. We map the memorization-to-generalization boundary across 384 configurations of two-hidden-layer MLPs on modular arithmetic, fitting a power-law scaling relation for generalization onset time:

T_{\mathrm{grok}} \propto H^{-0.27}\, D^{-2.04}\, η^{-0.50}\, λ^{-0.64}

(

R^2 = 0.732

;

0.821

with interactions). The exponent hierarchy reveals that data complexity (

D^{-2.04}

) is the dominant driver of regime transition, not model capacity (

H^{-0.27}

): doubling data accelerates generalization by

{\sim}4\times

, while doubling width yields only

{\sim}1.2\times

. A sharp phase boundary at weight decay

λ\gtrsim 1.0

separates grokking from non-grokking configurations, and weight norm trajectories show monotonic compression during the transition, consistent with implicit regularization selecting low-complexity solutions. These results provide a quantitative foundation for predicting and controlling regime transitions in overparameterized networks.

Anish Kataria

Princeton University, Princeton, NJ, USA.

Multilayer Perceptrons Neural Network Memorization Neural Network Training Dynamics Neural Scaling Laws Grokking

Ask PDF

Jul 13, 2026

How to Tame Grokking: Representation Geometry as a Control Signal

Grokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization. Despite extensive recent study, the factors influencing the emergence and timing of grokking remain incompletely understood. We investigate the relationship between representation geometry and delayed generalization. We find that dimensionality collapse consistently precedes the onset of grokking in all evaluated settings. Motivated by these observations, we introduce Geometric Dimensionality Regularization (GeomDR), a simple spectral regularizer that modifies the effective dimensionality of hidden representations during training. Across modular addition, modular division, and permutation composition tasks, GeomDR consistently alters grokking dynamics and can substantially accelerate the onset of generalization depending on the intervention schedule and target dimensionality. In several settings, grokking is accelerated by up to 52 times relative to standard AdamW training. Similar qualitative effects are observed in both multilayer perceptrons and transformers. Together, these results suggest that representation geometry can serve as an effective control signal for grokking and provide evidence that geometric interventions offer a practical approach for studying and influencing delayed generalization in neural networks.

Maksim A Kazanskii

Independent Researcher

Neural Network Generalization Spectral Regularization Representation Collapse Neural Representation Geometry Neural Network Training Dynamics Grokking

Ask PDF

Mar 5, 2026

The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

Grokking -- the sudden generalisation that appears long after a model has perfectly memorised its training data -- has been widely observed but lacks a quantitative theory explaining the length of the delay. We show that grokking is a norm-driven representational phase transition in regularised training dynamics, and establish the Norm-Separation Delay Law:

T_{\mathrm{grok}} - T_{\mathrm{mem}} = Θ(γ_{\mathrm{eff}}^{-1} \log(\|θ_{\mathrm{mem}}\|^2 / \|θ_{\mathrm{post}}\|^2))

, where

γ_{\mathrm{eff}}

is the optimiser's effective contraction rate (

γ_{\mathrm{eff}} = ηλ

for SGD,

γ_{\mathrm{eff}} \ge ηλ

for AdamW). The upper bound follows from a discrete Lyapunov contraction argument; the matching lower bound from dynamical constraints of regularised first-order optimisation. Across 293 training runs spanning modular addition, modular multiplication, and sparse parity, we confirm three falsifiable predictions: inverse scaling with weight decay (

R^2 = 0.97

), inverse scaling with learning rate (

R^2 = 0.92

), and logarithmic dependence on the norm ratio (Pearson

r = 0.91

). A fourth finding reveals that grokking requires an optimiser capable of decoupling memorisation from contraction: SGD fails entirely at the same hyperparameters where AdamW reliably groks. These results reframe grokking not as a mysterious optimisation artefact but as a predictable consequence of norm separation between competing interpolating representations. We further derive a practical three-input algorithm that predicts grokking delay at memorisation time with 34.6% mean absolute error (bootstrap 95% CI [30.0%, 39.4%],

N=60

seeds), enabling principled early stopping.

Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1

H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam

Grokking Neural Network Generalization Gradient Descent Dynamics Neural Network Memorization

Ask PDF