Authors: Yongding Tian, Zaid Al-Ars, Maksim Kitsak, Peter Hofstee
Organizations: Computer Engineering Lab Delft University of Technology Delft, NL · HDL TypeTech Delft, NL · Network and Architecture Service Delft University of Technology Delft, NL · IBM Infrastructure TX, USA
Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed in small algorithmic tasks trained with transformers, its underlying mechanism remains unsettled. In this work, we develop an analysis framework based on mode connectivity and the geometry of low-loss regions. The framework predicts that the standard modular-arithmetic setting does not always produce grokking: under a symmetry-preserving train/validation split, we observe a stable anti-grokking case in which validation performance does not recover. This counterexample challenges several existing correlational explanations of grokking. More broadly, our analysis framework and results further suggest that grokking arises when the low-loss regions induced by the training and validation partitions are misaligned. Once these regions become well aligned, training hyperparameters alone cannot produce grokking and the observed dynamics collapse to either trainable or non-trainable behavior.
Figures & tables
Figure 1: Two-panel illustration of the grokking setup. (a) Illustration of the modular dataset Dmod(x+y,10) . The x-axis and y-axis represent the values of x and y for the mapping function f(x,y)=x+y . Each square indexed by (x,y) contains two triangles: the upper-left triangle gives the value of f(x,y) , and the lower-right triangle indicates whether the sample belongs to the training or validation partition. This small example is shown only for illustration purposes, and our experiments use the larger dataset Dmod(x+y,97) . (b) Illustration of grokking. Validation accuracy increases only long after the model has overfit the training partition. The result is reproduced with Mgrok on Dmod(x+y,97) .
Figure 2: Two-panel illustration of grokking indicators. (a) Slingshot effect: red dots mark spike locations on the training-loss curve. The curve is taken from the experiment in Figure 1 (b) and plotted on a log scale to make the spikes visible. Prior work reports that such spikes are positively correlated with grokking ( Thilak et al., 2022 ) . (b) Fourier components of the neuron-logit map for a well-trained grokked model. The model is obtained at the end of the experiment in Figure 1 (b). Prior work suggests that the formation of this Fourier circuit can be used as a progress measure for grokking ( Nanda et al., 2023 ) .
Figure 3: Geometric illustration of LLR(Mgrok,Dmod(x+y,97)) in the weight space. Motivated by Conjecture 1 , we draw connected paths representing LLR(Dtrain) and LLR(Dval) . Equations 3 and 4 indicate that both overlap and non-overlap regions exist for the grokking setup, so the two low-loss regions are not aligned. The four learning trajectories ( Comprehension , Grokking , Memorization , and Confusion ) correspond to the learning phases identified in ( Liu et al., 2022 ) .
Figure 4: Anti-grokking setup. (a) Dataset partition for the anti-grokking example on Dmod(x+y,97) . Unlike a random partition, samples located at (a,b) and (b,a) always belong to the same partition. The numbers 4705 and 4704 indicate the sample counts in the two partitions. (b) Training Mgrok with the same hyperparameters on the split shown in Figure 4 (a) does not produce grokking: validation accuracy remains low, and validation loss does not decrease. We repeat this experiment with ten random seeds and different splits. The shaded region indicates the standard deviation across runs.
Figure 5: Anti-grokking diagnostics. (a) A Slingshot pattern similar to Figure 2 (a) is observed in the anti-grokking example. Thus, the Slingshot effect is not by itself a faithful explanation of grokking. (b) Compared with Figure 2 (b), several frequencies still have larger components than others, but the concentration is much weaker than in the grokking case.
Figure 6: Low-loss regions defined by the training and validation partitions become increasingly aligned as partition size increases. The overlap region is illustrated as a connected path because mode connectivity is empirically verified to persist for LLR(Mgrok,Dtrain+Dval) in Appendix B .
Figure 7: Phase diagrams for Dmod1997(xy,n) as the operand-domain size n increases. Each cell represents one learning-rate/weight-decay pair and is classified as Comprehension , Grokking , Memorization , or Confusion according to the criteria in Appendix C . As n increases from 97 to 1997 , the number of Memo. cells decreases and eventually vanishes, while Comp. cells emerge and become more frequent. This shift indicates that fewer training trajectories remain in the training-only low-loss region and more directly reach the overlap of the training and validation low-loss regions, suggesting that the two regions become increasingly aligned. No Grokking cells are observed, consistent with the anti-grokking analysis that the effective information carried by the training partition remains below the threshold required for grokking. Appendix D explains why phase-cell counts provide an indirect probe of alignment.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Results of applying LLPF to connect independently trained Mgrok models. The left panel reports training accuracy and loss on the full Dmod(x+y,97) dataset along the discovered path. The right panel reports layer-wise L2 distance along the same path. The training loss remains below 1.5×10−3 , and the L2 distance decreases gradually, supporting mode connectivity for Mgrok on Dmod(x+y,97) . The final point has nearly the same training accuracy and loss as the starting point because both endpoints are obtained through SGD. The experiment is repeated ten times, and the shaded region in the top panel indicates the standard deviation across runs.
Criteria
Phase
Final train. acc. ≥95%
Final val. acc. ≥95%
Val. acc. at ttrain95%≥50%
Comprehension
Yes
Yes
Yes
Grokking
Yes
Yes
No
Memorization
Yes
No
Not applicable
Confusion
No
No
Not applicable
Appendix
Table 1: Empirical definitions of the four learning phases.
Figure 9: Schematic explanation of why phase-cell counts can serve as an indirect probe of low-loss-region alignment. The upper panels show a misaligned case, in which the training and validation low-loss regions have a relatively large training-only portion, and an aligned case, in which their overlap is relatively larger. Each cell in the corresponding phase diagram represents one learning-rate/weight-decay pair and therefore one training trajectory. As the nested dataset grows, the low-loss regions contract while the optimization problem remains otherwise controlled. The resulting changes in the phase-cell distribution provide a hyperparameter-accessible proxy for alignment rather than a literal measurement of low-loss-region volume.
Figure 10: Phase diagrams for the conventional modular datasets Dmod(xy,m) as the modulus m increases. Each cell represents one learning-rate/weight-decay pair and is classified using the criteria in Appendix C . As m increases, Memorization cells decrease and eventually disappear, while Comprehension cells emerge and become more prominent; Grokking cells are mainly observed at intermediate values of m . The resulting shift toward Comprehension and Confusion is qualitatively consistent with increasing low-loss-region alignment. Because the datasets are not nested across m , however, this figure should be interpreted as a cross-dataset comparison rather than direct evidence of monotonic low-loss-region shrinkage.
Figure 11: Comparison of phase diagrams for Dmod(x+y,997) and Dmod(xy,997) . The visible trainable region is larger for Dmod(x+y,997) , indicating a larger low-loss region.
Figure 12: Grokking gradually recovers as the same-partition ratio decreases from 100% toward lower ratios.
Figure 13: Training dynamics for the normal grokking case on Dmod(x+y,97) . With a 40% training partition (a), validation accuracy remains below one and no complete grokking transition is observed. With a 45% training partition (b), training accuracy reaches one first and validation accuracy rises substantially later, producing a clear grokking trajectory.
Grokking case
Anti-grokking case
Train partition (%)
Training coverage
Same-partition ratio (%)
Training coverage
50
0.747
100
0.500
45
0.695
90
0.550
40
0.637
80
0.599
35
0.575
70
0.648
30
0.508
60
0.680
Appendix
Table 2: Symmetry-aware training coverage for the random-partition grokking experiments and the same-partition-ratio anti-grokking experiments. For the random partitions, the reported values are the expected coverages p/m+(1−1/m)(2p−p2) with m=97 and training fraction p ; for the symmetry-controlled partitions, coverage is computed directly from the realized split using the same symmetry-closure definition. In the random-partition setting, the transition from no complete grokking to grokking occurs between 40% and 45% training, corresponding to coverages of 0.637 and 0.695 . In the anti-grokking setting, the corresponding transition occurs between same-partition ratios of 70% and 60% , corresponding to coverages of 0.648 and 0.680 . The close coverage values suggest that symmetry-aware coverage is a more informative comparison across the two experimental controls than the raw partition parameters.
Figure 14: Dataset illustration (left) and phase diagram (right) for Dmod(x+y,997) .
Figure 15: Dataset illustration (left) and phase diagram (right) for Dmod(xy,997) .
Figure 16: Dataset illustration (left) and phase diagram (right) for Dmod(x2+2y2,997) .
Figure 17: Dataset illustration (left) and phase diagram (right) for Dmod(x2+2y2+xy,997) .
Figure 18: Dataset illustration (left) and phase diagram (right) for Dmod(x+y,97) .
Figure 19: Dataset illustration (left) and phase diagram (right) for Dmod(x2+y2,97) .
Figure 20: Dataset illustration (left) and phase diagram (right) for Dmod(2x2+y,97) .
Figure 21: Smaller model with 1 decoder layer, 2 attention heads, and width 64 . The fraction of Comprehension cells is higher than in the baseline model, indicating that the low-loss regions are more aligned.
Figure 22: Baseline model with 2 decoder layers, 4 attention heads per layer, and width 128 .
Figure 23: Larger model with 4 decoder layers, 8 attention heads per layer, and width 256 . Comprehension cells are rare, indicating that the low-loss regions are more misaligned than in the baseline model.
Grokking -- the delayed onset of generalization long after a network has fit its training set - -is usually studied in models too large to read completely and reported from single training runs. We instead study a publicly released ~11,856-parameter Llama-style transformer (Glimmer-1-Base) on modular arithmetic, small enough to enumerate its weights, attention, and full input-output map, and we measure grokking as a multi-seed rate rather than a single outcome. In this fully-tractable regime grokking is a conditional, fragile phase transition. It is gated by training-set coverage, whose threshold tracks output cardinality (the modulus) more than task structure, an ordering that holds above the transition and across a ten-fold change in domain size. Weight decay reproduces the Omnigrok inverted-U at 12K parameters, a positive control on the rate measurement. Grokking also sits on a numerical knife-edge: two perturbations of the floating-point environment -- CPU thread count (reduction order) and CPU-versus-GPU execution -- each flip a minority of same-seed outcomes without a detectable shift in the aggregate rate. Decomposition into sub-task specialists helps chiefly by making coverage cheap rather than by adding supervision. Methodologically, multi-seed control under a fixed numerical environment overturns three dramatic single-run narratives in our own data, each a seed confound. The unit of evidence for grokking must therefore be a multi-seed rate under a pinned numerical environment, checked where possible against a direct reading of the model.
Grokking is the delayed generalisation phenomenon where a transformer trained on modular arithmetic abruptly transitions from near-chance to near-perfect validation accuracy. It has been attributed to a Fourier-based algorithmic circuit, but its timing, causal structure, and controllability remain poorly understood. We introduce the Frequency Synchronization Degree (FSD), a normalised, permutation-tested metric for Fourier circuit synchronisation requiring no prior knowledge of the circuit. Across nine modular addition configurations (five primes, three seeds), FSD reaches its post-grokking level 500 to 3000 steps before grokking (mean lead 1722 steps, every configuration positive, sign-test p approx 0.004), and synchronises before a restricted-logit loss baseline in all nine cases, making it the earliest available predictor. We give direct causal evidence that the inter-phase gap is a regularisation phenomenon: forking training at the FSD-ceiling step and varying weight decay lambda produces monotonically earlier grokking, with delta-t proportional to 1/lambda. This law replicates across three primes (R-squared 0.89 to 0.99 on seed-averaged delta-t); per-run R-squared is unstable due to the chaotic transition, so we report error bars rather than single runs. Grokking occurs at a near-constant memorisation norm across lambda, grounding the constant in a threshold mechanism. This is not an artefact of applying a Fourier detector to a Fourier circuit: on the non-abelian group S5, a basis-faithful generalisation of FSD precedes grokking on all six seeds, while the original Fourier FSD does not. Using the FSD ceiling to schedule a weight-decay increase also accelerates grokking over a fixed schedule without destabilising training. An attention-only variant groks with a strong FSD precursor while an MLP-only model never groks.
On modular arithmetic, a network's embedding keeps compressing for tens of thousands of steps after it has already generalized. Reading effective rank at the grokking transition overstates the converged value by 3-5x on an MLP, and by 1.3-1.5x on a transformer trained to convergence; on the MLP it also erases which cells compress at all. Compression lags the accuracy transition by an amount on the order of the time-to-grok, at least 10,000 steps, rather than coinciding with it. A one-variable ablation shows what sets the lag size: adding LayerNorm to an otherwise identical transformer moves the fraction of compression done by the grok step from 0.87 to 0.25, and a pre-registered control rules out scale invariance as the mechanism. We package this as an audit that separates onset from compression, flags censoring, excludes boundary cells that never fully generalize, and checks that the reference floor has plateaued, with an adversarial suite that caught a false-confidence bug in our own branch. A secondary, MLP-specific depth law linking norm budget to converged floor fails a generality test on a transformer and flips sign under free weight decay. Code and the toolkit are released.