cs.LGNov 2, 2026

Dissecting Geometric Constraints in Grokking: Interaction-Dependent Acceleration in Modular Arithmetic

Authors: Zachary A. Hampton, Wasiq Khan, Bilal Khan

Organizations: School of Computer Science and Engineering, California State University, San Bernardino · School of Computer Science and Mathematics, Liverpool John Moores University, UK

Abstract

Spherical transformer configurations can accelerate grokking through simultaneous modifications of several model properties. The present study investigated the contributions of these modifications in the published Sphere+Fourier configuration using 960 controlled modular-arithmetic runs. A complete factorial analysis varied residual projection, readout weight normalization, logit scale, and weight decay. Across four matched settings with weight decay, the successful-run mean crossing times without projection were 8-37 times those observed with projection. However, residual projection was insufficient to induce grokking without additional interventions. Various other unprojected configurations also generalized. Since projection altered the magnitude entering the readout, these comparisons could not separate geometry from output scale. A separate 100-run fixed-horizon study demonstrated faster first crossing for the complete configuration across five train/test partitions. Faster crossing did not necessarily correspond to greater persistence over extended training. Control experiments demonstrated rapid grokking without Fourier initialization. Task and depth sweeps also identified conditions in which the configuration failed to reach the criterion by 50K epochs. Accordingly, the observed acceleration depended on interactions among the components and did not extend uniformly across the tested tasks or depths.

Explore similar work

Jun 11, 2026

Learned Structure Shows Up Earlier in the Task's Own Basis: A Controlled Look at a Synchronisation Precursor in Grokking

Analyses of grokking usually read hidden activations in one basis, most often the Fourier basis modulo pp. We ask how much the answer to "has the network organised yet?" depends on that choice. The measure FSDgen_{\rm gen} averages each unit's activation over inputs with the same answer and counts how many units share the same strongest irreducible representation. For modular multiplication we read one answer-averaged profile in three orders: discrete-logarithm order, in which the characters of the multiplicative group become Fourier modes; natural order; and random orders. Across 60 networks at p∈{59,97,113}p\in\{59,97,113\}, the discrete-log reading reached a fixed threshold before both other readings in 60 of 60 networks (59 of 60 when it had to hold 500 steps). Because untrained networks already score higher in discrete-log order, we also required an equal rise above each reading's untrained level; the discrete-log reading was still first in at least 16 of 20 networks at every prime. At p=59p=59 and p=97p=97 this happened during the memorisation plateau in 35 of 39 networks; at p=113p=113 in 12 of 20. Memorisation preceded grokking in all 179 runs that grokked, by more than any synchronisation signal on average, so a lead over grokking is not by itself evidence of a detected circuit. Synchronisation time is only weakly related to grokking time (pooled ρ=0.18ρ=0.18). On six finite groups, read with the same measure in every basis, the irreducible-representation reading came before random element orders on all six; with equal-rise thresholds this held in every case on four groups, in 7 of 10 on Z5×Z5\mathbb{Z}_5\times\mathbb{Z}_5 and in none on D16D_{16}, where about half the profile power lies in one-dimensional representations the measure leaves out. We also correct a conjugate-pair artefact that roughly halves the measure on abelian groups. This version withdraws the causal and predictive claims of v1.
Feb 18, 2026

Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking

Grokking -- the delayed transition from memorization to generalization in small algorithmic tasks -- remains poorly understood. We present a geometric analysis of optimization dynamics in transformers trained on modular arithmetic. PCA of attention weight trajectories reveals that training evolves predominantly within a low-dimensional execution subspace, with a single principal component capturing 68-83% of trajectory variance. To probe loss-landscape geometry, we measure commutator defects -- the non-commutativity of successive gradient steps -- and project them onto this learned subspace. We find that curvature grows sharply in directions orthogonal to the execution subspace while the trajectory remains largely confined to it. Importantly, curvature growth consistently precedes generalization across learning rates and hyperparameter regimes, with the lead time obeying a power law in the grokking timescale. Causal intervention experiments show that motion along the learned subspace is necessary for grokking, while artificially increasing curvature is insufficient. Together, these results support a geometric account in which grokking reflects escape from a metastable regime characterized by low-dimensional confinement and transverse curvature accumulation. All findings replicate across this learning-rate range, a qualitatively different slow regime (lr=5e-5, wd=0.1, 3 layers), and three random seeds, though alignment dynamics differ quantitatively between regimes. Causal intervention experiments establish that orthogonal gradient flow is necessary but not sufficient for grokking: suppressing it prevents generalization with a monotonic dose-response across four operations, while artificially boosting curvature defects has no effect.
Feb 19, 2026

Early-Warning Signals of Grokking via Loss-Landscape Geometry

Grokking -- the abrupt transition from memorization to generalization after prolonged training -- has been linked to confinement on low-dimensional execution manifolds in modular arithmetic. Whether this mechanism extends beyond arithmetic remains open. We study two sequence-learning benchmarks: SCAN compositional generalization and Dyck-1 depth prediction. Across both tasks and a wide range of learning rates, the commutator defect -- a curvature measure derived from non-commuting gradient updates -- rises well before generalization, with lead times following a superlinear power law (alpha approximately 1.18 for SCAN, approximately 1.13 for Dyck), consistent with prior results on modular arithmetic. Weight-space PCA reveals that spectral concentration is not a universal precursor; the commutator defect is. Causal interventions demonstrate a mechanistic role: amplifying non-commutativity accelerates grokking (roughly 32% on SCAN, roughly 50% on Dyck), while suppressing orthogonal gradient flow delays or prevents it. The three task families form a spectrum of causal sensitivity -- modular arithmetic is rigid, Dyck is responsive, SCAN is intermediate -- yet suppression delays or prevents grokking in all cases, establishing necessity as a universal finding. These results identify the commutator defect as a robust, architecture-agnostic, causally implicated early-warning signal for delayed generalization in transformers.