Grokking

Follow this topic Ask about this topic

Grokking studies why a model can fit its training data while performing poorly on new examples, then later generalize, often on tasks such as modular arithmetic, parity, and formal-language recognition. Researchers explain this delay through theories of weight decay, parameter norms, optimization noise, and competing memorization and generalization timescales, alongside analyses of learned circuits and representations using Fourier features, symmetry, and spectral geometry. Experiments also test ways to predict or control the transition, including geometric diagnostics, regularization, architectural changes, and structure-specific priors. A central open question is whether these findings reflect one general mechanism or several task- and optimizer-dependent processes; signals that predict grokking do not always establish its cause.

Latest papers

77 papers5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Jun 11, 2026

The Weight Norm Sets the Grokking Timescale: A Causal Delay Law

Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data. Whether the weight norm causes this delay is disputed: some studies report a critical norm at the transition, others observe grokking with no fixed norm at all. We settle this by intervening on the norm during training rather than only observing it. Under free training with weight decay, networks grok when the weight norm reaches a value Wc that varies little across seeds and learning rates (CV 1 to 2 percent) and grows with the modular base as a power law. When we instead clamp the norm to a fixed multiple rho of Wc and hold it there, the network still groks, but the delay follows T_grok proportional to exp(alpha rho). One exponent, alpha near 7.5, fits this delay across four moduli (R^2 = 0.996). Over the swept ranges the held norm moves the delay by about 19x and the learning rate by only about 2x, and holding the norm above Wc slows grokking rather than preventing it. A final LayerNorm removes the dependence by decoupling weight scale from the network function; without it the exponential law returns. This pinned-norm delay is the exponential counterpart to the logarithmic delay predicted for a freely contracting norm.

Truong Xuan Khanh, Doan Hoang Viet, Luu Duc Trung +1

H&K Research Studio / Clevix LLC, Hanoi, Vietnam · Bac A Bank, Hanoi · Banking Academy of Vietnam

Neural Network Generalization Neural Network Training Dynamics Grokking

Jun 11, 2026

Learned Structure Shows Up Earlier in the Task's Own Basis: A Controlled Look at a Synchronisation Precursor in Grokking

Analyses of grokking usually read hidden activations in one basis, most often the Fourier basis modulo

p

. We ask how much the answer to "has the network organised yet?" depends on that choice. The measure FSD

_{\rm gen}

averages each unit's activation over inputs with the same answer and counts how many units share the same strongest irreducible representation. For modular multiplication we read one answer-averaged profile in three orders: discrete-logarithm order, in which the characters of the multiplicative group become Fourier modes; natural order; and random orders. Across 60 networks at

p\in\{59,97,113\}

, the discrete-log reading reached a fixed threshold before both other readings in 60 of 60 networks (59 of 60 when it had to hold 500 steps). Because untrained networks already score higher in discrete-log order, we also required an equal rise above each reading's untrained level; the discrete-log reading was still first in at least 16 of 20 networks at every prime. At

p=59

and

p=97

this happened during the memorisation plateau in 35 of 39 networks; at

p=113

in 12 of 20. Memorisation preceded grokking in all 179 runs that grokked, by more than any synchronisation signal on average, so a lead over grokking is not by itself evidence of a detected circuit. Synchronisation time is only weakly related to grokking time (pooled

ρ=0.18

). On six finite groups, read with the same measure in every basis, the irreducible-representation reading came before random element orders on all six; with equal-rise thresholds this held in every case on four groups, in 7 of 10 on

\mathbb{Z}_5\times\mathbb{Z}_5

and in none on

D_{16}

, where about half the profile power lies in one-dimensional representations the measure leaves out. We also correct a conjugate-pair artefact that roughly halves the measure on abelian groups. This version withdraws the causal and predictive claims of v1.

Achyuthan Sivasankar

New York University

Weight Decay Frequency-Domain Feature Learning Neural Network Training Dynamics Grokking

Jun 8, 2026

Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic

While neural collapse (NC) predicts that a

K

-class-balanced classifier should organize terminal representations as a

(K-1)

-dimensional simplex equiangular tight frame (ETF), modular addition consistently enters a different regime: networks compress to a two-dimensional cyclic geometry in which both classifier weights and token embeddings lie on circles. We refine the explanation of this phenomenon in three directions. First, we formalize a layerwise non-uniform training mechanism: downstream classifier weights are driven by dense cross-entropy gradients into a rank-2 equiangular configuration before upstream embeddings fully reorganize, and once this classifier plane forms, backpropagated feature gradients constrain embedding motion to the same plane while weight decay suppresses orthogonal components. Second, after this subspace locking, the induced in-plane dynamics admit an entropy-regularized transport interpretation on

S^1

; combined with modular-addition labels, this reduces embedding formation to phase alignment, whose minimizers are single-frequency characters of

\mathbb{Z}/P\mathbb{Z}

and hence equal-angle points on a circle. Third, we quantify why this solution prevails over NC: a simplex ETF gains only an

O(1)

advantage in cross-entropy, whereas the cyclic rank-2 solution enjoys a

Θ(K)

advantage under Schatten or weight-decay surrogates, yielding a critical threshold

λ_{\mathrm{crit}} = Θ(1/K)

. Our results explain both why classifier weights move first and why embeddings subsequently align with them, showing that grokking on modular arithmetic is governed not by maximal separation alone but by a task-structured trade-off between separation, symmetry, and complexity.

Hu Tan, Kuo Gai, Shihua Zhang

Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; and School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China · Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China; and Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Hangzhou 310024, China

Neural Representation Geometry Classification Modular Addition Neural Collapse Neural Network Training Dynamics Modular Arithmetic+1

Jun 4, 2026

Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction

Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales. We formalize this phenomenon by separating the fast decay of the classification loss from the slower simplification of the learned representation, and we call the resulting pair of stopping times two training clocks. For deep linear networks, we show that a post-margin gap-growth or one-step tail-contraction condition reduces the cross-entropy loss to level epsilon on a logarithmic time scale. In contrast, when layerwise weight decay is present, the induced regularization on the end-to-end map can be expressed as a Schatten-type penalty; under a sharp late-time Kurdyka-Lojasiewicz tail, this structural energy closes on a polynomial time scale. The two clocks, therefore, separate fitting from representation simplification. We then explain how the same mechanism can appear in ReLU MLPs. In regions where the activation patterns on the training set remain fixed, the network reduces to a linear model in the active coordinates. In a two-layer ReLU embedding model, chain-rule estimates further show that the classifier head can receive larger effective gradients than the embedding block under controlled downstream norms. This supports a two-stage mechanism in which the classifier fits first, while the representation continues to simplify later. We use modular addition as the main experimental setting. The deep linear theory provides the rigorous core of the analysis. But the ReLU results are formulated as conditional reductions that account for empirical behavior without claiming a global proof for nonlinear training dynamics.

Hu Tan, Kuo Gai, Shihua Zhang

1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1

ReLU Neural Networks Deep Linear Networks Neural Network Training Dynamics Grokking

Jun 3, 2026

Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View

Modern Transformer architectures frequently employ normalization mechanisms such as RMSNorm and Query-Key Normalization, making parts of the model approximately scale-invariant with respect to weight magnitudes. In this regime, standard Frobenius-norm weight decay acts purely along the radial direction of the weight space and cannot directly simplify the function represented by the normalized layer. We study grokking in small algorithmic tasks through this lens and propose Low-Rank Decay (LRD), a nuclear-norm-like spectral regularizer whose subgradient -- the polar factor

UV^\top

-- retains a tangential component even in the scale-invariant setting. This distinction has a concrete dynamical consequence: after the model memorizes the training set and task gradients vanish, L2 decay can no longer reshape the weight spectrum, whereas LRD continues to compress singular values in an

\ell_1

-like fashion. On modular arithmetic tasks, we find that LRD induces rapid effective-rank collapse in Query/Key matrices and expands the data-fraction boundary at which delayed generalization (grokking) occurs. We further provide a spectral-geometric interpretation through the ``needle-to-fan'' expansion of the nuclear-norm subdifferential near low-rank strata.

Mingyu Li

Beijing Normal University

Spectral Regularization Weight Decay Transformer Nuclear Norm Minimization Grokking

May 26, 2026

Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent

Training loss and accuracy are the standard signals used to monitor generalization during deep neural network training. Two well-documented phenomena complicate this picture: in grokking, train loss falls rapidly while test performance improves abruptly only after a long delay; in epoch-wise double descent, train loss decreases monotonically while test loss or error rises and falls. Existing accounts are often task-specific, and a task-agnostic analysis framework for diagnosing and explaining these phenomena across realistic tasks and architectures is missing. We address this challenge by analyzing two competing processes that underlie learning dynamics: representation learning in the encoder and readout calibration in the final classifier. Using tools from representational geometry, neural tangent kernels, and linear probing, we show that both processes are active throughout training, with the fluctuations of their relative speed giving rise to seemingly anomalous generalization dynamics. Applying the representation-readout decomposition to grokking across a wide range of tasks and architectures, we find that the readout is train-biased before grokking onset, and representation learning is gradual but not absent, contrary to the lazy-to-rich account. The framework further provides diagnostic signatures distinguishing spurious from genuine generalization: in a previously reported MNIST grokking example and an epoch-wise double descent example, apparent delayed or non-monotone generalization is shown to arise from representation degradation and readout misalignment induced by non-standard training recipes. Together, these results establish the representation-readout decomposition as a top-down framework for understanding learning dynamics and revealing underlying algorithms for interpretability research.

Chi-Ning Chou, Oscar Uzdelewicz, Neng-Chun Chiu +2

Center for Computational Neuroscience, Flatiron Institute, New York, NY, USA. · Department of Physics, Harvard University, Cambridge, MA, USA. · Kempner Institute, Harvard University, Allston, MA, USA. +1

Double Descent Neural Network Generalization Representation Learning Neural Network Training Dynamics Grokking

May 19, 2026

Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics

Transformers trained on modular arithmetic exhibit sharp transitions between memorization, generalization, and collapse. We show that weight decay acts as a scalar empirical control parameter for these regimes, and introduce two cheap online diagnostics, mean pairwise attention-head cosine similarity and entropy standard deviation, that track training dynamics from attention activations alone and complement loss-landscape diagnostics at lower compute cost. Across eleven experimental conditions and three model scales (0.82M to 85M parameters), the weight-decay axis separates memorization, developmental grokking, and collapse. A near-transition logistic fit localizes the memorization-to-developmental boundary at

λ_c=0.0158

(95% CI [0.0109, 0.0200], N=210); a power-law fit gives an empirical exponent

ν=0.757

(CI [0.725, 0.799]). Reference exponents

ν=1/2

and 3D Ising

ν\approx 0.63

lie outside this empirical CI under our four-bin grid, so we report

ν

as empirical and defer universality-class identification to denser finite-size-scaling work. A horizon-matched multi-task replication (n=280, four modular operations) preserves the weight-decay control pattern; a paired attention-head re-initialization experiment at

λ=0.05

changes Phase-2 amplitude (Cohen's

d=-1.190

, n=10,

p_t=4.5 \times 10^{-3}

), while matched weight-norm clipping does not. Three cross-architecture probes (4L MLP, 4L LSTM, and 4L Mamba; each n=70) replicate the weight-decay-controlled transition with architecture-specific

λ_c

values. Main diagnostic claims are scoped to modular arithmetic in small transformer attention models; the non-attention experiments are scope probes, and architecture-wide, language-model, and universality-class claims are out of scope.

Lucky Verma

Independent Researcher

Weight Decay Neural Network Memorization Phase Transitions Neural Network Training Dynamics Grokking Transformer Training Dynamics

May 15, 2026

Grokking as Structural Inference: Transformers Need Bayesian Lottery Tickets

Why does a Transformer that has memorized its training set wait thousands of steps before it generalizes? Existing accounts locate this delay in norm minimization, feature emergence, or the late discovery of sparse subnetworks. These explanations capture important parts of the transition, but ignore a constraint unique to attention-based models: if attention discards an informative token, no bounded downstream computation can recover it. We formalize attention as an implicit Bayesian posterior over the task dependency graph and prove that generalization requires two separable conditions: a familiar Goldilocks bound on MLP capacity, coinciding with norm-based theories of grokking, and a novel Bayesian structural condition requiring attention to place sufficient mass on every informative token. This decoupling explains delayed generalization as delayed structural inference. Early in training, the MLP memorizes through unaligned features, drives the cross-entropy loss near zero, and thereby starves attention of structural gradient. Weight decay must then erode memorization before the missing graph becomes learnable, yielding the known inverse-weight-decay delay, which we derive as a structural waiting time. We then prove that this explaining-away delay can be bypassed by a KL-based structural intervention, yielding an inverse-intervention-strength scaling law for the grokking time. Experiments on algorithmic sequence tasks isolate structure from capacity and show that this Bayesian ticket matches or outperforms lottery-ticket transfer.

Kai Hidajat, Solden Stoll, Joseph An

Department of Computer Science University of Washington Seattle, WA 98195

Neural Network Generalization Lottery Ticket Hypothesis Transformer Self-Attention Attention Mechanisms Grokking

May 14, 2026

Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning

Critical-data-size accounts of grokking suggest a natural post-threshold intuition: once training data is sufficient to identify the underlying rule, additional data should accelerate validation convergence. We show that this intuition can fail in a controlled structured-output task. In Needleman--Wunsch (NW) matrix generation, small Transformers reach high validation exact-match accuracy fastest at an intermediate dataset size, not at the largest one. Past this dataset-size sweet spot, generalization remains achievable but requires more gradient updates. Conversely, in the regime where partial validation competence first appears, larger datasets can require fewer updates to reach high training accuracy, suggesting that emerging rule structure can accelerate fitting beyond example-wise memorization. A multiplication baseline does not show the same post-threshold slowdown. These results separate the critical data size for the onset of generalization from the dataset size that optimizes update-based convergence, and identify structured-output tasks where learning the rule and completing exact-fitting can diverge.

Shin So, Kyelim Lee, Albert No

Yonsei University

Neural Network Generalization Neural Network Training Dynamics Grokking Algorithmic Generalization

May 13, 2026

First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation

We give the first quantitative prediction of grokking delay under AdamW. Treating the delay as a first-passage time, we derive a closed-form law T_grok - T_mem = (1 / 2 kappa_LL eta lambda) log(V_mem / V_star), where V_t = ||theta_t||^2 is the squared parameter norm, V_star is an architecture-dependent threshold, and kappa_LL absorbs the AdamW correction to the clean-SGD contraction rate 2 eta lambda. Calibrating (kappa_LL, V_star) on a single hyperparameter cell predicts grokking delays on 26 held-out runs with MAPE 17.7% over a 41x delay range; the law generalises to MLPs (MAPE 18.0%, N=34) and degrades to 23.3% on cross-task extension (N=46, 43.5x range), with a structured residual in which V_star / V_mem stays comparatively stable within architecture (CV about 14% on the 1L transformer). First-passage of V_t is necessary but not sufficient. A quantile-margin theorem establishes that positive delay requires both norm separation V_mem > V_post and angular reachability of a threshold alpha_star = arcsin(C / V_T_mem^(1/2)), where C is computable from the empirical NTK feature map and the validation-margin quantile. Calibrating C on modulus p=89 predicts alpha_star = 47.2 degrees at p=97 (observed 47.8 degrees, error 1.3%) as a prior cross-cell prediction. Causal interventions that freeze the norm or remove weight decay at memorisation eliminate grokking (0/6 vs. 3/3 baseline), trapping the angular displacement near 12 degrees. kappa_LL is empirically measured per architecture rather than derived from (beta_1, beta_2, epsilon); within-architecture CV stays at most 15% across four architectures, but values differ by about 2x between architectural variants beyond depth alone. Empirical scope is algorithmic tasks (modular arithmetic, sparse parity) under AdamW; whether the law transfers to natural-language scale models is open.

Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1

H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam

Neural Network Memorization Neural Network Training Dynamics Algorithmic Reasoning Grokking

May 12, 2026

Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory

Training Neural Networks (NNs) without overfitting is difficult; detecting that overfitting is difficult as well. We present a novel Random Matrix Theory method that detects the onset of overfitting in deep learning models without access to train or test data. For each model layer, we randomize each weight matrix element-wise,

\mathbf{W} \to \mathbf{W}^{\mathrm{rand}}

, fit the randomized empirical spectral distribution with a Marchenko-Pastur distribution, and identify large outliers that violate self-averaging. We call these outliers Correlation Traps. During the onset of overfitting, which we call the "anti-grokking" phase in long-horizon grokking, Correlation Traps form and grow in number and scale as test accuracy decreases while train accuracy remains high. Traps may be benign or may harm generalization; we provide an empirical approach to distinguish between them by passing random data through the trained model and evaluating the JS divergence of output logits. Our findings show that anti-grokking is an additional grokking phase with high train accuracy and decreasing test accuracy, structurally distinct from pre-grokking through its Correlation Traps. More broadly, we find that some foundation-scale LLMs exhibit the same Correlation Traps, indicating potentially harmful overfitting.

Hari K. Prakash, Charles H Martin

University of California San Diego Data Science and Engineering · Calculation Consulting San Francisco, CA, USA

Neural Network Generalization Neural Network Training Dynamics Grokking

May 10, 2026

Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds

Existing accounts of grokking explain the phenomena in terms of mechanistic frameworks such as circuit efficiency or lazy-to-rich transitions. However, despite a known dependence between grokking and model size, how model capacity shapes grokking remains an open question. We give an information-theoretic account of this relationship on the task of modular arithmetic, showing that grokking does not immediately occur when a model becomes large enough to memorise the training set, but rather emerges as the outcome of a competition between two measurable timescales: a memorisation speed

T_{\text{mem}}(P)

and a generalisation speed

T_{\text{gen}}(P)

, both of which are functions of model parameter count

P

. Adapting the information capacity framework of Morris et al. (2025), we estimate

T_{\text{mem}}(P)

on random-label data of equivalent complexity and

T_{\text{gen}}(P)

on the modular task itself, and show that grokking emerges close to the parameter scale where these timescales intersect. The framework also suggests an empirical model for predicting memorisation speed given model capacity and dataset complexity, recovering the previously reported empirical observation that larger models memorise faster. Overall, we motivate the formalisation of different learning timescales as important abstractions to study when explaining how model capacity shapes grokking on algorithmic tasks.

Yiding Song, Hanming Ye

Harvard College Cambridge, MA 02138

Neural Network Generalization Neural Network Memorization Grokking

May 7, 2026

Topological Signatures of Grokking

We study the grokking phenomenon through the lens of topology. Using persistent homology on point clouds derived from the embedding matrices of a range of models trained on modular arithmetic with varying primes, we identify a clear and consistent topological signature of grokking: a sharp increase in both the maximum and total persistence of first homology (

H_1

). Persistence diagrams reveal the emergence of a dominant long-lived topological feature together with increasingly structured secondary features, reflecting the underlying cyclic structure of the task. Compared to existing spectral and geometric diagnostics -- specifically, Fourier analysis and local intrinsic dimension -- persistent homology provides a unified geometric and topological characterization of representation learning, capturing both local and global multi-scale structure. Ablations across data regimes and control settings show that these topological transitions are tied to generalization rather than memorization. Our results suggest that persistent homology offers a principled and interpretable framework for analyzing how neural networks internalize latent structure during training.

Yifan Tang, Qiquan Wang, Inés García-Redondo +1

Imperial College London · Queen Mary University of London · University of Fribourg

Neural Network Generalization Neural Representation Geometry Grokking

May 7, 2026

Distributional Spectral Diagnostics for Localizing Grokking Transitions

In grokking, a model first fits the training data while test accuracy remains low, and only later begins to generalize. We ask whether this transition can be localized from observed training trajectories before the test accuracy rises, and formulate grokking transition localization as a diagnostic problem with an explicit threshold/FPR/lead-time trade-off. Task-dependent observables are summarized as empirical distributions, mapped to Wasserstein/quantile coordinates, and analyzed by Hankel dynamic mode decomposition (DMD); the resulting reconstruction residual, together with spectrum and effective rank, forms the diagnostic output. On held-out modular-addition Transformer runs, the residual achieves AUROC

\approx

0.93 for grokking-vs-non-grokking discrimination at the run level; under a fixed sustained-threshold operating rule, true-positive alarms can precede onset, with lead time reported jointly with false-alarm rate and uncertainty intervals. Perturbation experiments show that, in the tested

wd=1

pool, high-residual windows exhibit about

3\times

larger short-horizon perturbation deviation than low-residual windows. In a same-data norm-window control, perturbation sensitivity aligns with the residual ordering rather than total-parameter-norm ordering, suggesting that the residual is not merely a total-norm proxy at the window level in the studied

wd=1

dynamics. Norm signals remain strong run-level regime indicators, and log-probability performs best among the observables tested under the current protocol. We position the residual as a window-level monitoring and localization signal in the studied modular-arithmetic Transformer settings, not a universal early-warning predictor or an intervention rule.

Ziyue Wang, Yufeng Ying, Takafumi Kanamori

Institute of Science Tokyo · University of Science and Technology of China

Change-Point Detection Grokking Dynamic Mode Decomposition

Apr 28, 2026

Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking

Tian (2025) proves a repulsion theorem (Theorem 6) for the matrix

B = (\widetilde{F}^\top \widetilde{F} + ηI)^{-1}

during the interactive feature-learning stage of grokking: similar features have negative off-diagonal entries

B_{j\ell}

, producing an effective repulsive force that drives them apart. However, the theorem does not specify when this mechanism becomes empirically observable, nor whether it leaves a measurable spectral signature in the parameter updates. We test this directly on Tian's modular addition setup (

M = 71

,

K = 2048

, MSE loss) and observe a clear structure-mechanism dissociation. The predicted sign rule holds robustly on the top-200 most-similar feature pairs across activations (empirical sign-match rising from 0.865 to 0.985 on

σ= x^2

across 5 seeds, and saturating at 1.000 on

σ= \operatorname{ReLU}

). However, the spectral signature in the parameter updates is strongly activation-dependent. With

σ= x^2

, a simple slope detector on the rolling eigengap

σ_2 / σ_3

of

ΔW

fires in 15/15 grokking seeds at epoch 174 (IQR [173,174]) and in 0/15 non-grokking controls, with 229

\times

late-stage magnitude separation; the spectrum is rank-2. In contrast, with

σ= \operatorname{ReLU}

, the detector never fires and the spectrum remains effectively rank-1. This dissociation aligns with Tian's Theorem 5 distinction between focused (power-law) and spreading (ReLU) memorization: while the sign structure of

B

depends only on

\widetilde{F}^\top \widetilde{F}

, how feature repulsion translates into weight updates critically depends on the activation derivative

σ'

.

Yongzhong Xu

Neural Network Training Dynamics Modular Arithmetic Grokking

Apr 28, 2026

Gradient-Direction Sensitivity Reveals Linear-Centroid Coupling Hidden by Optimizer Trajectories

We show that replacing the rolling SVD of AdamW updates with a rolling SVD of loss gradients changes the diagnostic by 1-2 orders of magnitude. Performing SVD on the loss gradient instead of the AdamW update increases the measured perturbative coupling between SED directions and Linear Centroid Hypothesis (LCH) features from

\bar{R}_k \approx 3

--

9\times

to

100

--

330\times

across four single-task modular arithmetic operations, eliminating the apparent operation dependence in the original measurement. On a multitask transformer with a shared encoder, update-based SED gives

\bar{R}_k \leq 1

-- an apparent failure of the diagnostic -- while per-operation gradient-based SED recovers

\bar{R}_k = 20

--

45\times

across all four operations. Gradient aggregation across competing tasks is the main obstruction; performing SVD on per-task gradients resolves it. A causal intervention shows that constraining attention updates to any rank-3 subspace (whether SED-derived or random) accelerates grokking by approximately

2.3\times

across random seeds and operations, while removing the rank-3 component has negligible effect under proper gradient-projection methodology. The SED-LCH coupling is therefore a strong diagnostic of where feature formation concentrates in parameter space, but it is not a unique causal pathway: the natural full-rank AdamW attention update is highly rank-redundant under our hyperparameters.

Yongzhong Xu

Weight-Space Analysis Grokking Gradient Descent Dynamics

Apr 22, 2026

ILDR: Geometric Early Detection of Grokking

Grokking describes a delayed generalization phenomenon in which a neural network achieves perfect training accuracy long before validation accuracy improves, followed by an abrupt transition to strong generalization. Existing detection signals are indirect: weight norm reflects parameter-space regularization and consistently lags the transition, while GrokFast's slow gradient EMA, used without gradient amplification, is unstable across seeds with standard deviation exceeding mean lead time. We propose the Inter/Intra-class Distance Ratio (ILDR), a geometric metric computed on second-to-last layer representations as the ratio of inter-class centroid separation to intra-class scatter. ILDR provides an early detection signal: it rises and crosses a threshold at 2.5 times its baseline before the grokking transition appears in validation accuracy, indicating early geometric reorganization in representation space. Grounded in Fisher's linear discriminant criterion, ILDR requires no eigendecomposition and runs in O(|C|^2 + N). It is evaluated exclusively on held-out data, making it robust to memorization effects. Across modular arithmetic and permutation group composition (S5), ILDR leads the grokking transition by 9 to 73 percent of the training budget, with lead time increasing with task algebraic complexity. Over eight random seeds, ILDR leads by 950 +/- 250 steps with a coefficient of variation of 26 percent, and post-grokking variance drops by 1696 times, consistent with a sharp phase transition in representation space. Using ILDR as an early stopping trigger reduces training by 18.6 percent on average. Optimizer interventions triggered at the ILDR threshold demonstrate bidirectional control over the transition, suggesting ILDR tracks representational conditions underlying generalization rather than a downstream correlate.

Shreel Golwala

Virginia Tech

Neural Network Generalization Early Stopping Neural Representation Geometry Grokking

Apr 20, 2026

A Mechanistic Study of Algorithmic Generalization in Diffusion Models on Modular Arithmetic

Despite their empirical success, how diffusion models generalize remains poorly understood from a mechanistic perspective. In this work, we provide a controlled mechanistic account of algorithmic learning in diffusion models. By utilizing the grokking phenomenon on an image-based modular addition task as a tractable testbed, we dissect the model's internal dynamics across two distinct visual regimes. In a single-image regime, mechanistic dissection reveals that the model implements modular addition by encoding operands as periodic representations and composing them at the self-attention--feedforward interface in a way that recovers the trigonometric addition identities. In a diverse-image regime with high intraclass variability, we find that the iterative sampling process partitions the task into an arithmetic computation phase and a subsequent visual refinement phase, with their transition precisely captured by an internal frequency-entropy signal. Together, our analysis clarifies the mechanics of algorithmic generalization of diffusion models in a controlled setting, while leaving open how broadly these mechanisms transfer beyond modular addition.

Joon Hyeok Kim, Yong-Hyun Park, Mattis Dalsætra Østby +1

Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, USA

Pixel-Space Diffusion Models Mechanistic Interpretability Modular Addition Grokking Diffusion Models

Apr 14, 2026

A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learning

In-context learning enables transformers to adapt to new tasks from a few examples at inference time, while grokking highlights that this generalization can emerge abruptly only after prolonged training. We study task generalization and grokking in in-context learning using a Bayesian perspective, asking what enables the delayed transition from memorization to generalization. Concretely, we consider modular arithmetic tasks in which a transformer must infer a latent linear function solely from in-context examples and analyze how predictive uncertainty evolves during training. We combine approximate Bayesian techniques to estimate the posterior distribution and we study how uncertainty behaves across training and under changes in task diversity, context length, and context noise. We find that epistemic uncertainty collapses sharply when the model groks, making uncertainty a practical label-free diagnostic of generalization in transformers. Additionally, we provide theoretical support with a simplified Bayesian linear model, showing that asymptotically both delayed generalization and uncertainty peaks arise from the same underlying spectral mechanism, which links grokking time to uncertainty dynamics.

Abdessamed Qchohi, Simone Rossi

EURECOM, France

In-Context Learning Grokking Epistemic Uncertainty

Apr 13, 2026

Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin

Grokking - the delayed transition from memorisation to generalisation in neural networks - remains poorly understood. We study this phenomenon through the geometry of learned representations and identify a consistent empirical signature preceding generalisation: collapse of the spectral entropy of the representation covariance matrix. Across modular arithmetic tasks and multiple random seeds, spectral entropy decreases gradually during training and crosses a stable task-specific threshold before test accuracy rises. A representation-mixing intervention that delays this collapse also delays grokking, including under norm-matched controls, indicating that the effect is not explained by parameter norm alone. We further show that the entropy gap predicts the remaining time until grokking with useful out-of-sample accuracy. To probe the structure underlying this transition, we introduce a Fourier-alignment observable for cyclic-group tasks. Entropy collapse is strongly coupled to the emergence of Fourier-aligned representations, suggesting that spectral entropy tracks concentration of the representation into task-structured directions rather than generic compression alone. The same qualitative dynamics appear in non-abelian group composition tasks, while MLP controls show that entropy collapse by itself is insufficient for grokking in the absence of appropriate inductive bias. Taken together, the results support a view of grokking as a representational phase transition with an observable geometric signature. We discuss the scope and limitations of this interpretation, connections to recent feature-learning and spectral-dynamics work, and directions for testing whether similar transitions appear in larger-scale learning systems.

Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1

H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam

Grokking Neural Network Training Dynamics Representation Learning Neural Network Memorization

Apr 8, 2026

The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression

We decompose the spectral edge -- the dominant direction of the Gram matrix of parameter updates -- into its gradient and weight-decay components during grokking in two sequence tasks (Dyck-1 and SCAN). We find a sharp two-phase lifecycle: before grokking the edge is gradient-driven and functionally active; at grokking, gradient and weight decay align, and the edge becomes a compression axis that is perturbation-flat yet ablation-critical (>4000x more impactful than random directions). Three universality classes emerge (functional, mixed, compression), predicted by the gap flow equation. Nonlinear probes show information is re-encoded, not lost (MLP

R^2=0.99

where linear

R^2=0.86

), and removing weight decay post-grok reverses compression while preserving the algorithm.

Yongzhong Xu

Neural Network Training Dynamics Grokking Weight Decay Neural Network Compression Spectral Methods

Apr 6, 2026

Spectral Edge Dynamics Reveal Functional Modes of Learning

Training dynamics during grokking concentrate along a small number of dominant update directions -- the spectral edge -- which reliably distinguishes grokking from non-grokking regimes. We show that standard mechanistic interpretability tools (head attribution, activation probing, sparse autoencoders) fail to capture these directions: their structure is not localized in parameter or feature space. Instead, each direction induces a structured function over the input domain, revealing low-dimensional functional modes invisible to representation-level analysis. For modular addition, all leading directions collapse to a single Fourier mode. For multiplication, the same collapse appears only in the discrete-log basis, yielding a 5.9x improvement in concentration. For subtraction, the edge spans a small multi-mode family. For

x^2+y^2

, no single harmonic basis suffices, but cross-terms of additive and multiplicative features provide a 4x variance boost, consistent with the decomposition (a+b)^2 - 2ab. Multitask training amplifies this compositional structure, with the

x^2+y^2

spectral edge inheriting the addition circuit's characteristic frequency (2.3x concentration increase). These results suggest that training discovers low-dimensional functional modes over the input domain, whose structure depends on the algebraic symmetry of the task. These results suggest that spectral edge dynamics identify low-dimensional functional subspaces governing learning, whose representation depends on the algebraic structure of the task. Simple harmonic structure emerges only when the task admits a symmetry-adapted basis; more complex tasks require richer functional descriptions.

Yongzhong Xu

Neural Network Training Dynamics Grokking Compositional Representation Learning Frequency-Domain Feature Learning

Apr 6, 2026

Dimensional Criticality at Grokking Across MLPs and Transformers

Abrupt transitions between distinct dynamical regimes are a hallmark of complex systems. Grokking in deep neural networks provides a striking example -- an abrupt transition from memorization to generalization long after training accuracy saturates -- yet robust macroscopic signatures of this transition remain elusive. Here we introduce TDU--OFC (Thresholded Diffusion Update--Olami-Feder-Christensen), an offline avalanche probe that converts gradient snapshots into cascade statistics and extracts a macroscopic observable -- the time-resolved effective cascade dimension

D(t)

-- via grokking-aligned finite-size scaling. Across Transformers trained on modular addition and MLPs trained on XOR, we discover a localized dynamical crossing of the Gaussian diffusion baseline

D=1

precisely at the generalization transition. The crossing direction is task-dependent: modular addition descends through

D=1

(approaching from

D>1

), while XOR ascends (from

D<1

). This opposite-direction convergence is consistent with attraction toward a candidate shared critical manifold, rather than trivial residence near

D \approx 1

. Negative controls confirm this picture: ungrokked runs remain supercritical (

D>1

) and never enter the post-transition regime. In addition, avalanche distributions exhibit heavy tails and finite-size scaling consistent with the dimensional exponent extracted from

D(t)

. Shadow-probe controls (

α_{\mathrm{train}}=0

) confirm that

D(t)

is non-invasive, and grokked trajectories diverge from ungrokked ones in

D(t)

some

100

--

200

epochs before the behavioral transition.

Ping Wang

Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China

Neural Network Training Dynamics Phase Transitions in Neural Networks Grokking Statistical Physics of Learning

Apr 6, 2026

Grokking as Dimensional Phase Transition in Neural Networks

Neural network grokking -- the abrupt memorization-to-generalization transition -- challenges our understanding of learning dynamics. Through finite-size scaling of gradient avalanche dynamics across eight model scales, we find that grokking is a dimensional phase transition: effective dimensionality~

D

crosses from sub-diffusive (subcritical,

D < 1

) to super-diffusive (supercritical,

D > 1

) at generalization onset, exhibiting self-organized criticality (SOC). Crucially,

D

reflects gradient field geometry, not network architecture: synthetic i.i.d.\ Gaussian gradients maintain

D \approx 1

regardless of graph topology, while real training exhibits dimensional excess from backpropagation correlations. The grokking-localized

D(t)

crossing -- robust across topologies -- offers new insight into the trainability of overparameterized networks.

Ping Wang

Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China

Phase Transitions in Neural Networks Grokking Neural Network Training Dynamics Statistical Physics of Learning Gradient Descent Dynamics

Mar 31, 2026

Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernels

Grokking occurs when a model achieves high training accuracy but generalization to unseen test points happens long after that. This phenomenon was initially observed on a class of algebraic problems, such as learning modular arithmetic (Power et al., 2022). We study grokking on algebraic tasks in a class of feature learning kernels via the Recursive Feature Machine (RFM) algorithm (Radhakrishnan et al., 2024), which iteratively updates feature matrices through the Average Gradient Outer Product (AGOP) of an estimator in order to learn task-relevant features. Our main experimental finding is that generalization occurs only when a certain symmetry in the training set is broken. Furthermore, we empirically show that RFM generalizes by recovering the underlying invariance group action inherent in the data. We find that the learned feature matrices encode specific elements of the invariance group, explaining the dependence of generalization on symmetry.

Marcel Tomàs Bernal, Neil Rohit Mallinar, Mikhail Belkin

CFIS UPC · UC San Diego

Symmetry Breaking Kernel Methods Invariant Representation Learning Grokking

Mar 31, 2026

Grokking From Abstraction to Intelligence

Grokking in modular arithmetic has established itself as the quintessential fruit fly experiment, serving as a critical domain for investigating the mechanistic origins of model generalization. Despite its significance, existing research remains narrowly focused on specific local circuits or optimization tuning, largely overlooking the global structural evolution that fundamentally drives this phenomenon. We propose that grokking originates from a spontaneous simplification of internal model structures governed by the principle of parsimony. We integrate causal, spectral, and algorithmic complexity measures alongside Singular Learning Theory to reveal that the transition from memorization to generalization corresponds to the physical collapse of redundant manifolds and deep information compression, offering a novel perspective for understanding the mechanisms of model overfitting and generalization.

Junjie Zhang, Zhen Shen, Gang Xiong +1

Grokking Neural Network Generalization Neural Network Memorization

Mar 30, 2026

The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior

Grokking in transformers trained on algorithmic tasks is characterized by a long delay between training-set fit and abrupt generalization, but the source of that delay remains poorly understood. In encoder-decoder arithmetic models, we argue that this delay reflects limited access to already learned structure rather than failure to acquire that structure in the first place. We study one-step Collatz prediction and find that the encoder organizes parity and residue structure within the first few thousand training steps, while output accuracy remains near chance for tens of thousands more. Causal interventions support the decoder bottleneck hypothesis. Transplanting a trained encoder into a fresh model accelerates grokking by 2.75 times, while transplanting a trained decoder actively hurts. Freezing a converged encoder and retraining only the decoder eliminates the plateau entirely and yields 97.6% accuracy, compared to 86.1% for joint training. What makes the decoder's job harder or easier depends on numeral representation. Across 15 bases, those whose factorization aligns with the Collatz map's arithmetic (e.g., base 24) reach 99.8% accuracy, while binary fails completely because its representations collapse and never recover. The choice of base acts as an inductive bias that controls how much local digit structure the decoder can exploit, producing large differences in learnability from the same underlying task.

Laura Gomezjurado Gonzalez

Stanford University

Grokking Neural Algorithmic Reasoning Mathematical Reasoning

Mar 30, 2026

Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training

We develop the spectral edge analysis: phase transitions in neural network training -- grokking, capability gains, loss plateaus -- are controlled by the spectral gap of the rolling-window Gram matrix of parameter updates. In the extreme aspect ratio regime (parameters

P \sim 10^8

, window

W \sim 10

), the classical BBP detection threshold is vacuous; the operative structure is the intra-signal gap separating dominant from subdominant modes at position

k^* = \mathrm{argmax}\, σ_j/σ_{j+1}

. From three assumptions we derive: (i) gap dynamics governed by a Dyson-type ODE with curvature asymmetry, damping, and gradient driving; (ii) a spectral loss decomposition linking each mode's learning contribution to its Davis--Kahan stability coefficient; (iii) the Gap Maximality Principle, showing that

k^*

is the unique dynamically privileged position -- its collapse is the only one that disrupts learning, and it sustains itself through an

α

-feedback loop requiring no assumption on the optimizer. The adiabatic parameter

\mathcal{A} = \|ΔG\|_F / (η\, g^2)

controls circuit stability:

\mathcal{A} \ll 1

(plateau),

\mathcal{A} \sim 1

(phase transition),

\mathcal{A} \gg 1

(forgetting). Tested across six model families (150K--124M parameters): gap dynamics precede every grokking event (24/24 with weight decay, 1/24 without), the gap position is optimizer-dependent (Muon:

k^*=1

, AdamW:

k^*=2

on the same model), and 19/20 quantitative predictions are confirmed. The framework is consistent with the edge of stability, Tensor Programs, Dyson Brownian motion, the Lottery Ticket Hypothesis, and neural scaling laws.

Yongzhong Xu

Phase Transitions in Neural Networks Neural Network Training Dynamics Phase Transitions Grokking

Mar 26, 2026

A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization

Grokking the delayed transition from memorization to generalization in neural networks remains poorly understood, in part because prior empirical studies confound the roles of architecture, optimization, and regularization. We present a controlled study that systematically disentangles these factors on modular addition (mod 97), with matched and carefully tuned training regimes across models. Our central finding is that grokking dynamics are not primarily determined by architecture, but by interactions between optimization stability and regularization. Specifically, we show: (1) depth has a non-monotonic effect, with depth-4 MLPs consistently failing to grok while depth-8 residual networks recover generalization, demonstrating that depth requires architectural stabilization; (2) the apparent gap between Transformers and MLPs largely disappears (1.11

\times

delay) under matched hyperparameters, indicating that previously reported differences are largely due to optimizer and regularization confounds; (3) activation function effects are regime-dependent, with GELU up to 4.3

\times

faster than ReLU only when regularization permits memorization; and (4) weight decay is the dominant control parameter, exhibiting a narrow ``Goldilocks'' regime in which grokking occurs, while too little or too much prevents generalization. Across 3--5 seeds per configuration, these results provide a unified empirical account of grokking as an interaction-driven phenomenon. Our findings challenge architecture-centric interpretations and clarify how optimization and regularization jointly govern delayed generalization.

Shalima Binta Manir, Anamika Paul Rupa

Howard University

Grokking Neural Network Training Dynamics Neural Network Generalization Neural Network Optimization

Mar 25, 2026

Grokking as a Falsifiable Finite-Size Transition

Grokking -- the delayed onset of generalization after early memorization -- is often described with phase-transition language, but that claim has lacked falsifiable finite-size inputs. Here we supply those inputs by treating the group order

p

of

\mathbb{Z}_p

as an admissible extensive variable and a held-out spectral head-tail contrast as a representation-level order parameter, then apply a condensed-matter-style diagnostic chain to coarse-grid sweeps and a dense near-critical addition audit. Binder-like crossings reveal a shared finite-size boundary, and susceptibility comparison strongly disfavors a smooth-crossover interpretation (

Δ\mathrm{AIC}=16.8

in the near-critical audit). Phase-transition language in grokking can therefore be tested as a quantitative finite-size claim rather than invoked as analogy alone, although the transition order remains unresolved at present.

Yuda Bi, Chenyu Zhang, Qiheng Wang +1

Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS Center), Atlanta, Georgia 30303, USA · Beijing Institute of Technology, Beijing 100081, China · Georgia Institute of Technology, Atlanta, Georgia 30332, USA

Grokking Phase Transitions Statistical Physics of Learning