Grokking

Follow this topic Ask about this topic

Grokking studies why a model can fit its training data while performing poorly on new examples, then later generalize, often on tasks such as modular arithmetic, parity, and formal-language recognition. Researchers explain this delay through theories of weight decay, parameter norms, optimization noise, and competing memorization and generalization timescales, alongside analyses of learned circuits and representations using Fourier features, symmetry, and spectral geometry. Experiments also test ways to predict or control the transition, including geometric diagnostics, regularization, architectural changes, and structure-specific priors. A central open question is whether these findings reflect one general mechanism or several task- and optimizer-dependent processes; signals that predict grokking do not always establish its cause.

Latest papers

77 papers5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

The Weight Norm Sets the Grokking Timescale: A Causal Delay Law
Jun 11, 2026Truong Xuan Khanh, Doan Hoang Viet, Luu Duc Trung +1Neural Network Generalization Neural Network Training Dynamics
H&K Research Studio / Clevix LLC, Hanoi, Vietnam · Bac A Bank, Hanoi · Banking Academy of Vietnam
Learned Structure Shows Up Earlier in the Task's Own Basis: A Controlled Look at a Synchronisation Precursor in Grokking
Jun 11, 2026Achyuthan SivasankarWeight Decay Frequency-Domain Feature Learning
New York University
Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic
Jun 8, 2026Hu Tan, Kuo Gai, Shihua ZhangNeural Representation Geometry Classification
Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; and School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China · Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China; and Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Hangzhou 310024, China
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
Jun 4, 2026Hu Tan, Kuo Gai, Shihua ZhangReLU Neural Networks Deep Linear Networks
1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1
Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View
Jun 3, 2026Mingyu LiSpectral Regularization Weight Decay
Beijing Normal University
Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent
May 26, 2026Chi-Ning Chou, Oscar Uzdelewicz, Neng-Chun Chiu +2Double Descent Neural Network Generalization
Center for Computational Neuroscience, Flatiron Institute, New York, NY, USA. · Department of Physics, Harvard University, Cambridge, MA, USA. · Kempner Institute, Harvard University, Allston, MA, USA. +1
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
May 19, 2026Lucky VermaWeight Decay Neural Network Memorization
Independent Researcher
Grokking as Structural Inference: Transformers Need Bayesian Lottery Tickets
May 15, 2026Kai Hidajat, Solden Stoll, Joseph AnNeural Network Generalization Lottery Ticket Hypothesis
Department of Computer Science University of Washington Seattle, WA 98195
Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning
May 14, 2026Shin So, Kyelim Lee, Albert NoNeural Network Generalization Neural Network Training Dynamics
Yonsei University
First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation
May 13, 2026Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1Neural Network Memorization Neural Network Training Dynamics
H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam
Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory
May 12, 2026Hari K. Prakash, Charles H MartinNeural Network Generalization Neural Network Training Dynamics
University of California San Diego Data Science and Engineering · Calculation Consulting San Francisco, CA, USA
Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds
May 10, 2026Yiding Song, Hanming YeNeural Network Generalization Neural Network Memorization
Harvard College Cambridge, MA 02138
Topological Signatures of Grokking
May 7, 2026Yifan Tang, Qiquan Wang, Inés García-Redondo +1Neural Network Generalization Neural Representation Geometry
Imperial College London · Queen Mary University of London · University of Fribourg
Distributional Spectral Diagnostics for Localizing Grokking Transitions
May 7, 2026Ziyue Wang, Yufeng Ying, Takafumi KanamoriChange-Point Detection Grokking
Institute of Science Tokyo · University of Science and Technology of China
Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking
Apr 28, 2026Yongzhong XuNeural Network Training Dynamics Modular Arithmetic
Gradient-Direction Sensitivity Reveals Linear-Centroid Coupling Hidden by Optimizer Trajectories
Apr 28, 2026Yongzhong XuWeight-Space Analysis Grokking
ILDR: Geometric Early Detection of Grokking
Apr 22, 2026Shreel GolwalaNeural Network Generalization Early Stopping
Virginia Tech
A Mechanistic Study of Algorithmic Generalization in Diffusion Models on Modular Arithmetic
Apr 20, 2026Joon Hyeok Kim, Yong-Hyun Park, Mattis Dalsætra Østby +1Pixel-Space Diffusion Models Mechanistic Interpretability
Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, USA
A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learning
Apr 14, 2026Abdessamed Qchohi, Simone RossiIn-Context Learning Grokking
EURECOM, France
Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin
Apr 13, 2026Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1Grokking Neural Network Training Dynamics
H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam
The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression
Apr 8, 2026Yongzhong XuNeural Network Training Dynamics Grokking
Spectral Edge Dynamics Reveal Functional Modes of Learning
Apr 6, 2026Yongzhong XuNeural Network Training Dynamics Grokking
Dimensional Criticality at Grokking Across MLPs and Transformers
Apr 6, 2026Ping WangNeural Network Training Dynamics Phase Transitions in Neural Networks
Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China
Grokking as Dimensional Phase Transition in Neural Networks
Apr 6, 2026Ping WangPhase Transitions in Neural Networks Grokking
Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China
Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernels
Mar 31, 2026Marcel Tomàs Bernal, Neil Rohit Mallinar, Mikhail BelkinSymmetry Breaking Kernel Methods
CFIS UPC · UC San Diego
Grokking From Abstraction to Intelligence
Mar 31, 2026Junjie Zhang, Zhen Shen, Gang Xiong +1Grokking Neural Network Generalization
The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior
Mar 30, 2026Laura Gomezjurado GonzalezGrokking Neural Algorithmic Reasoning
Stanford University
Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training
Mar 30, 2026Yongzhong XuPhase Transitions in Neural Networks Neural Network Training Dynamics
A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization
Mar 26, 2026Shalima Binta Manir, Anamika Paul RupaGrokking Neural Network Training Dynamics
Howard University
Grokking as a Falsifiable Finite-Size Transition
Mar 25, 2026Yuda Bi, Chenyu Zhang, Qiheng Wang +1Grokking Phase Transitions
Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS Center), Atlanta, Georgia 30303, USA · Beijing Institute of Technology, Beijing 100081, China · Georgia Institute of Technology, Atlanta, Georgia 30332, USA

Jun 11, 2026J/K move · Enter open · S save

The Weight Norm Sets the Grokking Timescale: A Causal Delay Law

Truong Xuan Khanh, Doan Hoang Viet, Luu Duc Trung, Phan Thanh Duc

H&K Research Studio / Clevix LLC, Hanoi, Vietnam · Bac A Bank, Hanoi · Banking Academy of Vietnam

Neural Network Generalization Neural Network Training Dynamics Grokking

Open paper AskPDF arXiv

Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data. Whether the weight norm causes this delay is disputed: some studies report a critical norm at the transition, others observe grokking with no fixed norm at all. We settle this by intervening on the norm during training rather than only observing it. Under free training with weight decay, networks grok when the weight norm reaches a value Wc that varies little across seeds and learning rates (CV 1 to 2 percent) and grows with the modular base as a power law. When we instead clamp the norm to a fixed multiple rho of Wc and hold it there, the network still groks, but the delay follows T_grok proportional to exp(alpha rho). One exponent, alpha near 7.5, fits this delay across four moduli (R^2 = 0.996). Over the swept ranges the held norm moves the delay by about 19x and the learning rate by only about 2x, and holding the norm above Wc slows grokking rather than preventing it. A final LayerNorm removes the dependence by decoupling weight scale from the network function; without it the exponential law returns. This pinned-norm delay is the exponential counterpart to the logarithmic delay predicted for a freely contracting norm.