Overview
Grokking studies why a model can fit its training data while performing poorly on new examples, then later generalize, often on tasks such as modular arithmetic, parity, and formal-language recognition. Researchers explain this delay through theories of weight decay, parameter norms, optimization noise, and competing memorization and generalization timescales, alongside analyses of learned circuits and representations using Fourier features, symmetry, and spectral geometry. Experiments also test ways to predict or control the transition, including geometric diagnostics, regularization, architectural changes, and structure-specific priors. A central open question is whether these findings reflect one general mechanism or several task- and optimizer-dependent processes; signals that predict grokking do not always establish its cause.
Latest papers
83 papers7 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.
Dissecting Geometric Constraints in Grokking: Interaction-Dependent Acceleration in Modular Arithmetic
Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking
Subspace Crystallization in Grokking: Rank Compression Tracks the Phase Transition
Mapping the Solution-Set Structure of Grokking on Latin Squares
Generalization in Learning: Eyring Formula and Predator-prey Model
Flat Loss, Evolving Features: Grokking in Modular Addition
Statistical Vignettes on the Grokking Phenomenon
Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking
Misalignment of Low-Loss Regions Causes Grokking
Grokking through the Lens of Minimum-Norm Interpolation
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
An Exploratory Replica-Overlap Probe of the Grokking Transition
Beyond Embedding Transfer: Architectural Interfaces, Iso-Parameter Scaling, and Preconditioner-Governed Consolidation in Grokking
Structured Features Overfit Where Random Features Grok
Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking
Biologically Inspired Mechanisms for Facilitating Grokking in Multilayer Perceptrons
Mapping the Emergence of Regularization-Driven Dynamics in Grokking
Is Grokking a Loss of Normal Hyperbolicity of the Interpolation Manifold?
Algorithmic Information Dynamics of Learning: A Certified, Differentiable Complexity Controller for Grokking
Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
Tunneling the Loss Landscape: Bypassing Memorization with Monte Carlo Parameter Swapping
fast cooling' where the training loss is reduced so quickly that a glass state is formed, followed by a slow relaxation' towards final generalization. Although providing a unifying framework for representative grokking theories, this perspective has remained largely at the theoretical on macroscopic level without direct empirical validation on training dynamics. Here we introduce a three-component framework to directly characterize the training dynamics via parameter mobility (PM), and two representative measurements from glassy dynamics: replica correlation (RC) and fractal dimension (FD). We demonstrate that standard optimization presents clear signatures of glass dynamics and inherently traps the grokking network in a kinetic arrested memorization state with a collapsed mobility, strong history dependence, and channel-like motions. This quantitative agreement motivates us to introduce State-Aware Monte Carlo Parameter Swapping (SAM-Swap), an optimization plug-in that can accelerate generalization, inspired by swap Monte Carlo algorithm widely used in glass dynamics. Comparing SAM-Swap, weight decay, and Gaussian gradient noise, we find that accelerated generalization is consistently associated with random exploration in the parameter space, similar to diffusion in physics.