Overview
Grokking studies why a model can fit its training data while performing poorly on new examples, then later generalize, often on tasks such as modular arithmetic, parity, and formal-language recognition. Researchers explain this delay through theories of weight decay, parameter norms, optimization noise, and competing memorization and generalization timescales, alongside analyses of learned circuits and representations using Fourier features, symmetry, and spectral geometry. Experiments also test ways to predict or control the transition, including geometric diagnostics, regularization, architectural changes, and structure-specific priors. A central open question is whether these findings reflect one general mechanism or several task- and optimizer-dependent processes; signals that predict grokking do not always establish its cause.
Latest papers
77 papers5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.