cs.LGNov 14, 2026

Grokking Can Be Steered: Selecting Desirable Solutions By Choice Of Learning Rule

Authors: Pascal Sager, Benjamin Grewe, Pau Vilimelis Aceituno, Thilo Stadelmann

Organizations: Centre for Artificial Intelligence, ZHAW · Institute of Neuroinformatics ETH Zürich and University of Zürich · ETH AI Center, Zürich, Switzerland · Centre for Artificial Intelligence, ZHAW Winterthur, Switzerland · 

Abstract

During grokking, a network transitions from memorization to generalization, yet what shapes the neural implementation (circuit) of the generalizing solution and specifically the number of participating neurons (its size) remains unclear. We find that the learning rule, the combination of credit assignment algorithm and optimizer, controls this: On modular addition, every learning rule we study that groks recovers the same Fourier-based algorithm, yet implemented by circuits that differ in size. Why? A neuron stays in use when its credit is large relative to the non-gradient term of its own update, and this single condition unifies three routes to unused neurons: the coupled weight decay, Adam's εε and the decay of a trainable per-neuron gain. Equalizing the magnitude of credit assigned to each neuron, or scaling each neuron's weight decay by its credit, expands the effective circuit size, measured from the outgoing weights, from 60% to 99% of the layer's neurons at similar training and test accuracy. We vary which neurons share one equalization factor, from the whole layer to individual neurons, and find that the circuit stays concentrated for groups of eight or more and fully distributes only when every neuron has its own factor. Credit magnitudes are usually set by the normalization of standard optimizers: Nero, which normalizes per neuron, distributes the computation, whereas Adam and seven others, including RMSProp, Muon, and Shampoo, use other normalization schemes and retain small, concentrated circuits. Distributed circuits tolerate neuron deletion better, while concentrated circuits can be pruned further and transfer about twice as fast to a task that recombines the same features, largely by recruiting their unused neurons. Learning rule choice thus steers circuit size, trading robustness against compressibility and transfer speed without changing the learned algorithm.

Figures & tables

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. A Stochastic--Geometric Theory of Scaling Laws in Grokking

    Jun 29, 2026Róisín Luo, Christian Gagné, Jonas Ngnawé +2Neural Network GeneralizationNeural Network Optimization

  2. Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking

    Oct 26, 2026Min Chen, Junqi Wang, Zeguan Wu +2Renormalization GroupGrokking