Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking

Min Chen; Junqi Wang; Zeguan Wu; Wenhao Huang; Junyu Liu

cs.LGOct 26, 2026

Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking

Authors: Min Chen, Junqi Wang, Zeguan Wu, Wenhao Huang, Junyu Liu

Organizations: Department of Computer Science, University of Pittsburgh, Pittsburgh, PA, USA

Abstract

Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN, an ensemble of trained models defines an order parameter (the squared weight norm) and fluctuation proxies for the heat capacity and the susceptibility, with training epoch and weight decay serving as thermal and field control coordinates. To bridge neural architectures to real-space RG, we construct an exact coupling matrix via a second-order expansion of the loss with respect to per-neuron norms and use its Fiedler vector to define an effective geometry. This spectral ordering yields a monotonic spatial decay of coupling strength that is absent under random permutations. We then execute real-space RG by block-spin aggregating adjacent neurons and parameterizing the coupling matrix into off-diagonal band strengths. Linearizing this local RG map yields relevant scaling exponents and five derived critical exponents, with direct empirical estimates matching RG predictions within 1-10%. Overall, our framework provides a consistent effective description of grokking, suggesting the scaling and renormalization group formalism as a tool to investigate NN dynamics.

Figures & tables

Figure 1: Overview. From trained NNs (left), the critical exponents of grokking are obtained via two complementary routes. RG route (top): each hidden neuron carries a spin

s_{i}

, its squared weight norm, while each neuron pair is connected by a coupling

J_{ij}

derived from the loss Hessian. Ordering the neurons on a chain using the Fiedler vector of this coupling graph enables adjacent neurons to be merged into blocks, where a linear fit of the coarsened couplings yields

y_{t}

and

y_{h}

, from which five critical exponents follow (Sections 3.2 – 3.3 ). Direct route (bottom): a physical dictionary uses the training epoch as an effective temperature, weight decay as conjugate field, and ensemble fluctuations as proxies for susceptibility and heat capacity, with power-law fits of these observables yielding the direct exponents (Sections 3.1 and 4 ). The two independent estimates agree within

1

–

10\%

(right).

Figure 2: Ensemble dynamics. (A) Train and test accuracy and the order parameter

\langle S\rangle

over the 30 runs. (B) Susceptibility

\chi_{S}=\operatorname{Var}_{r}S

and heat-capacity proxy

C_{H}=\operatorname{Var}_{r}H_{\mathrm{loss}}

. The dotted line marks the

\chi_{S}

peak at

e=118

; the dash-dotted line marks

e_{c}=145

Panel of Figure 3 — Figure 3: Validation of the constructed real space at $e_{c}=145$ . Left pair: $|J_{ij}|$ averaged over $20\times 20$ -neuron blocks, in the original neuron order and after Fiedler ordering, on a common colour scale. Right: mean $|J_{ij}|$ between neuron pairs at a given normalized Fiedler distance, compared with a random ordering of the same matrix.

Figure 4: The measured local RG flow. Colour gives the size of the step. Black dots are the 65 measured coordinates. The orange and blue lines are the magnetic and thermal eigendirections of $\mathbf{T}$ ; both point away from the center because $|\Lambda|>1$ . The star is the grokking center $(e_{c},\lambda_{c})=(145,15)$ . The regression that produced $\mathbf{T}$ is shown in Figure 9 (Appendix E ).

Exp.	Definition	How the direct value is obtained	Direct	RG	Rel. diff.
$\alpha$	$C\propto\|t\|^{-\alpha}$	$C_{H}=\operatorname{Var}_{r}H_{\mathrm{loss}}$ , $t<0$ , $e\in[107,133]$	0.549	0.556	1.2%
$\beta$	$m\propto\|t\|^{\beta}$	$\|\langle S\rangle_{r}-S_{c}\|$ , $t>0$ , $e\in[180,256]$	0.545	0.503	8.3%
$\gamma$	$\chi\propto\|t\|^{-\gamma}$	$\chi_{S}=\operatorname{Var}_{r}S$ , $t>0$ , $e\in[153,171]$	0.480	0.438	9.7%
$\nu$	$\xi\propto\|t\|^{-\nu}$	$\operatorname{std}_{r}e_{50}/\langle e_{50}\rangle_{r}\propto p^{-1/\nu}$ , seven moduli	1.400	1.444	3.1%
$\delta$	$m\propto\|h\|^{1/\delta}$ , $t=0$	$\|\langle S/N\rangle_{h}-\langle S/N\rangle_{h=0}\|\propto\|h\|^{1/\delta}$ at $e_{c}$	2.001	1.870	7.0%

Table 1: Critical exponents from direct estimates and from the local RG. Relative difference is

|\mathrm{direct}-\mathrm{RG}|/\mathrm{RG}

Panel of Figure 5 — Figure 5: Direct fits for the five critical exponents: $\alpha$ , $\beta$ , and $\gamma$ (top), $\nu$ (bottom left), and $\delta$ (bottom right).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Task, model, and training
$p$	modulus of the task, $p=97$ except in the modulus scan; the size variable of the finite-size analysis
$x_{1},x_{2}$ ; $y$	operands; label $y=(x_{1}+x_{2})\bmod p$ , the cyclic task coordinate
$x_{n}$ , $y_{n}$	one-hot input pair and one-hot target of training example $n$
$f_{\theta}$ , $\theta$	NN function; all trainable parameters
$W_{1}$ , $W_{2}$	first- and second-layer weights, $W_{1}\in\mathbb{R}^{N\times 2p}$ , $W_{2}\in\mathbb{R}^{p\times N}$

Appendix

Table 2: Notation.

Figure 6: Steps 1–8 on a toy graph of ten vertices (an illustration, not data from the trained NN). The toy weights decay with the distance between hidden positions on a line, up to multiplicative noise, and the vertex labels are shuffled. (A) The weighted graph; the width and darkness of an edge increase with

A_{ij}

, and the colour of a vertex is its Fiedler coordinate

v_{2,i}

. (B) The adjacency matrix in the original order (darker is larger; the diagonal is not used). (C) The Fiedler vector places the vertices on a line with unequal spacing (top); keeping only the ranks

\pi(i)

gives a chain of equally spaced sites (bottom). (D) The same matrix in Fiedler order, in which the large entries lie near the diagonal. (E) Adjacent sites of the chain are merged into the blocks

B_{1},\dots,B_{5}

(

b=2

);

K_{1}

and

K_{2}

are the mean couplings between first and second neighbours on the chain (green and pink arcs), and

K^{\prime}_{1}

and

K^{\prime}_{2}

are the same quantities on the chain of blocks. (F) In matrix form, with

\lvert J_{ij}\rvert=A_{ij}

, the blocks are the

2\times 2

cells separated by white lines, and

K_{1}

and

K_{2}

are the means of the first and second off-diagonal bands (green and pink outlines). (G) The block couplings

J^{\prime}_{BB^{\prime}}

of Equation ( 8 ), each the sum of one

2\times 2

cell, with the bands that give

K^{\prime}_{1}

and

K^{\prime}_{2}

. For the NN, the corresponding matrices are shown in Figure 3 .

Figure 7: Distance profile at

e_{c}=145

: mean

\lvert J_{ij}\rvert

versus normalized distance across 40 equal bins for Fiedler and random orderings.

Figure 8: Finite-size collapse of the modulus scan across 350 confirmation runs (

N=500

). (A) Mean test accuracy against reduced epoch

t

, centered at mean pseudo-transition

\langle e_{50}\rangle_{p}

. (B) Collapse achieved against rescaled coordinate

x=tp^{1/\nu_{\mathrm{RG}}}

. (C) Collapse cost function versus candidate scaling exponents, highlighting the minimum near

\nu_{\mathrm{RG}}

. (D) Scaling of mean pseudo-transition epoch against

p

, comparing unconstrained power-law fitting with a fixed

1/\nu_{\mathrm{RG}}

decay exponent.

Figure 9: The regression behind the local RG map

\mathbf{T}

of equation ( 11 ). (A) The 65 coupling coordinates

\mathbf{K}_{n}=(K_{1},K_{2})_{n}

of the parameter cloud (13 epochs

\times

5 weight decays), coloured and sized by the Gaussian weight

w_{n}

of equation ( 10 ); the ring marks the grokking center

(e_{c},\lambda_{c})=(145,15)

, which carries weight 1. (B, C) Observed block coordinates

K^{\prime}_{1}

and

K^{\prime}_{2}

against the values

c+\mathbf{T}\mathbf{K}_{n}

returned by the weighted affine fit, with the identity line. The fit is local: points far from the center have negligible weight and are shown only to indicate the extent of the cloud.

Figure 10: Eigendirections of

\mathbf{T}

and control tangents on the coupling plane. Grey points: the 65-point parameter cloud; star: the grokking center

(e_{c},\lambda_{c})=(145,15)

. The orange and blue lines are the right eigendirections

v_{+}

and

v_{-}

through the center; the arrows are the unit tangents

\partial\mathbf{K}/\partial e

and

\partial\mathbf{K}/\partial\lambda

obtained by regressing the cloud on the two control coordinates.

Quantity	Statistical physics	NN (this work)
Thermodynamic quantities
Reduced temperature $t$	$(T-T_{c})/T_{c}$	$(e-e_{c})/e_{c}$ , $e_{c}=145$ (epoch as an inverse-temperature-like coordinate, so that $t\approx-t_{T}$ )
Field $h$	$H=\cdots-h\sum_{i}s_{i}$	$h=-(\lambda-15)/2$ (weight decay, conjugate to $S$ )
Magnetization $M$	$\sum_{i}s_{i}$ , $m=M/N$	$S=\sum_{i}\lVert w_{i}\rVert_{2}^{2}$ , measured relative to $S_{c}$
Susceptibility $\chi$	$\partial m/\partial h\propto\operatorname{Var}(M)$	$\chi_{S}=\operatorname{Var}_{r}[S]$ over 30 seeds (proxy)
Specific heat $C$	$-T\,\partial^{2}f/\partial T^{2}\propto\operatorname{Var}(E)$	$C_{H}=\operatorname{Var}_{r}[H_{\mathrm{loss}}]$ over 30 seeds (proxy)

Appendix

Table 3: Dictionary between the standard scaling analysis and the trained NN. Every entry in the right column is an operational definition, not an identity; the assumptions behind each are discussed in the text.

Grokking

Grokking is delayed generalization in which neural networks memorize training data before abruptly learning a rule that works on unseen examples.

See all 81 papers

Apr 6, 2026

Grokking as Dimensional Phase Transition in Neural Networks

Neural network grokking -- the abrupt memorization-to-generalization transition -- challenges our understanding of learning dynamics. Through finite-size scaling of gradient avalanche dynamics across eight model scales, we find that grokking is a dimensional phase transition: effective dimensionality~

D

crosses from sub-diffusive (subcritical,

D < 1

) to super-diffusive (supercritical,

D > 1

) at generalization onset, exhibiting self-organized criticality (SOC). Crucially,

D

reflects gradient field geometry, not network architecture: synthetic i.i.d.\ Gaussian gradients maintain

D \approx 1

regardless of graph topology, while real training exhibits dimensional excess from backpropagation correlations. The grokking-localized

D(t)

crossing -- robust across topologies -- offers new insight into the trainability of overparameterized networks.

Ping Wang

Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China

Phase Transitions in Neural Networks Grokking Neural Network Training Dynamics Statistical Physics of Learning Gradient Descent Dynamics

Ask PDF

Sep 9, 2026

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

Neural networks trained past memorization frequently undergo a delayed transition to generalization, a phenomenon known as grokking. Despite theoretical progress on why this transition occurs, the quantitative structure of when it occurs in hyperparameter space remains uncharacterized. We map the memorization-to-generalization boundary across 384 configurations of two-hidden-layer MLPs on modular arithmetic, fitting a power-law scaling relation for generalization onset time:

T_{\mathrm{grok}} \propto H^{-0.27}\, D^{-2.04}\, η^{-0.50}\, λ^{-0.64}

(

R^2 = 0.732

;

0.821

with interactions). The exponent hierarchy reveals that data complexity (

D^{-2.04}

) is the dominant driver of regime transition, not model capacity (

H^{-0.27}

): doubling data accelerates generalization by

{\sim}4\times

, while doubling width yields only

{\sim}1.2\times

. A sharp phase boundary at weight decay

λ\gtrsim 1.0

separates grokking from non-grokking configurations, and weight norm trajectories show monotonic compression during the transition, consistent with implicit regularization selecting low-complexity solutions. These results provide a quantitative foundation for predicting and controlling regime transitions in overparameterized networks.

Anish Kataria

Princeton University, Princeton, NJ, USA.

Neural Network Training Dynamics Neural Scaling Laws Grokking Neural Network Memorization Multilayer Perceptrons Phase Transitions in Neural Networks+1

Ask PDF

Jun 29, 2026

A Stochastic--Geometric Theory of Scaling Laws in Grokking

Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell--core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model's parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and

\ell_2

regularization coefficient, which are further validated through experiments and shown to recover results from prior literature.

Róisín Luo, Christian Gagné, Jonas Ngnawé +2

Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval +2

Neural Network Generalization Neural Network Optimization Neural Network Memorization Neural Scaling Laws Grokking

Ask PDF

Abstract

Grokking

Explore similar work

Grokking as Dimensional Phase Transition in Neural Networks

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

A Stochastic--Geometric Theory of Scaling Laws in Grokking