We study how the layers in deep neural networks (NNs) converge, using Wilsonian Renormalization-Group (RG) ideas to organize the spectral picture. During training, these layers often develop heavy-tailed power-law (PL) spectra, and the strongest checkpoints often appear near the PL exponent α≈2. This pattern raises a simple question: why should this value mark a well-trained layer? The answer is both statistical and physical. Our statistical hypothesis is that α=2 marks the boundary between stable if not optimal learning and overfitting via non-self-averaging. The physical answer is that the same value makes the spectral energy gain scale evenly across logarithmic spectral scales, which is the proposed Wilsonian marginal condition. We then apply the ERG trace-log condition, the proposed SETOL theory, which we justify by taking volume-preserving, scale-invariant transformations to remove arbitrary layer scale along the proposed RG flow. The resulting spectral RG flow connects the HTSR and SETOL theories to optimizers such as Muon, and motivates a new optimizer extension, WW-PGD. Together, these results point toward a spectral Renormalization-Group view of learning, in which NNs may approach shared spectral structure in the settings studied here.
Figures & tables
Figure 1: Empirical Spectral Density (ESD) of a well-trained dense layer. Panel (a) shows the ESD of X=N−1WTW in log-log coordinates. The tail is fit by a finite-window MLE/KS power law with α≃2 . Panel (b) shows the trace-log-normalized ECS, where TrlogXR≃0 ( R denotes the ECS space). The red and purple vertical lines denote the PL tail, λ~PL+ , and the ECS, λ~R+ , resp. Plots have been generated using the open-source WeightWatcher tool. [ 14 ]
Figure 2: Layerwise HTSR α histograms from linear layers, computed with the open-source WeightWatcher tool. Left: GPT-OSS 20B/120B models. Right: Qwen2.5 instruct models. The warning bands are heuristic: small α marks overfit or trap risk, while large α marks weakly correlated or underfit tails. Optimizer notebook details are listed in Appendix A .
Figure 3: HTSR/SETOL phases during training. The HTSR power-law tail TPL and the SETOL Effective Correlation Space RECS are selected independently. During training, they overlap. At convergence, they align RECS⋆≃TPL⋆ , at α≃2 .
Figure 4: Self-averaging and the failure of a shrinking bound. Here μM=E[OM] . In the self-averaging case, the normalized observable OM/μM concentrates around one and the relative variance VM(O) tends to zero, allowing concentration bounds to tighten. In the illustrated non-self-averaging case, finite relative deviations retain nonzero probability as the system grows, so no valid upper bound on that probability can shrink to zero.
Figure 5: The randomized MP test. After W→rand(W) , the bulk follows the MP fit. The dashed line marks the far bulk plus TW edge (λMP++ΔTW) ; persistent eigenvalues to its right are counted as correlation traps.
Figure 6: Long-horizon grokking. Top: test accuracy falls as the randomized-spectrum trap count rises. Bottom: the fitted exponent subsequently crosses below α=2 . The aligned horizontal axes show the order of the warnings [ 12 , 13 ] .
Figure 8: Heavy-Tailed Mechanistic Universality.
Regime
spectral energy per scale
Scale-flow meaning
Statistical meaning
Learning meaning
α>2
λ~2ρ(λ)∼λ~2−α→0
Spectral band gain decays; weak-tail sector or approach toward the trivial fixed point F0 .
First moment self-averages; dominant tail components stay controlled.
Stable but potentially weakly correlated or undertrained tail.
α=2
λ~2ρ(λ)∼λ~0
Scale-balanced logarithmic-band state; candidate FERG when ECS aligns with the fitted tail.
Table 2: Scale counting, statistics, and empirical meaning of the three spectral regimes.
Term
Notation
Meaning here
RG step
RECS
Project to the retained ECS and evaluate the ESD on the SETOL scale.
Fixed point
RRG[ρ⋆]=ρ⋆
A SETOL-normalized ESD that is unchanged by the RG step.
Weak-tail branch
F0,gEundefined
The trivial solution HTSR Random-like phase and nearby ESDs.
Correlated branch
FERG
The empirical RG fixed point candidate.
Band Energy
GE(Λ,b)
Spectral energy carried by one logarithmic spectral band.
Band gain
gE(λ~R):=λ~R2ρR(λ~R)
Spectral energy per logarithmic spectral scale.
Table 3: Special RG notation used in this section.
Figure 12: Trained checkpoints show approximate matching and substantial exceptions. (a) Five FC1 tails at learning rate 0.32 have α^=2.0036 but unequal log-band energies. Rj divides each band’s energy by that checkpoint’s mean band energy; error bars are between-seed standard deviations. (b) ERG rank gaps for both hidden layers of all 30 checkpoints; labels mark the paired seed-4 FC1 examples. (c) The same five FC1 tails have different mean-log offsets under the stored, covariance, and declared full-spectrum conventions. The archive is an independent MNIST protocol replication.
Figure 15: MNIST optimizer comparison: AdamW versus Muon. Same three-layer MNIST MLP trained with AdamW and Muon. Short-horizon panel compares early epochs; long-horizon panel tracks the Muon αℓ(t)→2 .
Figure 16: WW-PGD stabilizes layerwise HTSR exponents while preserving test accuracy. A three-layer MNIST MLP is trained with AdamW alone or AdamW plus WW-PGD. Top: layerwise fitted HTSR exponents for FC1–FC3. Bottom: ordinary MNIST test accuracy. Curves show mean ± standard deviation across five runs.
Figure 17: WW-PGD corrects overfit layers. Extending training on an anti-grokking checkpoint from Figure 6 . [ 15 ]
Figure 18: RG flow in the space of effective correlations. Different nonlinear learning algorithms follow different microscopic trajectories, but coarse-graining can suppress irrelevant directions and carry retained spectral structure toward a candidate fixed point. Candidate universal quantities are read off near the fixed point, not from the microscopic update rule.
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
We study the evolution of hidden-weight spectra in wide neural networks trained by (stochastic) gradient descent. We develop a two-level dynamical mean-field theory (DMFT) that jointly tracks bulk and outlier spectral dynamics for spiked ensembles whose spike directions remain statistically dependent on the random bulk. We apply this framework to two settings: (1) infinite-width nonlinear networks in mean-field/μP scaling and (2) deep linear networks in the proportional high-dimensional limit, where width, input dimension, and sample size diverge with fixed ratios. Our theory predicts how outliers evolve with training time, width, output scale, and initialization variance. In deep linear networks, μP yields width-consistent outlier dynamics and hyperparameter transfer, including width-stable growth of the leading NTK mode toward the edge of stability (EoS). In contrast, NTK parameterization exhibits strongly width-dependent outlier dynamics, despite converging to a stable large-width limit. We show that this bulk+outlier picture is descriptive of simple tasks with small output channels, but that tasks involving large numbers of outputs (ImageNet classification or GPT language modeling) are better described by a restructuring of the spectral bulk. We develop a toy model with extensive output channels that recapitulates this phenomenon and show that edge of the spectrum still converges for sufficiently wide networks.
Clarissa Lauditi, Cengiz Pehlevan, Blake Bordelon
John A. Paulson School of Engineering and Applied Sciences, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Center for Mathematical Sciences and Applications, Harvard University +1
Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN, an ensemble of trained models defines an order parameter (the squared weight norm) and fluctuation proxies for the heat capacity and the susceptibility, with training epoch and weight decay serving as thermal and field control coordinates. To bridge neural architectures to real-space RG, we construct an exact coupling matrix via a second-order expansion of the loss with respect to per-neuron norms and use its Fiedler vector to define an effective geometry. This spectral ordering yields a monotonic spatial decay of coupling strength that is absent under random permutations. We then execute real-space RG by block-spin aggregating adjacent neurons and parameterizing the coupling matrix into off-diagonal band strengths. Linearizing this local RG map yields relevant scaling exponents and five derived critical exponents, with direct empirical estimates matching RG predictions within 1-10%. Overall, our framework provides a consistent effective description of grokking, suggesting the scaling and renormalization group formalism as a tool to investigate NN dynamics.
Min Chen, Junqi Wang, Zeguan Wu +2
Department of Computer Science, University of Pittsburgh, Pittsburgh, PA, USA
Why does gradient descent reliably find good solutions in non-convex neural network optimization, despite the landscape being NP-hard in the worst case? We show that gradient flow on L-layer ReLU networks without bias preserves L-1 conservation laws C_l = ||W_{l+1}||_F^2 - ||W_l||F^2, confining trajectories to lower-dimensional manifolds. Under discrete gradient descent, these laws break with total drift scaling as eta^alpha where alpha is approximately 1.1-1.6 depending on architecture, loss function, and width. We decompose this drift exactly as eta^2 * S(eta), where the gradient imbalance sum S(eta) admits a closed-form spectral crossover formula with mode coefficients c_k proportional to e_k(0)^2 * lambda{x,k}^2, derived from first principles and validated for both linear (R=0.85) and ReLU (R>0.80) networks. For cross-entropy loss, softmax probability concentration drives exponential Hessian spectral compression with timescale tau = Theta(1/eta) independent of training set size, explaining why cross-entropy self-regularizes the drift exponent near alpha=1.0. We identify two dynamical regimes separated by a width-dependent transition: a perturbative sub-Edge-of-Stability regime where the spectral formula applies, and a non-perturbative regime with extensive mode coupling. All predictions are validated across 23 experiments.