A Spectral Renormalization-Group View of Learning
Organizations: Calculation Consulting, San Francisco, CA
Abstract
We study how the layers in deep neural networks (NNs) converge, using Wilsonian Renormalization-Group (RG) ideas to organize the spectral picture. During training, these layers often develop heavy-tailed power-law (PL) spectra, and the strongest checkpoints often appear near the PL exponent . This pattern raises a simple question: why should this value mark a well-trained layer? The answer is both statistical and physical. Our statistical hypothesis is that marks the boundary between stable if not optimal learning and overfitting via non-self-averaging. The physical answer is that the same value makes the spectral energy gain scale evenly across logarithmic spectral scales, which is the proposed Wilsonian marginal condition. We then apply the ERG trace-log condition, the proposed SETOL theory, which we justify by taking volume-preserving, scale-invariant transformations to remove arbitrary layer scale along the proposed RG flow. The resulting spectral RG flow connects the HTSR and SETOL theories to optimizers such as Muon, and motivates a new optimizer extension, WW-PGD. Together, these results point toward a spectral Renormalization-Group view of learning, in which NNs may approach shared spectral structure in the settings studied here.
Figures & tables
| Regime | spectral energy per scale | Scale-flow meaning | Statistical meaning | Learning meaning |
|---|---|---|---|---|
| Spectral band gain decays; weak-tail sector or approach toward the trivial fixed point . | First moment self-averages; dominant tail components stay controlled. | Stable but potentially weakly correlated or undertrained tail. | ||
| Scale-balanced logarithmic-band state; candidate when ECS aligns with the fitted tail. | Marginal self-averaging boundary. Proposed universal critical exponent. | Maximal stable structured-variation capture before tail-component domination. | ||
| Spectral band gain grows toward the dominant tail components. | Tail components and fluctuations can dominate the first moment. | Overfit risk; correlation traps; held-out degradation. |
| Term | Notation | Meaning here |
|---|---|---|
| RG step | Project to the retained ECS and evaluate the ESD on the SETOL scale. | |
| Fixed point | A SETOL-normalized ESD that is unchanged by the RG step. | |
| Weak-tail branch | The trivial solution HTSR Random-like phase and nearby ESDs. | |
| Correlated branch | The empirical RG fixed point candidate. | |
| Band Energy | Spectral energy carried by one logarithmic spectral band. | |
| Band gain | Spectral energy per logarithmic spectral scale. |
Neural Network Training Dynamics
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
See all 549 papers