cs.LGMay 22, 2026

Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent

Authors: Anherutowa Calvo

Abstract

The curvature exponent αα in hkσkαh_k \propto σ_k^α -- governing how Hessian eigenvalues scale with gradient singular values -- varies systematically across layer types (α2α\approx 2 for convolutions, 1\approx 1 for transformer attention, <1< 1 for MLP up-projections). Why? We prove the Spectral Alignment Decomposition: α=2+dlogΦk/dlogσkα= 2 + d\logΦ_k / d\logσ_k, where ΦkΦ_k measures alignment between Kronecker factor eigenbases and gradient singular directions. This reduces "why does αα vary?" to a geometric question we answer for LayerNorm, residual connections, and softmax heads. The decomposition implies a spectral transfer identity s=αγs = αγ linking curvature exponent, effective gradient rank-decay γγ, and Hessian decay exponent ss. The identity is algebraic; its empirical content is that αα and γγ, fit on independent data (HVPs vs. SVD), recover ss to ~2% median error across 93 layers, five architectures, and three datasets -- with no free parameters. A zeta-function bound on participation ratio shows curvature concentrates onto effectively one direction per layer. As a proof of concept, we derive the architecture-adaptive preconditioner T(σ;α)T(σ;α) and show that Spectral Newton -- implementing TT in the gradient singular basis -- outperforms AdamW on vision benchmarks where α2α\approx 2.

Explore similar work

CardsList