stat.MLMay 5, 2026

Free Decompression with Algebraic Spectral Curves

Authors: Siavash Ameli, Chris van der Heide, Liam Hodgkinson, Michael W. Mahoney

Organizations: 1ICSI and Department of Statistics, University of California, Berkeley · 2Dept. of Electrical and Electronic Engineering, University of Melbourne · School of Mathematics and Statistics, University of Melbourne · 4ICSI, LBNL, and Department of Statistics, University of California, Berkeley

Abstract

Tools from random matrix theory have become central to deep learning theory, using spectral information to provide mechanisms for modeling generalization, robustness, scaling, and failure modes. While often capable of modeling empirical behavior, practical computations are limited by matrix size, often imposing a restriction to models that are too small to be realistic. This motivates the inference of properties of larger models from the behavior of smaller ones. Free decompression (FD) is a recently proposed method for extrapolating spectral information across matrix sizes, but its utility is currently limited by strong assumptions that preclude its implementation on more realistic machine learning (ML) models. We use algebraic spectral curve theory to provide a general FD methodology for spectral densities whose Stieltjes transform satisfies an algebraic relation, a modeling assumption that is more likely to hold in practice. This recasts FD as an evolution along spectral curves which can be readily integrated. Our framework enables the expansion of spectral densities that have multiple or multi-modal bulks, that exist at multiple scales, and that contain atoms, all characteristic of real-world data and popular ML models. We demonstrate the efficacy of our framework on models of interest in modern ML, including Hessian and activation matrices associated with neural networks and large-scale diffusion models.

Explore similar work

CardsList
Jun 16, 2025stat.ML

Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models

Modern Machine Learning (ML) and Deep Neural Networks (DNNs) often operate on high-dimensional data and rely on overparameterized models, where classical low-dimensional intuitions break down. In particular, the proportional regime where the data dimension, sample size, and number of model parameters are all large and comparable, gives rise to novel and sometimes counterintuitive behaviors. This paper extends traditional Random Matrix Theory (RMT) beyond eigenvalue-based analysis of linear models to address the challenges posed by nonlinear ML models such as DNNs in this regime. We introduce the concept of High-dimensional Equivalent, which unifies and generalizes both Deterministic Equivalent and Linear Equivalent, to systematically address three technical challenges: high dimensionality, nonlinearity, and the need to analyze generic eigenspectral functionals. Leveraging this framework, we provide precise characterizations of the training and generalization performance of linear models, nonlinear shallow networks, and deep networks. Our results capture rich phenomena, including scaling laws, double descent, and nonlinear learning dynamics, offering a unified perspective on the theoretical understanding of deep learning in high dimensions.
Zhenyu Liao, Michael W. Mahoney
High-DimensionalOver-Parameterized Neural NetworksLargest EigenvalueLow-Data RegimesRegression
AskPDF
May 22, 2026cs.LG

Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent

The curvature exponent αα in hk∝σkαh_k \propto σ_k^α -- governing how Hessian eigenvalues scale with gradient singular values -- varies systematically across layer types (α≈2α\approx 2 for convolutions, ≈1\approx 1 for transformer attention, <1< 1 for MLP up-projections). Why? We prove the Spectral Alignment Decomposition: α=2+dlog⁡Φk/dlog⁡σkα= 2 + d\logΦ_k / d\logσ_k, where ΦkΦ_k measures alignment between Kronecker factor eigenbases and gradient singular directions. This reduces "why does αα vary?" to a geometric question we answer for LayerNorm, residual connections, and softmax heads. The decomposition implies a spectral transfer identity s=αγs = αγ linking curvature exponent, effective gradient rank-decay γγ, and Hessian decay exponent ss. The identity is algebraic; its empirical content is that αα and γγ, fit on independent data (HVPs vs. SVD), recover ss to ~2% median error across 93 layers, five architectures, and three datasets -- with no free parameters. A zeta-function bound on participation ratio shows curvature concentrates onto effectively one direction per layer. As a proof of concept, we derive the architecture-adaptive preconditioner T(σ;α)T(σ;α) and show that Spectral Newton -- implementing TT in the gradient singular basis -- outperforms AdamW on vision benchmarks where α≈2α\approx 2.
Anherutowa Calvo
Hessian EigenvalueHessian MatrixLocal CurvatureConvex LossSingular Value DecompositionNewton+1
AskPDF
May 12, 2026stat.ML

Minimax Rates and Spectral Distillation for Tree Ensembles

Tree ensembles such as random forests (RFs) and gradient boosting machines (GBMs) are among the most widely used supervised learners, yet their theoretical properties remain incompletely understood. We adopt a spectral perspective on these algorithms, with two main contributions. First, we derive minimax-optimal convergence for RF regression, showing that, under mild regularity conditions on tree growth, the eigenvalue decay of the induced kernel operator governs the statistical rate. Second, we exploit this spectral viewpoint to develop compression schemes for tree ensembles. For RFs, leading eigenfunctions of the kernel operator capture the dominant predictive directions; for GBMs, leading singular vectors of the smoother matrix play an analogous role. Learning nonlinear maps for these spectral representations yields distilled models that are orders of magnitude smaller than the originals while maintaining competitive predictive performance. Our methods compare favorably to state of the art algorithms for forest pruning and rule extraction, with applications to resource constrained computing.
Binh Duc Vu, David S. Watson
Random ForestSpectral AlgorithmsOptimal Sample ComplexitySupervised LearningOptimalityDataset Distillation
AskPDF