Optimal scaling laws in learning hierarchical multi-index models
Organizations: Departement d’Informatique, École Normale Supérieure, PSL & CNRS · Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · INRIA Paris
Abstract
In this work, we provide a sharp theory of scaling laws for two-layer neural networks trained on a class of hierarchical multi-index targets, in a genuinely representation-limited regime. We derive exact information-theoretic scaling laws for subspace recovery and prediction error, revealing how the hierarchical features of the target are sequentially learned through a cascade of phase transitions. We further show that these optimal rates are achieved by a simple, target-agnostic spectral estimator, which can be interpreted as the small learning-rate limit of gradient descent on the first-layer weights. Once an adapted representation is identified, the readout can be learned statistically optimally, using an efficient procedure. As a consequence, we provide a unified and rigorous explanation of scaling laws, plateau phenomena, and spectral structure in shallow neural networks trained on such hierarchical targets.
Figures & tables
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Neural Scaling Laws
Neural scaling laws describe how model performance changes with training data, compute, model capacity, and other resources.
See all 91 papers