stat.MLFeb 5, 2026

Optimal scaling laws in learning hierarchical multi-index models

Authors: Leonardo Defilippis, Florent Krzakala, Bruno Loureiro, Antoine Maillard

Organizations: Departement d’Informatique, École Normale Supérieure, PSL & CNRS · Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · INRIA Paris

Abstract

In this work, we provide a sharp theory of scaling laws for two-layer neural networks trained on a class of hierarchical multi-index targets, in a genuinely representation-limited regime. We derive exact information-theoretic scaling laws for subspace recovery and prediction error, revealing how the hierarchical features of the target are sequentially learned through a cascade of phase transitions. We further show that these optimal rates are achieved by a simple, target-agnostic spectral estimator, which can be interpreted as the small learning-rate limit of gradient descent on the first-layer weights. Once an adapted representation is identified, the readout can be learned statistically optimally, using an efficient procedure. As a consequence, we provide a unified and rigorous explanation of scaling laws, plateau phenomena, and spectral structure in shallow neural networks trained on such hierarchical targets.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model

    May 14, 2026Arie Wortsman-Zurich, Hugo Tabanelli, Yatin Dandi +2Hierarchical Representation LearningScaling Laws

  2. The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning

    Jul 3, 2026Joan BrunaSingle-Index ModelsHierarchical Representation Learning