cs.LGNov 6, 2026

The Multiscale Single-Index Model: An MSE Analysis of Online SGD Dynamics

Authors: Gordon Dai, Joan Bruna

Organizations: New York University

Abstract

We consider the Multiscale Single-Index Model (MSIM), first introduced by Oymak and Soltanolkotabi (2021), as a stylized model for hierarchical feature learning with scale separation. Each layer extracts a shared single-index feature at one physical scale and passes it to the next, giving a tractable setting in which to study how deep architectures learn local multiscale representations. Under non-degeneracy and delocalization assumptions on the link function and planted features respectively, for the two hidden-layer case (K=2K=2), we analyze joint online spherical SGD under MSE loss, which is a canonical regression objective, in discrete time and show that it achieves weak recovery from random initialization in O(dlog⁡2d)O(d\log^2 d) samples, followed by strong recovery to accuracy ε\varepsilon in O(dlog⁡(1/ε))O(d\log(1/\varepsilon)) additional samples. The main technical challenge arises from the non-Gaussian learned features observed by the second layer. We address this challenge through a quantitative Gaussian comparison using delocalization, combined with sharp trajectory-level control of the coupled SGD dynamics. This work is a companion to Bruna (2026, arXiv:2607.03347), which studies recovery under a correlation objective for general KK.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning

    Jul 3, 2026Joan BrunaSingle-Index ModelsHierarchical Representation Learning

  2. Interactive Learning of Single-Index Models via Stochastic Gradient Descent

    Feb 19, 2026Nived Rajaraman, Yanjun HanSingle-Index ModelsGeneralized Linear Bandits

  3. Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model

    May 14, 2026Arie Wortsman-Zurich, Hugo Tabanelli, Yatin Dandi +2Hierarchical Representation LearningScaling Laws