Overview

Neural scaling laws study how performance changes as models, datasets, training time, and compute grow, often fitting power laws to loss or task metrics. Empirical methods use controlled model and data sweeps, compute-allocation curves, and scaling-law fits to forecast performance and choose training budgets; theory connects these trends to statistics, optimization, spectral structure, and learned representations. Researchers are extending the framework beyond language models to vision, scientific, medical, and sequential tasks, while studying how architecture, data composition, optimizers, and training choices alter the results. A central challenge is that simple fits can fail across data-limited regimes, task metrics, or extrapolations, so reliable predictions need to account for these dependencies and their uncertainty.

Latest papers

91 papers9 papers in the last four weeks, down 47% on the four weeks before. 0.1% of all new papers.

Jun 16, 2025

Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models

Modern Machine Learning (ML) and Deep Neural Networks (DNNs) often operate on high-dimensional data and rely on overparameterized models, where classical low-dimensional intuitions break down. In particular, the proportional regime where the data dimension, sample size, and number of model parameters are all large and comparable, gives rise to novel and sometimes counterintuitive behaviors. This paper extends traditional Random Matrix Theory (RMT) beyond eigenvalue-based analysis of linear models to address the challenges posed by nonlinear ML models such as DNNs in this regime. We introduce the concept of High-dimensional Equivalent, which unifies and generalizes both Deterministic Equivalent and Linear Equivalent, to systematically address three technical challenges: high dimensionality, nonlinearity, and the need to analyze generic eigenspectral functionals. Leveraging this framework, we provide precise characterizations of the training and generalization performance of linear models, nonlinear shallow networks, and deep networks. Our results capture rich phenomena, including scaling laws, double descent, and nonlinear learning dynamics, offering a unified perspective on the theoretical understanding of deep learning in high dimensions.