Overview

Neural scaling laws study how performance changes as models, datasets, training time, and compute grow, often fitting power laws to loss or task metrics. Empirical methods use controlled model and data sweeps, compute-allocation curves, and scaling-law fits to forecast performance and choose training budgets; theory connects these trends to statistics, optimization, spectral structure, and learned representations. Researchers are extending the framework beyond language models to vision, scientific, medical, and sequential tasks, while studying how architecture, data composition, optimizers, and training choices alter the results. A central challenge is that simple fits can fail across data-limited regimes, task metrics, or extrapolations, so reliable predictions need to account for these dependencies and their uncertainty.

Latest papers

CardsList

91 papers9 papers in the last four weeks, down 47% on the four weeks before. 0.1% of all new papers.

  1. Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models

    Jun 16, 2025Zhenyu Liao, Michael W. MahoneyRandom Matrix TheoryNeural Network Generalization