Neural Scaling Laws

Ask about this topic

Overview

Neural scaling laws study how performance changes as models, datasets, training time, and compute grow, often fitting power laws to loss or task metrics. Empirical methods use controlled model and data sweeps, compute-allocation curves, and scaling-law fits to forecast performance and choose training budgets; theory connects these trends to statistics, optimization, spectral structure, and learned representations. Researchers are extending the framework beyond language models to vision, scientific, medical, and sequential tasks, while studying how architecture, data composition, optimizers, and training choices alter the results. A central challenge is that simple fits can fail across data-limited regimes, task metrics, or extrapolations, so reliable predictions need to account for these dependencies and their uncertainty.

Latest papers

CardsList

83 papers15 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

  1. Prescriptive Scaling Laws for Data Constrained Training

    May 2, 2026Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2Weight DecayLanguage Model Scaling Laws

    Department of Computer Science Cornell University Ithaca, NY

  2. A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws

    Apr 27, 2026Jun Shu, Junxiong Jia, Deyu Meng +1Neural Scaling LawsAI Scaling Laws

    School of Mathematics and Statistics, Xi’an Jiaotong University.

  3. OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens

    Apr 20, 2026Konstantin F. Willeke, Polina Turishcheva, Alex Gilbert +18Neural DecodingVisual Neuroscience

    Department of Ophthalmology, Byers Eye Institute, Stanford University · 2Stanford Bio-X, Stanford University · 3Wu Tsai Neurosciences Institute, Stanford University +6

  4. How Much Data is Enough? The Zeta Law of Discoverability in Biomedical Data, featuring the enigmatic Riemann zeta function

    Apr 19, 2026Paul M. ThompsonNeural Scaling LawsData-Efficient Learning

    Stevens Institute for Neuroimaging & Informatics · University of Southern California · Los Angeles, CA, USA

  5. Generalization and Scaling Laws for Mixture-of-Experts Transformers

    Apr 10, 2026Mansour Zoubeirou a MayakiMixture-of-Experts ModelsAI Scaling Laws

    Univ Lyon, Lyon 2, LIRIS UMR 5205 F-69676, Lyon, France

  6. Towards Scaling Law Analysis For Spatiotemporal Weather Data

    Apr 6, 2026Alexander Kiefer, Prasanna Balaprakash, Xiao WangWeather ForecastingSpatiotemporal Forecasting

    Bredesen Center, University of Tennessee, Knoxville, Tennessee · Oak Ridge National Laboratory, Oak Ridge, Tennessee

  7. On Neural Scaling Laws for Weather Emulation through Continual Training

    Mar 26, 2026Shashank Subramanian, Alexander Kiefer, Arnur Nigmetov +3Neural Scaling LawsWeather Forecasting

    Lawrence Berkeley National Laboratory · Oak Ridge National Laboratory · ICSI and UC Berkeley +1

  8. Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits

    Mar 21, 2026Eric Czech, Zhiwei Xu, Yael Elmatad +2Scaling LawsNeural Scaling Laws

    Open Athena AI Foundation · Department of Statistics, University of Michigan, Ann Arbor

  9. Scaling Laws and Pathologies of Single-Layer PINNs: Network Width and PDE Nonlinearity

    Mar 13, 2026Faris ChaudhryPhysics-Informed MLNeural Scaling Laws

    Department of Computer Science, Imperial College London

  10. Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia

    Mar 12, 2026Haining Pan, Nakul Aggarwal, J. H. PixleyPhase Transitions in Neural NetworksNeural Network Pruning

    Department of Physics and Astronomy, Center for Materials Theory, Rutgers University, Piscataway, NJ 08854 USA · Center for Computational Quantum Physics, Flatiron Institute, New York, New York 10010, USA

  11. Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design

    Mar 11, 2026Junzhuo Li, Peijie Jiang, Changxin Tian +3Mixture-of-Experts ModelsAI Scaling Laws

    The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · Ant Group

  12. Scaling Laws in the Tiny Regime: How Small Models Change Their Mistakes

    Mar 7, 2026Mohammed Alnemari, Rizwan Qureshi, Nader BegrazadahNeural Scaling LawsTiny ML

    Faculty of Computer Science and Information Technology, University of Tabuk, Tabuk, Saudi Arabia · AIST Research Center, University of Tabuk, Tabuk, Saudi Arabia · Department of Computer Science, Salim Habib University, Karachi, Pakistan +1

  13. When Bigger is Worse: A Practitioner's Guide to Model Selection Under Data Scarcity

    Mar 2, 2026Kwame Mbobda-Kuate, Gabriel KasmiModel SelectionEfficient Object Detection

    ENSAE Paris, Palaiseau, 91120, France. · Centre O.I.E., Mines Paris – PSL University, Sophia-Antipolis, 06904, France.

  14. Efficient Decoder Scaling Strategy for Neural Routing Solvers

    Feb 28, 2026Qing Luo, Fu Luo, Ke Li +1Neural Combinatorial OptimizationNeural Scaling Laws

    Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China. · Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China.

  15. Scaling View Synthesis Transformers

    Feb 24, 2026Evan Kim, Hyunwoo Ryu, Thomas W. Mitchel +1Novel View SynthesisNeural Scaling Laws

    MIT · Adobe

  16. Scaling Law of Neural Koopman Operators

    Feb 23, 2026Abulikemu Abuduweili, Yuyang Pang, Feihan Li +1Koopman Operator LearningRobot Dynamics Learning

    Authors are with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA.

  17. Neural Scaling Laws for Boosted Jet Tagging

    Feb 17, 2026Matthias Vigl, Nicole Hartman, Michael Kagan +1Neural Scaling LawsAI Scaling Laws

    Technical University of Munich · SLAC National Accelerator Laboratory

  18. Scaling Laws for Masked-Reconstruction Transformers on Single-Cell Transcriptomics

    Feb 16, 2026Ihor KendiukhovNeural Scaling LawsTransformer

    Department of Computer Science, University of Tübingen

  19. Divine Benevolence is an x2x^2: GLUs scale asymptotically faster than MLPs

    Feb 16, 2026Alejandro Francisco QueirugaNeural Scaling LawsNeural Network Approximation Theory

  20. μμpscaling small models: Principled warm starts and hyperparameter transfer

    Feb 11, 2026Yuxin Ma, Nan Chen, Mateo Díaz +3Neural Network OptimizationHyperparameter Transfer

    Department of Applied Mathematics and Statistics Johns Hopkins University

  21. Deriving Neural Scaling Laws from the statistics of natural language

    Feb 7, 2026Francesco Cagnetta, Allan Raventós, Surya Ganguli +1Language Model Scaling LawsNeural Scaling Laws

    Theoretical and Scientific Data Science, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Applied Physics, Stanford University, Stanford, California · Department of Physics and Astronomy, Johns Hopkins University, Baltimore, Maryland +1

  22. From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis

    Dec 15, 2025Łukasz DębowskiZipf's LawNeural Scaling Laws

    Institute of Computer Science, Polish Academy of Sciences, ul. Jana Kazimierza 5, 01-248 Warszawa, Poland

  23. Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models

    Jun 16, 2025Zhenyu Liao, Michael W. MahoneyRandom Matrix TheoryNeural Network Generalization

    School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, China · ICSI, LBNL, and Department of Statistics, University of California, Berkeley, USA