Neural Scaling Laws

Ask about this topic

Overview

Neural scaling laws study how performance changes as models, datasets, training time, and compute grow, often fitting power laws to loss or task metrics. Empirical methods use controlled model and data sweeps, compute-allocation curves, and scaling-law fits to forecast performance and choose training budgets; theory connects these trends to statistics, optimization, spectral structure, and learned representations. Researchers are extending the framework beyond language models to vision, scientific, medical, and sequential tasks, while studying how architecture, data composition, optimizers, and training choices alter the results. A central challenge is that simple fits can fail across data-limited regimes, task metrics, or extrapolations, so reliable predictions need to account for these dependencies and their uncertainty.

Latest papers

CardsList

83 papers15 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

  1. Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks

    Oct 26, 2026Taishi Watanabe, Ryo Karakida, Jun-nosuke TeramaeScaling LawsNeural Network Training Dynamics

    Kyoto University · AIST · Institute of Science Tokyo

  2. Ambiguous Fits: The Rashomon Effect and Challenges in Scaling Law Estimation

    Oct 25, 2026Bruno Mlodozeniec, Spencer FreiNeural Scaling LawsScaling Laws

    University of Cambridge · Google DeepMind

  3. Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity

    Oct 24, 2026Datong Zhou, Blake BordelonNeural Scaling LawsResidual Learning

    Laboratoire Jacques-Louis Lions (LJLL) & Laboratoire de Probabilités, Statistique et Modélisation (LPSM) · Oden Institute and Department of Neuroscience University of Texas at Austin

  4. An Empirical Study of Reinforcement Learning Scaling Laws

    Oct 24, 2026Zeyu Zheng, Clare Lyle, Khimya Khetarpal +1Reinforcement LearningNeural Scaling Laws

    Google DeepMind

  5. A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks

    Oct 15, 2026Pankhil Gawade, Kirsten Fischer, Guy Wolf +1Neural Representation GeometryNeural Scaling Laws

    Institute of AI for Health, Helmholtz Munich · School of Computation, Information and Technology, Technical University of Munich · Université de Montréal +3

  6. Exact Scaling Laws for a Random-Features Bigram Model

    Oct 13, 2026Julian Enoch, Elvis DohmatobZipf's LawRandom Feature Methods

    Concordia University · Mila – Quebec AI Institute

  7. Chain Dimension Scaling: Communication-Efficient Loop-Style Scaling for Industrial Short-Video Ranking

    Oct 12, 2026Zhimeng Jiang, Yuening Li, Ruixiao Sun +4Communication-Efficient Distributed TrainingLearning to Rank

    Google

  8. Memorization in image classification: test set behavior and scaling laws

    Oct 12, 2026Jeevesh Attri, Florentin GuthNeural Network MemorizationNeural Network Training Dynamics

    Center for Data Science New York University · Center for Computational Neuroscience Flatiron Institute · Work done while partially affiliated with Center for Data Science, New York University.

  9. Emergent Inverse-Depth Scaling From Nonlinearity In Attention

    Oct 8, 2026Zirui Peng, Yizhou Liu, Ziming Liu +1Neural Scaling LawsScaling Laws

    Department of Physics, Peking University · Department of Physics, Massachusetts Institute of Technology, Cambridge, MA · College of AI, Tsinghua University

  10. A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling

    Oct 5, 2026Itay Lavie, Clarissa Lauditi, Cengiz PehlevanEdge of StabilityNeural Scaling Laws

    John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA · Center for Brain Science, Harvard University, Cambridge, MA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Cambridge, MA

  11. A Unified Scaling Law for Time Series Foundation Models

    Oct 4, 2026Xilin Dai, Yiding Liu, Zewei Dong +2Time Series ForecastingNeural Scaling Laws

    Ant International · The Chinese University of Hong Kong

  12. Overcoming Kernel Redundancy for Scaling Logic Gate Networks

    Oct 1, 2026Sejin Park, Hongjae Lee, Changwoo Han +1Neural Scaling LawsConditional Computation

    Department of Electrical Engineering Korea University

  13. Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks

    Oct 1, 2026Tilen Cadez, Sanghoon Lee, Kyoung-Min KimNeural Network Activation FunctionsKolmogorov-Arnold Networks

    Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk, 37673, Republic of Korea · Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Republic of Korea

  14. Effective Does Not Mean Useful: Conditional Functional Substitutability for Redundancy and Scaling in Transformers

    Sep 30, 2026Jiaheng Chen, Jiaxing Li, Yucheng Xiao +4Neural Scaling Laws

    Harbin Institute of Technology, Shenzhen, China · Northeastern University, Shenyang, China · Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China +2

  15. Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

    Sep 29, 2026Hyunseok Lee, Mihir Basil, Yizhou Liu +1Neural Network Training DynamicsNeural Scaling Laws

    Department of Physics, Massachusetts Institute of Technology, Cambridge, MA · Department of Physics, University of Pennsylvania, Philadelphia, PA

  16. SINO: Scale-Invariant Neural Operator

    Sep 29, 2026Kaichen Ouyang, Chenglei Yu, Chuanrui Wang +1Neural OperatorsNeural Scaling Laws

    Westlake University · University of Science and Technology of China

  17. Neural Scaling Laws of Transformer Operator Network

    Sep 27, 2026Haoran Yan, Zhongjie Shi, Yuanzhe Xi +2Neural Network GeneralizationNeural Network Approximation Theory

    School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332, USA · Department of Mathematics, Emory University, Atlanta, GA 30322, USA · School of Computational Science and Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA

  18. Neural Dynamics as the Composition of Quantized Units

    Sep 26, 2026Jacopo Minniti, Aravinth Kulanthaivelu, Richard SproatNeural Network Training DynamicsCompositional Representation Learning

    Minerva University · Sakana AI

  19. Emergent One-Third Scaling Law as Attention Tries to Concentrate

    Sep 26, 2026Yizhou Liu, Sara Kangaslahti, Jeff GoreLanguage Model Scaling LawsScaling Laws

    Massachusetts Institute of Technology · Harvard University

  20. The Capability Manifold and ML Scaling Laws

    Sep 23, 2026Syed Ali Raza Zaidi, Maryam HafeezScaling LawsNeural Scaling Laws

    SPRINT Lab University of Leeds Leeds, LS2 9JT, UK.

  21. A Scaling Study for fMRI Foundation Models

    Sep 23, 2026Wenhao Ye, Xuanye Pan, Junfeng Xia +3Medical Imaging Foundation ModelsfMRI Foundation Models

    Southern University of Science and Technology · Shenzhen University · SLAI +1

  22. Do Quantum Models Scale Like LLMs?

    Sep 17, 2026David S. Berman, Ying-Jer Kao, Roger G. Melko +1Neural Scaling LawsTransformer

    Centre for Theoretical Physics, Queen Mary University of London, Mile End Road, London E1 4NS, United Kingdom · Department of Physics, National Taiwan University, Taipei 10607, Taiwan · Department of Physics and Astronomy, University of Waterloo, Waterloo, Ontario N2L 3G1, Canada +1

  23. Symmetry without a manifold: intrinsic dimension on orbits

    Sep 15, 2026Chon-Fai Kam, Miloud Bessafi, Frédéric CadetIntrinsic DimensionalityModular Addition

    Dipartimento di Fisica e Chimica, Universit`a degli Studi di Palermo, via Archirafi 36, I-90123 Palermo, Italy · EnergyLab, Universit´e de La R´eunion, F-97715 Saint-Denis, France · Universit´e Paris Cit´e and Universit´e de La R´eunion, BIGR, INSERM UMR S1134, F-75014 Paris, France +1

  24. Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

    Sep 9, 2026Anish KatariaNeural Network Training DynamicsNeural Scaling Laws

    Princeton University, Princeton, NJ, USA.

  25. Coupled Scaling: A Representational Accessibility Framework for Neural Scaling Laws

    Sep 3, 2026Jie WangNeural Representation GeometryNeural Scaling Laws

    School of Civil and Commercial Law, Southwest University of Political Science and Law

  26. Linear RNN Scaling Laws: When Longer Sequences Beat More Sequences

    Aug 23, 2026Ziyan Chen, Zhongzhu Zhou, Linhe Zheng +2Neural Scaling LawsLanguage Model Scaling Laws

    The University of Sydney · Together AI · The Pennsylvania State University

  27. Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws

    Aug 13, 2026Liu Ziyin, Yizhou Xu, Tomaso Poggio +1Dynamical SystemsNeural Network Training Dynamics

    1Massachusetts Institute of Technology · 2 ´Ecole Polytechnique F´ed´erale de Lausanne

  28. Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

    Aug 7, 2026Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz +1Language Model Scaling LawsNeural Scaling Laws

    1FAIR at Meta

  29. Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension

    Jul 27, 2026Jinhao Zhang, Zeyu Liu, Zicheng Yan +4Neural Network GeneralizationResidual Learning

    Beijing University of Posts and Telecommunications · Institute of Computing Technology, Chinese Academy of Sciences · University of Science and Technology of China +1

  30. EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

    Jul 6, 2026Deyao Zhu, Xin Zhou, Shengling Qin +44Long-Horizon Agent EvaluationAI Agent Benchmarks