Neural Scaling Laws

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 53

All topics

A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling
Oct 5, 2026Itay Lavie, Clarissa Lauditi, Cengiz PehlevanEdge of Stability Neural Scaling Laws
John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA · Center for Brain Science, Harvard University, Cambridge, MA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Cambridge, MA
A Unified Scaling Law for Time Series Foundation Models
Oct 4, 2026Xilin Dai, Yiding Liu, Zewei Dong +2Time Series Forecasting Neural Scaling Laws
Ant International · The Chinese University of Hong Kong
Overcoming Kernel Redundancy for Scaling Logic Gate Networks
Oct 1, 2026Sejin Park, Hongjae Lee, Changwoo Han +1Neural Scaling Laws Conditional Computation
Department of Electrical Engineering Korea University
Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks
Oct 1, 2026Tilen Cadez, Sanghoon Lee, Kyoung-Min KimNeural Network Activation Functions Kolmogorov-Arnold Networks
Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk, 37673, Republic of Korea · Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Republic of Korea
Effective Does Not Mean Useful: Conditional Functional Substitutability for Redundancy and Scaling in Transformers
Sep 30, 2026Jiaheng Chen, Jiaxing Li, Yucheng Xiao +4Neural Scaling Laws
Harbin Institute of Technology, Shenzhen, China · Northeastern University, Shenyang, China · Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China +2
Neural Scaling Laws of Transformer Operator Network
Sep 27, 2026Haoran Yan, Zhongjie Shi, Yuanzhe Xi +2Neural Network Generalization Neural Network Approximation Theory
School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332, USA · Department of Mathematics, Emory University, Atlanta, GA 30322, USA · School of Computational Science and Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA
The Capability Manifold and ML Scaling Laws
Sep 23, 2026Syed Ali Raza Zaidi, Maryam HafeezScaling Laws Neural Scaling Laws
SPRINT Lab University of Leeds Leeds, LS2 9JT, UK.
A Scaling Study for fMRI Foundation Models
Sep 23, 2026Wenhao Ye, Xuanye Pan, Junfeng Xia +3Medical Imaging Foundation Models fMRI Foundation Models
Southern University of Science and Technology · Shenzhen University · SLAI +1
Symmetry without a manifold: intrinsic dimension on orbits
Sep 15, 2026Chon-Fai Kam, Miloud Bessafi, Frédéric CadetIntrinsic Dimensionality Neural Scaling Laws
Dipartimento di Fisica e Chimica, Universit`a degli Studi di Palermo, via Archirafi 36, I-90123 Palermo, Italy · EnergyLab, Universit´e de La R´eunion, F-97715 Saint-Denis, France · Universit´e Paris Cit´e and Universit´e de La R´eunion, BIGR, INSERM UMR S1134, F-75014 Paris, France +1
Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking
Sep 12, 2026Anish KatariaMultilayer Perceptrons Neural Network Memorization
Princeton University, Princeton, NJ, USA.
Coupled Scaling: A Representational Accessibility Framework for Neural Scaling Laws
Sep 3, 2026Jie WangNeural Representation Geometry Neural Scaling Laws
School of Civil and Commercial Law, Southwest University of Political Science and Law
Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
Aug 13, 2026Liu Ziyin, Yizhou Xu, Tomaso Poggio +1Dynamical Systems Neural Network Training Dynamics
1Massachusetts Institute of Technology · 2 ´Ecole Polytechnique F´ed´erale de Lausanne
Skaling: Chinchilla's Exponents Meet Kaplan's Coupling
Aug 7, 2026Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz +1Language Model Scaling Laws Neural Scaling Laws
1FAIR at Meta
Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension
Jul 27, 2026Jinhao Zhang, Zeyu Liu, Zicheng Yan +4Neural Network Generalization Residual Learning
Beijing University of Posts and Telecommunications · Institute of Computing Technology, Chinese Academy of Sciences · University of Science and Technology of China +1
EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments
Jul 6, 2026Deyao Zhu, Xin Zhou, Shengling Qin +44Long-Horizon Agent Evaluation AI Agent Benchmarks
Hyperparameter Transfer in Graph Neural Networks
Jul 6, 2026Gage DeZoort, Boris HaninGraph Neural Networks Neural Network Optimization
Department of Operations Research and Financial Engineering · School of Engineering and Applied Science · Princeton University +1
How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
Jul 1, 2026Fabian SchaippLanguage Model Scaling Laws Scaling Laws
Inria, ´Ecole Normale Sup´erieure, PSL Research University, Paris
From Approximation to Emergence: A Theory of Deep Learning
Jul 1, 2026Zhilin ZhaoNeural Network Generalization Deep Learning Optimization
Sun Yat-sen University
A Stochastic--Geometric Theory of Scaling Laws in Grokking
Jun 29, 2026Róisín Luo, Christian Gagné, Jonas Ngnawé +2Neural Network Generalization Neural Network Optimization
Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval +2
How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
Jun 26, 2026Julius Girardin, Emanuele Troiani, Yizhou Xu +3Neural Network Generalization Scaling Laws
1Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · 2Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Jun 23, 2026Yizhou Liu, Jeff GoreLanguage Model Scaling Laws Neural Scaling Laws
Massachusetts Institute of Technology Cambridge, MA 02139
Statistical Properties of Training & Generalization
Jun 18, 2026Itay Lavie, Noam Levi, Yonatan KahnNeural Network Generalization Scientific ML
John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Tel Aviv University, Tel Aviv, Israel · Department of Physics, University of Toronto and Vector Institute, Toronto, ON, Canada
Towards Engineering Scaling Laws with Pretraining Data Composition
Jun 18, 2026Jan-Lucas Uslu, Kevin Greif, Daniel Whiteson +1Synthetic Data Pretraining High-Energy Physics
Department of Applied Physics Stanford University Stanford, CA 94305, USA · Department of Physics and Astronomy University of California, Irvine Irvine, CA 92697, USA · Department of Particle Physics and Astrophysics SLAC National Accelerator Laboratory Stanford University Menlo Park, CA 94025, USA
Explaining Data Mixing Scaling Laws
Jun 6, 2026Rui Dai, Shuran ZhengData Mixture Optimization Neural Scaling Laws
Beijing Institute of Technology · IIIS, Tsinghua University
Compute-Optimal Network Design for Echocardiography Myocardial Segmentation and Perfusion Quantification using Neural Scaling Laws
Jun 4, 2026Clara Rodrigo González, Matthieu Toulemonde, Lasha Gvinianidze +5Ultrasound Image Segmentation Medical Image Analysis
Department of Bioengineering Imperial College London London, UK · National Heart and Lung Institute Imperial College London Guy’s and St. Thomas’ NHS Foundation Trust
Neuron Populations Exhibit Divergent Selectivity with Scale
Jun 2, 2026Amil Dravid, Yasaman Bahri, Alexei A. Efros +1Neural Network Interpretability Neural Encoding
UC Berkeley · TTIC
Structure and Scale in Simplicial Sequence Modelling
May 31, 2026Matthew Farrugia-RobertsRepresentation Geometry Transformer
Department of Computer Science, University of Oxford
Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail
May 29, 2026Konstantin Nikolaou, Jonas Scheunemann, Sven Krippendorf +2Representation Learning Neural Network Training Dynamics
Institute for Computational Physics, University of Stuttgart, Stuttgart, Germany · Department of Applied Mathematics and Theoretical Physics and Department of Physics, University of Cambridge, Cambridge, UK
How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions
May 28, 2026Jeff A. Bilmes, Gantavya Bhatt, Arnav M. DasSubmodular Optimization Neural Scaling Laws
Department of Electrical & Computer Engineering 2Paul G. Allen School of Computer Science & Engineering University of Washington, Seattle Seattle, WA 98195
On the Optimizer Dependence of Neural Scaling Laws
May 28, 2026Vansh Ramani, Shourya Vir JainNatural Gradient Methods Neural Network Optimization
Department of Computer Science and Engineering, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi

Oct 5, 2026, stat.MLJ/K move · Enter open · S save

A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling

Itay Lavie, Clarissa Lauditi, Cengiz Pehlevan

John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA · Center for Brain Science, Harvard University, Cambridge, MA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Cambridge, MA

Edge of Stability Neural Scaling Laws Stochastic Gradient Descent

Open paper AskPDF arXiv

A recurring design principle in modern optimizers is to decouple update magnitude from the raw gradient norm, yet its consequences for learning-curve and resource scaling remain unclear. We isolate this mechanism by studying normalized SGD in a random-feature model with power-law teacher and data covariance. Fixed-norm updates induce an effective learning rate that grows as gradients shrink. We derive a dynamical mean-field theory (DMFT) describing the joint dependence of the loss on training time, model width and batch size. Normalization initially accelerates SGD, mapping the power-law exponent

r_{\rm SGD}<1

2r_{\rm SGD}/(1-r_{\rm SGD})

, with exponential convergence at

r_{\rm SGD}=1

and formal finite-time convergence for

r_{\rm SGD}>1

. At finite step size, however, the same feedback ultimately breaks the acceleration and leads to marginal stability. The late-time theory yields width-limited, edge-of-stochastic-stability (EoSS), and deterministic edge-of-stability (EoS) regimes. These phases determine when larger batches or wider models reduce serial training time at comparable compute. We quantify in which of these phases increased batch size or width can compensate the excess compute use per step by fewer optimization steps to target loss. Linearized ResNet experiments on CIFAR-5M support the predicted acceleration, breakdown, and resource-scaling trends. Together, these results connect normalization-induced acceleration, EoS effects, and width--batch allocation within a solvable theory.

Momentum

Related topics

Latest papers 53