Scaling Laws

Latest papers 50

14 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Emergent Inverse-Depth Scaling From Nonlinearity In Attention
Oct 8, 2026Zirui Peng, Yizhou Liu, Ziming Liu +1Neural Scaling Laws Scaling Laws
Department of Physics, Peking University · Department of Physics, Massachusetts Institute of Technology, Cambridge, MA · College of AI, Tsinghua University
How to scale your HEP ML models: A recipe for robust architecture comparisons at scale
Oct 5, 2026Matthias Vigl, Nikita Pond, Jackson Barr +5High-Energy Physics Scaling Laws
Technical University of Munich · SLAC National Accelerator Laboratory · University College London +4
Writing as a Self-Organized Critical Process
Oct 4, 2026Nikolay MikhaylovskiyScaling Laws
NTR Labs / Moscow, Russia · Higher IT School of Tomsk State University / Tomsk, Russia
Less is more: error-distance scaling relation for data-efficient kilometer-scale downscaling of extreme heat
Sep 30, 2026Ahmed Marey, Henry Lu, Abhishek Gaur +6Spatial Generalization Scaling Laws
Centre for Zero Energy Building Studies, Department of Building, Civil and Environmental Engineering, Concordia University, Montreal, H3G 1M8 Canada · Building and Climate Interface, Construction Research Centre, National Research Council Canada, Ottawa, ON, K1A 0R6, Canada · Department of Architecture, School of Sciences and Engineering, The American University in Cairo, New Cairo 11835, Egypt +3
When do data mixtures improve scaling laws? Insights from high-dimensional regression
Sep 29, 2026Diyuan Wu, Lehan Chen, Theodor Misiakiewicz +1Data Mixture Optimization Scaling Laws
Institute of Science and Technology Austria (ISTA). · Department of Applied and Computational Mathematics, Yale University. · Department of Statistics and Data Science, Yale University.
Search Dimension in Unlabeled Projection Pursuit: A Scaling Law for Subspace Restriction
Sep 29, 2026Rares Dimitrie Grozavescu, Mark GirolamiScaling Laws
University of Cambridge
Paired Multimodal Scaling Laws
Sep 28, 2026Marcus Ma, Shrikanth NarayananMultimodal Classification Scaling Laws
University of Southern California
ScAn-Bench: Evaluating Scaling Analysis Methodology
Sep 28, 2026Artin Sermaxhaj, Nastaran Alipour, Donat Sinani +4VLM Evaluation LLM Evaluation
University of Freiburg · Zuse School ELIZA · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)
How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
Sep 28, 2026Lin Chen, Bolin Ni, Qi Yang +6Multimodal Pretraining Multimodal Large Language Models
CASIA · UCAS · Foundation Model Department, Tencent
Emergent One-Third Scaling Law as Attention Tries to Concentrate
Sep 26, 2026Yizhou Liu, Sara Kangaslahti, Jeff GoreLanguage Model Scaling Laws Scaling Laws
Massachusetts Institute of Technology · Harvard University
The Capability Manifold and ML Scaling Laws
Sep 23, 2026Syed Ali Raza Zaidi, Maryam HafeezScaling Laws Neural Scaling Laws
SPRINT Lab University of Leeds Leeds, LS2 9JT, UK.
Scaling Laws for Physics-Aware ACOPF Surrogate Learning
Sep 14, 2026Yijiang Li, Emon Dey, Stefano Fenu +3Neural Surrogate Modeling Optimal Power Flow
Argonne National Laboratory Lemont, USA · Oak Ridge National Laboratory Oak Ridge, USA
One Spectrum, Two Resources: Data-Memory Scaling in Autoregressive Prediction
Sep 11, 2026Chiwun Yang, Xiaoyu LiScaling Laws Autoregressive Modeling
City University of Hong Kong · University of New South Wales
Hyperparameter Scaling Laws Across MoE Sparsity
Sep 8, 2026Changxin Tian, Kunlong Chen, Jia Liu +3Language Model Scaling Laws Sparse Mixture-of-Experts
Ling Team, Ant Group
SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws
Sep 7, 2026Konstantinos Christopher Tsiolis, Denny Wu, Christos Thrampoulidis +1Logistic Regression Multiclass Classification
University of Toronto, Vector Institute · New York University, Flatiron Institute · University of British Columbia +1
On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashing
Sep 7, 2026Yuval Koren, Assaf Ben-Kish, Raja Giryes +2Mechanistic Interpretability Scaling Laws
Blavatnik School of Computer Science and AI, Tel Aviv University
Scaling Laws, Tabular Data and Actuarial Ratemaking Models
Sep 2, 2026Ronald RichmanTabular ML Scaling Laws
insureai and Bayes Business School, Johannesburg, South Africa
Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
Aug 28, 2026Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4Language Model Pretraining Learning Rate Scheduling
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +6
Optimal Training-Time Scaling in Gradual Adaptation
Aug 5, 2026Zonghuan Xu, Krishna HarishDomain Adaptation Scaling Laws
Fudan University Shanghai, China · Lawrence E. Elkins High School Missouri City, Texas, USA
Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers
Jul 30, 2026Chongjian Ge, Hanwen Jiang, Tianyu Wang +9Video Diffusion Models Diffusion Transformer
Adobe Research
Bridging Compute- and Data-Optimal Pretraining
Jul 28, 2026Tian Qin, Kimia Hamidieh, David Alvarez-MelisLanguage Model Pretraining Language Model Scaling Laws
Harvard University · MIT CSAIL
Scaling Native Multimodal Pre-Training From Scratch
Jul 24, 2026Haoyuan Wu, Aoqi Wu, Hai Wang +3Cross-Modal Learning Multimodal Pretraining
The Chinese University of Hong Kong · LLM Department, Tencent
Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study
Jul 23, 2026Kaihua DingTabular ML Scaling Laws
University of Pennsylvania
How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
Jul 1, 2026Fabian SchaippLanguage Model Scaling Laws Scaling Laws
Inria, ´Ecole Normale Sup´erieure, PSL Research University, Paris
How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
Jun 26, 2026Julius Girardin, Emanuele Troiani, Yizhou Xu +3Neural Network Generalization Scaling Laws
1Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · 2Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)
Data-driven Machine Learning Cannot Reach Symbolic-level Logical Reasoning -- The Limit of the Scaling Law
Jun 24, 2026Tiansi Dong, Mateja Jamnik, Pietro LiòLogical Reasoning Symbolic Reasoning
The Alan Turing Institute, 96 Euston Road, London, NW1 2DB, UK. · Department of Computer Science and Technology, University of Cambridge, 15 JJ Thomson Ave., Cambridge, CB3 0FD, UK.
Solve for the Hyperparameter, Skip the Search: Kolmogorov-Optimal Scaling Laws for Spline Regression
Jun 22, 2026Yong Yi Bay, Kathleen A. YearickModel Selection Scaling Laws
PhD, University of Illinois at Urbana-Champaign
How Well Can Your Video Model Remember? Measuring Memory-Budget Trade-offs in Long Video Understanding
Jun 17, 2026Yixian TianVLM Evaluation Streaming Video Understanding
1Fudan University · 2Beijing University of Posts and Telecommunications
Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
Jun 5, 2026Zhiwei Xu, Shihao Wu, Hanseul Cho +2Language Model Pretraining Language Model Scaling Laws
University of Michigan · KAIST AI
Streaming Communication in Multi-Agent Reasoning
Jun 3, 2026Zhen Yang, Xiaogang Xu, Wen Wang +3Multi-Agent LLM Systems Multi-Agent System Optimization
HKUST(GZ) · ZJU · Alibaba Group +1

Oct 8, 2026, cs.LGJ/K move · Enter open · S save

Emergent Inverse-Depth Scaling From Nonlinearity In Attention

Zirui Peng, Yizhou Liu, Ziming Liu, Jeff Gore

Department of Physics, Peking University · Department of Physics, Massachusetts Institute of Technology, Cambridge, MA · College of AI, Tsinghua University

Neural Scaling Laws Scaling Laws Attention Mechanisms Transformer Attention

Open paper AskPDF arXiv

Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the parameter count scaling, existing theory posits power-law scaling with model depth. In linear-attention models, this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones. Large language models, however, can be strongly nonlinear. Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra. Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel. Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth. Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant.