Language Model Scaling Laws

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 95

All topics

Deriving Neural Scaling Laws from the statistics of natural language
Feb 7, 2026Francesco Cagnetta, Allan Raventós, Surya Ganguli +1Language Model Scaling Laws Neural Scaling Laws
Theoretical and Scientific Data Science, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Applied Physics, Stanford University, Stanford, California · Department of Physics and Astronomy, Johns Hopkins University, Baltimore, Maryland +1
BRIDGE: Predicting Human Task Completion Time From Model Performance
Feb 6, 2026Fengyuan Liu, Jay Gala, Nilaksh +3LLM Evaluation Language Model Scaling Laws
Mila - Quebec AI Institute · McGill University · Polytechnique Montréal +3
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Jan 12, 2026Xin Cheng, Rui Tian, Wangding Zeng +18Memory-Augmented Language Models Language Model Scaling Laws
Peking University · DeepSeek-AI
A Large Scale Investigation of Scaling Limits in Chemical Language Models
Aug 19, 2025Roshan Balaji, Kamran Chitsaz, Quentin Fournier +2Language Model Scaling Laws AI-Assisted Scientific Research
BioSystems Engineering and Control Lab · Wadhwani School of Data Science and AI, IIT Madras · The Centre for Integrative Biology and Systems medicinE (IBSE) +5
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Date pendingShaowen Wang, Ge Zhang, Kairong Luo +6Language Model Scaling Laws Transformer Attention
Tsinghua University · ByteDance Seed · M-A-P

Feb 7, 2026, cs.LGJ/K move · Enter open · S save

Deriving Neural Scaling Laws from the statistics of natural language

Francesco Cagnetta, Allan Raventós, Surya Ganguli, Matthieu Wyart

Theoretical and Scientific Data Science, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Applied Physics, Stanford University, Stanford, California · Department of Physics and Astronomy, Johns Hopkins University, Baltimore, Maryland+1

Language Model Scaling Laws Neural Scaling Laws

Open paper AskPDF arXiv

Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset. We provide the first such theory in the case of data-limited scaling laws. We isolate two key statistical properties of language that alone can predict neural scaling exponents: (i) the decay of pairwise token correlations with time separation between token pairs, and (ii) the decay of the next-token conditional entropy with the length of the conditioning context. We further derive a simple formula in terms of these statistics that predicts data-limited neural scaling exponents from first principles without any free parameters or synthetic data models. Our theory exhibits a remarkable match with experimentally measured neural scaling laws obtained from training GPT-2 and LLaMA style models from scratch on two qualitatively different benchmarks, TinyStories and WikiText.

Momentum

Related topics

Latest papers 95

Deriving Neural Scaling Laws from the statistics of natural language

BRIDGE: Predicting Human Task Completion Time From Model Performance

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

A Large Scale Investigation of Scaling Limits in Chemical Language Models

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers