Language Model Scaling Laws

Follow this topic Ask about this topic

Language model scaling laws study how predictive loss and downstream abilities change with parameter count, training tokens, compute, and test-time effort, helping forecast performance and allocate costly resources. Classic power-law and Chinchilla-style fits guide compute-optimal model and data trade-offs, while newer mixture-aware and information-theoretic laws account for data quality, repetition, batch size, tokenization, and architectures such as Mixture-of-Experts and recurrent models. Researchers use controlled IsoFLOP experiments and smaller proxy runs to fit these relationships, then test their predictions on held-out loss and capability benchmarks. A central open tension is whether laws extrapolate beyond measured scales: data scarcity and duplication can shift optimal allocations, and better loss or greater scale does not guarantee uniformly better, safer, or more reliable task performance.

Latest papers

141 papers27 papers in the last four weeks, up 145% on the four weeks before. 0.2% of all new papers.

Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints
Oct 22, 2026Nicholas Roberts, Ahmed Ahmed, Suhas Kotha +4Synthetic Data Pretraining Language Model Scaling Laws
Princeton University · Stanford University · Jane Street
Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law
Oct 21, 2026Ziqing Qiao, Xiaoyue Xu, Sikui Zhang +4Long-Context Language Modeling Continued Pretraining
Department of Computer Science and Technology, Tsinghua University, Beijing, China · OpenBMB
Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
Oct 19, 2026Xuyuan Xiong, Siyu Chen, Zihan Shao +5Truncated Backpropagation Through Time Efficient Language Model Training
MMLab, The University of Hong Kong · Yale University
Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation
Oct 14, 2026Joel Dag, Nikit Srivastava, Hamada M. Zahera +1Multilingual Language Models Mixture-of-Experts Language Models
Data Science Group, Heinz Nixdorf Institute, Paderborn University, Germany
Articulation Upper Bounds for Language Models
Oct 14, 2026Alexander Spangher, Daniel E. Ho, Diyi Yang +2LLM Evaluation Language Model Scaling Laws
Stanford University
Exact Scaling Laws for a Random-Features Bigram Model
Oct 13, 2026Julian Enoch, Elvis DohmatobZipf's Law Random Feature Methods
Concordia University · Mila – Quebec AI Institute
Mixed Precision Scaling Laws for Quantized Pretraining
Oct 13, 2026Jinuk Kim, Rajarshi Saha, Muhammed Fatih Balin +1Mixed-Precision Quantization LLM Quantization
Seoul National University · Amazon Annapurna Labs
SLDBench v2: Active Scaling Law Discovery
Oct 12, 2026Lingbei Meng, Rui Huang, Zhirong Shen +5AI Agents for Scientific Discovery Language Model Scaling Laws
The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · University of Electronic Science and Technology of China +2
Compute-Optimal Scaling Laws for Spiking Language Models
Oct 10, 2026Dan Saunders, Benjamin Ren, Catherine HeSpiking Language Models Language Model Scaling Laws
Luma AI · X School
Scaling Laws for Multimodal Data Mixtures
Oct 10, 2026Aditi Khandelwal, Ayush K. Tarun, Yixuan Xu +5Multimodal Pretraining AI Scaling Laws
MILA - Quebec AI Institute, McGill University · EPFL · ETH +1
Neurograf v1: An End-to-End Romanian-First Language Model Family
Oct 9, 2026Ionut Adrian RotaruLanguage Modeling LLM Training
Extreme Light Infrastructure Nuclear Physics, Horia Hulubei National Institute for R & D in Physics and Nuclear Engineering, 077125 Bucharest-Magurele, Romania
How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies
Oct 8, 2026Bowei He, Yankai Chen, Xiaokun Zhang +4Agentic RL Synthetic Data Generation for LLM Agents
MBZUAI · McGill University · CityUHK +1
Fault-tolerant foundation models
Oct 7, 2026Trevor McCourt, Ila R. Fiete, Isaac L. ChuangLLM Training Language Model Scaling Laws
Department of Electrical Engineering and Computer Science, MIT · Department of Brain and Cognitive Sciences, McGovern Institute, MIT · Department of Physics, MIT
Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements
Oct 6, 2026Jeremy CanaleLLM Alignment Language Model Safety Evaluation
Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models
Oct 5, 2026Joe DwyerLanguage Model Scaling Laws Efficient Language Model Training
ECPI University
What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining
Oct 4, 2026Yekun Chai, Haoyi XiongLanguage Model Pretraining Language Model Scaling Laws
FloatAI · ETH Zurich · Independent Researcher
Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining
Oct 4, 2026Ziyue WANG, T. KanamoriLanguage Model Pretraining Language Model Scaling Laws
Institute of Science Tokyo
A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinese
Oct 4, 2026Hongao Zhu, Muxiaoqiao Xu, Yikang Liu +4Language Model Scaling Laws Language Modeling
Department of Linguistics, UC San Diego · School of Foreign Languages, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University +3
Capability Scaling-Down Laws for LLM Compression
Oct 1, 2026Xueqi Cheng, Liang Wu, Kelly Wan +2LLM Compression LLM Pruning
Florida State University · Nokia Applied Research
Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference
Oct 1, 2026Xinye Zhao, Yunkai Dang, Yunchen Wu +1Vision-Language Models Efficient VLM Inference
School of Intelligence Science and Technology, Nanjing University
Scaling Laws for Looped Mixture of Experts
Sep 30, 2026Yanbei Chen, Anirudh Goyal, Raghuraman KrishnamoorthiMixture-of-Experts Language Models Language Model Scaling Laws
Meta AI
How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
Sep 30, 2026Jenna Russell, Ben Glickenhaus, Katherine Thai +4Language Model Pretraining Synthetic Data Pretraining
University of Maryland · Pangram Labs
From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes
Sep 30, 2026Yichen Wang, Fanghui Liu, Yudong ChenLanguage Model Pretraining Deep Learning Optimization
Department of Computer Sciences, University of Wisconsin–Madison, USA. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.
Emergent phases of superposition: from partial to full representation
Sep 29, 2026Lihao Guo, Yizhou Liu, Jeff GorePhase Transitions in Neural Networks Feature Superposition
Massachusetts Institute of Technology Cambridge, MA 02139, USA
ScAn-Bench: Evaluating Scaling Analysis Methodology
Sep 28, 2026Artin Sermaxhaj, Nastaran Alipour, Donat Sinani +4VLM Evaluation LLM Evaluation
University of Freiburg · Zuse School ELIZA · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)
X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths
Sep 28, 2026Bowen Dong, Yilong Fan, Tengyu Pan +6Language Model Scaling Laws Mixture-of-Experts Models
Tsinghua University · Tianjin University
Training and Inference Dynamics of PLDR-LLMs: Row-Map Collapse, Renormalization, and Predictive Reduction
Sep 28, 2026Burc GokdenLLM Inference Language Model Scaling Laws
Fromthesky Research Labs LLC, Oregon, USA
DepthBench: Measuring How Residual Connections Enable More Computational Depth
Sep 26, 2026Keyu Wang, Yangyi Huang, Jiale Kang +3Attention Residual Connections Residual Learning
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +1
Emergent One-Third Scaling Law as Attention Tries to Concentrate
Sep 26, 2026Yizhou Liu, Sara Kangaslahti, Jeff GoreLanguage Model Scaling Laws Scaling Laws
Massachusetts Institute of Technology · Harvard University
In-Context Binding Capacity in Language Models
Sep 24, 2026Manas Venkata Sai Ravulapalli, Samrath Singh ChadhaWorking Memory in Language Models Language Model Scaling Laws
Efficient Computation Inc.

Oct 22, 2026J/K move · Enter open · S save

Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

Nicholas Roberts, Ahmed Ahmed, Suhas Kotha, Percy Liang+3

Princeton University · Stanford University · Jane Street

Synthetic Data Pretraining Language Model Scaling Laws Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Synthetic rewrites, where a language model rewrites a natural document e.g. into a target style, are now a standard ingredient of pretraining corpora. However, critical aspects of their use remain poorly understood, such as how much compute should be spent rewriting training data and whether a model trained on rewrites can outperform the rewriter. We study these two questions by proposing a rewriting scaling law that predicts loss from model size, training tokens, the fraction of training tokens that are rewrites, and the number of epochs over the natural data, charging the compute spent generating rewrites. We fit this scaling law in two settings. In the compute-limited setting, natural data is plentiful and rewrites come from a third-party rewriter trained with more compute than our largest run. We find that the estimated optimal rewrite fraction increases with compute, from under

10\%

3\times10^{18}

FLOPs to

32\%

10^{22}

FLOPs. Our scaling forecast extrapolates

33\times

in FLOPs to train a 22B-parameter model, whose final loss is within

1.1\%

of our prediction. This model achieves a better BPB than its rewriter on several web domains at one-tenth the rewriter's training compute. In the data-limited setting, we assume we only have a fixed seed set of natural data and that our rewriter is a model of the same size as the target, pretrained on that seed set. Again, we find the optimal rewrite fraction increases with the compute budget. We also demonstrate that a model trained on rewrites of its own pretraining data, generated by a rewriter of the same size, beats the best run on natural data alone at every model size up to 3B. Our results show that synthetic rewriting can be studied through the scaling law lens: that optimal methodological choices are predictable and scaling does not necessarily break down beyond the rewriter's capabilities.