Language Model Scaling Laws

Follow this topic Ask about this topic

Language model scaling laws study how predictive loss and downstream abilities change with parameter count, training tokens, compute, and test-time effort, helping forecast performance and allocate costly resources. Classic power-law and Chinchilla-style fits guide compute-optimal model and data trade-offs, while newer mixture-aware and information-theoretic laws account for data quality, repetition, batch size, tokenization, and architectures such as Mixture-of-Experts and recurrent models. Researchers use controlled IsoFLOP experiments and smaller proxy runs to fit these relationships, then test their predictions on held-out loss and capability benchmarks. A central open tension is whether laws extrapolate beyond measured scales: data scarcity and duplication can shift optimal allocations, and better loss or greater scale does not guarantee uniformly better, safer, or more reliable task performance.

Latest papers

141 papers27 papers in the last four weeks, up 145% on the four weeks before. 0.2% of all new papers.

Time is Not Compute: Scaling Laws for Wall-Clock Constrained Training on Consumer GPUs
Mar 29, 2026Yi LiuScaling Laws Language Model Scaling Laws
Independent Researcher
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
Mar 23, 2026Weilin Wan, Jingtao Han, Debing Zhang +2Mixture-of-Experts Models AI Scaling Laws
College of Computer Science and Artificial Intelligence, Fudan University · Hi Lab, Xiaohongshu Inc. · Project Numina +1
Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits
Mar 21, 2026Eric Czech, Zhiwei Xu, Yael Elmatad +2Scaling Laws Neural Scaling Laws
Open Athena AI Foundation · Department of Statistics, University of Michigan, Ann Arbor
Optimal Splitting of Language Models from Mixtures to Specialized Domains
Mar 19, 2026Skyler Seto, Pierre Ablin, Anastasiia Filippova +4Efficient Language Model Training Domain-Adaptive Pretraining
Apple · National University of Singapore
Data-efficient pre-training by scaling synthetic megadocs
Mar 19, 2026Konwoo Kim, Suhas Kotha, Yejin Choi +3Language Model Pretraining Synthetic Data Pretraining
Stanford University
ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws
Mar 18, 2026Xuyang Cao, Qianying Liu, Chuan Xiao +7Language Model Scaling Laws Multilingual Language Models
NII LLMC · The University of Osaka · Nara Institute of Science and Technology +2
The Institutional Scaling Law: Non-Monotonic Fitness, Capability-Trust Divergence, and Symbiogenetic Scaling in Generative AI
Mar 14, 2026Mark Baciak, Thomas A. CellucciTrustworthy AI Language Model Scaling Laws
Ekta Inc.
As Language Models Scale, Low-order Linear Depth Dynamics Emerge
Mar 13, 2026Buddhika Nettasinghe, Geethu JosephLLM Interpretability Language Model Scaling Laws
University of Iowa, Iowa City, USA · 2Delft University of Technology, Delft, Netherlands
IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL
Mar 12, 2026Zhoujun Cheng, Yutao Xie, Yuxiao Qu +12Reinforcement Learning RL Post-Training
UC San Diego · MBZUAI-IFM · Carnegie Mellon University
Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover
Mar 11, 2026Indranil Halder, Annesya Banerjee, Cengiz PehlevanAdversarial Attacks on LLMs Prompt Injection Attacks on LLMs
John A. Paulson School of Engineering And Applied Sciences, Harvard University · Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology · Speech and Hearing Bioscience and Technology, Harvard Medical School +2
Brevity Constraints Reverse Performance Hierarchies in Language Models
Mar 11, 2026MD Azizul HakimLLM Evaluation Language Model Scaling
Department of Computer Science Bangladesh Sweden Polytechnic Institute Chattogram, Bangladesh
Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization
Mar 9, 2026Jingwei Li, Xinran Gu, Jingzhao ZhangData Mixture Optimization LLM Training
Institute for Interdisciplinary Information Sciences, Tsinghua University · Shanghai Qizhi Institute
Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA
Feb 26, 2026Hai Huang, Yann LeCun, Randall BalestrieroEfficient Language Model Training LLM Training
Atlassian · NYU · Brown
Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple
Feb 25, 2026Amirhossein Bozorgkhoo, Igor MolybogSpeculative Decoding LLM Inference Acceleration
Independent Researcher · University of Hawai’i at Manoa, Honolulu, HI, USA.
Scale Dependent Data Duplication
Feb 18, 2026Joshua Kazdan, Noam Levi, Rylan Schaeffer +6Data Repetition in Language Model Training Language Model Pretraining
Department of Statistics, Stanford University · AI4Science, EPFL · Department of Computer Science, Stanford University +2
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
Feb 17, 2026Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis +1Language Model Performance Prediction Language Model Scaling Laws
Harvard University · Stanford University
Stabilizing Native Low-Rank LLM Pretraining
Feb 12, 2026Paul Janson, Edouard Oyallon, Eugene BelilovskyLanguage Model Pretraining Low-Rank Matrix Decomposition
Concordia University, Montreal, Canada · Mila Quebec AI Institute, Montreal, Canada · Sorbonne University, CNRS, Paris, France
Deriving Neural Scaling Laws from the statistics of natural language
Feb 7, 2026Francesco Cagnetta, Allan Raventós, Surya Ganguli +1Language Model Scaling Laws Neural Scaling Laws
Theoretical and Scientific Data Science, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Applied Physics, Stanford University, Stanford, California · Department of Physics and Astronomy, Johns Hopkins University, Baltimore, Maryland +1
BRIDGE: Predicting Human Task Completion Time From Model Performance
Feb 6, 2026Fengyuan Liu, Jay Gala, Nilaksh +3LLM Evaluation Language Model Scaling Laws
Mila - Quebec AI Institute · McGill University · Polytechnique Montréal +3
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Jan 12, 2026Xin Cheng, Rui Tian, Wangding Zeng +18Memory-Augmented Language Models Language Model Scaling Laws
Peking University · DeepSeek-AI
A Large Scale Investigation of Scaling Limits in Chemical Language Models
Aug 19, 2025Roshan Balaji, Kamran Chitsaz, Quentin Fournier +2Chemical Language Models Language Model Scaling Laws
BioSystems Engineering and Control Lab · Wadhwani School of Data Science and AI, IIT Madras · The Centre for Integrative Biology and Systems medicinE (IBSE) +5

Mar 29, 2026J/K move · Enter open · S save

Time is Not Compute: Scaling Laws for Wall-Clock Constrained Training on Consumer GPUs

Yi Liu

Independent Researcher

Scaling Laws Language Model Scaling Laws Efficient Language Model Training

Open paper AskPDF arXiv

Scaling laws relate model quality to compute budget (FLOPs), but practitioners face wall-clock time constraints, not compute budgets. We study optimal model sizing under fixed time budgets from 5 minutes to 24 hours on consumer GPUs (RTX 4090). Across 70+ runs spanning 50M--1031M parameters, we find: (1)~at each time budget a U-shaped curve emerges where too-small models overfit and too-large models undertrain; (2)~optimal model size follows

N^* \propto t^{0.60}

, growing faster than Chinchilla's

N^* \propto C^{0.50}

, with

α= 0.60 \pm 0.07

robustly exceeding compute-optimal across all sensitivity analyses; (3)~a dual U-shape mechanism: short-budget U-curves arise from compute bottlenecks, while long-budget U-curves emerge from data bottlenecks (overfitting), with an intermediate regime where the U-curve temporarily disappears. These findings have immediate implications for researchers training on consumer hardware, where wall-clock time -- not FLOPs -- is the binding constraint. We release all code, logs, and 70+ experimental configurations.