Language Model Scaling Laws

Overview

Language model scaling laws study how predictive loss and downstream abilities change with parameter count, training tokens, compute, and test-time effort, helping forecast performance and allocate costly resources. Classic power-law and Chinchilla-style fits guide compute-optimal model and data trade-offs, while newer mixture-aware and information-theoretic laws account for data quality, repetition, batch size, tokenization, and architectures such as Mixture-of-Experts and recurrent models. Researchers use controlled IsoFLOP experiments and smaller proxy runs to fit these relationships, then test their predictions on held-out loss and capability benchmarks. A central open tension is whether laws extrapolate beyond measured scales: data scarcity and duplication can shift optimal allocations, and better loss or greater scale does not guarantee uniformly better, safer, or more reliable task performance.

Latest papers

CardsList

151 papers8 papers in the last four weeks, down 73% on the four weeks before. 0.1% of all new papers.

  1. A Large Scale Investigation of Scaling Limits in Chemical Language Models

    Aug 19, 2025Roshan Balaji, Kamran Chitsaz, Quentin Fournier +2Chemical Language ModelsLanguage Model Scaling Laws