Language Model Scaling Laws

Overview

Language model scaling laws study how predictive loss and downstream abilities change with parameter count, training tokens, compute, and test-time effort, helping forecast performance and allocate costly resources. Classic power-law and Chinchilla-style fits guide compute-optimal model and data trade-offs, while newer mixture-aware and information-theoretic laws account for data quality, repetition, batch size, tokenization, and architectures such as Mixture-of-Experts and recurrent models. Researchers use controlled IsoFLOP experiments and smaller proxy runs to fit these relationships, then test their predictions on held-out loss and capability benchmarks. A central open tension is whether laws extrapolate beyond measured scales: data scarcity and duplication can shift optimal allocations, and better loss or greater scale does not guarantee uniformly better, safer, or more reliable task performance.

Latest papers

151 papers8 papers in the last four weeks, down 73% on the four weeks before. 0.1% of all new papers.

Aug 19, 2025

A Large Scale Investigation of Scaling Limits in Chemical Language Models

Chemical Language Models (CLMs) are increasingly used in de novo drug design, driven by recent growth in model scale, compute, and dataset size. However, the relationship between design choices, training dynamics, and downstream generation quality remains poorly understood. We present a compute-controlled scaling study of CLMs comprising more than 30,000 experiments across molecular representations (SMILES, SELFIES, SAFE), tokenizations (atom-level and byte-pair encoding), model scales (0.5M-1B parameters), leakage-controlled datasets (MOSES, ChEMBL, PubChem, ZINC-22), and architectures (decoder-only and encoder-decoder). By fitting IsoFLOP profiles, we establish clear scaling trends in pretraining loss, but find that these improvements do not translate into comparable gains in goal-directed molecular design. Layer-wise probing and sparse autoencoder analysis reveal continued development of chemical representations: chemical syntax saturates early, while semantic properties emerge more slowly and become increasingly accessible with further training and model scale. These representational gains coexist with diminishing improvements in goal-directed generation under the evaluated protocols and oracle budgets. Our resulting suite of models, NovoMolGen, achieves state-of-the-art results, outperforming prior CLMs and specialized generative models in goal-directed molecular generation across drug discovery tasks. These findings expose a disconnect between chemical representation learning and downstream molecular design, motivating the development of pretraining paradigms that more directly learn chemical semantics.