Scaling Laws

Scaling laws study how performance changes as models, datasets, compute budgets, or inference resources grow, helping predict results and allocate limited resources. Researchers fit power-law relationships to controlled training runs, compare compute-optimal trade-offs such as model size versus training data, and use theory from high-dimensional regression, optimization, and attention to explain observed patterns. These methods are applied across language, multimodal, scientific, and structured-data models, with growing attention to training steps, batch size, numerical precision, and test-time compute. A central open tension is whether a small set of scaling forms can generalize across tasks and training regimes, since data quality, model architecture, optimizer, and experimental design can change both the fitted laws and the best allocation of resources.