Predicting Large Model Test Losses with a Noisy Quadratic System

Chuning Li; Chris J. Maddison

cs.LGMay 9, 2026

Predicting Large Model Test Losses with a Noisy Quadratic System

Authors: Chuning Li, Chris J. Maddison

Organizations: University of Toronto · Vector Institute

Abstract

We introduce a predictive model that estimates the pre-training loss of large models from model size (N), batch size (B) and number of weight updates (K). This is the first loss prediction model that can handle changing batch size. The model outperforms Chinchilla's loss model, a model of the test loss using the batch size and number of tokens, in terms of projecting the loss at extrapolated compute budgets (up to 1000 folds). A natural use of the model is to find optimal N, B, K configurations under explicit and compound resource constraints like time, memory and compute. In our experiments, the model-selected configurations are close to ground-truth optimal. Our work advocates for loss prediction as a better alternative to heuristic-based laws, which are growing in complexity. The implementation is available on https://github.com/chuningxdy/Noisy-Quadratic-System.

Explore similar work

A Defense of the Quadratic Model
Jul 23, 2026Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +3Large Models Condition Number
Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
Aug 28, 2026Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4Learning Rates Batch Size
Predict before you train: Scaling Laws for particle physics foundation models
Jul 25, 2026Jan-Lucas Uslu, Benjamin Nachman, Christopher ReScaling Laws Finetuning

Jul 23, 2026 · cs.LGJ/K move · Enter open · S save

A Defense of the Quadratic Model

Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan+2

1Kempner Institute at Harvard University · †: Work done while at Harvard. · 2MIT

Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically tractable the model is, and how accurately it describes the true optimization dynamics. In this work, we stress test the simplest possible model of optimization -- the quadratic model -- and show that it can be surprisingly predictive in an LLM setting with 150M parameters and 3B training tokens. Specifically, we show that Taylor expanding the model and the loss function at intermediate checkpoints through training can accurately predict the optimization dynamics over windows that can last up to 10% of training. Having established this agreement, we then turn to analyzing the structure of these local quadratic optimization problems through two lenses: the Hessian spectrum and local stability. Using Lanczos quadrature with extremely deep probes, we are able to estimate the Hessian spectrum deep into the tail, and we find a surprising amount of structure in both the eigenvalues and eigenvectors, which depends on the batch size, preconditioner, and training time. We also empirically test local linear stability at intermediate checkpoints and compare it to theoretical predictions to demonstrate that optimization in LLMs typically occurs at a stochastic edge of stability, whose nature is also determined by batch size. Our results indicate the quadratic model may be a theoretically tractable proxy for pretraining optimization dynamics.

Large Models Condition Number Loss Landscape Neural Network Optimization Polynomials

Open paper Ask about it PDF

Abstract

Explore similar work

A Defense of the Quadratic Model

Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss

Predict before you train: Scaling Laws for particle physics foundation models