Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits

Eric Czech; Zhiwei Xu; Yael Elmatad; Yixin Wang; William Held

cs.LGMar 21, 2026

Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits

Authors: Eric Czech, Zhiwei Xu, Yael Elmatad, Yixin Wang, William Held

Organizations: Open Athena AI Foundation · Department of Statistics, University of Michigan, Ann Arbor

Abstract

Chinchilla Approach 2 is among the most widely used methods for fitting neural scaling laws. Its parabolic approximation introduces systematic biases in compute-optimal allocation estimates, even on noise-free synthetic data. Applied to published Llama 3 IsoFLOP data at open frontier compute scales, these biases imply a parameter underallocation corresponding to 6.5% of the $3.8\times10^{25}$ FLOP training budget and $1.4M (90% CI: $412K-$2.9M) in unnecessary compute at 50% H100 MFU. Simulated multimodal model misallocations show even greater opportunity costs due to higher loss surface asymmetry. Three sources of this error are examined: IsoFLOP sampling grid width (Taylor approximation accuracy), uncentered IsoFLOP sampling, and loss surface asymmetry ( $α\neq β$ ). Chinchilla Approach 3 largely eliminates these biases but is often regarded as less data-efficient, numerically unstable, prone to local minima, and harder to implement. Each concern is shown to be unfounded or addressable, especially when the partially linear structure of the objective is exploited via Variable Projection, enabling unbiased inference on all five loss surface parameters through a two-dimensional optimization that is well-conditioned, analytically differentiable, and amenable to dense, or even exhaustive, grid search. It may serve as a more convenient replacement for Approach 2 or a more scalable alternative for adaptations of Approach 3 to richer scaling law formulations. See https://github.com/Open-Athena/vpnls for details and https://openathena.ai/scaling-law-analysis for other results from this study.

Figures & tables

Figure 1: Approach 2 misallocation costs extrapolated to

3.8\times 10^{25}

FLOPs. Left: Deadweight Compute Loss (DCL) as a percentage of budget; dollar cost ranges for empirical rows are 90% bootstrap CIs. Right: allocation details including true vs. inferred token counts and model sizes, loss penalty, and dollar cost. “Empirical” rows use Approach 2 power laws fit to digitized Llama 3 IsoFLOP data, evaluated against VPNLS and Approach 3 surfaces. “Simulated” rows use synthetic IsoFLOP data generated from each published surface with the XS grid and

3\times

drift bias used in the Drifting Bias simulation section (Section 4.3.2 ).

Figure 2: Effect of progressive quality control filtering on Approach 2 Deadweight Compute Loss, measured against an Approach 3 surface fit to unfiltered Llama 3 data. Each row cumulatively applies one QC stage. Nearly all DCL reduction comes from the off-center and weak-curvature filters, which target specific Approach 2 biases. The right table reports points and budgets removed at each stage, along with DCL as a percentage of the

3.8\times 10^{25}

FLOP evaluation budget and dollar cost.

Figure 3: Approach 2 applied to a symmetric loss surface. Left: IsoFLOP curves with fitted parabolas. True (

\times

) and inferred (

+

) optima are indistinguishable. Right: Power-law fit recovers the exact scaling exponent.

Parameter	True Value	Inferred Value	Relative Error
$b$ ( $D^{*}$ exponent)	0.500000	0.500000	$+6.2\times 10^{-12}$ %
$b_{0}$ ( $D^{*}$ intercept)	$-0.389076$	$-0.389076$	$-1.4\times 10^{-10}$ %

Table 1: Approach 2 parameter recovery on the symmetric surface.

Figure 4: Approach 2 on asymmetric loss surfaces. Note the visible gap between true (dashed) and inferred (solid) power-law lines in the Asymmetric case. The exponents match perfectly, but the intercepts differ.

Surface	Parameter	True Value	Inferred Value	Relative Error
Chinchilla	$b$ ( $D^{*}$ exponent)	0.548387	0.548387	$\approx 0$ %
Chinchilla	$b_{0}$ ( $D^{*}$ intercept)	$-0.555357$	$-0.578092$	$-4.1$ %
Asymmetric	$b$ ( $D^{*}$ exponent)	0.750000	0.750000	$\approx 0$ %
Asymmetric	$b_{0}$ ( $D^{*}$ intercept)	$-1.345791$	$-1.459957$	$-8.5$ %

Table 2: Approach 2 parameter recovery on asymmetric surfaces.

Grid Name	$\pm k\times$	Sampling Range	Total Ratio	Decade Span
Extra Small (XS)	$\pm 2\times$	$\frac{1}{2}\times$ to $2\times$	$4\times$	0.60
Small (S)	$\pm 4\times$	$\frac{1}{4}\times$ to $4\times$	$16\times$	1.20
Large (L)	$\pm 8\times$	$\frac{1}{8}\times$ to $8\times$	$64\times$	1.81
Extra Large (XL)	$\pm 16\times$	$\frac{1}{16}\times$ to $16\times$	$256\times$	2.41

Table 3: Sampling grid widths used in simulations.

Figure 5: Relative error in compute-optimal token prediction when extrapolating from the training range (

10^{17}

–

10^{21}

FLOPs) to

10^{24}

FLOPs. Negative values indicate underestimation: the inferred scaling law predicts fewer tokens than optimal. Bars are grouped by sampling grid width. Annotations for the Chinchilla surface show

D^{*}

(true compute-optimal token count) versus

\hat{D}^{*}

(the Approach 2 estimate); the Small and Large grid annotations are emphasized (thicker borders) as they fall within the realistic 1–2 decade range typical of scaling law experiments, while Extra Small and Extra Large bracket either side as more extreme configurations.

Figure 6: Effect of a constant

3\times

offset in sampling centers on the symmetric surface. Top left: IsoFLOP curves at the Large grid (

\pm 8\times

), with black diamonds marking the (off-center) sampling center, red

\times

the true

D^{*}

, and blue

+

the inferred

D^{*}

. Top right: extrapolation error in compute-optimal token prediction at

10^{24}

FLOPs for each grid width. Bottom row: exponent and intercept errors across grid widths from XS (

\pm 2\times

) to XL (

\pm 16\times

), plotted on the same y-axis scale. The exponent is recovered perfectly (flat at zero) while the intercept shows systematic bias that varies with grid width.

Figure 7: Effect of a linear drift in sampling centers (centered at true optimum for lowest budget, drifting to

3\times

at highest budget) on the symmetric surface. Unlike the constant bias case, the exponent error (bottom left) is now non-zero: the slope of

\log D^{*}

\log C

is distorted because the offset varies across compute budgets.

Figure 8: IsoFLOP curves from three published scaling law studies. Left: Chinchilla (loss vs parameters). Center: Llama 3 (loss vs tokens). Right: DeepSeek (bits-per-byte vs parameters). Each panel shows curves at multiple compute budgets, fit using Approach 2.

Figure 9: Relative error in

D^{*}

10^{24}

FLOPs with off-center sampling on all three loss surfaces. Left: constant

3\times

center offset at every budget. Right: linear drift to

3\times

at the highest compute budget. Bars are grouped by sampling grid width (XS through XL). Negative values indicate underestimation; positive values indicate overestimation. On the symmetric surface, the constant offset results correspond to Figure 6 and the drift results correspond to Figure 7 ; the asymmetric surfaces reveal how these sampling biases interact with the inherent asymmetry bias.

Figure 10: Comparison of six fitting methods on noise-free synthetic data across three loss surfaces and 20 sampling ranges (60 fits total per method). Left: geometric mean of $|\text{relative error}|$ (%) pooled across all surfaces, grid widths, and parameters, with horizontal bars spanning the min-to-max range. Right: maximum $|\text{relative error}|$ (%) per parameter, on a log-scale colormap. Methods are sorted by geometric mean error, with the worst at top. VPNLS [ $+\nabla$ ] uses analytical gradients (L-BFGS-B with OLS inner solve); VPNLS [ $\lnot\nabla$ ] is gradient-free (Nelder-Mead with NNLS inner solve).

Figure 11: Comparison of five fitting methods on noisy synthetic data, pooled across three noise levels, three budget counts, four points-per-curve settings, and 256 noise realizations (9,216 fits per method). Left: geometric mean of $|\text{relative error}|$ (%) with min-to-max range bars, rug ticks for individual errors, and kernel density estimate. Right: maximum $|\text{relative error}|$ (%) per exponent. Methods sorted by worst-case error, worst at top. Surface: Asymmetric ( $\alpha=0.465$ , $\beta=0.155$ ) with $3\times$ drift, $\pm 8\times$ grid, compute budgets $10^{17}$ – $10^{21}$ FLOPs.

Figure 12: Estimator variance for the compute-optimal exponent $b=\alpha/(\alpha+\beta)$ on a symmetric surface ( $\alpha=\beta=0.31$ , $A=B=400$ , $E=1.69$ , true $b=0.5$ ) with perfectly centered IsoFLOP sampling. Left: overall variance pooled across noise levels. Right: variance by noise level. Sweep over points per curve $\in\{21,31,41\}$ , budget counts $\in\{3,5,7\}$ , log-widths $\in\{\pm 2\times,\pm 4\times,\pm 8\times\}$ , noise levels $\sigma\in\{0.01,0.02,0.05\}$ , with 10 trials per configuration. Lower variance is better. Approach 3 and VPNLS achieve comparable precision; Approach 2 is consistently less efficient.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$E$	$A$	$B$	$\alpha$	$\beta$	$a$	$b$
VPNLS	1.9051	4.0005	1.0509	0.3511	0.4587	0.5665	0.4335
Approach 3 (log-scaled loss)	1.8926	4.0146	1.0571	0.3522	0.4430	0.5571	0.4429
Approach 3 (linear loss)	1.9051	4.0005	1.0509	0.3511	0.4587	0.5665	0.4335
ml-scalefit (MSE)	1.9051	4.0001	1.0509	0.3510	0.4588	0.5665	0.4335
ml-scalefit (Huber)	1.8267	3.7066	1.0386	0.3240	0.4080	0.5573	0.4427

Appendix

Table A1: VPNLS validation against ml-scalefit on the Chinchilla dataset. VPNLS and ml-scalefit (MSE) agree to three decimal places (

1\times 10^{-3}

). Approach 3 with linear loss is identical to VPNLS; only the log-scaled loss variant diverges.

Figure A1: IsoFLOP curves for six experiments spanning different datasets and model families. Clean points (circles) and outlier points (markers by reason) are shown alongside parabolic fits (dashed) and Approach 3 predictions with per-budget FLOP factor adjustment (solid). Experiments shown: Epoch AI/Chinchilla, Llama 3, Marin/CoMMA, Marin/DCLM, Marin/Nemotron, and Misfitting/FineWeb-C4.

Figure A2: Noisy IsoFLOP samples used in the exponent inference comparison (Figure 11 ). Columns correspond to noise levels (

\sigma=0.05,0.1,0.2

); rows show loss versus

\log_{10}(N)

(top) and

\log_{10}(D)

(bottom). Scatter points are noisy observations; solid curves show the noiseless reference surface. Red

\times

marks the true compute-optimal point at each budget; black diamonds mark the sampling centers, which drift away from the true optima at higher compute budgets. Surface: Asymmetric (

\alpha=0.465

\beta=0.155

), 4 budgets (

10^{17}

–

10^{21}

FLOPs), 32 points per curve,

\pm 8\times

grid.

Figure A3: Per-parameter recovery error for six fitting methods across three loss surfaces and 20 sampling ranges (baseline, no bias). Each panel shows absolute relative error (%) on a log scale versus sampling range, with one curve per method. Rows correspond to loss surfaces (symmetric, Chinchilla, high imbalance); columns correspond to parameters (

E

A

B

\alpha

\beta

Figure A4: Relative error in compute-optimal token count

D^{*}

when extrapolating from the fitting range (

10^{17}

–

10^{21}

FLOPs) to higher compute budgets (

10^{22}

–

10^{25}

FLOPs), with asymmetry and sampling biases acting simultaneously. Columns correspond to loss surfaces (symmetric, Chinchilla, Asymmetric); rows correspond to sampling ranges (narrow

\pm 2\times

, medium

\pm 16\times

, wide

\pm 100\times

). The wide row uses an extreme grid width not employed in the main text, included here to further illustrate how far results can deviate with a misconfigured experiment. Each curve represents a different sampling bias configuration: baseline (no bias), two linear drift rates (drift_0.2 and drift_0.4, where the value is the

\log_{10}

offset at the highest compute budget), and two constant center offsets (scale_1.5 and scale_2.0, where the value is the multiplicative factor applied at every budget). On symmetric surfaces, errors are driven entirely by off-center sampling; on asymmetric surfaces, the inherent surface bias adds a constant offset visible as the non-zero baseline curve. Drift-based biases produce errors that grow with extrapolation distance (steeper curves), while constant offsets and surface asymmetry produce flat errors.

Figure A5: Per-condition breakdown of exponent inference errors from Figure 11 , showing 4 of 5 methods (excludes Naive Approach 3, which serves only as a negative control in the main figure). Rows correspond to number of compute budgets (2, 3, 4); left columns show the lowest noise level (

\sigma=0.05

) and right columns show the highest (

\sigma=0.2

), each split by exponent (

a

and

b

). Within each panel, boxplots show absolute relative error (%) on a log scale for each method at each points-per-curve setting (4, 8, 16, 32), over 256 noise realizations.

Figure A6: Per-noise-level breakdown of signed exponent errors from Figure 12 . Boxplots show the distribution of

\hat{b}-b_{\mathrm{true}}

on a symlog scale for each method at each noise level. Pooled summary statistics (sample size, variance, IQR) are shown below each method group.

Source	Domain	Config	Method	$a$	$b$	$\beta/\alpha$
DeepSeek ( Bi et al., 2024 )	Web text	OpenWebText2 ‡	Appr. 2	0.73	0.27	2.70
		MassiveText ‡		0.49	0.51	0.96
		DeepSeek (early)		0.450	0.550	0.82
		DeepSeek (current)		0.524	0.476	1.10
		DeepSeek (OWT2)		0.578	0.422	1.37
EHR ( Jiang et al., 2025 )	Health records	Transformer	Appr. 2	0.58	0.44	1.32

Appendix

Table A2: Published compute-allocation exponents from scaling law studies across domains. Columns

a

and

b

are the allocation exponents governing optimal model size (

N^{*}\propto C^{a}

) and dataset size (

D^{*}\propto C^{b}

). The

\beta/\alpha

column gives the ratio of loss-surface exponents, computed as

a/b

for Approach 2 and 3 estimates or reported directly for Kaplan estimates.

Figure A7: Progressive quality control filtering on Chinchilla data, analogous to Figure 2 for Llama 3. DCL is measured against an Approach 3 surface fit to unfiltered data and extrapolated to

5.8\times 10^{23}

FLOPs. Starting from 37.9% DCL on raw data, the off-center and weak-curvature filters reduce DCL to 12.5%.

Figure A8: Per-budget residual distributions after Approach 3 fitting with per-budget FLOP factor adjustment, across six experiments. Each row shows the residuals for one compute budget as a layered rug plot, boxplot, and KDE, with an overall row pooling all clean points. The zero-residual reference line (dashed) indicates perfect prediction.

Figure A9: Per-budget residual standard deviation (left) and summary statistics (right) for six experiments. Kruskal-Wallis tests whether residual locations differ across budgets. Levene tests whether residual variances differ across budgets. High p-values in most experiments indicate that a budget-independent noise model is a reasonable approximation for simulation purposes.

Scaling Laws

Scaling laws describe how model performance and learning dynamics change with model size, data, compute, training time, and other resources.

See all 72 papers

May 9, 2026

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World

The scaling laws guiding modern model training were calibrated for a single regime: data-rich, single-epoch pretraining. The dominant such scaling law form, Chinchilla's

L = E + A/N^α+ B/D^β

, has three structural limitations outside that regime: it diverges as unique data shrinks instead of saturating at the uninformed baseline; it cannot represent overfitting when capacity exceeds the data; and it conflates total examples seen with unique examples available. We propose a closed-form extension,

L(N, D, T) = E + (L_0 - E)\,h/(1+h)

with

h = a/N^α+ b/T^β+ c\,N^γ/D^δ

, that decomposes loss into undercapacity, undertraining, and overfitting terms. It saturates between the irreducible loss

E

and an uninformed baseline

L_0

fixed by the loss type, and reduces to Chinchilla in the data-rich, single-epoch limit. We validate it on four multi-epoch experiments spanning four architecture families (MLPs, ResNets, Fourier neural operators, and transformers) across vision, scientific ML, and language domains, and refit it to five published LLM scaling-law grids. Extrapolating to higher compute and larger unique data than seen at fit time, our form achieves state-of-the-art RMSE on every published LLM grid we evaluate and on most cells of our constructed experiments. Once calibrated, the form admits a cost-aware allocation that recovers Chinchilla's optimum when data is free and shifts toward smaller corpora and more epochs as data grows expensive.

Christopher M. Bryant, Hao Liu

Arena Physica

Language Model Scaling Laws Neural Scaling Laws

Ask PDF

Sep 4, 2026

Amortizing Scaling Law Construction Costs

Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to

10\text{--}100\times

Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6

ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +4

Language Model Scaling Laws Bayesian Optimization Hyperparameter Optimization

Ask PDF

May 2, 2026

Prescriptive Scaling Laws for Data Constrained Training

Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay (

λ=1.0

) reduces this coefficient by approximately 70%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.

Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2

Department of Computer Science Cornell University Ithaca, NY

Weight Decay Language Model Scaling Laws Neural Scaling Laws

Ask PDF

Abstract

Scaling Laws

Explore similar work

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World

Amortizing Scaling Law Construction Costs

Prescriptive Scaling Laws for Data Constrained Training