Amortizing Scaling Law Construction Costs

Abhash Kumar Jha; Diana Alexandra Onuţu; Neeratyoy Mallik; Swagatam Haldar; Sam Laing; Niccolò Ajroldi; Shiwei Liu; Joaquin Vanschoren

cs.LGSep 4, 2026

Amortizing Scaling Law Construction Costs

Authors: Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik, Swagatam Haldar, Sam Laing, Niccolò Ajroldi, Shiwei Liu, Joaquin Vanschoren,

Organizations: ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology · University of Freiburg ·

Abstract

Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to $10\text{--}100\times$ .

Figures & tables

Figure 1: Scaling law

L=E+AC^{\alpha}{}

fit to different loss subsets. Left : the full Pareto envelope based all 14 different observations and its fitted law; Middle : a poorly selected four-point subset produces a deviated fit; Right : a different 4-point subset closely recovers the full-data fit. Both sparse fits use the same number of observations, but their accuracy depends strongly on which points are selected, motivating the problem of finding a small subset that reliably recovers the full scaling law.

Figure 2: Results on

L=E+AC^{\alpha}

comparing the same BO surrogate model, acquiring data by optimizing over the Full search space or a growing Window conditioned on the observations so far. Panels 1-3 report absolute error in each fitted parameter against the full data fit (sans held-out ). Panel 4 records the MSE of the loss predicted by the fitted parameters against the actual held-out loss. Panel 5 tracks the % recovery of the optimal-loss envelope. Shaded band is 1 S.D. over 10 seeds.

Compute $C$ (in FLOPs):	$10^{25}$	$10^{27}$	$10^{29}$
$1\%$ of Total Budget	$1.09\pm 1.84$	$1.37\pm 2.35$	$1.56\pm 2.67$
$5\%$ of Total Budget	$0.22\pm 0.34$	$0.29\pm 0.41$	$0.36\pm 0.45$
$10\%$ of Total Budget	$0.04\pm 0.04$	$0.05\pm 0.04$	$0.06\pm 0.10$

Table 1: Extending the results of Figure 2 ’s window + fantasization to much larger extrapolation lengths. Entries are the relative % difference in predicted loss against the full data fit (sans held-out), at different fractions of the compute to exhaust the grid. S.D. over 10 seeds.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition
$N$ , $D$	Model size (parameter count) and training tokens
$\lambda$	Hyperparameter configuration, $\lambda\in\Lambda\subseteq\mathbb{R}^{k}$
$\mathcal{N}$ , $\mathcal{D}$	Sets of candidate model sizes and token budgets
$\mathcal{G}$	Full grid of configurations, $\mathcal{G}=\mathcal{N}\times\mathcal{D}\times\Lambda$
$f(N,D,\lambda)$	Loss of a training run
$g(N,D)$	Compute cost in FLOPs, $C(N,D)=g(N,D)\approx 6ND$

Appendix

Table 2: Summary of notations used.

Figure 3: Overview of the framework. From the search grid we draw initial samples, then run the HPO loop: at each iteration a surrogate

M_{t}

is fit on the observations

O_{t}

, the next query is selected over the expanding window

W_{t+1}

, its loss is evaluated, and

O_{t}

is updated, until the compute budget

C_{\mathrm{total}}

is reached. The surrogate optionally fantasizes losses over the grid, and the resulting observed (and fantasized) points are used to fit and evaluate the scaling law against the full-grid reference. Compute slicing and fantasization are the two additions to vanilla BO.

Dataset	# Runs	# $N$	# $D$	# $(N,D)$	# $\eta$	# $b$	# $(\eta,b)$	# $(N,D,\eta,b)$
OELLM-English	942	6	9	51	5	8	36	942
Porian-HP	566	7	39	39	7	7	46	237
StepLaw	1911	5	15	17	14	13	175	1911
Warmstart	150	6	23	23	3	7	14	44
OELLM-Multilingual	332	4	9	36	5	8	30	332

Appendix

Table 3: Overview of the benchmark datasets and their configuration grids.

Dataset	$N$ range	$D$ range	$C_{\mathrm{total}}$
OELLM-English	$47.6\mathrm{M}$ – $1.71\mathrm{B}$	$6\mathrm{B}$ – $300\mathrm{B}$	$1.45\times 10^{23}$
Porian-HP	$5.2\mathrm{M}$ – $221\mathrm{M}$	$103\mathrm{M}$ – $4.4\mathrm{B}$	$2.88\times 10^{20}$
StepLaw	$215\mathrm{M}$ – $1.07\mathrm{B}$	$4\mathrm{B}$ – $100\mathrm{B}$	$1.45\times 10^{23}$
Warmstart	$32.3\mathrm{M}$ – $1.22\mathrm{B}$	$32.3\mathrm{M}$ – $60.9\mathrm{B}$	$1.66\times 10^{21}$
OELLM-Multilingual	$206\mathrm{M}$ – $1.26\mathrm{B}$	$6\mathrm{B}$ – $300\mathrm{B}$	$6.57\times 10^{22}$

Appendix

Table 4: Scale and compute coverage of the benchmark datasets.

Budget	$E$	$A$	$\alpha$
$1\%$ of Total Budget	$0.4655\pm 0.0309$	$6.833\pm 0.332$	$-0.1559\pm 0.0087$
$5\%$ of Total Budget	$0.4536\pm 0.0054$	$6.686\pm 0.024$	$-0.1521\pm 0.0007$
$10\%$ of Total Budget	$0.4502\pm 0.0014$	$6.674\pm 0.006$	$-0.1517\pm 0.0001$
$25\%$ of Total Budget	$0.4502\pm 0.0014$	$6.674\pm 0.006$	$-0.1517\pm 0.0001$
$50\%$ of Total Budget	$0.4502\pm 0.0014$	$6.674\pm 0.006$	$-0.1517\pm 0.0001$
$100\%$ of Total Budget	$0.4499$	$6.679$	$-0.1518$

Appendix

Table 5: Incumbent coefficients of the loss law

L(C)=E+A\,C^{\alpha}

under LCB acquisition, at increasing fractions of the compute needed to exhaust the grid. Mean

\pm

S.D. over 10 seeds.

Figure 4: Effect of our proposed search (window) methodology with other models: Random Search (top) as a surrogate-free approach, and two other choices of surrogates for BO: TabPFN (middle) and xgboost (bottom). Our proposed method emperically improve over the standard BO definition (Full).

Figure 5: We provide

L(C)

scaling law recovery plots on applying our approach to other scaling law studies (each row). Here we keep standard BO settings fixed: GP surrogate with Matérn-5/2 kernel and EI as the acquisition function and show the results for plain (Full) and slabbed (Window) variants (dashed: only observed points used, solid: observed plus fantasized points used to construct

L(C)

Figure 6: Different kernel choices for the GP surrogate with EI acquisition function. We experiment with

2

standard kernels and

2

additive kernels, with one applying different kernels to different variable types.

Figure 7: Three standard choices for the acquisition function used in BO. LCB performs slightly better than EI and PI on OELLM-English dataset.

Figure 8: Recovery of the

L(N,D)

parametric fit on OELLM-English under different BO variants. We compare acquisition over the full search space ( Full ) and a growing compute window ( Window ), with scaling laws fitted using either observed points only or observed and fantasized points. Panels report parameter recovery for

\alpha

\beta

, and

E

, held-out MSE, and recovery of the minimum-loss configuration at each

(N,D)

pair. Shaded regions show

\pm 1

S.D. over 10 seeds.

Figure 9:

L(N,D)

scaling law recovery across benchmark datasets. Columns report recovery of

\alpha

\beta

, and

E

, held-out MSE, and per-

(N,D)

envelope recovery. Lines compare Full and Window acquisition with observed-only and observed-plus-fantasized fits.

Language Model Scaling Laws

Language model scaling laws describe how model size, training data, compute, and inference effort jointly shape loss, capabilities, and efficiency.

See all 141 papers

Prescriptive Scaling Laws for Data Constrained Training
May 2, 2026Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2Weight Decay Language Model Scaling Laws
Department of Computer Science Cornell University Ithaca, NY
Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World
May 9, 2026Christopher M. Bryant, Hao LiuLanguage Model Scaling Laws Neural Scaling Laws
Arena Physica
Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection
Apr 24, 2026Sijie Li, Shanda Li, Haowei Lin +3Active Experimental Design Scaling Laws
Peking University · Carnegie Mellon University · Datadog

May 2, 2026J/K move · Enter open · S save

Prescriptive Scaling Laws for Data Constrained Training

Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar+1

Department of Computer Science Cornell University Ithaca, NY

Weight Decay Language Model Scaling Laws Neural Scaling Laws

Open paper AskPDF arXiv

Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay (

λ=1.0

) reduces this coefficient by approximately 70%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.

Abstract

Language Model Scaling Laws

Explore similar work

Prescriptive Scaling Laws for Data Constrained Training

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World

Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection