Predictable Scaling: Efficient Model Exploration via Scaling-Law Prediction

Aishwarya Wesanekar; Sheng-Tao Yang; Jiawei Zang; Flavio Sales Truzzi; Chen Chen; Nader Beigiparast; Shengyuan Huang; Roshanak Zilouchian Moghaddam

cs.LGOct 16, 2026

Predictable Scaling: Efficient Model Exploration via Scaling-Law Prediction

Authors: Aishwarya Wesanekar, Sheng-Tao Yang, Jiawei Zang, Flavio Sales Truzzi, Chen Chen, Nader Beigiparast, Shengyuan Huang, Roshanak Zilouchian Moghaddam,

Organizations: Meta Platforms, Inc.

Abstract

Evaluating candidate changes (hypotheses) to recommendation models requires full-scale training runs consuming thousands of GPU-hours, severely limiting the hypothesis space engineers can feasibly explore. We introduce Predictable Scaling (PS), a systematic framework that exploits empirically verified regularity conditions to predict full-scale model performance from small-scale proxies. PS constructs scaled-down variants via composable scaling functions along distinct resource axes, and projects metrics to the target scale via curve fitting with calibrated uncertainty estimates. We compare three prediction strategies of increasing expressiveness: even the simplest (raw proxy) achieves prediction error within the empirical noise floor of repeated-run variance; the most expressive (Gaussian Process) provides an indicative 15% relative MAE improvement and calibrated uncertainty. Evaluation on two ranking models of increasing complexity---a task-specific model and a foundation model---demonstrates that PS correctly ranks actionable hypotheses across diverse hypothesis types at 79-88% compute reductions.

Figures & tables

Fn	Description	Scaled	Dim	$r_{k}(\alpha)$
DD	Uniform subsampling, hash-seeded	Data volume	FLOPs	$\alpha$
HSR	Reduce table rows (min 5K)	Embedding rows	Memory	$\alpha$
ES	Shrink vector width, aligned	Embedding width	FLOPs+Mem	$\alpha$
FS	Prune by importance scores	Input features	FLOPs	$\alpha$
WAS	Scale all dims uniformly	All hidden dims	FLOPs+Mem	$\alpha^{2}$
DS	Truncate to $\lfloor\alpha L\rfloor$ layers	Layers	FLOPs	$\alpha$

Table 1: Scaling functions in PS. Each parameterized by

\alpha\in(0,1]

Model	Primary Task(s)	FLOPs/fwd (M)	Trainers	Samples (B)	Hardware	Proxy Budget Constraint
TS-1 (Task-Specific)	Feed CTR + CVR	$3{,}403$	$112$	$80$	NVIDIA H100	$\Delta$ Trainers $\leq-50\%$
FM-1 (Foundation)	Multi-task CTR + CVR	$117{,}734$	$1{,}024$	$122$	NVIDIA H100	$\Delta$ Samples $\leq-50\%$

Table 2: Summary of the two evaluation models. FLOPs denote the Floating-point Operations per forward pass and QPS denotes the queries per second for each model type.

Predictor	MAE ( $\times\tau_{\text{AA}}$ )	SE	$\Delta$ MAE	$\rho$	$\Delta\rho$
Raw proxy	$0.865$	$\pm 0.776$	—	$0.755$	—
Linear	$0.846$	$\pm 0.738$	$-4\%$	$0.755$	$0\%$
GP (EGP)	$\bm{0.738}$	$\pm 0.657$	$-15\%$	$\bm{0.800}$	$+6\%$

Table 3: TS-1 prediction accuracy (CTR head, 14 held-out hypotheses).

Head	Predictor	MAE ( $\times\tau_{\text{AA}}$ )	SE	$\Delta$ MAE	$\rho$	$\Delta\rho$
CTR-1	Raw proxy	$2.150$	$\pm 1.692$	—	$0.607$	—
CTR-1	GP (EGP)	$\bm{0.701}$	$\pm 0.552$	$-67\%$	$\bm{0.714}$	$+18\%$
CTR-2	Raw proxy	$1.750$	$\pm 1.466$	—	$0.643$	—
CTR-2	GP (EGP)	$\bm{0.753}$	$\pm 0.631$	$-57\%$	$\bm{0.714}$	$+11\%$
CVR-1	Raw proxy	$0.950$	$\pm 0.556$	—	$0.714$	—
CVR-1	GP (EGP)	$\bm{0.854}$	$\pm 0.500$	$-10\%$	$0.714$	$0\%$

Table 4: FM-1 prediction accuracy (7 held-out hypotheses).

\Delta

columns: change vs. raw proxy.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	DD	HSR	ES	Other	Constraint
TS-1	$\alpha{=}0.317$	$\alpha{=}0.5$	$\alpha{=}0.25$	—	$\Delta$ Trainers $\leq-50\%$
FM-1	$\alpha{=}0.5$	$\alpha{=}0.2$	$\alpha{=}0.2$	FS $0.2$ , DS $0.5$	$\Delta$ Samples $\leq-50\%$

Appendix

Table 5: Scaling configurations per model.

Model	$\Delta$ QPS	$\Delta$ Samples	$\Delta$ Trainers	Compute Red.
TS-1	$-76\%$	$-80\%$	$-86\%$	$88\%$
FM-1	$-70\%$	$-50\%$	$-88\%$	$79\%$

Appendix

Table 6: Resource efficiency of selected proxy configurations.

Component	ES	WAS
Embedding tables	✓	✓
User-history embeddings	✓	✓
Interaction $d_{\text{model}}$	✓	✓
Dense layers	—	✓
Specialized sub-networks	—	✓
Task-specific heads	—	✓

Appendix

Table 7: Components scaled by ES and WAS.

Metric	ES	WAS	$\Delta$
NE Gap	$-0.0090$	$-0.0056$	$38\%\!\downarrow$
QPS (k)	194.9	244.1	$+25.2\%$
FLOPs/sample (M)	109,357.9	62,440.4	$-42.9\%$
FLOPs reduction vs. full	68.4%	82.0%	—

Appendix

Table 8: ES vs. WAS at

\alpha{=}0.5

on Foundation Model. NE gap: signed difference (negative = proxy underperforms).

Config	HSR	ES	DD
hash_only	$0.5$	$1.0$	$1.0$
emb_only	$1.0$	$0.5$	$1.0$
data_downsample	$1.0$	$1.0$	$0.5$
hash_emb	$0.5$	$0.5$	$1.0$
hash_data	$0.5$	$1.0$	$0.5$
all_scaled	$0.5$	$0.5$	$0.5$

Appendix

Table 9: Scaling configurations for public benchmark evaluation. Each configuration scales the indicated dimensions by a factor of

0.5

, targeting overall training compute reduction.

Config	Predictor	MAE ( $\times\tau_{\text{AA}}$ )	SE ( $\times\tau_{\text{AA}}$ )	$\Delta$ MAE	Spearman $\rho$	$\Delta\rho$
hash_only	Raw proxy	$0.731$	$\pm 0.617$	—	$0.417$	—
	Linear	$0.731$	$\pm 0.617$	$0\%$	$0.417$	$0.000$
	GP (EGP)	$0.728$	$\pm 0.479$	$0\%$	$0.417$	$0.000$
emb_only	Raw proxy	$1.174$	$\pm 0.494$	—	$-0.050$	—
	Linear	$0.991$	$\pm 0.875$	$-16\%$	$-0.050$	$0.000$
	GP (EGP)	$0.947$	$\pm 0.771$	$-19\%$	$-0.050$	$0.000$

Appendix

Table 10: Public DLRM prediction accuracy across six scaling configurations. MAE and SE reported as multiples of

\tau_{\text{AA}}

\Delta

MAE: relative % change vs. raw proxy;

\Delta\rho

: absolute difference vs. raw proxy.

Config	Predictor	MAE ( $\times\tau_{\text{AA}}$ )	SE ( $\times\tau_{\text{AA}}$ )	$\Delta$ MAE	Spearman $\rho$	$\Delta\rho$
hash_only	Raw proxy	$0.381$	$\pm 0.303$	—	$0.939$	—
	Linear	$0.381$	$\pm 0.303$	$0\%$	$0.939$	$0.000$
	GP (EGP)	$\bm{0.285}$	$\pm 0.304$	$-25\%$	$\bm{0.939}$	$0.000$
emb_only	Raw proxy	$0.848$	$\pm 0.487$	—	$0.115$	—
	Linear	$0.848$	$\pm 0.487$	$0\%$	$0.115$	$0.000$
	GP (EGP)	$0.795$	$\pm 0.460$	$-6\%$	$0.115$	$0.000$

Appendix

Table 11: Public DeepFM prediction accuracy across six scaling configurations. MAE and SE reported as multiples of

\tau_{\text{AA}}

\Delta

MAE: relative % change vs. raw proxy;

\Delta\rho

: absolute difference vs. raw proxy.

Config	Predictor	MAE ( $\times\tau_{\text{AA}}$ )	SE ( $\times\tau_{\text{AA}}$ )	$\Delta$ MAE	Spearman $\rho$	$\Delta\rho$
hash_only	Raw proxy	$0.814$	$\pm 1.575$	—	$0.836$	—
	Linear	$0.814$	$\pm 1.575$	$0\%$	$0.836$	$0.000$
	GP (EGP)	$0.791$	$\pm 1.609$	$-3\%$	$0.836$	$0.000$
emb_only	Raw proxy	$0.699$	$\pm 0.517$	—	$0.518$	—
	Linear	$0.699$	$\pm 0.517$	$0\%$	$0.518$	$0.000$
	GP (EGP)	$0.564$	$\pm 0.435$	$-19\%$	$0.518$	$0.000$

Appendix

Table 12: Public xDeepFM prediction accuracy across six scaling configurations. MAE and SE reported as multiples of

\tau_{\text{AA}}

\Delta

MAE: relative % change vs. raw proxy;

\Delta\rho

: absolute difference vs. raw proxy.

Config	Predictor	MAE ( $\times\tau_{\text{AA}}$ )	SE ( $\times\tau_{\text{AA}}$ )	$\Delta$ MAE	Spearman $\rho$	$\Delta\rho$
hash_only	Raw proxy	$0.686$	$\pm 0.706$	—	$0.006$	—
	Linear	$0.686$	$\pm 0.706$	$0\%$	$0.006$	$0.000$
	GP (EGP)	$0.653$	$\pm 0.618$	$-5\%$	$0.006$	$0.000$
emb_only	Raw proxy	$0.581$	$\pm 0.323$	—	$0.224$	—
	Linear	$0.581$	$\pm 0.323$	$0\%$	$0.224$	$0.000$
	GP (EGP)	$0.545$	$\pm 0.302$	$-6\%$	$0.333$	$+0.109$

Appendix

Table 13: Public DCN-v2 prediction accuracy across six scaling configurations. MAE and SE reported as multiples of

\tau_{\text{AA}}

\Delta

MAE: relative % change vs. raw proxy;

\Delta\rho

: absolute difference vs. raw proxy.

Panel of Figure 5 — Figure 5: Hash-only scaling

Panel of Figure 12 — Figure 12: Hash-only scaling

Panel of Figure 19 — Figure 19: Hash-only scaling

Panel of Figure 26 — Figure 26: Hash-only scaling

Model Selection

Model selection chooses models, features, or configurations that balance predictive fit, generalization, complexity, and deployment constraints.

See all 96 papers

Amortizing Scaling Law Construction Costs
Sep 4, 2026Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6Language Model Scaling Laws Bayesian Optimization
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +4
Foundations of the Scaling Law
Oct 20, 2026Tianyu Zhao, Zhixiang RenScaling Laws Test-Time Scaling
LingQiong AI Lab, Shanghai Smart Logic Technology
SLDBench v2: Active Scaling Law Discovery
Oct 12, 2026Lingbei Meng, Rui Huang, Zhirong Shen +5AI Agents for Scientific Discovery Language Model Scaling Laws
The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · University of Electronic Science and Technology of China +2

Sep 4, 2026J/K move · Enter open · S save

Amortizing Scaling Law Construction Costs

Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik, Swagatam Haldar+5

ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology+4

Language Model Scaling Laws Bayesian Optimization Hyperparameter Optimization

Open paper AskPDF arXiv

Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to

10\text{--}100\times

Abstract

Model Selection

Explore similar work

Amortizing Scaling Law Construction Costs

Foundations of the Scaling Law

SLDBench v2: Active Scaling Law Discovery