Foundations of the Scaling Law

Tianyu Zhao; Zhixiang Ren

cs.LGOct 20, 2026

Foundations of the Scaling Law

Authors: Tianyu Zhao, Zhixiang Ren

Organizations: LingQiong AI Lab, Shanghai Smart Logic Technology

Abstract

Empirical scaling laws relate loss to parameters $N$ , training tokens $D$ and compute $C$ . This technical note explains three mechanisms behind their exponents. Local approximation bounds function error at rate $N_e^{-γs/d}$ , with effective parameters $N_e$ , smoothness $s$ , reuse exponent $γ$ and dimension $d$ ; the loss function determines the risk bound. Independent feature coverage gives $α_D=(β-1)/ζ$ from predictive-weight decay $β$ and occurrence rarity $ζ$ . Separable-loss optimization determines training allocations and deployment crossovers. Nonlinear scoring can produce apparent emergent abilities, while grokking concerns delayed generalization and verification constrains test-time scaling. For regular molecular configurations, the $\mathrm{SE}(3)$ quotient gives $d_{\mathrm{int}}=d-6$ and relative approximation-exponent gain $6/d_{\mathrm{int}}$ . Optimized dense-to-equivariant excess-loss ratios scale as $C^{p_{\mathrm{eq}}-p_{\mathrm{dense}}}$ , where $p_j$ are compute-loss exponents and the prefactor depends on loss amplitudes and FLOP costs. Domain examples also cover visual resolution, conformational coverage and correlated-sample precision.

Figures & tables

Figure 1: Pythia scaling across tokens, parameters and compute. Zero-shot LAMBADA loss

L=\ln\mathrm{PPL}

(nats per target word) from Pythia [ 1 ] . (a) Small models plateau as tokens

D

accumulate; the 12B run improves. (b) Subtracting fitted offsets aligns three token-budget slices of

L-L_{0}(D)

around a shared fit. (c) Estimated FLOPs

C\simeq 6ND

compare compute use. Orange dashes trace the 12B late-stage fit (a), shared fit (b) and frontier guide (c).

Figure 2: Scaling has a finite power-law regime. As training data

D

increase, error passes from a best-guess plateau through power-law decay to the irreducible error floor

L_{\infty}

, the remaining prediction uncertainty at large data. Redrawn after Hestness et al. [ 4 , Fig. 6] ; a single exponent describes the middle interval rather than both plateaus.

Figure 3: Extrapolation magnifies uncertainty in the loss forecast. Synthetic pilot runs (circles) fit excess loss

\epsilon_{C}:=L(C)-L_{\infty}

above a fixed floor; solid and dashed lines show the fit and extrapolation to the diamond forecast. The approximate 90% band for mean log-excess loss widens with extrapolation distance, conditional on the floor and power-law model.

Model	Assumptions	Prediction or decision
Geometry	Compact support, regular cells, local approximation class	Finer cells reduce function error; loss determines the risk bound
Predictive spectrum	Independent draws, aligned feature weights, learning on observation	Error from useful tail features not yet observed
Training allocation	Separable loss, fixed dense FLOP factor	Parameter–data split at fixed training compute
Candidate search	Query-specific proposals and a calibrated selection protocol	Candidate coverage and delivered-answer accuracy

Table 1: How representation, sampling and compute budgets determine error and resource allocation.

Figure 4: Finer cells and smoother targets reduce approximation error. (a) A

d

-dimensional manifold is covered by

K

regular cells with size

h\propto K^{-1/d}

. (b) A quadratic Taylor approximation (dashed orange) to a smooth target (navy) leaves a uniform error

\eta

(maximum red residual) bounded by a constant times

h^{3}

. More generally, smoothness

s

gives a bound scaling as

K^{-s/d}

Panel of Figure 5 — Figure 5: Feature visibility converts spectral tails into data scaling. Rank $r$ carries predictive mass $\lambda(r)\propto r^{-\beta}$ and occurrence probability $q(r)\propto r^{-\zeta}$ ; sample count $D$ sets the boundary $Dq(r_{v})=1$ . For the shallow spectrum, hard-tail shading and smooth unseen mass $\lambda(r)[1-q(r)]^{D}$ (blue dashed) yield the same large-data exponent $(\beta-1)/\zeta$ in the coverage model.

Figure 7: Training-optimal allocation differs from serving-oriented allocation. Gray contours hold excess loss fixed across active parameters

N

and training tokens

D

; the balanced path (blue) meets the fixed-compute line

C=\kappa ND

(dashed, FLOP coefficient

\kappa

) at minimum-loss tangency. Moving toward smaller

N

and larger

D

reduces serving work per token but departs from this training optimum.

Figure 8: Nonlinear scoring can make smooth progress appear abrupt. As model scale

\log X

grows, per-step success

q(X)

improves smoothly and latent loss

-\log q

(blue) falls. Exact-match success

q^{r}

for an

r

-step chain (red) stays low until per-step reliability is high, producing sharper apparent thresholds for longer chains.

Figure 9: Finding a correct candidate does not ensure selecting it. For

k

independent proposals with per-path success

p

, coverage

P_{\mathrm{cov}}(k)=1-(1-p)^{k}

(solid) approaches one with diminishing gains. The red dashed example multiplies the

p=0.10

coverage by conditional selection success

v=0.88

, giving delivered accuracy

A_{\mathrm{sel}}=vP_{\mathrm{cov}}

and a selector-imposed ceiling.

Stage	Evidence/target	Output construction
AF1 [ 59 ]	Sequence-derived evidence; protein structure	Predicted distance distributions guide structure optimization
AF2 [ 57 ]	Alignments, pair representations; protein structure	Evoformer with an end-to-end structure module
AF3 [ 58 ]	Joint biomolecular complexes	Diffusion-based coordinate generation

Table 2: AlphaFold stages expand the prediction target and change the mechanism used to construct structures.

Figure 10: Dimension dilutes geometric gains; compute cost shifts crossovers. (a) Removing

d_{G}=6

rigid-pose coordinates from total dimension

d

leaves internal dimension

d_{\mathrm{int}}=d-d_{G}

; both relative and normalized exponent gains decline. (b) Optimal excess loss scales as

L_{\mathrm{opt}}-L_{\infty}\propto C^{-p}

: the costlier quotient model wins above each crossing

C_{i}^{\star}

, while a smaller loss-exponent gap delays that crossing.

Figure 11: Rare conformations can remain poorly covered. (a) Example observations occupy basin A more densely than B on a free-energy surface in collective variables $s_{1},s_{2}$ . (b) Along path parameter $\lambda$ , $\Delta F_{\mathrm{path}}$ marks the maximum above A; equilibrium rarity depends on integrated basin weights. More records need not resolve rare basins.

Scaling Laws

Scaling laws describe how model performance and learning dynamics change with model size, data, compute, training time, and other resources.

See all 74 papers

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World
May 9, 2026Christopher M. Bryant, Hao LiuLanguage Model Scaling Laws Neural Scaling Laws
Arena Physica
Prescriptive Scaling Laws for Data Constrained Training
May 2, 2026Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2Weight Decay Language Model Scaling Laws
Department of Computer Science Cornell University Ithaca, NY
Amortizing Scaling Law Construction Costs
Sep 4, 2026Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6Language Model Scaling Laws Bayesian Optimization
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +4

May 9, 2026J/K move · Enter open · S save

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World

Christopher M. Bryant, Hao Liu

Arena Physica

Language Model Scaling Laws Neural Scaling Laws

Open paper AskPDF arXiv

The scaling laws guiding modern model training were calibrated for a single regime: data-rich, single-epoch pretraining. The dominant such scaling law form, Chinchilla's

L = E + A/N^α+ B/D^β

, has three structural limitations outside that regime: it diverges as unique data shrinks instead of saturating at the uninformed baseline; it cannot represent overfitting when capacity exceeds the data; and it conflates total examples seen with unique examples available. We propose a closed-form extension,

L(N, D, T) = E + (L_0 - E)\,h/(1+h)

with

h = a/N^α+ b/T^β+ c\,N^γ/D^δ

, that decomposes loss into undercapacity, undertraining, and overfitting terms. It saturates between the irreducible loss

E

and an uninformed baseline

L_0

fixed by the loss type, and reduces to Chinchilla in the data-rich, single-epoch limit. We validate it on four multi-epoch experiments spanning four architecture families (MLPs, ResNets, Fourier neural operators, and transformers) across vision, scientific ML, and language domains, and refit it to five published LLM scaling-law grids. Extrapolating to higher compute and larger unique data than seen at fit time, our form achieves state-of-the-art RMSE on every published LLM grid we evaluate and on most cells of our constructed experiments. Once calibrated, the form admits a cost-aware allocation that recovers Chinchilla's optimum when data is free and shifts toward smaller corpora and more epochs as data grows expensive.

Abstract

Scaling Laws

Explore similar work

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World

Prescriptive Scaling Laws for Data Constrained Training

Amortizing Scaling Law Construction Costs