A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents

Indranil Halder; Rastri Dey; Cengiz Pehlevan

cs.LGSep 26, 2026

A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents

Authors: Indranil Halder, Rastri Dey, Cengiz Pehlevan

Organizations: John A. Paulson School of Engineering and Applied Sciences at Harvard University · PACCAR Inc. · Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University · Center for Brain Science, Harvard University

Abstract

Pre-training data poisoning of large language models is usually studied using targeted backdoors and their survival through safety post-training, which leaves open a more basic question: how does a model's clean data performance degrade as the poison rate $\varepsilon$ grows? Motivated by our controlled pre-training runs of OLMo-style models, in which the relative clean data validation perplexity increase $Δ$ between poisoned and clean models matched in architecture, token budget, and optimization schedule is well fit by a power law $Δ\approx C\varepsilon^{a}$ with a non-integer exponent, we ask what such a law requires theoretically. We first prove an analyticity barrier: whenever the contaminated objective depends analytically on $\varepsilon$ around a nondegenerate clean data optimum, $Δ$ is generically quadratic in $\varepsilon$ , so a generic non-integer exponent is a signature of genuinely singular structure. We then supply that structure in solvable truncated ridge regression with heavy-tailed covariates, controlled by $q_\star$ , and a label-shift poisoning. Our central result is that the excess risk scaling exponent depending on the order of limits: in the higher dimensional proportional regime it is $ε^{q_\star/(q_\star+2)}$ , whereas taking the ample-data limit first gives $ε^{2-2/q_\star}$ , and the limits do not commute. We confirm this prediction through several numerical simulations. Finally, we argue that finite training time plays the role of truncation on the curvature spectrum in local LLM pre-training, deriving the observed scaling law under heavy tailed inverse curvature spectrum as a modeling hypothesis.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Random-poisoning scaling curves at multiple training checkpoints for one model ( $774$ M): over-training the slope and intercept drift systematically. (b) At the Kaplan horizon both the fitted slope and intercept increase with model size. Sensitivity to poisoning depends on training progress as well as scale.

Panel of Figure 2 — Figure 2: ( 2(a) ) The measured optimized exponent follows $q_{\star}/(q_{\star}+2)$ (proportional) and $2-2/q_{\star}$ (ample) across $q_{\star}\in[1.5,4]$ ; the branches are well separated. ( 2(b) ) For $q_{\star}=2$ , fitting the exponent over a fixed $\epsilon$ -window as $\phi=P/D$ varies shows a clean cross-over from the ample value $1.0$ to the proportional value $0.5$ .

Panel of Figure 3 — Figure 3: At $q_{\star}=2$ the optimized degradation follows the predicted proportional ( 3(a) ) and ample ( 3(b) ) slopes simulation (points) against the exact prediction (dashed).

Figure 4: The plot corresponds to an experiment performed on OLMo-style model with total parameters

P=1.677\times 10^{8}

pre-trained upto

3

B tokens. The graph on the left shows that heavy tailed curvature spectrum exists and, the graph on the right determines the exponent. Red shades show where numerical analysis is unreliable. For the details of

R,\hat{\alpha}_{2}

see Appendix E , these are a specially constructed function of

z

; smaller

z

probes a smaller eigenvalue spectrum. The ratio test in Corollary 32 shows that

R\to 1

asserts a heavy-tailed spectrum of

X=\lambda^{-1}

with hard edge exponent

\alpha\leq 1

. On the other hand Remark 5 shows

\hat{\alpha}_{2}

approaches the value of

\alpha\approx 0.33

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Scaling law for a small amount of random poisoning $\varepsilon$ . We see that the scaling law for the larger models is more stable. This fact might be explained by saying stronger models have smaller $\phi_{\mathrm{LLM}}$ and hence the ample data regime extends up to smaller values of $\varepsilon$ .

Panel of Figure 6 — Figure 6: Scaling law for a small amount of language switching poisoning $\varepsilon$ . We see that the scaling law for the larger models is more stable. This fact might be explained by saying stronger models have smaller $\phi_{\mathrm{LLM}}$ and hence the ample data regime extends up to smaller values of $\varepsilon$ .

Panel of Figure 7 — Figure 7: Random poisoning scaling curves at multiple training checkpoints for each model. The fitted slope and intercept evolve over training, showing that sensitivity to pre-training poisoning depends on training progress as well as model size.

Panel of Figure 8 — Figure 8: Language switching poisoning scaling curves at multiple training checkpoints for each model. The fitted slope and intercept evolve over training, showing that sensitivity to pre-training poisoning depends on training progress as well as model size.

Panel of Figure 9 — Figure 9: Ratio of random poisoned-model to clean-model perplexity, evaluated on clean data. A poisoned model is trained with per-token poison probability $\varepsilon$ ; its clean baseline shares the same architecture, token budget, and optimization schedule. The perplexity ratio increases with $\varepsilon$ over the range shown.

Panel of Figure 10 — Figure 10: Ratio of language switching poisoned-model to clean-model perplexity, evaluated on clean data. A poisoned model is trained with per-token poison probability $\varepsilon$ ; its clean baseline shares the same architecture, token budget, and optimization schedule. The perplexity ratio increases with $\varepsilon$ over the range shown.

Label	$d_{\mathrm{model}}$	heads	layers	MLP ratio	parameters
main-135M	1024	16	8	4	$135.349$ M
main-388M	1600	16	12	4	$387.611$ M
main-490M	1600	16	16	4	$490.011$ M
main-774M	2048	32	16	4	$774.015$ M
main-1287M	3072	32	12	4	$1286.851$ M

Appendix

Table 1: Model configurations used in the poisoning scaling experiments.

Model	Learning rate	$D_{\mathrm{H}}$ tokens	$S_{\mathrm{H}}$ steps
$135.349$ M	$6.27\times 10^{-4}$	$8.7$ B	$66{,}090$
$387.611$ M	$4.80\times 10^{-4}$	$12.8$ B	$97{,}530$
$490.011$ M	$4.47\times 10^{-4}$	$13.9$ B	$106{,}364$
$774.015$ M	$3.83\times 10^{-4}$	$16.5$ B	$125{,}959$
$1286.851$ M	$3.13\times 10^{-4}$	$39.9$ B	$304{,}031$

Appendix

Table 2: Learning rates and Kaplan et al. (2020) horizon training budgets.

Figure 11: The plot on the left shows that heavy tailed curvature spectrum exists and the plot on the right determines the exponent. Smaller

z

probes a smaller eigenvalue spectrum.

Figure 12: Every mechanism that limits the measurement performed.

Data Poisoning Attacks

Data poisoning attacks manipulate training or retrieval data to corrupt model behavior, implant backdoors, or undermine reliability.

See all 126 papers

Jun 23, 2026

On the Smallness of the Large Language Models Scaling Exponents

We discuss reasons why the scaling exponents of current Large Language Models (LLMs) applications are indicating an unsustainable regime in terms of energy resources. We further show that attributing the smallness of such exponents to a numerical bias due to the neglect of a non-zero value of the loss function in the limit of infinite data (``pedestal effect") does not remove the unsustainability issue. Finally, the effects of the smoothness (roughness) of the data on the scaling exponents is commented upon based on an analogy with phenomenological models of fluid turbulence.

Sauro Succi, Peter V. Coveney, Alex Hansen

Italian Institute of Technology, Viale Regina Elena, 291, 00161, Rome, Italy · PoreLab, Physics Department, Norwegian University of Science and Technology, 7491 Trondheim, Norway · Centre for Computational Science, Chemistry Department, University College of London, 20 Gordon Street, WC1H 0AJ, London, United Kingdom

Energy-Efficient ML Language Model Scaling Laws

Ask PDF

May 4, 2026

InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition

Upweighting high-quality data in LLM pretraining often improves performance, but in datalimited regimes, especially under overtraining, stronger upweighting increases repetition and can degrade performance. However, standard scaling laws do not reliably extrapolate across mixture recipes or under repetitions, making the selection for optimal data recipes at scaling underdetermined. To solve this, we introduce InfoLaw (Information Scaling Laws), a data-aware scaling framework that predicts loss from consumed tokens, model size, data mixture weights, and repetition. The key idea is to model pretraining as information accumulation, where quality controls information density and repetition induces scaledependent diminishing returns. We first collect the model performance after training on datasets that vary in scale, quality distribution, and repetition level. Then we build up the modeling for information so that information accurately predicts those model performance. InfoLaw predicts performance on unseen data recipes and larger scale runs (up to 7B, 425B tokens) with 0.15% mean and 0.96% max absolute error in loss, and it extrapolates reliably across overtraining levels, enabling efficient data-recipe selection under varying compute budgets.

Fengze Liu, Weidong Zhou, Binbin Liu +7

1ByteDance · 2UC Santa Cruz

Language Model Pretraining Language Model Scaling Laws Data Mixture Optimization Data Repetition in Language Model Training

Ask PDF

May 9, 2026

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World

The scaling laws guiding modern model training were calibrated for a single regime: data-rich, single-epoch pretraining. The dominant such scaling law form, Chinchilla's

L = E + A/N^α+ B/D^β

, has three structural limitations outside that regime: it diverges as unique data shrinks instead of saturating at the uninformed baseline; it cannot represent overfitting when capacity exceeds the data; and it conflates total examples seen with unique examples available. We propose a closed-form extension,

L(N, D, T) = E + (L_0 - E)\,h/(1+h)

with

h = a/N^α+ b/T^β+ c\,N^γ/D^δ

, that decomposes loss into undercapacity, undertraining, and overfitting terms. It saturates between the irreducible loss

E

and an uninformed baseline

L_0

fixed by the loss type, and reduces to Chinchilla in the data-rich, single-epoch limit. We validate it on four multi-epoch experiments spanning four architecture families (MLPs, ResNets, Fourier neural operators, and transformers) across vision, scientific ML, and language domains, and refit it to five published LLM scaling-law grids. Extrapolating to higher compute and larger unique data than seen at fit time, our form achieves state-of-the-art RMSE on every published LLM grid we evaluate and on most cells of our constructed experiments. Once calibrated, the form admits a cost-aware allocation that recovers Chinchilla's optimum when data is free and shifts toward smaller corpora and more epochs as data grows expensive.

Christopher M. Bryant, Hao Liu

Arena Physica

Language Model Scaling Laws Neural Scaling Laws

Ask PDF

Abstract

Data Poisoning Attacks

Explore similar work

On the Smallness of the Large Language Models Scaling Exponents

InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World