Data Attribution Should Steer Pretraining
Abstract
Pretraining data curation decides what a model learns from, yet what makes a document worth learning from depends on what the model already knows. Influence-driven selectors already rank documents to improve one run; we argue that training-data attribution should steer pretraining by identifying testable data properties that become reusable curation criteria, evaluated on new data against cheaper selectors at matched total cost, discovery included. A document ranking is consumed by the run it was computed for, whereas an explained data property can be tested across corpora and checkpoints, especially where familiar quality signals diverge from measured usefulness. Because historical contribution, an understandable explanation, and future data value require distinct evidence, we specify a feedback loop that interprets attributed sources, tests candidate data changes, and states for five claim types what evidence each requires and what none establishes. We close with a falsifiable comparison against perplexity, classifier, and mixing baselines at equal total compute.
Figures & tables
| Question | Relevant evidence | What it does not establish |
|---|---|---|
| 1. Which data contributed? | Attribution estimates under a stated target and counterfactual. | That influential data will be useful in a later training decision. |
| 2. What might explain the contribution? | Inspection of content, lexical and structural patterns, and competing explanations. | That an understandable pattern faithfully explains the model’s dependence on the data. |
| 3. Does the relationship predict unseen outcomes? | Predictions evaluated on evidence distinct from that used to construct them. | That a forecast of attribution scores predicts the effect of changing training data. |
| 4. Does the candidate data change help? | A comparison isolating the intervention across target and off-target effects. | That the motivating explanation was right, or that the policy transfers. |
| 5. How does the data shape computation? | A separately validated account linking data, representations, and behavior. | That this account identifies the best future curation policy. |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Protocol component | Illustrative specification |
|---|---|
| 1. Corpora and split | Discovery pool ( ): 10B-token stratified slice of DCLM-Baseline or Dolma labeled with WebOrganizer topic-by-format bins [ Li et al., 2024 , Wettig et al., 2025 ] . Held-out transfer pool ( ): disjoint 20B-token DCLM-Pool slice untouched during discovery. |
| 2. Model scale and checkpoints | Discovery run: 410M- or 1B-parameter decoder-only model trained for 10B tokens on , saving 10 evenly spaced checkpoints with projected gradients ( – ) indexed via TRAK or EK-FAC [ Park et al., 2023 , Coalson et al., 2025 ] . Transfer run: 1B-parameter model trained from scratch for 10B selected tokens from (plus a mid-training branch at 50% budget for stage-conditional tests). |
| 3. Criterion extraction | Contrast top- and bottom-attributed regions or token clusters on across competing hypotheses (discourse structure, lexical overlap, source metadata), screen candidate edits via bounded end-of-training ablations [ Blakeney et al., 2024 ] or unlearning, and compile the surviving property into a standalone classifier or token filter [ Wettig et al., 2024 , Rathi and Radford, 2026 ] that runs on without gradients. |
| 4. Baseline selectors | (a) Small-model perplexity pruning and perplexity correlations [ Ankner et al., 2025 , Thrush et al., 2025 ] ; (b) reference-corpus classifier and quality raters [ Penedo et al., 2024 , Wettig et al., 2024 ] ; (c) embedding-centroid similarity to discovery queries; and (d) domain reweighting or online mixing [ Xie et al., 2023a , Albalak et al., 2023 ] . |
| 5. Matched total cost ( ) | Let be all accelerator FLOPs (or GPU-hours) spent on beyond the base run, paired with an explicit cap on human inspection (e.g., analyst-hours, or 0 hours when extraction uses automated lexical and discourse contrasts). Baseline arms pay only their selection cost on and invest the difference into extra training tokens or mixture-weight sweeps on so every arm expends . |
| 6. Probe suite and guardrails | Evaluated under OLMES [ Gu et al., 2025a ] with above-chance admission floors. Discovery queries: e.g., SocialIQA (social reasoning), GSM8K (math), or RealToxicityPrompts (safety). Disjoint held-out target probes: unseen tasks and phrasings testing the same construct (e.g., social pragmatics, MATH-elementary, ToxiGen). Off-target guardrails: MMLU macro-average, ARC-Challenge, HellaSwag, and Paloma validation perplexity. |
Training Data Selection
Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.
See all 196 papers