stat.MLOct 21, 2026
SaveHigh-Value Training Data from Noisy, Diverse Sources
Organizations: Hiddenweights · Arena · UCLA · Stanford ·
Abstract
Training state-of-the-art models increasingly requires learning from large, heterogeneous datasets whose examples vary in quality, domain relevance, and source of supervision. Dataweighting---assigning different weights to training examples---offers a general way to control how such data influences training, but learning effective weights at scale remains challenging. We develop a scalable approach to learning dataweights: given a small, high-quality target dataset, we use metagradients through the training process to efficiently optimize training-example weights directly for performance on the target domain. To scale beyond a fixed dataset and model, we learn parametric weighting rules that generalize to new and streaming examples and transfer from smaller models to substantially larger ones. We study reward modeling as a prototypical application, where preference data may be noisy, adversarial, synthetic, or mismatched with the target domain. Learned dataweights improve held-out performance, automatically adapt reward models to specialized domains, and allow mixtures of human and synthetic supervision to outperform either source alone.
Figures & tables
Figure 1: Dataweighting outperforms attribution-based selection. Held-out STEM test loss after training on Human-155k . Left: Baselines across different selection fractions, with the unweighted baseline and a gray band showing standard deviation over three seeds. Dataweighting reweights the full pool (keeping of data points). Right: Each method at its best selection fraction (in parentheses); the leftmost bar is the unweighted baseline. Dataweighting reaches , compared with for TRAK, the strongest selection baseline. See Appendix E.4 for other training datasets.
Figure 2: Dataweighting improves the cost–performance tradeoff. Held-out STEM loss versus compute for Human-30k , comparing dataweighting, single-step differentiation, and attribution-based selection. Point labels indicate meta-epochs (i.e., complete runs of the algorithm ); black rings mark Pareto-optimal observations. The gray band shows the unweighted baseline mean and standard deviation across three seeds. Additional results appear in Appendix E.2 .
Figure 3: A small STEM target set complements direct training on STEM data. Target loss (left) and held-out test loss (right) with a Human-33k training base. Target-domain ( STEM ) pairs are allocated entirely to ordinary training (Pure A), entirely to the meta-optimization target (Pure B), or split, with target pairs and the rest used for training (A+B). Full denotes pairs; Quarter, Half, and Threequarter are fractions of the full target set .
Figure 4: Dataweighting consistently improves reward models across a variety of settings. Each plot depicts the test loss of reward models trained with and without dataweighting across different training data sources and evaluation sets. In (a), we apply dataweights to a heterogeneous corpus of human preference data, targeting the curated Expert target set, and show that dataweighting improves test-set performance. In (b), we apply dataweighting to a training set comprising heterogeneous human data that has been “poisoned” with data suspected to be from bots; dataweighting allows us to recover performance. In (c), we plot the test loss as a function of the training segment for dataweights that are further and further optimized for the STEM target set. Finally, in (d), we plot the test loss of models trained with and without dataweighting on real, synthetic, and mixed data; dataweighting enables us to leverage human and synthetic sources to outperform either source in isolation.
Figure 5: Parametric dataweights transfer to larger models and new training data. (a) STEM test loss when training on Human-45k plus STEM examples. Curves show means over three seeds; error bars show their minimum and maximum. (b) Transferring dataweights learned at model size 0.5B to 1.5B, 3B, and 7B reward models on Human-45k plus STEM examples ( k total). (c) Same setup as (b), transferring the linear dataweighting map to the full Human-155k training base, with approximately times as many human examples; the new examples were not seen while learning the dataweighting map.
Figure 6: Parametric rank-dataweights transfer to unseen training points. A parametric dataweight map is optimized against the soft-Spearman objective on a small subset ( k/ k/ k pairs), then used to predict weights for all k training pairs; a new reward model is then trained once under those weights. Held-out leaderboard Spearman improves with the meta-training size and approaches non-parametric, per-sample dataweights, while unweighted training lags well behind.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: BT and Spearman: standard vs. meta-optimized reward models. Rank MAE (lower is better) and Spearman (higher is better) on held-out test models.
Figure 8: Rank-targeted meta-optimization improves held-out ranking across splits. Held-out Spearman (left, higher is better) and Rank MAE (right, lower is better) across nine model-disjoint holdout splits, comparing unweighted Bradley-Terry training, BT-targeted meta-optimization, and Spearman-targeted meta-optimization. The Spearman-targeted objective shifts the whole distribution toward better rankings (higher , lower MAE) relative to both baselines.
Figure 9: Test and target loss by segment for model trained on Human-155k , targeting Expert . This example shows notable overfitting. Although the first few metagradient steps improve test loss, further steps overfit the target.
Figure 10: Test and target loss by segment for model trained on Human-45k , targeting Expert .
Figure 11: Test and target loss by segment for model trained on Human-77k , targeting Expert .
Figure 12: Dataweighting automatically separates bot and human data. Here, we visualize a KDE ridge plot of the learned dataweight distribution for human and bot data points throughout the dataweight learning procedure. Each horizontal ridge shows a kernel density estimate of the dataweights at one checkpoint. Bot data points decrease in weight over time.
Figure 13: Kernel density estimate of learned dataweights for Bot versus Human examples, targeting STEM .
Figure 14: Test and target loss by segment for model trained on Human-10k + Bot-10k , targeting Expert .
Figure 15: Test and target loss by segment for model trained on Human-30k + Bot-10k , targeting Expert .
Figure 16: Test and target loss by segment for model trained on Human-45k + Bot-10k , targeting Expert .
Figure 17: Test and target loss by segment for model trained on Human-60k + Bot-10k , targeting Expert . Notable overfitting to target observed.
Figure 18: Test and target loss by segment for model trained on Human-100k + Bot-10k , targeting Expert . Notable overfitting to target observed.
Figure 19: Target loss by segment for model trained on Human-155k , targeting STEM .
Figure 20: Test loss curves on Expert domain, while training on pure synthetic data ( Gemini-77k ), with and without dataweights. Test loss from training on Human-77k is shown for reference.
Figure 21: Test and target loss by segment for model trained on Gemini-110k , targeting Expert .
Figure 22: Test and target loss for model trained on Human-45k + Gemini-110k , targeting Expert .
Figure 23: Test loss comparison for equal-sized Human , Synthetic , and mixed training sets, targeting Expert .
Figure 24: Test and target loss by segment for model trained on Gemini-77k , targeting Expert .
Figure 25: Test and target loss by segment for model trained on Human-77k + Gemini-77k , targeting Expert .
Figure 26: Per-sample dataweights learned at 0.5B transfer to a 1.5B reward model. Target and test loss curves on the STEM domain for a 1.5B reward model, with and without dataweights learned at the 0.5B scale.
Figure 27: Transfer of learned dataweights to a 1.5B-parameter reward model trained on Human data, targeting Expert .
Figure 28: Transfer of learned dataweights to a 1.5B-parameter reward model trained on Gemini data, targeting Expert .
Figure 29: Dataweighting’s cost–performance trajectory on the STEM target. Held-out test loss versus compute for two training pools; lower and left is better. Each dataweighting point is one completed meta-epoch, labelled by its zero-based index , including the continuation run. At each budget, we report the test loss of the checkpoint with the lowest selection-set loss available so far. Baselines are shown at their best keep fraction, using the original scoring-plus-one-retrain cost estimates. Black rings mark the non-dominated points among the plotted observations. The grey band is standard deviation over three unweighted seeds. Dataweighting requires no separate retraining stage.
Figure 30: Aggressive selection skews response-length preferences. Audit of Human-155k subsets retained by TRAK, influence functions, TracIn, and random selection across eleven keep fractions (one seed). Left and middle: fraction of retained pairs whose chosen response is shorter than the rejected response, measured in characters and words, respectively; equal-length pairs remain in the denominator. Horizontal lines show the full human pool and the STEM target and test distributions. Right: corresponding test loss at the checkpoint selected using the selection set . TracIn and influence functions increasingly favor shorter-response preferences at low retention. These curves describe the selected data, rather than directly measuring the trained models’ reliance on response length.
Figure 31: The main-paper comparison, on every pool. Figure 1 repeated for all four training pools against the STEM target, same protocol and same panels throughout. Rows 2–3 are a paired pool: Human-30k is the poisoned pool with its bot rows removed and nothing else changed, so reading them adjacently separates what the contamination costs each scorer from what pool size costs it. Dataweighting attains the lowest test loss on all four pools, followed by TRAK and influence functions. Dataweighting’s margin over the strongest baseline is (155k, poisoned 40k, clean 30k, 77k).
| Hyperparameter | Setting |
|---|---|
| Base model | Qwen2.5-Instruct, 0.5B by default; larger models for transfer |
| Reward head | Linear scalar head on the <|cls|> hidden state |
| Training objective | Bradley–Terry loss |
| Optimizer | AdamW |
| Learning rate | |
| Batch size | 128 preference pairs by default |
Table 3: Reward-model training hyperparameters (inner optimization).
| Setup | Figure | Optimizer | LR | |||
|---|---|---|---|---|---|---|
| STEM, per-sample | Parametric comparison(a) | AdamW | 0.1 | 10 | 10 | 1 |
| STEM, linear source | Parametric comparison(a–c) | AdamW | 0.1 | 10 | 10 | 1 |
| Ranking, linear source | Ranking transfer | AdamW | 0.1 | 10 | 10 | 1 |
| Ranking, per-sample | Ranking distribution | SignSGD | 0.1 | 10 | 10 | 1 |
| Selection, per-sample | Selection comparison; all pools | SignSGD | 0.1 | 10 | 5–10 | 1 |
| Human-30k, full-pass | Cost–performance | SignSGD | 0.1 | 1 | 15 | 1 |
Table 4: Dataweight optimization settings and the corresponding figures.
| Component | Setting |
|---|---|
| Embedding model | OpenAI text-embedding-3-small , frozen |
| Input texts | Prompt and the two candidate responses, embedded separately |
| Embedding dimensions | per preference pair |
| Feature normalization | normalization of the concatenated feature vector |
| Weighting head | Single affine map ; no hidden layers |
| Trainable parameters | 4,609: 4,608 coefficients and one bias |
Table 5: Parametric linear dataweight model used for the parametric comparison and ranking transfer.