Data selection for language model pretraining faces a fundamental tension between quality and diversity. While quality filtering is empirically effective, it often induces diversity collapse: by favoring texts similar to high-quality reference corpora (e.g., educational or QA-style data), it systematically excludes valuable data from underrepresented domains. In contrast, diversified selection preserves domain balance and encourages robust downstream performance, yet existing methods either focus on coverage-oriented objectives that indirectly enhance diversity, or directly optimize for diversity via costly covariance matrix recomputation that limits scalability. To address these issues, we introduce Leverage Score Sampling (Lev), which iteratively selects samples that maximally expand the determinantal volume of the embedded data via leverage scores, a computationally efficient criterion that eliminates matrix recomputation and enables scalable selection. Empirically, Lev delivers up to 72× speedup and improves dataset diversity, measured by the Vendi score, by 9.2% over the strong diversification baseline DiSF. On CommonCrawl (CC) web data selection, Lev improves accuracy across seven downstream tasks by up to 1.31% over existing baselines. For domains where robust quality criteria are inherently difficult to define (e.g., code), Lev serves as an effective unsupervised curation alternative: on StarCoderData, the selected subset reduces bits-per-byte by 3.08% over DiSF. Notably, we uncover a cross-domain collapse of quality filtering: CC data filtered by DCLM-fastText fail to retain sufficient code-related content, yielding inferior code performance relative to Lev-selected data. These findings advocate for integrating diversity-aware practices into quality filtering for more effective data curation in language model pretraining.
Figures & tables
Figure 1: Lev outperforms existing diversification baseline DiSF across all fronts.
Figure 2: Leverage score is efficiently computed via a quadratic form (left). It quantifies the relative volume expansion of the candidate sample once selected (right), enabling effective diversification.
Figure 3: The statistical mechanism of leverage score functions: underrepresented semantic components are assigned higher weights, highlighting samples that stretch along these components.
Figure 4: Diversity comparison.
Method
50% Selection Ratio
40% Ratio
ARC-e
ARC-c
WG
HS
PIQA
SIQA
CSQA
Mean ↑
Mean ↑
Random
48.50
27.82
52.09
46.50
69.40
46.30
43.00
47.66 (3)
47.66 (4)
DSIR
53.50
27.73
53.04
42.00
65.20
44.30
42.26
46.86 (5)
44.84 (5)
Density
51.70
27.99
50.75
48.20
66.40
45.00
42.51
47.51 (4)
47.98 (3)
DiSF
52.90
28.16
52.49
46.40
67.80
44.50
42.59
47.83 (2)
48.35 (2)
Table 1: Downstream performance on CommonCrawl subset selection. Higher is better. Accuracy(%) is reported with subscripts indicating the ranking. (Best in bold. Second best in italics.)
Figure 5: Code performance of different diversity-based methods. Lower is better.
Table 2: Comparison between DCLM-fastText (Qual) and Lev under two data recipe. BPB( ×10−2 ) for Code Mean and Accuracy(%) for Base Mean. Subscripts indicate the ranking.
Figure 6: Quality-filtered CC + Lev-selected code data achieves the best performance.
Step size
CommonCrawl
StarCoderData
Lev
DiSF
Speedup
Lev
DiSF
Speedup
1
3.03h
217.28h
72 ×
1.03h
67.45h
65 ×
2
1.67h
108.48h
65 ×
34min
33.68h
59 ×
4
55min
56.12h
61 ×
19min
17.05h
54 ×
8
35min
26.88h
46 ×
11min
8.58h
44 ×
Table 3: Runtime comparison in two real scenarios: web and code selection.
Figure 7: Runtime comparison between Lev and DiSF with respect to batch size B .
Figure 8: Sensitivity to hyperparameters.
Figure 9: Code mean vs. Code selection ratio. Lower is better.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter name
Value
Parameter number
1,143,556,096
Hidden size
2048
Intermediate Hidden Size
4096
Context Len
2048
Heads
32
Layers
20
Appendix
Table 4: Detailed configurations of the 1.1B model of Llama 3.1 architecture.
Method
CluIne
LogDet
CosDist
Entropy
KNN
NovelSum
Vendi
Lev (ours)
0.767
1.638
0.853
5.317
1.076
0.640
203.805
DiSF
0.757
1.623
0.832
5.229
1.071
0.626
186.629
Density
0.744
1.612
0.833
5.181
1.056
0.627
177.802
DCLM-fastText
0.747
1.611
0.835
5.187
1.055
0.626
178.855
DSIR
0.732
1.599
0.821
5.122
1.053
0.611
167.730
Appendix
Table 5: Dataset diversity comparison at 40% selection ratio. Higher is better.
Method
40% web data selection
ARC-e
ARC-c
WG
HS
PIQA
SIQA
CSQA
Mean ↑
Random
48.50 (5)
27.82 (4)
52.09 (3)
46.50 (3)
69.40 (1)
46.30 (1)
43.00 (3)
47.66 (4)
DSIR
49.10 (4)
29.69 (2)
51.22 (4)
39.90 (5)
64.40 (5)
43.70 (5)
35.87 (5)
44.84 (5)
Density
51.50 (3)
28.50 (3)
53.12 (1)
47.40 (2)
68.50 (3)
44.40 (4)
42.42 (4)
47.98 (3)
DiSF
53.70 (2)
30.12 (1)
51.07 (5)
45.20 (4)
67.80 (4)
45.60 (2)
44.96 (2)
48.35 (2)
Appendix
Table 6: Downstream performance at 40% and 50% web data selection. Accuracies (%) are Reported. Higher is Better.
DCLM-fastText for the web data.
Method
50% code data selection
Python
Go
PHP
Ruby
Java
JS
HumanEval
Mean ↓
Random
92.65 (3)
59.22 (2)
52.15 (1)
98.89 (4)
38.90 (2)
68.34 (4)
53.66 (3)
66.26 (4)
Density
92.28 (2)
61.26 (3)
53.71 (3)
91.98 (2)
40.68 (4)
63.55 (2)
57.44 (4)
65.84 (3)
DiSF
96.19 (4)
61.58 (4)
54.90 (4)
91.08 (1)
38.48 (1)
64.35 (3)
53.22 (1)
65.68 (2)
Appendix
Table 7: Downstream performance at 50% and 40% code data selection. Bits-per-byte( ×10−2 ) is reported. Lower is better.
Code Data
Web Data
Code Mean ↓
Base Mean ↑
Random
Lev (top 40%)
65.64 (1)
49.72 (3)
Lev (top 50%)
65.96 (2)
49.85 (2)
Qual (top 40%)
66.26 (3)
50.46 (1)
Random
66.48 (4)
48.76 (4)
Lev (top 50%)
Lev (top 40%)
62.48 (1)
49.90 (2)
Appendix
Table 8: Comparison between DCLM-fastText and leverage score sampling under different code data. BBP( ×10−2 ) for Code Mean and Accuracy(%) for Base Mean.
Method
Number of
Reduction of Code Docs
Code Mean ↓
Code Docs
relative to Lev
Lev
1879
-0.00%
62.48
DCLM-fastText
1544
-17.83%
63.66
NemoCurater
1378
-26.66%
63.82
Random
1003
-46.62%
64.46
Appendix
Table 9: Recall of code-related docs and corresponding code modeling ability under different quality scorers. BBP( ×10−2 ) for Code Mean.
Diversity plays a critical role in data selection, improving performance under fixed data budgets by reducing redundancy and repetition. However, optimizing for diversity is inherently challenging, as it is a set-level property that depends on interactions between data points rather than individual examples. As a result, existing approaches typically rely on proxies or approximations, which often fail to ensure sufficiently diverse subsets. In this work, we directly optimize diversity by introducing a probabilistic diversification framework based on the G-Vendi score, optimized via exponentiated gradient descent. Our method produces subsets that are substantially more diverse than those obtained via random sampling, achieving a +489 increase in G-Vendi score on a 500k-sample subset. We evaluate our approach on FineWeb and DCLM, where it consistently outperforms existing methods. Notably, SPOKES (diversity-only) improves average downstream performance by +0.4 and +0.5 points over random sampling on DCLM and FineWeb, respectively. More importantly, jointly optimizing for both quality and diversity yields the strongest results: SPOKES achieves gains of +1.5 and +1.4 points on DCLM and FineWeb, outperforming all baselines, including semantic deduplication and quality filtering.
Clarence Lee, Yejin Choi, Luke Zettlemoyer +2
DSO National Laboratories · Stanford University · University of Washington
Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quality Filtering (CQF), which trains a binary classifier to distinguish between pretraining data and a small, high-quality set. It assigns each pretraining document a quality score defined as the classifier's score and retains only the top-scoring ones. We provide an in-depth analysis of CQF. We show that while CQF improves downstream task performance, it does not necessarily enhance language modeling on the high-quality set. Importantly, we find that training on CQF-selected data can outperform training directly on the high-quality set, even when the latter is sufficiently large. This finding alone is particularly striking, given the substantial effort and cost recently devoted to augmenting high-quality data. We explain this paradox by the fact that CQF implicitly filters the high-quality dataset as well as the low-quality one. Finally, we introduce an optimization-driven notion of data quality and demonstrate that it can be reliably estimated using small-scale proxy experiments. Altogether, our results both elucidate the mechanisms behind CQF and deepen our understanding of data selection methods widely used in practice.
Thiziri Nait Saada, Louis Bethune, Michal Klein +3
Work done as an intern at Apple · University of Oxford · Apple
Recent research has shown that filtering massive English web corpora into high-quality subsets significantly improves training efficiency. However, for high-resource non-English languages like German, French, or Japanese, aggressive filtering creates a strategic dilemma: should practitioners prioritize diversity by training once on large amounts of lightly filtered web data, or prioritize quality by strictly filtering for a high-quality core and repeating it over multiple epochs? We investigate this trade-off for German by constructing hierarchical quality filters applied to 500M web documents, comparing multi-epoch training on the filtered subsets against single-pass training on a diverse corpus. Our experiments across multiple model scales and token budgets show that repeating high-quality data consistently outperforms single-pass training on larger, less filtered sets. Notably, the performance gap persists even after 7 epochs. Our findings suggest that for non-English LLMs, semantic concentration through quality filtering offers a more viable path to efficient language modeling than simply maximizing unique data volume. We release our German language models (called Boldt), as well as our cleaned evaluation benchmarks to the research community. Our experiments indicate that they achieve state-of-the-art results despite training on 10-360x fewer tokens than comparable models.