FLUX: Data Worth Training On

Gowtham; Sai Rupesh; Sanjay Kumar; Saravanan; Venkata Chaithanya

cs.CLMar 14, 2026

FLUX: Data Worth Training On

Authors: Gowtham, Sai Rupesh, Sanjay Kumar, Saravanan, Venkata Chaithanya

Abstract

Modern large language model training is no longer limited by data availability, but by the inability of existing preprocessing pipelines to simultaneously achieve massive scale and high data quality. Current approaches are forced to sacrifice one for the other: either aggressively filtering to improve quality at the cost of severe token loss, or retaining large volumes of data while introducing substantial noise. In this work, we introduce FLUX, a preprocessing pipeline specifically designed to break this long-standing trade-off by maximizing token retention while enforcing rigorous quality control. Models trained on FLUX-curated data consistently outperform prior methods. A 3B-parameter model trained on 60B tokens with FLUX achieves 32.14% MMLU accuracy, surpassing the previous state-of-the-art pipeline DCLM (31.98%) and significantly outperforming FineWeb (29.88%). FLUX achieves the same aggregate score as a model trained on DCLM data using only 39B tokens, resulting in a 34.4% reduction in training compute. At the data level, FLUX extracts 50B usable tokens from a single dump (CC-MAIN-2025-51), compared to 40B from DCLM (+25% retention). FLUX-Base yields 192B tokens, exceeding FineWeb's 170B while still maintaining superior quality. Overall, FLUX establishes a new state of the art in web-scale data preprocessing by demonstrating that high retention, strong quality control, and computational efficiency can be achieved simultaneously, redefining the limits of scalable dataset construction for modern language models.

Figures & tables

Figure 1: Compute Savings at Equal Performance (3B scale). DCLM requires $1.227\times 10^{21}$ FLOPs to reach 50.48% aggregate score, whereas FLUX reaches the same performance with $8.044\times 10^{20}$ FLOPs (about 34.4% lower compute).

Pipeline	Input	Wall-clock	CPU-hrs	CPU-hrs/TB	Vs. FLUX
DCLM	1.20 TB	11 hr 7 min	355.7	296.4	6.26x
FineWeb	0.60 TB	5 hr 39 min	181	301.3	6.36x
FLUX	1.03 TB	1 hr 31 min	48.5	47.3	1.00x

Table 1: Pipeline-level preprocessing efficiency comparison.

Component	Classification time (in mins)
FastText (Single-bin)	30
FastText (Dual-bin)	60
BERT (110M)	1290

Table 2: Time taken by the components to classify 100B tokens from CC-MAIN-2025-51 snapshot post deduplication. The time taken by the FastText also depends on the size of the bin. We used bin sizes of 1.7GB for both single and dual-bin as mention in Section 3.4 .

Parser	Total Tokens (Multilingual)	Total Tokens (English)	Aggregate	Compute Cost (hrs)
Resiliparse	279.4B	54.6B	0.4089	6.63
Apex	279.5B	54.0B	0.4123	4.31
Trafilatura	136.7B	35.5B	0.4330	47.33

Table 3: Parser Comparison with Runtime Cost (10K WARC files from CC-MAIN-2025-51 snapshot; 530M scale; 10.6B tokens). Total multilingual/English token yield, aggregate downstream score, and compute cost (hours) across Resiliparse, Apex, and Trafilatura.

Figure 4: Parser-Level Downstream Performance Comparison (530M scale; 10.6B tokens). This figure reports aggregated downstream benchmark performance for models trained on corpora produced by each parser configuration (Apex, Resiliparse, and Trafilatura) under the same training setup. It highlights the quality–retention trade-off across parsers and motivates the final parser choice used in the FLUX pipeline.

Pipeline	Post-Dedup Tokens	Aggregate (530M)
FineWeb	24.50B	44.31
DCLM	24.81B	44.40
FLUX	27.22B	45.45

Table 4: FLUX vs DCLM-RefinedWeb and FineWeb — filtering stage results (530M scale; 10.6B tokens).

Figure 5: Heuristic filtering summary (530M scale; 10.6B tokens). Overview figure for the FLUX filtering design and its retention–quality behavior.

Pipeline	Tokens Retained
DCLM	40B
FLUX	50B

Table 5: Token retention after Dual-Bin FastText classification (CC-MAIN-2025-51). FLUX retains 50B tokens from this dump compared to 40B for DCLM—a 25% improvement—while simultaneously achieving higher downstream model quality.

Figure 6: Dual-Bin Threshold Sweep and Final Operating Point (530M scale; 10.6B tokens). Aggregate learning curves across all evaluated $(\tau_{\mathrm{DCLM}},\tau_{\mathrm{BETR}})$ settings on a fixed 100B post-deduplication input from CC-MAIN-2025-51. The selected operating point $(\tau_{\mathrm{DCLM}}=0.025119,\tau_{\mathrm{BETR}}=0.76)$ consistently leads after early training and yields the best final Aggregate while retaining 26.88% of input tokens.

Figure 7: Progressive improvement across FLUX pipeline stages (530M scale; 10.6B tokens). Each successive pipeline stage — text extraction, heuristic filtering, deduplication, and model based classification — yields consistent aggregate score gains, confirming that all four components contribute meaningfully to final dataset quality.

Benchmark	Unique Contaminated Documents	Contaminated Evaluation Instances
MMLU	117	322
HellaSwag	73	82
ARC-Challenge	23	29
ARC-Easy	21	37
CSQA	9	34
PIQA	7	8

Table 6: Decontamination statistics against evaluation benchmarks.

Pipeline	MMLU	ARC-Easy	ARC-Challenge	CSQA	HellaSwag	OpenBookQA	PIQA	SocialIQA	WinoGrande	Aggregate
FineWeb	0.298	0.602	0.337	0.548	0.515	0.400	0.706	0.471	0.518	0.4883
DCLM	0.319	0.673	0.343	0.548	0.524	0.454	0.703	0.467	0.513	0.5048
FLUX	0.321	0.666	0.372	0.570	0.541	0.450	0.722	0.489	0.542	0.5192

Table 7: Performance across the nine-benchmark evaluation suite (3B model, 60B training tokens). FLUX achieves the highest aggregate score (51.92) compared to DCLM (50.48) and FineWeb (48.83). FLUX also obtains the best results on 7 of the 9 benchmarks in the evaluation suite.

Figure 8: Final Comparison Overview (3B scale; 60B tokens). Aggregate learning-curve comparison across FLUX, DCLM, and FineWeb under a matched token budget.

Pipeline	MMLU	ARC-Easy	ARC-Challenge	CSQA	HellaSwag	OpenBookQA	PIQA	SocialIQA	WinoGrande	Aggregate
FineWeb	$0.294\,729$	$0.599$	$0.313\,993$	$0.503\,686$	$0.524$	$0.386$	$0.691$	$0.481$	$0.534\,333$	$0.481\,415\,667$
FLUX-Base	$0.297\,655$	$0.596$	$0.314\,846$	$0.542\,998$	$0.522$	$0.380$	$0.703$	$0.486$	$0.525\,651$	$0.484\,794\,444$

Table 8: Comparison of FLUX-Base and FineWeb across the nine-benchmark evaluation suite (1B model, 20B training tokens). FLUX-Base achieves a slightly higher aggregate score (48.53) than FineWeb (48.05), and outperforms FineWeb on 5 of the 9 benchmarks.

Figure 9: FLUX-Base vs FineWeb (1B scale; 20B tokens). Comparison between FLUX-Base and FineWeb, emphasizing quality-retention behavior when FLUX is evaluated without the final classifier stage.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Scale	Layers (L)	Heads (H)	$d_{model}$	$d_{head}$	Peak LR	Weight decay	Final LR frac ( $\alpha_{f}$ )	Global batch
530M	16	16	1,344	84	$2.8\times 10^{-3}$	0.1	0.1	448
1B	16	16	2,048	128	$2.1\times 10^{-3}$	0.1	0.1	768
3B	24	16	2,752	172	$1.0\times 10^{-4}$	0.1	0.05	960

Appendix

Table 9: Model scales and training hyperparameters. For each model scale, we report the number of transformer layers (L), attention heads (H), hidden dimension (

d_{model}

), per-head dimension (

d_{head}

), peak learning rate, weight decay, final learning rate fraction (

\alpha_{f}

), and global batch size. Batch sizes are reported in sequences of length 2,048 tokens.

Parser	Multilingual (B tokens)	English (B tokens)
Parser V1	391.3	50.7
Parser V2	327.4	49.6
Parser V3	153.4	41.0
Apex	504.5	57.9

Appendix

Table 10: Token yield per parser across multilingual and English corpora.

Component	Parser V1	Parser V2	Parser V3	Apex
WARC Reader	Custom streaming	Custom streaming	FastWARC (C)	FastWARC (C)
HTML Parser	Selectolax/Lexbor	Selectolax/Lexbor	Resiliparse	Resiliparse
Extraction Strategy	Container-first	Container-first	Element-wise traversal	Statistical main-content
Encoding Detection	Multi-stage heuristic	Multi-stage heuristic	Statistical inference	C-level detection

Appendix

Table 11: Architectural comparison of the four parser implementations. The four parsers differ primarily in their content extraction strategy — from container-first (V1/V2) and element-wise traversal (V3) to Apex’s statistical main-content detection — which is the principal driver of performance and yield differences.

Figure 10: Parser-stage comparison across configurations (530m scale; 10.6B tokens). Combined view of parser outputs used in the parser analysis.

#	Component	Functional Group	Type
1	Document Parsing	URL & Normalization	Structural preprocessor
2	UT1 Domain Blocklist	URL & Normalization	Document filter
3	URL Strict Substring	URL & Normalization	Document filter
4	URL Hard Substring	URL & Normalization	Document filter
5	URL Soft Substring	URL & Normalization	Document filter
6	URL Token Removal & Newline Norm.	URL & Normalization	Content modifier (no rejection)

Appendix

Table 12: FLUX filtering pipeline: 13 stages in execution order.

Filter (Stage)	Docs Removed	Tokens Removed	% Total Tokens
UT1 Domain Blocklist (2)	12,057	16,752,457	0.26%
URL Strict Substring (3)	343	514,779	0.01%
URL Hard Substring (4)	2,545	3,723,647	0.06%
URL Soft Substring (5)	396	619,897	0.01%
URL Token Removal $\dagger$ (6)	—	73,019,753	1.13%

Appendix

Table 13: URL-stage removal statistics (10,000 WARC files from CC-MAIN-2025-13 snapshot).

\dagger

Content modifier; no document rejection.

Criterion	Min Threshold	Max Threshold
Document word count	50	100,000
Mean word length (chars)	3.0	10.0
Symbol-to-word ratio	—	0.10
Bullet-line ratio	—	0.90
Ellipsis-line ratio	—	0.30
Alphabetic word ratio	0.80	—

Appendix

Table 14: Gopher Quality Filter thresholds.

Criterion	Max Threshold
Non-alphanumeric character ratio	0.25
Numeric character ratio	0.15
URL character ratio	0.20
Whitespace character ratio	0.25
Parenthesis character ratio	0.10

Appendix

Table 15: Nemo Filter thresholds.

Metric	Granularity	Max Threshold
Duplicate line fraction	Line	0.30
Duplicate line character fraction	Line	0.20
Duplicate paragraph fraction	Paragraph	0.30
Duplicate paragraph character fraction	Paragraph	0.20
Top 2-gram character fraction	Word n-gram	0.20
Top 3-gram character fraction	Word n-gram	0.18

Appendix

Table 16: Gopher Repetition Filter thresholds.

Criterion	Bound	Threshold
Minimum token count	Min	50
Stop-word ratio	Min	0.20
Unclosed bracket ratio	Max	0.05

Appendix

Table 17: Custom Quality Filter thresholds.

Heuristic	Removal Criterion
Minimum word count	Line contains fewer than 2 whitespace-delimited words
Maximum uppercase ratio	Uppercase characters exceed 50% of total line characters
Maximum numeric ratio	Numeric characters exceed 99.9999% of total line characters
Social engagement counters	Pattern: [0-9]+(K\|M\|B)? with likes, shares, comments, retweets, reposts, quotes, bookmarks, upvotes, downvotes, downloads, views, followers
Substring modifiers	Line $\leq 10$ words and matches boilerplate markers such as “items in cart”, “Read more…”, “Sign-in”
JS/CSS artifacts	Line-start code patterns (e.g., function( , var , let , const , $. , @media , => )

Appendix

Table 18: Line-level cleaning heuristics: removal criteria for all eleven heuristic classes. Each class targets a distinct boilerplate pattern; lines matching any criterion are excised individually rather than triggering whole-document rejection.

Filter Stage	Type	Docs Removed	Tokens Removed	% Total
Initial Input	—	—	424,893,496,759	100.00%
UT1 Domain Blocklist (2)	Filter	12,057	16,752,457	0.26%
URL Strict Substring (3)	Filter	343	514,779	0.01%
URL Hard Substring (4)	Filter	2,545	3,723,647	0.06%
URL Soft Substring (5)	Filter	396	619,897	0.01%
URL Token Removal $\dagger$ (6)	Modifier	—	73,019,753	1.13%

Appendix

Table 19: Per-filter document and token removal statistics for FLUX over 10,000 WARC Files from CC-MAIN-2025-13 snapshot.

Filter Component	URL+LID	DocFilter	LineClean	FLUX
URL Blocklist	Y	Y	Y	Y
URL Substring Filtering	Y	Y	Y	Y
URL Token Removal	Y	Y	Y	Y
Newline Normalization	Y	Y	Y	Y
Language Identification	Y	Y	Y	Y
Gopher Repetition	—	Y	Y	Y

Appendix

Table 20: Filter composition across pipeline configurations. Y = active; — = not applied.

Figure 11: Aggregate learning curves for URL+LID against DCLM-RefinedWeb and FineWeb (530M scale; 10.6B tokens).

Figure 12: Aggregate learning curves for DocFilter against DCLM-RefinedWeb and FineWeb (530M scale; 10.6B tokens).

Figure 13: Aggregate learning curves for LineClean against DCLM-RefinedWeb and FineWeb (530M scale; 10.6B tokens).

Figure 14: Aggregate learning curves for FLUX-Base against DCLM-RefinedWeb and FineWeb (530M scale; 10.6B tokens).

Configuration	Tokens	vs DCLM- RefinedWeb	vs FineWeb	Aggregate	vs DCLM- RefinedWeb	vs FineWeb
URL+LID	42.39B	+70.9%	+73.0%	41.76	-2.64 pp	-2.55 pp
DocFilter	23.39B	-5.7%	-4.5%	44.24	-0.16 pp	-0.07 pp
LineClean	21.00B	-15.4%	-14.3%	44.68	+0.28 pp	+0.37 pp
FLUX-Base	27.22B	+9.7%	+11.1%	45.45	+1.05 pp	+1.14 pp
FineWeb	24.50B	—	—	44.31	—	—
DCLM-RefinedWeb	24.81B	—	—	44.40	—	—

Appendix

Table 21: Post-deduplication token retention and downstream aggregate score for all four pipeline configurations with DCLM and FineWeb reference points. FLUX is the only configuration that exceeds both DCLM-RefinedWeb and FineWeb on token retention and downstream quality simultaneously; all other configurations sacrifice one for the other.

Document Composition	Whole-Doc Score	Weighted Score	Retention Decision
800 B English ( $p=0.95$ ) + 200 B French ( $p=0.10$ )	0.88	0.78	Both retain
400 B English ( $p=0.80$ ) + 600 B French ( $p=0.10$ )	0.70	0.38	Whole retains; Weighted rejects

Appendix

Table 22: LID scores for two mixed-language documents under each strategy. Weighted average line LID rejects a majority-French document that whole-document LID retains, illustrating how line-level scoring can discard documents that still carry useful English-language modeling signal.

LID Strategy	Documents Retained	Tokens Retained	Avg. Docs/JSONL	Avg. Tokens/JSONL
Whole-Document	49.4M	48.1B	5,512	5,360,516
Weighted Avg. Line	47.0M	45.0B	5,239	5,088,583

Appendix

Table 23: Retention statistics for each LID strategy across 10,000 WARC Files from CC-MAIN-2025-13 snapshot. Whole-document LID retains 2.4M more documents and 3.1B more tokens than weighted average line LID, motivating its adoption as the FLUX default.

Figure 15: Whole-document LID (black) versus Weighted Average line LID (red) (530M scale; 10.6B tokens).

LID Strategy	Aggregate
Whole-Document LID	0.4191
Weighted Average Line LID	0.4136

Appendix

Table 24: Aggregate accuracy (530M scale; 10.6B tokens). Whole-document LID outperforms weighted average line LID on aggregate score, confirming that its retention advantage translates to better downstream model quality.

Pipeline	Input	Wall-clock	CPU-hrs	CPU-hrs/TB	vs. FLUX
DCLM	1.20 TB	11 hr 7 min	355.7	296.4	6.26 $\times$
FineWeb	0.60 TB	5 hr 39 min	181.0	301.3	6.36 $\times$
FLUX	1.03 TB	1 hr 31 min	48.5	47.3	1.00 $\times$

Appendix

Table 25: Filtering-stage compute cost (c8a.8xlarge, 32 vCPUs, 10,000 WARC Files from CC-MAIN-2025-13 snapshot). FLUX is 6.26

\times

more efficient than DCLM and 6.36

\times

more efficient than FineWeb on a per-TB basis, completing filtering of 1.03 TB in under 2 hours.

FP Rate	Expected n-grams	Retention (GB)	Time	Bin Size	Sparsity
$10^{-13}$	30B	944	2 hr 15 min	216	0.914
$10^{-13}$	50B	944	1 hr 58 min	362	0.771
$10^{-6}$	70B	944	54 min	234	0.716
$10^{-3}$	30B	936	42 min	50	0.911
$\mathbf{10^{-3}}$	100B	944	37 min	179	0.516
$10^{-2}$	100B	944	23 min	56	0.578

Appendix

Table 26: Ablation results over FP rate and expected n-gram count. The selected configuration (FP =

10^{-3}

, 100B expected n-grams) achieves a post-run sparsity of 0.516 — closest to the theoretically optimal 0.5 — while retaining 944 GB and completing in 37 minutes.

$\tau_{\mathrm{DCLM}}$	$\tau_{\mathrm{BETR}}$	Retention %	MMLU	Aggregate
0.2000	0.6350	26.43%	0.2887	0.5028
0.025119	0.7600	26.88%	0.2982	0.5065
0.0900	0.6350	29.25%	0.2920	0.5022
0.018112	0.7347	31.10%	0.2949	0.5025
0.018112	0.7000	33.32%	0.2915	0.5012

Appendix

Table 27: Full Dual-Bin FastText classification threshold sweep (100B tokens post-dedup input). The selected operating point

(\tau_{\mathrm{DCLM}}=0.025119,\tau_{\mathrm{BETR}}=0.76)

achieves the highest Aggregate and MMLU scores across all evaluated pairs; configurations retaining more tokens yield strictly lower scores on both metrics.

Figure 16: Overview of corpus composition in the FLUX dataset. The figure provides a high-level characterization of the FLUX corpus, showing how documents are distributed across major topical domains and structural formats. The diversity observed reflects the broad coverage of web content captured through the FLUX preprocessing pipeline, with detailed domain and format analyses reported in Figures 17 and 18.

Figure 17: Domain category distribution. The FLUX corpus exhibits a diverse domain composition spanning informational, technical, and lifestyle-oriented categories. Health (9.53%), Finance & Business (8.94%), Entertainment (8.22%), and Science & Tech (7.09%) represent the largest segments. Additional domains such as Home & Hobbies, Food & Dining, and Education & Jobs contribute substantial coverage, while categories including Politics, Social Life, and History appear at moderate frequencies. Sensitive domains remain minimal (Adult: 0.63%), indicating effective suppression of undesirable content during dataset construction.

Figure 18: Format category distribution. The FLUX dataset contains a diverse mix of web content formats. Informational structures dominate the corpus, with Knowledge Articles (14.78%), Tutorials (14.57%), and Product Pages (14.36%) forming the largest shares. Narrative and blog-style formats, including Nonfiction Writing (12.15%) and Personal Blogs (7.72%), also contribute significantly. Additional formats such as Listicles (7.50%) and News Articles (4.87%) further enrich structural diversity. Low-value categories including Spam/Ads (0.51%) and Legal Notices (0.21%) remain minimally represented, reflecting the effectiveness of the filtering pipeline.

Figure 19: Learning curves for FLUX, DCLM, and FineWeb across all nine evaluation benchmarks (3B scale; 60B tokens).

Pipeline	MMLU	ARC- Easy	ARC- Challenge	CSQA	Hella Swag	OpenBook QA	PIQA	Social IQA	Wino Grande	Aggregate
(1B scale; 20B tokens)
FineWeb	$0.294\,729$	$0.599$	$0.313\,993$	$0.503\,686$	$0.524$	$0.386$	$0.691$	$0.486$	$0.534\,333$	$0.481\,415\,667$
DCLM	$0.312\,332$	$0.686$	$0.361\,775$	$0.559\,378$	$0.541$	$0.448$	$0.696$	$0.496$	$0.562\,747$	$0.518\,136\,889$
FLUX	$0.316\,814$	$0.671$	$0.363\,481$	$0.591\,319$	$0.554$	$0.496$	$0.728$	$0.498$	$0.540\,647$	$0.528\,806\,777\,8$
(530M scale; 10.6B tokens)
FineWeb	$0.280$	$0.545$	$0.283$	$0.453$	$0.418$	$0.360$	$0.670$	$0.447$	$0.512$	$0.4408$

Appendix

Table 28: Results across the nine-benchmark suite. FLUX outperforms both DCLM and FineWeb on aggregate score at both the 1B and 530M scales, confirming that the quality advantage observed at 3B is consistent across compute regimes.

Training Data Curation

Training data curation improves the quality, diversity, organization, and reliability of datasets used to train machine-learning models.

See all 155 papers

Efficient Pre-Training with Token Superposition
May 7, 2026Bowen Peng, Théo Gigant, Jeffrey QuesnelleLanguage Model Pretraining Efficient Language Model Training
Nous Research
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
Jul 27, 2026Zhen Huang, Yikun Wang, Shijie Xia +1Language Model Pretraining Training Data Curation
1Fudan University · 3SII · 4GAIR +1
ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset
Feb 16, 2026DatologyAI, :, Aldo Gael Carranza +32Multilingual Language Models Training Data Curation
DatologyAI

May 7, 2026J/K move · Enter open · S save

Efficient Pre-Training with Token Superposition

Bowen Peng, Théo Gigant, Jeffrey Quesnelle

Nous Research

Language Model Pretraining Efficient Language Model Training

Open paper AskPDF arXiv

Pre-training of Large Language Models is often prohibitively expensive and inefficient at scale, requiring complex and invasive modifications in order to achieve high data throughput. In this work, we present Token-Superposition Training (TST), a simple drop-in method that significantly improves the data throughput per FLOPs during pre-training without modifying the parallelism, optimizer, tokenizer, data, or model architecture. TST is done in two phases: (i) A highly efficient superposition phase where we combine many contiguous tokens into one bag and train using a multi-hot cross-entropy (MCE) objective, and (ii) a recovery phase where we revert back to standard training. We extensively evaluate TST on the scale of 270M and 600M parameters and validate on 3B and a 10B A1B mixture of experts model, demonstrating that it is highly robust in different settings. Ultimately, TST consistently outperforms baseline loss and downstream evaluations, and under equal-loss settings, TST yields up to a 2.5x reduction in total pre-training time at the 10B A1B scale.

Abstract

Training Data Curation

Explore similar work

Efficient Pre-Training with Token Superposition

DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset