How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

Joel Niklaus; Atsuki Yamaguchi; Michal Štefánik; Guilherme Penedo; Hynek Kydlíček; Elie Bakouch; Lewis Tunstall; Edward Emanuel Beeching

cs.CLApr 15, 2026

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

Authors: Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik, Guilherme Penedo, Hynek Kydlíček, Elie Bakouch, Lewis Tunstall, Edward Emanuel Beeching,

Organizations: Hugging Face · University of Sheffield, United Kingdom · National Institute of Informatics, Japan

Abstract

Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop FinePhrase, a 486-billion-token open dataset of rephrased web text. We show that FinePhrase outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community.

Figures & tables

Figure 1: Overview of our experimental methodology.

Format	Instructional Content Example
math	“Rewrite the document to create a mathematical word problem… Provide a step-by-step solution that shows the calculation process clearly.”
faq	“Rewrite the document as a comprehensive FAQ… Order questions logically from foundational to advanced… Ensure the FAQ works as a standalone document.”
table	“Rewrite the document as a structured table that organizes the key information, then generate one question-answer pair based on the table.”
tutorial	“Rewrite the document as a clear, step-by-step tutorial… Use numbered steps or bullet points where appropriate to enhance clarity.”

Table 1: Excerpts of prompt templates for structured pedagogical rephrasing. Full templates are available in Appendix B.4 .

	Dataset	Macro Avg
Non- synthetic	DCLM	13.77
	Ultra-FineWeb	13.00
	FineWeb-HQ	11.82
	FineWeb-LQ	8.83
Synthetic	Nemotron-HQ-Synth $\star$	13.54
	REWIRE $\star$	13.49

Table 2: Macro-averaged scores (

\times 100

) for different existing datasets at 10K steps. Best and second best in bold and underline .

\star

indicates a synthetic dataset. Full results are in Table 11 .

	Prompt	Macro Avg
	DCLM	13.77
Nemotron-CC	Diverse QA Pairs	14.58
	Distill	13.17
	Wikipedia	13.14
	Knowledge List	13.11
	Extract Knowledge	11.81

Table 3: Macro-averaged scores for different existing rephrasing approaches. Full results are in Table 12 .

Prompt	Macro Avg
DCLM	13.77
math	15.31
faq	14.45
table	14.83
tutorial	14.30

Table 4: New pedagogical structured prompts vs. DCLM. Full results are in Table 13 .

Figure 2: Macro-averaged scores for different Gemma 3 (using math ) and SmolLM2 (using tutorial ) model scales. Full results are in Table 14 .

	Generator Scale
Student Scale	270M	1B	4B	12B	27B
500M	10.96	12.14	11.50	11.49	11.50
1.2B	13.80	15.31	15.06	14.68	14.76
2.9B	13.80	15.97	14.75	16.40	15.72
6.2B	14.39	16.85	16.69	17.06	16.69

Table 5: Macro-averaged scores across student model scales (500M to 6.2B) and Gemma 3-based generator model scales (270M to 27B) trained for 21B tokens on a 50/50 FWHQ mixture using the math prompt.

Family	Macro Avg
DCLM	13.77
SmolLM2	16.55
Falcon 3	15.54
Qwen 3	14.49
Gemma 3	14.72
Granite 3.1	14.87

Table 6: Comparison of model families across the four new prompts (all within the 1B to 1.7B parameter order of magnitude). Full results are in Table 15 .

Prompt	Synthetic Only	Mixed
DCLM	13.77
math	15.20	15.31
faq	13.12	14.45
table	13.74	14.83
tutorial	12.32	14.30

Table 7: Comparison of synthetic-only and mixed training. Full results are in Table 16 .

Table 10

Figure 3: Fine-grained evaluation of pedagogical formats against the DCLM baseline. The best result in each task is marked with

\star

. We use the best-performing SmolLM2 1.7B as

\mathcal{G}

Panel of Figure 4 — Figure 4: GPU cost vs. performance. Symbols distinguish model variants; adjacent characters represent prompt formats (e.g., t : table, m : math).

Dataset	Generator	Tokens	GPU-hrs	Tok/GPU-hr
Cosmopedia	Mixtral 8x7B	25B	$>$ 10K	$<$ 2.5M
SYNTH	fine-tuned	80B	4K	20M
REWIRE	Llama 3.3 70B	400B	$\sim$ 352K	$\sim$ 1.1M
FinePhrase	SmolLM2 1.7B	486B	$\sim$ 14.7K	$\sim$ 33.1M

Table 10: Generation cost comparison (all H100 GPU-hours).

Figure 5: FinePhrase prompts vs. baselines. The best result in each task is marked with

\star

. All FinePhrase runs use SmolLM2 1.7B on FineWeb, mixed with FineWeb-HQ.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	General Knowledge		Reading Comp.		Reasoning		NLU			Math	Table Understanding
Approach	ARC	MMLU	DROP	SQuAD v2	OBQA	XCSQA	WG	PIQA	HS	GSM8K	WikiTQ	TQA
DCLM	15.26	12.78	16.55	42.56	2.94	4.84	3.90	7.42	9.25	10.87	19.31	13.76
Ultra-FineWeb	15.04	12.09	14.67	40.44	2.09	2.83	3.63	6.96	9.03	10.32	17.91	14.62
FineWeb-HQ	17.31	12.15	12.33	34.54	2.37	2.41	2.63	5.73	7.51	10.85	15.01	11.62
FineWeb-LQ	3.87	6.97	12.17	23.86	0.78	0.69	2.45	6.30	8.95	9.48	12.75	10.83
Nemotron-HQ-Synth $\star$	18.33	14.30	16.39	42.56	2.67	4.36	4.34	6.98	8.13	10.53	17.99	10.04

Appendix

Table 11: Breakdown scores (

\times 100

) for different existing datasets at 10K steps.

\star

indicates a synthetic dataset.

	General Knowledge		Reading Comp.		Reasoning		NLU			Math	Table Understanding
Approach	ARC	MMLU	DROP	SQuAD v2	OBQA	XCSQA	WG	PIQA	HS	GSM8K	WikiTQ	TQA
DCLM	15.26	12.78	16.55	42.56	2.94	4.84	3.90	7.42	9.25	10.87	19.31	13.76
Diverse QA Pairs	21.56	13.58	17.62	51.65	3.03	3.06	3.38	5.86	7.51	10.91	17.93	12.03
Distill	20.28	13.58	14.88	39.65	2.64	3.19	3.27	6.20	7.53	10.94	16.38	12.61
Wikipedia	19.46	13.52	14.24	42.90	2.61	2.91	3.60	5.90	7.40	10.41	16.38	11.77
Knowledge List	19.96	14.06	17.83	39.91	3.06	2.94	2.46	6.03	7.32	11.16	13.96	11.52

Appendix

Table 12: Breakdown scores for different existing rephrasing approaches.

	General Knowledge		Reading Comp.		Reasoning		NLU			Math	Table Understanding
Approach	ARC	MMLU	DROP	SQuAD v2	OBQA	XCSQA	WG	PIQA	HS	GSM8K	WikiTQ	TQA
DCLM	15.26	12.78	16.55	42.56	2.94	4.84	3.90	7.42	9.25	10.87	19.31	13.76
math	20.73	14.37	16.04	53.81	2.89	3.76	3.12	6.11	7.68	12.36	20.30	14.50
faq	21.57	14.22	16.50	50.20	3.47	3.27	2.74	6.09	7.70	10.85	16.80	13.46
table	22.76	14.66	14.64	47.37	3.23	3.91	2.88	5.94	7.38	11.24	21.61	15.45
tutorial	21.34	14.11	17.98	45.49	3.06	3.51	3.71	6.07	7.71	10.54	17.29	14.42

Appendix

Table 13: Breakdown scores for new pedagogical structured prompts vs. DCLM.

	General Knowledge		Reading Comp.		Reasoning		NLU			Math	Table Understanding
Approach	ARC	MMLU	DROP	SQuAD v2	OBQA	XCSQA	WG	PIQA	HS	GSM8K	WikiTQ	TQA
DCLM	15.26	12.78	16.55	42.56	2.94	4.84	3.90	7.42	9.25	10.87	19.31	13.76
270M	19.85	13.61	18.57	44.42	2.87	2.70	2.90	5.90	7.40	10.60	17.31	12.70
1B	20.73	14.37	16.04	53.81	2.89	3.76	3.12	6.11	7.68	12.36	20.30	14.50
4B	19.32	14.35	19.24	46.03	3.15	3.43	3.32	5.60	7.44	13.51	20.88	15.03
12B	19.77	13.97	17.30	43.39	3.15	3.01	3.83	5.67	7.44	13.96	19.86	14.84

Appendix

Table 14: Breakdown scores for different Gemma 3 model scales.

	General Knowledge		Reading Comp.		Reasoning		NLU			Math	Table Understanding
Approach	ARC	MMLU	DROP	SQuAD v2	OBQA	XCSQA	WG	PIQA	HS	GSM8K	WikiTQ	TQA
DCLM	15.26 (0.00)	12.78 (0.00)	16.55 (0.00)	42.56 (0.00)	2.94 (0.00)	4.84 (0.00)	3.90 (0.00)	7.42 (0.00)	9.25 (0.00)	10.87 (0.00)	19.31 (0.00)	13.76 (0.00)
SmolLM2	22.47 (0.46)	15.34 (0.14)	18.86 (1.96)	59.67 (1.58)	3.63 (0.16)	4.84 (0.11)	3.79 (0.18)	6.48 (0.13)	7.82 (0.15)	11.71 (1.06)	20.48 (0.81)	16.80 (2.64)
Falcon 3	22.20 (0.44)	15.37 (0.23)	17.29 (2.45)	51.55 (2.39)	3.30 (0.13)	4.59 (0.13)	3.73 (0.23)	6.33 (0.24)	7.75 (0.12)	11.33 (0.54)	20.45 (1.49)	16.00 (1.72)
Qwen 3	22.29 (2.07)	15.02 (0.79)	15.47 (1.07)	46.24 (3.00)	3.28 (0.15)	3.84 (0.56)	3.51 (0.21)	6.11 (0.22)	7.54 (0.16)	11.47 (0.68)	18.76 (0.72)	13.30 (1.56)
Gemma 3 1B	21.60 (0.74)	14.34 (0.21)	16.29 (1.19)	49.22 (3.14)	3.16 (0.21)	3.61 (0.24)	3.11 (0.37)	6.05 (0.06)	7.62 (0.14)	11.25 (0.69)	19.00 (2.02)	14.46 (0.70)

Appendix

Table 15: Breakdown scores for different model families across the four new prompts (all

\sim

1B scale) with standard deviations in parentheses.

	General Knowledge		Reading Comp.		Reasoning		NLU			Math	Table Understanding
Approach	ARC	MMLU	DROP	SQuAD v2	OBQA	XCSQA	WG	PIQA	HS	GSM8K	WikiTQ	TQA
DCLM	15.26	12.78	16.55	42.56	2.94	4.84	3.90	7.42	9.25	10.87	19.31	13.76
math w/o $\mathcal{D}_{\text{mix}}$	20.24	14.02	17.23	54.07	2.64	2.96	3.02	5.76	6.99	12.36	20.34	14.26
math	20.73	14.37	16.04	53.81	2.89	3.76	3.12	6.11	7.68	12.36	20.30	14.50
faq w/o $\mathcal{D}_{\text{mix}}$	21.34	13.02	16.28	40.95	2.47	3.24	2.70	5.55	6.63	10.83	16.24	11.11
faq	21.57	14.22	16.50	50.20	3.47	3.27	2.74	6.09	7.70	10.85	16.80	13.46

Appendix

Table 16: Breakdown scores between synthetic-only and mixed training.

	General Knowledge		Reading Comp.		Reasoning		NLU			Math	Table Understanding
Approach	ARC	MMLU	DROP	SQuAD v2	OBQA	XCSQA	WG	PIQA	HS	GSM8K	WikiTQ	TQA
FineWeb-LQ	3.87	6.97	12.17	23.86	0.78	0.69	2.45	6.30	8.95	9.48	12.75	10.83
FineWeb-HQ	17.31	12.15	12.33	34.54	2.37	2.41	2.63	5.73	7.51	10.85	15.01	11.62
Cosmopedia (LQ)	11.20	9.28	14.26	38.40	1.31	1.66	1.13	5.99	6.93	10.73	14.48	6.85
Cosmopedia (HQ)	19.53	12.74	13.24	43.39	2.38	2.85	2.20	6.33	7.37	11.26	16.26	10.97
DCLM (LQ)	12.37	11.20	16.54	37.59	2.70	3.76	3.99	7.28	8.97	10.36	17.12	13.83

Appendix

Table 17: Breakdown scores for different mix-in dataset

\mathcal{D}_{\text{mix}}

choices with FWHQ (HQ) or FWLQ (LQ) as

\mathcal{D}_{\text{source}}

. Parentheses indicate the mix-in dataset quality (HQ vs. LQ).

	General Knowledge		Reading Comp.		Reasoning		NLU			Math	Table Understanding
Approach	ARC	MMLU	DROP	SQuAD v2	OBQA	XCSQA	WG	PIQA	HS	GSM8K	WikiTQ	TQA
Cosmopedia (FWHQ)	20.33	14.17	15.01	45.23	2.84	3.69	3.65	6.95	8.26	10.79	16.44	12.91
Cosmopedia (Cosmopedia)	11.85	9.48	11.25	35.98	1.36	2.95	1.25	5.56	6.64	10.76	14.06	5.76
DCLM (FWHQ)	20.48	13.78	18.16	46.93	2.83	4.23	4.36	7.08	8.57	10.49	18.09	12.80
DCLM (DCLM)	16.26	12.45	14.17	42.69	2.27	5.37	4.39	7.50	9.16	10.28	20.11	14.36
FineWeb-HQ (FWHQ)	21.34	14.11	17.98	45.49	3.06	3.51	3.71	6.07	7.71	10.54	17.29	14.42

Appendix

Table 18: Breakdown scores for different source dataset

\mathcal{D}_{\text{source}}

choices. Parentheses indicate the mix-in dataset used (Cosmopedia, DCLM, or FWHQ/FWLQ).

Table 23

Student Model Size	FWHQ (Baseline)	FWHQ + Table	Absolute Gain	Gain / Std Ratio
0.5B	9.59 $\pm$ 0.44	14.30 $\pm$ 0.20	+4.71	23.55 $\times$
1.7B	12.09 $\pm$ 0.27	17.25 $\pm$ 0.16	+5.16	32.25 $\times$
2.9B	12.67 $\pm$ 0.22	17.89 $\pm$ 0.19	+5.22	27.47 $\times$
6.2B	13.38 $\pm$ 0.49	18.61 $\pm$ 0.20	+5.23	26.15 $\times$

Appendix

Table 21: Multi-seed grid results (aggregate macro scores

\pm

standard deviation across 9 seed combinations per scale) evaluating model-initialization and data-order randomness.

Synthetic Fraction	Average Macro Score
10%	15.1
20%	15.2
30%	15.8
40%	16.2
50% (Default)	16.4
60%	16.8

Appendix

Table 22: Downstream macro-averaged performance across synthetic token fractions (10% to 90%) mixed with FWHQ.

Model Family	Distinct Openings	Most Common Prefix Count
SmolLM2	1,897	3,235
Qwen 3	40	7,619
Llama 3.2	1,041	2,481
Gemma 3	1,857	3,559
Falcon 3	476	3,733
Granite 3	1,627	1,152

Appendix

Table 23: Prefix repetition (first 10 characters) and structural collapse metrics across 10,000 generations per 1B-class model family.

Metric	SmolLM2	Qwen 3	Llama 3.2	Gemma 3	Falcon 3	Granite 3
No Math Content (Zero Digits)	23.0	0.1	0.7	7.7	2.1	1.9
Per-Equation Arithmetic Accuracy	57.3	78.9	71.6	68.1	71.1	61.8

Appendix

Table 24: Format adherence and arithmetic accuracy across 10,000 generated samples (%).

Dataset	Avg Contamination (Docs/B Tokens)	Benchmark Share Breakdown
DCLM	365	WikiTQ (68%), MMLU-Redux (18%), HS (8%), Others (6%)
Ultra-FineWeb	785	MMLU-Redux (36%), HS (28%), WikiTQ (21%), Others (15%)
FineWeb-Edu-HQ	841	MMLU-Redux (34%), HS (30%), WikiTQ (20%), Others (16%)
FineWeb-Edu-LQ	814	MMLU-Redux (35%), HS (29%), WikiTQ (19%), Others (17%)
Nemotron-HQ-Synth	312	WikiTQ (65%), MMLU-Redux (19%), HS (9%), Others (7%)
REWIRE	$\sim$ 1,900	HS (38%), PIQA (38%), MMLU-Redux (12%), WikiTQ (8%)

Appendix

Table 25: Data contamination rates across existing datasets (10-gram overlaps per billion tokens).

Prompt	Avg Contamination (Docs/B Tokens)	Benchmark Share Breakdown
math	744	MMLU-Redux (34%), HS (30%), WikiTQ (20%), Others (16%)
faq	738	MMLU-Redux (34%), HS (30%), WikiTQ (20%), Others (16%)
table	731	MMLU-Redux (33%), HS (31%), WikiTQ (21%), Others (15%)
tutorial	728	MMLU-Redux (35%), HS (29%), WikiTQ (20%), Others (16%)
Our Framework (Average)	735	MMLU-Redux (34%), HS (30%), WikiTQ (20%), Others (16%)

Appendix

Table 26: Data contamination rates across our structured prompts.

Model Family	Avg Contamination (Docs/B Tokens)	Benchmark Share Breakdown
SmolLM2	651	MMLU-Redux (32%), WikiTQ (25%), HS (26%), Others (17%)
Falcon 3	633	MMLU-Redux (31%), WikiTQ (27%), HS (25%), Others (17%)
Qwen 3	746	MMLU-Redux (33%), HS (29%), WikiTQ (22%), Others (16%)
Gemma 3	801	MMLU-Redux (35%), HS (30%), WikiTQ (20%), Others (15%)
Granite 3.1	694	MMLU-Redux (33%), WikiTQ (23%), HS (27%), Others (17%)
Llama 3.2	858	MMLU-Redux (36%), HS (32%), WikiTQ (18%), Others (14%)

Appendix

Table 27: Data contamination rates across rephrasing model families.

Synthetic Data Pretraining

Synthetic data pretraining uses generated, simulated, or transformed examples to initialize models when real training data are scarce or costly.

See all 119 papers

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
Sep 29, 2026Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7Language Model Pretraining Synthetic Data Pretraining
PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab +6
Data-efficient pre-training by scaling synthetic megadocs
Mar 19, 2026Konwoo Kim, Suhas Kotha, Yejin Choi +3Language Model Pretraining Synthetic Data Pretraining
Stanford University
FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
Jan 29, 2026Ajay Patel, Colin Raffel, Chris Callison-BurchLanguage Model Pretraining Synthetic Data Pretraining
University of Pennsylvania · University of Toronto · Vector Institute +1

Sep 29, 2026J/K move · Enter open · S save

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov+6

PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab+6

Language Model Pretraining Synthetic Data Pretraining Efficient Language Model Training Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--for instance, they contain little explicit reasoning. Thus, many frontier labs have begun to develop their own internal datasets, starting from state-of-the-art models, to augment their pre-training data mix, eg, with reasoning traces to address cold-start problems. While demonstratively effective, none of these datasets are public, and the effect of this so-called synthetic data on knowledge and skill acquisition of language models, including small ones, remains poorly understood. We present SYNTH, the first open-source synthetic corpus derived from 58,698 Wikipedia articles that collapses pre-, mid-, and post-training into a single training stage via structured amplification of curated encyclopedic seeds. We evaluate SYNTH by training a suite of models: a 56M tiny model (Monad), 0.3B-0.6B dense models (Baguettotron), and a 13B / 1B-active MoE. At iso-compute, SYNTH outperforms filtered web data, and our models remain competitive with similarly-sized open-weight baselines. Because SYNTH is back-translated from grounded passages, SYNTH-trained models achieve high factual precision despite 10-140x fewer training tokens, with memorization targeted by the seed corpus. These results show that synthetic datasets, including our SYNTH dataset, are capable of producing competitive generalist models from a fraction of the training data, enabling rapid iteration as the frontier advances. These findings open up possibilities for both generalist models with significantly increased data efficiency, as well as domain-specific models where no instruction or conversational data is available. Finally, we publicly release our SYNTH dataset and the suite of Baguettotron models under a permissive license, thus supporting open-source language model development.

Abstract

Synthetic Data Pretraining

Explore similar work

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

Data-efficient pre-training by scaling synthetic megadocs

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale