AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining

Yunhao Wang; Feng Zhang; Tinghao Yu; Xing Wu; Jiang Zhou

cs.CLOct 14, 2026

AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining

Authors: Yunhao Wang, Feng Zhang, Tinghao Yu, Xing Wu, Jiang Zhou

Organizations: Hunyuan Team, Tencent

Abstract

Synthetic data is increasingly used in language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usually take a source passage or document segment as input and generate rewritten or QA-style samples from that text. This can improve surface form and prompt diversity, but it does not explicitly require fine-grained source facts -- especially secondary entities, list/table entries, and precise temporal attributes -- to appear in the synthesized corpus. We propose AFCS (Atomic-Fact-Conditioned Synthesis), a pipeline that first extracts source-supported atomic facts and then uses each retained fact to drive generation. AFCS uses structure-aware extraction over the document tree, source-grounded verification, local context expansion, and multi-format generation. In continued-pretraining experiments with OLMo-3-7B and a 32B base on the same FineWiki source corpus, AFCS outperforms WRAP and Nemotron-CC on SimpleQA, SimpleQA Verified, PopQA, and HotpotQA. Granularity ablations show that fact-level synthesis improves over chunk-level, paragraph-level, and entity-level synthesis; and a 100B-token mid-training integration improves factual benchmarks and raises the 12-task general benchmark average by +1.09.

Figures & tables

Figure 1: AFCS pipeline and source-fact coverage. Source-conditioned synthesis (top) segments the web corpus and asks an LLM to synthesize from each segment, so the generated corpus tends to preserve salient facts while under-covering fine-grained facts in lists, tables, and secondary mentions. Atomic-fact-conditioned synthesis (bottom) first extracts structure-aware atomic facts from the same source corpus and then conditions generation on source-anchored assertion cards. Each card marks the local anchor (e.g., a document scope, list item, table row, or date cell), the natural-language assertion, and the highlighted value structure (e.g., a role binding, multi-element set, compound clause, or bibliographic record). These cards are generation targets rather than canonical entity–relation KG triples.

Method	Data type	Gen. tok.	Train tok.	Mode
WRAP	4-prompt mix	16.6B	20B	source-cond.
Nemotron-CC	5-prompt mix	18.7B	20B	source-cond.
AFCS	multi-format	19.3B	20B	atomic-fact-cond.

Table 1: Compared methods. Gen. tok. counts generated tokens before packing; Train tok. is the matched continued-pretraining budget after packing, with corpora reshuffled/reused as needed. All runs use the same base model, source corpus, optimizer, and LR schedule.

Model	Size	Tokens	SimpleQA	SimpleQA Verified	PopQA	HotpotQA
Base	7B	–	3.63	3.95	23.06	27.78
WRAP	7B	20B	3.99	4.22	25.74	27.08
Nemotron-CC	7B	20B	3.99	4.16	26.42	27.40
AFCS	7B	20B	5.63	6.17	29.80	29.82
Base	32B	–	5.94	5.78	29.17	34.50
WRAP	32B	20B	7.10	7.20	32.46	35.07

Table 2: Main factual benchmark results for synthetic-data methods at 7B and 32B scale. Tokens denotes continued-pretraining tokens; Base is evaluated before continued pretraining. Values are judge-based accuracies (%).

Figure 2: Training dynamics across factual QA benchmarks. The x-axis is aligned by consumed training tokens: 1k steps correspond to 2B tokens for 7B models and 4B tokens for 32B models. Solid lines denote 7B models and dashed lines denote 32B models. AFCS maintains consistent gains over source-conditioned baselines across the average score and individual factual QA benchmarks.

Breakdown	Group	AFCS	WRAP	Nemo.	Chunk	Para.	Entity
Facts/doc	1–19	90.3	82.2	83.5	80.4	77.3	70.3
	20–99	88.9	82.6	84.6	78.9	73.3	65.2
	100+	82.5	61.2	66.5	68.8	69.6	58.4
Fact type	List/table	82.3	72.9	78.8	70.9	70.0	53.5
	Secondary entity	83.6	73.4	79.9	70.5	69.6	52.0
	Fine-grained temporal	80.8	72.5	78.1	69.8	69.5	53.4

Table 3: Source-fact coverage (%) by document fact density and for the fact types highlighted in the omission analysis.

Block	Variant	SQA	SQA-V	PopQA	Hotpot
Reference	Base	3.63	3.95	23.06	27.78
Unit	Fact (AFCS)	5.63	6.17	29.80	29.82
	Chunk	4.21	4.80	27.43	28.94
	Paragraph	4.52	5.10	26.35	27.61
	Entity/triple	3.85	4.30	25.10	27.20
Baseline	EntiGraph	3.91	4.56	23.53	26.56

Table 4: Granularity, adapted-baseline, and mechanism comparisons at 7B scale under the same 20B-token CPT budget. Values are percentages; SQA-V denotes SimpleQA Verified. Fact (AFCS) is the full multi-format mixture, and w/o verification removes source-grounded verification from it. AFCS-QA retains all facts with multiple QA realizations; 1 QA / fact reduces per-fact amplification; Half facts retains 50% of the facts.

Table 7

Model	Data	SQA	SQA-V	PopQA	Hotpot	12-task
Base	–	3.63	3.95	23.06	27.78	57.79
Midtrain	100B gen.	3.68	4.02	23.18	28.07	68.24
Mid.+AFCS	95B gen.+5B AFCS	4.36	4.69	26.06	32.14	69.33

Table 7: Integrating AFCS into OLMo-3-7B 100B-token mid-training. Values are percentages; SQA-V denotes SimpleQA Verified.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Value
Articles processed	$\sim$ 6.7M
Sections processed	78.4M
Total atomic facts extracted	312.7M
Mean facts / article	46.7
Median facts / article	18
P5 / P95 facts / article	3 / 92

Appendix

Table 8: Atomic-fact extraction statistics over the FineWiki training corpus.

Local anchor	Extracted source-supported assertion	Why it is non-canonical
Giovanni Boccaccio	observed that no history of Hippo’s ancestry or place of birth survived, other than from “books of the ancients”	evidential absence and reported observation, not a positive entity attribute
Edzard Ernst	wrote that zero balancing did not appear to have evidence published for its efficacy	negation plus attribution to a source
Ng Sui	trained in mainland China because equipment in Hong Kong was not up to standard and training time was insufficient	causal explanation with an embedded comparison
Nuclear industry	where wastewater discharges to rivers with drinking-water abstraction, harmful or long-half-life radioisotopes such as tritium form part of routine monitoring	conditional process statement with a full-clause value
Segre theorem	states that for any point in a convex hull that is not a vertex, at least four curve points have osculating planes passing through it	mathematical assertion whose anchor is a theorem, not a named-entity relation
List of power pop artists and songs	includes power pop songs by artists whose main music genre is not power pop	list-scope assertion anchored by a document section

Appendix

Table 9: Examples of AFCS atomic assertions that are source-local generation targets rather than canonical KG triples. The notation is compact, but the predicate and value may include negation, attribution, causality, conditionals, or full clauses.

Table 11

Facts/doc	AFCS (prose)	AFCS (all)	WRAP (prose)	Nemo. (prose)
1–19	89.1	90.3	82.2	83.5
20–99	88.3	88.9	82.6	84.6
100+	79.6	82.5	61.2	66.5

Appendix

Table 12: Source-fact coverage (%) on the natural prose-only subset. The all-document AFCS column is included as a reference, not as a structure-removal intervention.

Level	Count	%
L1: All questions	1,000	100
L2: Source article in corpus	776	77.6
L3: Answer verifiably in article	555	55.5
Gap A: no article (L1 $-$ L2)	224	22.4
Gap B: article but answer absent (L2 $-$ L3)	221	22.1

Appendix

Table 13: Source-availability funnel for SimpleQA Verified against FineWiki. Gap A contains questions whose source article is not found in the corpus; Gap B contains questions whose article is found but whose target answer is not verifiably present in that article.

Model	Size	SQA	SQA-V	PopQA	Hotpot
Base	7B	$3.63{\pm}0.17$	$3.95{\pm}0.35$	$23.06{\pm}0.20$	$27.78{\pm}0.22$
WRAP	7B	$3.99{\pm}0.15$	$4.22{\pm}0.37$	$25.74{\pm}0.19$	$27.08{\pm}0.21$
Nemotron-CC	7B	$3.99{\pm}0.22$	$4.16{\pm}0.39$	$26.42{\pm}0.20$	$27.40{\pm}0.22$
AFCS	7B	$\mathbf{5.63{\pm}0.30}$	$\mathbf{6.17{\pm}0.33}$	$\mathbf{29.80{\pm}0.21}$	$\mathbf{29.82{\pm}0.25}$
Base	32B	$5.94{\pm}0.35$	$5.78{\pm}0.36$	$29.17{\pm}0.21$	$34.50{\pm}0.24$
WRAP	32B	$7.10{\pm}0.33$	$7.20{\pm}0.38$	$32.46{\pm}0.21$	$35.07{\pm}0.25$

Appendix

Table 14: Factual benchmark accuracy (%) with bootstrap standard error. SQA and SQA-V denote SimpleQA and SimpleQA Verified.

Model	Size	SQA	SQA-V	PopQA	Hotpot
FineWiki Raw	7B	3.90	4.90	25.23	28.26
AFCS	7B	5.63	6.17	29.80	29.82
FineWiki Raw	32B	6.97	7.26	31.96	35.29
AFCS	32B	9.11	9.08	37.22	35.40

Appendix

Table 15: AFCS versus continued pretraining on raw FineWiki. SQA and SQA-V denote SimpleQA and SimpleQA Verified; values are judge-based accuracies (%).

Category	Benchmark	Base	Mid.	AFCS
Gen. Know.	ARC	77.47	84.98	85.24
Gen. Know.	MMLU Redux	60.30	65.30	66.12
Read. Comp.	SQuAD v2	42.27	48.59	52.96
Read. Comp.	DROP	40.51	66.51	65.93
Reasoning	OpenBookQA	76.40	85.20	82.40
Reasoning	CSQA	72.15	74.20	73.87

Appendix

Table 16: Per-benchmark breakdown for 100B-token mid-training integration. Mid. denotes the 100B general mid-training baseline.

Synthetic Data Generation for Language Models

Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.

See all 293 papers

WRAP++: Web discoveRy Amplified Pretraining
Apr 8, 2026Jiang Zhou, Yunhao Wang, Xing Wu +2Knowledge Augmentation for Language Models Synthetic Data Pretraining
EmbGen: Teaching with Reassembled Corpora
May 19, 2026Arun K Lenin, Kai Rouse, Andrea Nicastro +1Synthetic Instruction Generation Domain Adaptation for LLMs
Commonwealth Bank of Australia Sydney, Australia
How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
Apr 15, 2026Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9Synthetic Data Pretraining Synthetic Data Generation for Language Models
Hugging Face · University of Sheffield, United Kingdom · National Institute of Informatics, Japan

Apr 8, 2026J/K move · Enter open · S save

WRAP++: Web discoveRy Amplified Pretraining

Jiang Zhou, Yunhao Wang, Xing Wu, Tinghao Yu+1

Knowledge Augmentation for Language Models Synthetic Data Pretraining Synthetic Data Generation for Language Models Multi-Hop QA

Open paper AskPDF arXiv

Synthetic data rephrasing has emerged as a powerful technique for enhancing knowledge acquisition during large language model (LLM) pretraining. However, existing approaches operate at the single-document level, rewriting individual web pages in isolation. This confines synthesized examples to intra-document knowledge, missing cross-document relationships and leaving facts with limited associative context. We propose WRAP++ (Web discoveRy Amplified Pretraining), which amplifies the associative context of factual knowledge by discovering cross-document relationships from web hyperlinks and synthesizing joint QA over each discovered document pair. Concretely, WRAP++ discovers high-confidence relational motifs including dual-links and co-mentions, and synthesizes QA that requires reasoning across both documents. This produces relational knowledge absent from either source document alone, creating diverse entry points to the same facts. Because the number of valid entity pairs grows combinatorially, this discovery-driven synthesis also amplifies data scale far beyond single-document rewriting. Instantiating WRAP++ on Wikipedia, we amplify ~8.4B tokens of raw text into 80B tokens of cross-document QA data. On SimpleQA, OLMo-based models at both 7B and 32B scales trained with WRAP++ substantially outperform single-document approaches and exhibit sustained scaling gains, underscoring the advantage of cross-document knowledge discovery and amplification.

Abstract

Synthetic Data Generation for Language Models

Explore similar work

WRAP++: Web discoveRy Amplified Pretraining

EmbGen: Teaching with Reassembled Corpora

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data