WRAP++: Web discoveRy Amplified Pretraining

Jiang Zhou; Yunhao Wang; Xing Wu; Tinghao Yu; Feng Zhang

cs.CLApr 8, 2026

WRAP++: Web discoveRy Amplified Pretraining

Authors: Jiang Zhou, Yunhao Wang, Xing Wu, Tinghao Yu, Feng Zhang

Abstract

Synthetic data rephrasing has emerged as a powerful technique for enhancing knowledge acquisition during large language model (LLM) pretraining. However, existing approaches operate at the single-document level, rewriting individual web pages in isolation. This confines synthesized examples to intra-document knowledge, missing cross-document relationships and leaving facts with limited associative context. We propose WRAP++ (Web discoveRy Amplified Pretraining), which amplifies the associative context of factual knowledge by discovering cross-document relationships from web hyperlinks and synthesizing joint QA over each discovered document pair. Concretely, WRAP++ discovers high-confidence relational motifs including dual-links and co-mentions, and synthesizes QA that requires reasoning across both documents. This produces relational knowledge absent from either source document alone, creating diverse entry points to the same facts. Because the number of valid entity pairs grows combinatorially, this discovery-driven synthesis also amplifies data scale far beyond single-document rewriting. Instantiating WRAP++ on Wikipedia, we amplify ~8.4B tokens of raw text into 80B tokens of cross-document QA data. On SimpleQA, OLMo-based models at both 7B and 32B scales trained with WRAP++ substantially outperform single-document approaches and exhibit sustained scaling gains, underscoring the advantage of cross-document knowledge discovery and amplification.

Figures & tables

Figure 1: Overview of the WRAP++ pipeline. Unlike single-document WRAP, which rewrites individual documents, WRAP++ discovers cross-document entity relationships from web topology and amplifies them into pretraining data through joint QA synthesis.

Data Recipe	OLMo-3-7B	OLMo-3-32B
Pretrained Base	34.76	42.35
+ WRAP	39.55	44.43
+ Extended WRAP	43.69	47.91
+ WRAP++	49.13	53.97

Table 1: SimpleQA results after 1-epoch continued training on OLMo-3-7B and OLMo-3-32B using different data recipes. The metric reported is the empirical pass@128 rate (%).

Figure 2: SimpleQA pass@128 vs. training tokens. Single-document recipes (WRAP and Extended WRAP) reach a data bottleneck early, limiting further knowledge acquisition. In contrast, the combinatorial nature of WRAP++ allows it to scale effectively up to 80B tokens, improving performance without obviously plateauing.

Figure 3: Evolution of pass@ $k$ performance during training. The curves illustrate the unbiased pass@ $k$ of OLMo-3-7B (a) and OLMo-3-32B (b) on SimpleQA ( $k\in[1,128]$ in log scale). The color gradient (from light to dark blue) tracks the accumulation of consumed WRAP++ tokens (from 10B to 80B). The strictly monotonic upward shift across all values of $k$ indicates robust, unsaturated knowledge internalization.

Ablation Axis	Variant	pass@128
Pairing Strategy (§ 4.1 )	Random entity pairing	43.46
Pairing Strategy (§ 4.1 )	Topological relation discovery	45.11
Relation Topology (§ 4.2 )	Dual-link only	44.24
	Co-mention only	44.36
	Combined	45.11
Synthesis Method (§ 4.3 )	Raw concatenation (no QA)	35.43

Table 2: Ablation results across different design choices (OLMo-3-7B,

\sim

8B tokens). We report SimpleQA pass@128. WRAP++ (default) uses topological relation discovery, combined topologies, joint QA synthesis, and the Qwen3-30B-A3B synthesis model.

Single-Document Component	Proportion of WRAP++ in Mixture
Single-Document Component	0% (Baseline Only)	50% (1:1 Mix)
Pretrained Model (No further training)	34.76
Raw FineWiki	39.23	41.80
Distill	38.12	41.59
Extract Knowledge	38.69	43.35
Knowledge List	38.60	42.03

Table 3: Performance comparison of WRAP++ mixed with other single-document strategies. Each mixture contains

\sim

8B tokens total.

Setting	SimpleQA	Gen. Avg
	pass@128	(12 tasks)
Pretrained Base	34.76	57.79
+ Midtrain (100B)	34.74	68.24
+ WRAP++ Mix (100B)	37.58	68.16

Table 4: Integrating WRAP++ into OLMo-3-7B 100B-token mid-training. We report SimpleQA pass@128 and the average over 12 general benchmarks (including MMLU Redux, HellaSwag, etc. See Appendix F for the full list of tasks).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	OLMo-3-7B (8B)	OLMo-3-7B (80B)	OLMo-3-32B
Sequence length	8,192	8,192	8,192
Global batch size (tokens)	$\sim$ 2M ( $2^{21}$ )	$\sim$ 2M ( $2^{21}$ )	$\sim$ 4M ( $4\times 2^{20}$ )
Peak learning rate	$2.07\times 10^{-5}$	$2.07\times 10^{-5}$	$2.07\times 10^{-5}$
LR schedule	Linear $\rightarrow$ 0	Linear $\rightarrow$ 0	Linear $\rightarrow$ 0
Warmup steps	0	0	0
Training steps	4,000	40,000	10,000

Appendix

Table 5: Continued pretraining hyperparameters. All experiments use a linear decay schedule (no warmup, decaying to 0) and SkipStepAdamW ( Olmo et al., 2025 ) with

\beta_{1}\!=\!0.9

\beta_{2}\!=\!0.95

, weight decay

0.1

(embedding weights excluded), max gradient norm

1.0

, and auxiliary z-loss with multiplier

10^{-5}

. Parameters are stored in bfloat16 and gradients are reduced in float32 via HSDP.

Statistic	Value
Source corpus	FineWiki (English)
Raw corpus tokens	$\sim$ 8.4B
Wikipedia articles processed	$\sim$ 6.7M
Dual-link pairs ( $A\leftrightarrow B$ )	$\sim$ 9.6M
Co-mention pairs ( $A\rightarrow E\leftarrow B$ , $A\rightarrow B$ )	$\sim$ 232M
Dual-link synthesized tokens	$\sim$ 3B

Appendix

Table 6: Statistics of the hyperlink relations discovered from FineWiki (English Wikipedia).

Category	Benchmark	Base	Midtrain	WRAP++ Mix
General Knowledge	ARC	77.47	84.98	85.07
General Knowledge	MMLU Redux	60.30	65.30	66.58
Reading Comprehension	SQuAD v2	42.27	48.59	49.99
Reading Comprehension	DROP	40.51	66.51	65.36
Reasoning	OpenBookQA	76.40	85.20	82.40
Reasoning	CSQA	72.15	74.20	72.97

Appendix

Table 7: Per-benchmark breakdown for mid-training integration (OLMo-3-7B, full 100B-token schedule, 3-shot cloze format). Accuracy (%) is reported for all tasks except SQuAD v2 and DROP, which use token-level F1.

Figure 4: Per-benchmark score trajectories during mid-training with the WRAP++ Mix (100B tokens) on OLMo-3-7B. Each subplot tracks one of the 12 evaluation tasks over the course of training. All benchmarks exhibit a clear upward trend, with knowledge-intensive tasks (e.g., HellaSwag, GSM8K, CSQA) showing the most pronounced gains.

Co-mention Variant	SimpleQA pass@8
2-doc (entities $A$ , $B$ only)	15.74
3-doc (entities $A$ , $B$ + bridge $E$ )	15.42

Appendix

Table 8: Co-mention synthesis using 2 vs. 3 input documents (OLMo-3-7B,

\sim

8B tokens, SimpleQA pass@8).

Relation Type	QA Instances	Proportion
Co-mention	231,292,954	96.1%
Dual-link	9,365,111	3.9%
Total	240,658,065	100.0%

Appendix

Table 9: Composition of the WRAP++ synthesized QA dataset by relation type.

Figure 5: Question vs. answer length distributions at character level (left) and word level (right). Questions are tightly concentrated around a median of 203 characters (32 words), while answers exhibit a broader, right-skewed distribution with a median of 1,386 characters (212 words), reflecting the explicit factual chaining required by the synthesis prompt.

Character-level
Field	Mean	Std	Min	P5	P25	Median	P75	P95
Question	207	43	45	143	177	203	233	283
Answer	1,424	624	139	563	938	1,386	1,817	2,463
QA (combined)	1,651	637	242	766	1,161	1,610	2,051	2,716
Word-level
Question	33	7	7	23	28	32	37	45

Appendix

Table 10: Length statistics of WRAP++ synthesized QA data (aggregated over all 240.7M instances). “Chars” denotes character count; “Words” denotes whitespace-delimited token count.

Figure 6: Synthesized QA length distribution by relation type. Both subsets exhibit a similar unimodal shape, with the co-mention distribution (blue) shifted slightly rightward relative to dual-link (red), consistent with the longer factual chains required to bridge co-mentioned entities.

Relation Type	Question	Answer	QA Total
Co-mention	204 (32 words)	1,394 (213 words)	1,618
Dual-link	197 (32 words)	1,210 (189 words)	1,428
Overall	203 (32 words)	1,386 (212 words)	1,610

Appendix

Table 11: Median QA lengths (characters) by relation type. Word-level medians are shown in parentheses.

Figure 7: Violin plot comparison of question length, answer length, and answer word count between co-mention and dual-link subsets. Black horizontal lines indicate medians. Both motifs produce similarly distributed questions, while co-mention answers are moderately longer, reflecting the additional elaboration needed to bridge indirectly related entities.

Figure 8: Source document length distributions for the two input passages. Passage A (the referencing entity) is typically shorter (median

\approx

4,700 chars), while Passage B (the referenced entity) tends to be longer and more prominent (median

\approx

9,500 chars), with a visible mass accumulation at the 50K truncation boundary.

Figure 9: Bucketed length distributions computed over the full dataset of 240.7M instances. Left: combined QA length; right: answer-only length. The 1K–2K character bucket dominates both distributions (

>

55%), with a secondary concentration in the 2K–5K range for QA and the 500–1K range for answers. Extreme lengths (

>

5K or

<

200) are negligible.

Length Range (chars)	QA (combined)		Answer only
Length Range (chars)	Count	%	Count	%
$[0,200)$	0	0.0	2,479	${<}0.01$
$[200,500)$	456,399	0.2	6,771,177	2.8
$[500,1{,}000)$	38,609,776	16.0	61,411,377	25.5
$[1{,}000,2{,}000)$	135,127,249	56.1	132,242,235	55.0
$[2{,}000,5{,}000)$	66,336,557	27.6	40,114,788	16.7

Appendix

Table 12: Bucketed length distributions for the combined QA text and answer text (character-level).

WRAP (single-document QA on entity A)	WRAP++ (cross-document QA on entities A+B)
Q: Who is Ludwig Göransson? A: Ludwig Göransson is a Swedish composer known for film scores. Q: What films has Göransson scored? A: He has scored Black Panther , Tenet , and Oppenheimer .	Q: What films earned Nolan’s original Tenet composer and his replacement their second Oscars? A: Hans Zimmer turned down Tenet for Dune, which earned his second Oscar (after The Lion King). His replacement, Ludwig Göransson, later won his second Oscar for Oppenheimer (after Black Panther). Therefore, the answer is Dune and Oppenheimer.

Appendix

Table 13: Qualitative comparison between single-document WRAP and cross-document WRAP++ synthesis for the Ludwig Göransson and Oppenheimer example.

Knowledge Augmentation for Language Models

Knowledge augmentation for language models adds external or newly learned information to improve factuality, reasoning, and domain performance.

See all 178 papers

AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining
Oct 14, 2026Yunhao Wang, Feng Zhang, Tinghao Yu +2Synthetic Data Generation for Language Models Synthetic Data Pretraining
Hunyuan Team, Tencent
How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
Apr 15, 2026Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9Synthetic Data Pretraining Synthetic Data Generation for Language Models
Hugging Face · University of Sheffield, United Kingdom · National Institute of Informatics, Japan
It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
Sep 29, 2026Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7Language Model Pretraining Synthetic Data Pretraining
PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab +6

Oct 14, 2026J/K move · Enter open · S save

AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining

Yunhao Wang, Feng Zhang, Tinghao Yu, Xing Wu+1

Hunyuan Team, Tencent

Synthetic Data Generation for Language Models Synthetic Data Pretraining Synthetic Data Generation Factual Knowledge in Language Models Language Model Pretraining

Open paper AskPDF arXiv

Synthetic data is increasingly used in language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usually take a source passage or document segment as input and generate rewritten or QA-style samples from that text. This can improve surface form and prompt diversity, but it does not explicitly require fine-grained source facts -- especially secondary entities, list/table entries, and precise temporal attributes -- to appear in the synthesized corpus. We propose AFCS (Atomic-Fact-Conditioned Synthesis), a pipeline that first extracts source-supported atomic facts and then uses each retained fact to drive generation. AFCS uses structure-aware extraction over the document tree, source-grounded verification, local context expansion, and multi-format generation. In continued-pretraining experiments with OLMo-3-7B and a 32B base on the same FineWiki source corpus, AFCS outperforms WRAP and Nemotron-CC on SimpleQA, SimpleQA Verified, PopQA, and HotpotQA. Granularity ablations show that fact-level synthesis improves over chunk-level, paragraph-level, and entity-level synthesis; and a 100B-token mid-training integration improves factual benchmarks and raises the 12-task general benchmark average by +1.09.

Abstract

Knowledge Augmentation for Language Models

Explore similar work

AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs