Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG

Seungju Han; Konwoo Kim; Chanwoo Park; Benjamin Newman; Suhas Kotha; Jaehun Jung; James Zou; Yejin Choi

cs.LGMar 24, 2026

Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG

Authors: Seungju Han, Konwoo Kim, Chanwoo Park, Benjamin Newman, Suhas Kotha, Jaehun Jung, James Zou, Yejin Choi

Organizations: Stanford University · MIT · University of Washington

Abstract

Synthetic data augmentation helps language models learn new knowledge in data-constrained domains. However, naively scaling existing synthetic data methods by training on more synthetic tokens or using stronger generators yields diminishing returns below the performance of RAG. To break the RAG ceiling, we introduce Synthetic Mixed Training, which combines synthetic QAs and synthetic documents. This leverages their complementary training signals, and enables log-linear improvements as both synthetic data volume and generator strength increase. This allows the model to outperform RAG by a 2.6% relative gain on QuaLITY, a long-document reading comprehension benchmark. In addition, we introduce Focal Rewriting, a simple technique for synthetic document generation that explicitly conditions document generation on specific questions, improving the diversity of synthetic documents and yielding a steeper log-linear scaling curve. On QuaLITY, our final recipe trains a Llama 8B model that outperforms RAG by 4.4% relatively. Across models and benchmarks (QuaLITY, LongHealth, FinanceBench), our training enables models to beat RAG in five of six settings, outperforms by 2.6%, and achieves a 9.1% gain when combined with RAG.

Figures & tables

Figure 1: Naively scaling synthetic data plateaus, but our simple methods allows effective scaling and surpass RAG. We evaluate four synthetic data generation strategies using both 8B and 70B generators, scaling training data up to 700M tokens. Across all four baselines, performance saturates and remains below RAG, showing that simply increasing synthetic data or compute is insufficient. In contrast, our two simple techniques— Synthetic Mixed Training and Focal Rewriting —exhibit clear log-linear scaling with both more data and a stronger generator, ultimately surpassing RAG.

Panel of Figure 2 — Figure 2: (Left) Comparing the data scaling of existing methods: self-generated (8B) synthetic QAs and synthetic documents. This shows QuaLITY accuracy as a function of the number of synthetic training tokens; shaded areas indicate the standard deviation corresponding to the 95% confidence interval, estimated from $n=8$ inference runs. We use Llama 3.1 8B Inst for both data generation and model training. AR indicates Active Reading ( Lin et al., 2025a ) , EG indicates EntiGraph ( Yang et al., 2025b ) , and WRAP indicates rephrasing ( Maini et al., 2024 ) . On QuALITY, synth QA is substantially more efficient than all existing methods which generate synthetic documents. (Right) Scaling the generator to improve synthetic token efficiency. (1) Scaling the generator to 70B does not improve synth token efficiency for QA, only 0.1% gain over 8B generator at 88M tokens. (2) In contrast, document-based methods do benefit from scaling the generator, achieving 4.5% gain on average. (3) For all methods, even with a stronger generator, data scaling plateaus.

Figure 3: Mixing synthetic documents does not help. Mixing different kinds of synthetic documents (blue line) provides a minimal gain over just using AR documents (pink line).

Figure 4: Synthetic Mixed Training breaks the RAG ceiling. We combine 70B-generated synthetic QAs and AR documents at a 1:1 ratio, attempting to achieve the best of both worlds. This (skyblue line) yields performance comparable to RAG at 350M synthetic training tokens and ultimately surpasses RAG when scaled to 700M tokens.

Figure 5: Synthetic Mixed Training with mixture of domains. Here, the x-axis denotes the number of synthetic tokens grounded in the QuaLITY dataset. (1) Mixing 50% synthetic QAs grounded in a different domain with 50% synthetic documents grounded in the target domain yields a better scaling curve than training solely on target-domain synthetic documents. (2) The best performance comes from mixing target-domain synthetic QAs with target-domain synthetic documents, suggesting that synthetic QAs not only teaches recall behavior but also provides domain-specific knowledge.

Figure 6: Scaling synthetic document generation with Focal Rewriting. We apply Focal Rewriting to AR when generating synthetic documents. Synthetic documents generated with Focal Rewriting (purple line) exhibit better scaling behavior than those generated without it (skyblue line), as shown by the steeper slope of the fitted scaling curve. Data are all generated using 70B model.

Panel of Figure 7 — Figure 7: Training Qwen3 8B and Llama 3.1 8B Instruct using synthetic data (generated with 70B model) across three benchmarks (QuaLITY, LongHealth, FinanceBench). Using our recipe (purple line) enables beating RAG on 5/6 of the setups.

Dataset	#docs	#avg tokens/docs	#eval	Eval type	Domain
QuaLITY	265	6K	4609	MCQA	Fictional stories
LongHealth	400	12K	400	MCQA	Medical
FinanceBench	1367	16K	150	Free-form	Finance

Table 1: Dataset statistics.

\#

docs indicates the number of source documents used for data generation, and

\#

avg tokens/docs indicates the average number of tokens per source document.

\#

eval indicates the number of QA sets used in evaluation. For FinanceBench, we use Qwen3-14B to judge the model-generated answer using gold answer as a reference.

Figure 8: Training different-sized models from the Qwen3 family on the synthetic QuaLITY dataset. We train using our best recipe: Mixed Synthetic Training with synthetic QAs and Focal Rewriting AR documents, using 70B generator. All models exhibit log-linear scaling behavior, and larger models can match RAG performance with fewer synthetic tokens. Based on the fitted curves, we observe that the 14B model requires 102

\times

, the 8B model 142

\times

, the 4B model 177

\times

, and the 1.7B model 813

\times

more synthetic tokens than the original token count (shown as #synth tokens/#original tokens) .

Benchmark	Model	Ours	Ours + RAG	Vanilla RAG	$\Delta$
QuaLITY	Llama	68.2%	69.7%	65.3%	+4.4
QuaLITY	Qwen	70.1%	73.6%	69.2%	+4.4
LongHealth	Llama	71.2%	80.3%	70.0%	+10.3
LongHealth	Qwen	76.7%	82.3%	73.7%	+8.6
FinanceBench	Llama	52.6%	54.3%	48.0%	+6.3
FinanceBench	Qwen	55.5%	60.2%	58.8%	+1.4

Table 2: Our training complements retrieval augmentation. “Ours” denotes the trained model without RAG, “Ours + RAG” denotes the same trained model with RAG, and “Vanilla RAG” denotes the RAG baseline.

\Delta

denotes the absolute improvement of “Ours + RAG” over “Vanilla RAG”. Averaged across all settings, our training improves over Vanilla RAG by 5.9 points.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Here’s a learning strategy.

{strategy}

Apply this strategy to the document ‘‘{doc_name}’’ of {company}.

Output: {generated_text}

Appendix

Table 3: Example data format for synthetic training data with metadata. This is an example of using FinanceBench matadata.

### Question

{question}

### Choices

{options}

Choose the best answer from the following options after thinking step by step. There is only one correct choice.

Your answer format should be like this:

Explanation: [your explanation]

Appendix

Table 4: Prompt template used for multiple-choice QA evaluation. This is used for QuaLITY and LongHealth.

### Question

{question}

Answer the question using the document above.

Your answer format should be like this:

Explanation: [your explanation]

Answer: [your answer]

Table 5 : Prompt template used for open-ended QA evaluation. This is used for FinanceBench.

Appendix

Table 5: Prompt template used for open-ended QA evaluation. This is used for FinanceBench.

Generate question-answer pairs from the following article.

Article:

{article}

ONLY ‘Question: ...‘ and ‘Answer: ...‘ tags are allowed. DO NOT include any other text.

Table 6 : Prompt template used for question-answer pair generation from an article.

Appendix

Table 6: Prompt template used for question-answer pair generation from an article.

Rewrite the following document to help the user understand the document better.

{document}

</document>

Table 7 : Prompt template used for document rephrasing [ Maini et al., 2024 ] .

Appendix

Table 7: Prompt template used for document rephrasing [ Maini et al., 2024 ] .

As a knowledge analyzer, your task is to dissect and understand an article provided by the user.

You are required to perform the following steps:

1. Summarize the Article: Provide a concise summary of the entire article, capturing the main points and themes.

2. Extract Entities: Identify and list all significant "nouns" or entities mentioned within the article. These entities should include but not limited to:

* People: Any individuals mentioned in the article, using the names or references provided.

* Places: Both specific locations and abstract spaces relevant to the content.

* Object: Any concrete object that is referenced by the provided content.

Appendix

Table 8: Prompt template used to extract entities for EntiGraph [ Yang et al., 2025b ] .

You will act as a knowledge analyzer tasked with dissecting an article provided by the user. Your role involves two main objectives:

1. Rephrasing Content: The user will identify two specific entities mentioned in the article. You are required to rephrase the content of the article twice:

* Once, emphasizing the first entity.

* Again, emphasizing the second entity.

2. Analyzing Interactions: Discuss how the two specified entities interact within the context of the article.

Your responses should provide clear segregation between the rephrased content and the interaction analysis. Ensure each section of the output include sufficient context, ideally referencing the article’s title to maintain clarity about the discussion’s focus.

Here is the format you should follow for your response:

Appendix

Table 9: Prompt template used for entity linking for generating EntiGraph documents [ Yang et al., 2025b ] .

Consider the following document. What are some strategies specific to this document that I can use to help me learn and remember all of the information contained? Use markdown and prefix each strategy with ##.

{document}

</document>

Table 10 : Prompt template used for generating active reading strategies [ Lin et al., 2025a ] .

Appendix

Table 10: Prompt template used for generating active reading strategies [ Lin et al., 2025a ] .

Here’s a learning strategy.

{strategy}

Apply this strategy to the following document:

{document}

</document>

Table 11 : Prompt template used for active reading document generation with a provided learning strategy [ Lin et al., 2025a ] .

Appendix

Table 11: Prompt template used for active reading document generation with a provided learning strategy [ Lin et al., 2025a ] .

{document}

</document>

Here’s a learning strategy.

{strategy}

Apply this strategy to the document above, with the focus on the question: {query}

Table 12 : Prompt template used for Focal Rewriting active reading with a provided learning strategy.

Appendix

Table 12: Prompt template used for Focal Rewriting active reading with a provided learning strategy.

Figure 9: Average gradient similarity between datasets. We compute gradient embeddings for each data point and use cosine similarity between embeddings to measure the similarity of gradients across datasets. (1) QA examples from different datasets exhibit high gradient similarity ( $\geq 0.94$ ), whereas QA examples and documents (AR) from different datasets show low gradient similarity ( $\leq 0.25$ ). (2) Even QA and documents from the same dataset do not exhibit high gradient similarity ( $\leq 0.26$ ).

Panel of Figure 10 — Figure 10: How data diversity changes with different synthetic document generation methods. (Left) Semantic diversity of synthetic documents, measured by the Vendi score [ Friedman and Dieng, 2023 ] using embedding-based similarity to compute distances between data points. (Right) Lexical diversity of synthetic documents, measured as the ratio of unique 4-grams in the data. For both metrics, higher values indicate greater diversity. (1) Focal Rewriting increases both semantic and lexical diversity at all dataset sizes, and (2) scaling the generator does not significantly affect diversity.

Figure 11: Synthetic mixed training with different synthetic QA-document mixing ratios. We test four mixing ratios of QA and AR: (1:1), (1:8), (2:7), and (8:1). The remaining 10% is used for replay with FineWeb. Using 1:1 mixing gives the best result.

Knowledge Augmentation for Language Models

Knowledge augmentation for language models adds external or newly learned information to improve factuality, reasoning, and domain performance.

See all 171 papers

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
Sep 29, 2026Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7Language Model Pretraining Synthetic Data Pretraining
PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab +6
MASS-RAG: Multi-Agent Synthesis Retrieval-Augmented Generation
Apr 20, 2026Xingchen Xiao, Heyan Huang, Runheng Liu +1Retrieval-Augmented Generation Multi-Agent LLM Systems
School of Computer Science and Technology, Beijing Institute of Technology · Department of Mathematical Sciences, Tsinghua University
SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection
Mar 23, 2026Kexian Tang, Jiani Wang, Shaowen Wang +1Knowledge Augmentation for Language Models Synthetic Data Generation for Language Models
Institute for Interdisciplinary Information Sciences, Tsinghua University

Sep 29, 2026J/K move · Enter open · S save

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov+6

PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab+6

Language Model Pretraining Synthetic Data Pretraining Efficient Language Model Training Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--for instance, they contain little explicit reasoning. Thus, many frontier labs have begun to develop their own internal datasets, starting from state-of-the-art models, to augment their pre-training data mix, eg, with reasoning traces to address cold-start problems. While demonstratively effective, none of these datasets are public, and the effect of this so-called synthetic data on knowledge and skill acquisition of language models, including small ones, remains poorly understood. We present SYNTH, the first open-source synthetic corpus derived from 58,698 Wikipedia articles that collapses pre-, mid-, and post-training into a single training stage via structured amplification of curated encyclopedic seeds. We evaluate SYNTH by training a suite of models: a 56M tiny model (Monad), 0.3B-0.6B dense models (Baguettotron), and a 13B / 1B-active MoE. At iso-compute, SYNTH outperforms filtered web data, and our models remain competitive with similarly-sized open-weight baselines. Because SYNTH is back-translated from grounded passages, SYNTH-trained models achieve high factual precision despite 10-140x fewer training tokens, with memorization targeted by the seed corpus. These results show that synthetic datasets, including our SYNTH dataset, are capable of producing competitive generalist models from a fraction of the training data, enabling rapid iteration as the frontier advances. These findings open up possibilities for both generalist models with significantly increased data efficiency, as well as domain-specific models where no instruction or conversational data is available. Finally, we publicly release our SYNTH dataset and the suite of Baguettotron models under a permissive license, thus supporting open-source language model development.

Abstract

Knowledge Augmentation for Language Models

Explore similar work

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

MASS-RAG: Multi-Agent Synthesis Retrieval-Augmented Generation

SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection