cs.LGMar 24, 2026

Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG

Authors: Seungju Han, Konwoo Kim, Chanwoo Park, Benjamin Newman, Suhas Kotha, Jaehun Jung, James Zou, Yejin Choi

Organizations: Stanford University · MIT · University of Washington

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Synthetic data augmentation helps language models learn new knowledge in data-constrained domains. However, naively scaling existing synthetic data methods by training on more synthetic tokens or using stronger generators yields diminishing returns below the performance of RAG. To break the RAG ceiling, we introduce Synthetic Mixed Training, which combines synthetic QAs and synthetic documents. This leverages their complementary training signals, and enables log-linear improvements as both synthetic data volume and generator strength increase. This allows the model to outperform RAG by a 2.6% relative gain on QuaLITY, a long-document reading comprehension benchmark. In addition, we introduce Focal Rewriting, a simple technique for synthetic document generation that explicitly conditions document generation on specific questions, improving the diversity of synthetic documents and yielding a steeper log-linear scaling curve. On QuaLITY, our final recipe trains a Llama 8B model that outperforms RAG by 4.4% relatively. Across models and benchmarks (QuaLITY, LongHealth, FinanceBench), our training enables models to beat RAG in five of six settings, outperforms by 2.6%, and achieves a 9.1% gain when combined with RAG.

Figures & tables

Figure 1: Naively scaling synthetic data plateaus, but our simple methods allows effective scaling and surpass RAG. We evaluate four synthetic data generation strategies using both 8B and 70B generators, scaling training data up to 700M tokens. Across all four baselines, performance saturates and remains below RAG, showing that simply increasing synthetic data or compute is insufficient. In contrast, our two simple techniques— Synthetic Mixed Training and Focal Rewriting —exhibit clear log-linear scaling with both more data and a stronger generator, ultimately surpassing RAG.
Figure 2: (Left) Comparing the data scaling of existing methods: self-generated (8B) synthetic QAs and synthetic documents. This shows QuaLITY accuracy as a function of the number of synthetic training tokens; shaded areas indicate the standard deviation corresponding to the 95% confidence interval, estimated from n=8n=8 inference runs. We use Llama 3.1 8B Inst for both data generation and model training. AR indicates Active Reading ( Lin et al., 2025a ) , EG indicates EntiGraph ( Yang et al., 2025b ) , and WRAP indicates rephrasing ( Maini et al., 2024 ) . On QuALITY, synth QA is substantially more efficient than all existing methods which generate synthetic documents. (Right) Scaling the generator to improve synthetic token efficiency. (1) Scaling the generator to 70B does not improve synth token efficiency for QA, only 0.1% gain over 8B generator at 88M tokens. (2) In contrast, document-based methods do benefit from scaling the generator, achieving 4.5% gain on average. (3) For all methods, even with a stronger generator, data scaling plateaus.
Figure 3: Mixing synthetic documents does not help. Mixing different kinds of synthetic documents (blue line) provides a minimal gain over just using AR documents (pink line).
Figure 4: Synthetic Mixed Training breaks the RAG ceiling. We combine 70B-generated synthetic QAs and AR documents at a 1:1 ratio, attempting to achieve the best of both worlds. This (skyblue line) yields performance comparable to RAG at 350M synthetic training tokens and ultimately surpasses RAG when scaled to 700M tokens.
Figure 5: Synthetic Mixed Training with mixture of domains. Here, the x-axis denotes the number of synthetic tokens grounded in the QuaLITY dataset. (1) Mixing 50% synthetic QAs grounded in a different domain with 50% synthetic documents grounded in the target domain yields a better scaling curve than training solely on target-domain synthetic documents. (2) The best performance comes from mixing target-domain synthetic QAs with target-domain synthetic documents, suggesting that synthetic QAs not only teaches recall behavior but also provides domain-specific knowledge.
Figure 6: Scaling synthetic document generation with Focal Rewriting. We apply Focal Rewriting to AR when generating synthetic documents. Synthetic documents generated with Focal Rewriting (purple line) exhibit better scaling behavior than those generated without it (skyblue line), as shown by the steeper slope of the fitted scaling curve. Data are all generated using 70B model.
Figure 7: Training Qwen3 8B and Llama 3.1 8B Instruct using synthetic data (generated with 70B model) across three benchmarks (QuaLITY, LongHealth, FinanceBench). Using our recipe (purple line) enables beating RAG on 5/6 of the setups.
Dataset#docs#avg tokens/docs#evalEval typeDomain
QuaLITY2656K4609MCQAFictional stories
LongHealth40012K400MCQAMedical
FinanceBench136716K150Free-formFinance
Table 1: Dataset statistics. #\# docs indicates the number of source documents used for data generation, and #\# avg tokens/docs indicates the average number of tokens per source document. #\# eval indicates the number of QA sets used in evaluation. For FinanceBench, we use Qwen3-14B to judge the model-generated answer using gold answer as a reference.
Figure 8: Training different-sized models from the Qwen3 family on the synthetic QuaLITY dataset. We train using our best recipe: Mixed Synthetic Training with synthetic QAs and Focal Rewriting AR documents, using 70B generator. All models exhibit log-linear scaling behavior, and larger models can match RAG performance with fewer synthetic tokens. Based on the fitted curves, we observe that the 14B model requires 102 ×\times , the 8B model 142 ×\times , the 4B model 177 ×\times , and the 1.7B model 813 ×\times more synthetic tokens than the original token count (shown as #synth tokens/#original tokens) .
BenchmarkModelOursOurs + RAGVanilla RAGΔ\Delta
QuaLITYLlama68.2%69.7%65.3%+4.4
Qwen70.1%73.6%69.2%+4.4
LongHealthLlama71.2%80.3%70.0%+10.3
Qwen76.7%82.3%73.7%+8.6
FinanceBenchLlama52.6%54.3%48.0%+6.3
Qwen55.5%60.2%58.8%+1.4
Table 2: Our training complements retrieval augmentation. “Ours” denotes the trained model without RAG, “Ours + RAG” denotes the same trained model with RAG, and “Vanilla RAG” denotes the RAG baseline. Δ\Delta denotes the absolute improvement of “Ours + RAG” over “Vanilla RAG”. Averaged across all settings, our training improves over Vanilla RAG by 5.9 points.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Here’s a learning strategy.
{strategy}
 
Apply this strategy to the document ‘‘{doc_name}’’ of {company}.
 
Output: {generated_text}
Appendix
Table 3: Example data format for synthetic training data with metadata. This is an example of using FinanceBench matadata.
### Question
{question}
### Choices
{options}
Choose the best answer from the following options after thinking step by step. There is only one correct choice.
Your answer format should be like this:
Explanation: [your explanation]
Appendix
Table 4: Prompt template used for multiple-choice QA evaluation. This is used for QuaLITY and LongHealth.
### Question
{question}
Answer the question using the document above.
Your answer format should be like this:
Explanation: [your explanation]
Answer: [your answer]
Table 5 : Prompt template used for open-ended QA evaluation. This is used for FinanceBench.
Appendix
Table 5: Prompt template used for open-ended QA evaluation. This is used for FinanceBench.
Generate question-answer pairs from the following article.
Article:
{article}
ONLY ‘Question: ...‘ and ‘Answer: ...‘ tags are allowed. DO NOT include any other text.
Table 6 : Prompt template used for question-answer pair generation from an article.
Appendix
Table 6: Prompt template used for question-answer pair generation from an article.
Rewrite the following document to help the user understand the document better.
<document>
{document}
</document>
Table 7 : Prompt template used for document rephrasing [ Maini et al., 2024 ] .
Appendix
Table 7: Prompt template used for document rephrasing [ Maini et al., 2024 ] .
As a knowledge analyzer, your task is to dissect and understand an article provided by the user.
You are required to perform the following steps:
1. Summarize the Article: Provide a concise summary of the entire article, capturing the main points and themes.
2. Extract Entities: Identify and list all significant "nouns" or entities mentioned within the article. These entities should include but not limited to:
* People: Any individuals mentioned in the article, using the names or references provided.
* Places: Both specific locations and abstract spaces relevant to the content.
* Object: Any concrete object that is referenced by the provided content.
Appendix
Table 8: Prompt template used to extract entities for EntiGraph [ Yang et al., 2025b ] .
You will act as a knowledge analyzer tasked with dissecting an article provided by the user. Your role involves two main objectives:
1. Rephrasing Content: The user will identify two specific entities mentioned in the article. You are required to rephrase the content of the article twice:
* Once, emphasizing the first entity.
* Again, emphasizing the second entity.
2. Analyzing Interactions: Discuss how the two specified entities interact within the context of the article.
Your responses should provide clear segregation between the rephrased content and the interaction analysis. Ensure each section of the output include sufficient context, ideally referencing the article’s title to maintain clarity about the discussion’s focus.
Here is the format you should follow for your response:
Appendix
Table 9: Prompt template used for entity linking for generating EntiGraph documents [ Yang et al., 2025b ] .
Consider the following document. What are some strategies specific to this document that I can use to help me learn and remember all of the information contained? Use markdown and prefix each strategy with ##.
<document>
{document}
</document>
Table 10 : Prompt template used for generating active reading strategies [ Lin et al., 2025a ] .
Appendix
Table 10: Prompt template used for generating active reading strategies [ Lin et al., 2025a ] .
Here’s a learning strategy.
{strategy}
Apply this strategy to the following document:
<document>
{document}
</document>
Table 11 : Prompt template used for active reading document generation with a provided learning strategy [ Lin et al., 2025a ] .
Appendix
Table 11: Prompt template used for active reading document generation with a provided learning strategy [ Lin et al., 2025a ] .
<document>
{document}
</document>
Here’s a learning strategy.
{strategy}
Apply this strategy to the document above, with the focus on the question: {query}
Table 12 : Prompt template used for Focal Rewriting active reading with a provided learning strategy.
Appendix
Table 12: Prompt template used for Focal Rewriting active reading with a provided learning strategy.
Figure 9: Average gradient similarity between datasets. We compute gradient embeddings for each data point and use cosine similarity between embeddings to measure the similarity of gradients across datasets. (1) QA examples from different datasets exhibit high gradient similarity ( ≥0.94\geq 0.94 ), whereas QA examples and documents (AR) from different datasets show low gradient similarity ( ≤0.25\leq 0.25 ). (2) Even QA and documents from the same dataset do not exhibit high gradient similarity ( ≤0.26\leq 0.26 ).
Figure 10: How data diversity changes with different synthetic document generation methods. (Left) Semantic diversity of synthetic documents, measured by the Vendi score [ Friedman and Dieng, 2023 ] using embedding-based similarity to compute distances between data points. (Right) Lexical diversity of synthetic documents, measured as the ratio of unique 4-grams in the data. For both metrics, higher values indicate greater diversity. (1) Focal Rewriting increases both semantic and lexical diversity at all dataset sizes, and (2) scaling the generator does not significantly affect diversity.
Figure 11: Synthetic mixed training with different synthetic QA-document mixing ratios. We test four mixing ratios of QA and AR: (1:1), (1:8), (2:7), and (8:1). The remaining 10% is used for replay with FineWeb. Using 1:1 mixing gives the best result.
More in this topic

Knowledge Augmentation for Language Models

Knowledge augmentation for language models adds external or newly learned information to improve factuality, reasoning, and domain performance.

See all 171 papers

Explore similar work

CardsList
Sep 29, 2026

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--for instance, they contain little explicit reasoning. Thus, many frontier labs have begun to develop their own internal datasets, starting from state-of-the-art models, to augment their pre-training data mix, eg, with reasoning traces to address cold-start problems. While demonstratively effective, none of these datasets are public, and the effect of this so-called synthetic data on knowledge and skill acquisition of language models, including small ones, remains poorly understood. We present SYNTH, the first open-source synthetic corpus derived from 58,698 Wikipedia articles that collapses pre-, mid-, and post-training into a single training stage via structured amplification of curated encyclopedic seeds. We evaluate SYNTH by training a suite of models: a 56M tiny model (Monad), 0.3B-0.6B dense models (Baguettotron), and a 13B / 1B-active MoE. At iso-compute, SYNTH outperforms filtered web data, and our models remain competitive with similarly-sized open-weight baselines. Because SYNTH is back-translated from grounded passages, SYNTH-trained models achieve high factual precision despite 10-140x fewer training tokens, with memorization targeted by the seed corpus. These results show that synthetic datasets, including our SYNTH dataset, are capable of producing competitive generalist models from a fraction of the training data, enabling rapid iteration as the frontier advances. These findings open up possibilities for both generalist models with significantly increased data efficiency, as well as domain-specific models where no instruction or conversational data is available. Finally, we publicly release our SYNTH dataset and the suite of Baguettotron models under a permissive license, thus supporting open-source language model development.

Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7

PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab +6

Language Model PretrainingSynthetic Data PretrainingEfficient Language Model TrainingSynthetic Data Generation for Language Models
AskPDF
Apr 20, 2026

MASS-RAG: Multi-Agent Synthesis Retrieval-Augmented Generation

Large language models (LLMs) are widely used in retrieval-augmented generation (RAG) to incorporate external knowledge at inference time. However, when retrieved contexts are noisy, incomplete, or heterogeneous, a single generation process often struggles to reconcile evidence effectively. We propose MASS-RAG, a multi-agent synthesis approach to retrieval-augmented generation that structures evidence processing into multiple role-specialized agents. MASS-RAG applies distinct agents for evidence summarization, evidence extraction, and reasoning over retrieved documents, and combines their outputs through a dedicated synthesis stage to produce the final answer. This design exposes multiple intermediate evidence views, allowing the model to compare and integrate complementary information before answer generation. Experiments on four benchmarks show that MASS-RAG consistently improves performance over strong RAG baselines, particularly in settings where relevant evidence is distributed across retrieved contexts.

Xingchen Xiao, Heyan Huang, Runheng Liu +1

School of Computer Science and Technology, Beijing Institute of Technology · Department of Mathematical Sciences, Tsinghua University

Retrieval-Augmented GenerationMulti-Agent LLM SystemsMulti-Agent Reasoning
AskPDF
Mar 23, 2026

SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection

While large language models (LLMs) are pretrained on massive amounts of data, their knowledge coverage remains incomplete in specialized, data-scarce domains, motivating extensive efforts to study synthetic data generation for knowledge injection. We propose SPA (Scaling Prompt-engineered Augmentation), a simple but tough-to-beat baseline that uses a small set of carefully designed prompts to generate large-scale synthetic data for knowledge injection. Through systematic comparisons, we find that SPA outperforms several strong baselines. Furthermore, we identify two key limitations of prior approaches: (1) while RL-based methods may improve the token efficiency of LLM-based data augmentation at small scale, they suffer from diversity collapse as data scales, leading to diminishing returns; and (2) while multi-stage prompting may outperform simple augmentation methods, their advantages can disappear after careful prompt tuning. Our results suggest that, for knowledge injection, careful prompt design combined with straightforward large-scale augmentation can be surprisingly effective, and we hope SPA can serve as a strong baseline for future studies in this area. Our code is available at https://github.com/Tangkexian/SPA.

Kexian Tang, Jiani Wang, Shaowen Wang +1

Institute for Interdisciplinary Information Sciences, Tsinghua University

Knowledge Augmentation for Language ModelsSynthetic Data Generation for Language ModelsLarge Language Model-Based Data AugmentationLLM Prompting
AskPDF