cs.LGMar 19, 2026

Data-efficient pre-training by scaling synthetic megadocs

Authors: Konwoo Kim, Suhas Kotha, Yejin Choi, Tatsunori Hashimoto, Nick Haber, Percy Liang

Organizations: Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Synthetic data augmentation has emerged as a promising solution when pre-training is constrained by data rather than compute. We study how to design synthetic data algorithms that achieve better loss scaling: not only lowering loss at finite compute but especially as compute approaches infinity. We first show that pre-training on web data mixed with synthetically generated rephrases improves i.i.d. validation loss on the web data, despite the synthetic data coming from an entirely different distribution. With optimal mixing and epoching, loss and benchmark accuracy improve without overfitting as the number of synthetic generations grows, plateauing near 1.48×1.48\times data efficiency at 32 rephrases per document. We find even better loss scaling under a new perspective: synthetic generations from the same document can form a single substantially longer megadocument instead of many short documents. We show two ways to construct megadocs: stitching synthetic rephrases from the same web document or stretching a document by inserting rationales. Both methods improve i.i.d. loss, downstream benchmarks, and especially long-context loss relative to simple rephrasing, increasing data efficiency from 1.48×1.48\times to 1.80×1.80\times at 3232 generations per document. Importantly, the improvement of megadocs over simple rephrasing widens as more synthetic data is generated. Our results show how to design synthetic data algorithms that benefit more from increasing compute when data-constrained.

Figures & tables

Figure 1: Synthetic data by scaling generation count and utilizing megadocs. When training on 200M unique real tokens, the best 300M parameter model with tuned epoching and regularization achieves 3.553.55 loss ( Kim et al., 2025 ) . By mixing synthetically generated rephrases into pre-training, we can improve i.i.d. loss on the original distribution (orange points), monotonically as the number of generations per document grows. We consider two synthetic data algorithms that leverage multiple generations to produce a single megadoc: stitched rephrasing, which concatenates all generations from the same document (blue points), and latent thoughts, which stretches documents by adding rationales (gray points). Both algorithms improve i.i.d. loss and improve scaling in the number of generations per document. The scaling in synthetic token count is even better as the average latent thought is shorter than the average rephrase.
Figure 2: Scaling synthetic generations. Left: Given 200M DCLM tokens, we tune a 300M parameter baseline utilizing epoching and regularization which achieves 3.553.55 loss (purple point). We then measure the benefit of sampling GG rephrases per real doc and searching for locally optimal learning rate, weight decay, epoch count, and mixing fraction (orange points). We find that loss monotonically improves in the number of rephrases generated, appearing to plateau around 3232 generations at loss 3.413.41 . Right: The loss improvements and plateau are reflected on downstream benchmarks.
Figure 3: Synthetic data streams for megadocs. We visualize synthetic streams with 2 real docs and 3 generations. Simple rephrasing (Section 3 ) permutes all generations and real docs. Stitched rephrasing concatenates all rephrases from the same real doc and either prepends or appends each real doc. Latent thoughts fixes GG split points and synthesizes a rationale to derive each suffix from its prefix. Notably, megadocs can exceed the length of the model’s context. When training, masking is always disabled across document and megadocument boundaries.
Figure 4: Different ways to scale document length. We compare synthetic data algorithms (visualized in Figure 3 ) given 8 generations per pre-training doc. We run 3 seeds of each algorithm. Both variants of stitched rephrasing outperform simple rephrasing with real last slightly outperforming real first. Latent thoughts further improves both i.i.d and arxiv loss.
Figure 5: Scaling generation count for stitching and latent thoughts. We find that stitched rephrasing (blue points) and latent thoughts (gray points) improve i.i.d. loss, long-context loss, and downstream benchmark accuracy over simple rephrasing (orange points). The improvements scale in the number of generations per document and show less signs of plateauing.
Figure 6: Megadocs benefit from more steps on real and synthetic data. We attribute the favorable scaling of megadocs to different optimal hyperparameters enabling longer training. For different synthetic data pools, we plot the optimal loss at each real data epoch count (left). We find that more synthetic tokens (higher GG ) allows for training for more real epochs, and megadocs push this effect further. Controlling for longer training by using the best simple rephrasing hyperparameters (right) shows that megadocs still improve but the loss win doesn’t scale with GG . The scaling is less clean in the right plot since the hyperparameters are tuned for simple rephrasing.
Figure 7: Rephrasing composes with ensembling. We measure the loss of ensembling synthetically pre-trained models as we increase the member count. For a given synthetic data recipe, we use the asymptote (horizontal dotted line) of a power law to estimate the loss as the member count approaches infinity. Pre-training with self-distillation does not compose with ensembling as ensembling self-distilled models does not improve the asymptote. Meanwhile, simple rephrasing, stitched rephrasing, and latent thoughts all compose with ensembling and improve the asymptote as member count approaches infinity. Note that unlike simple rephrasing, the hyperparameters for stitched rephrasing and latent thoughts have not been tuned to improve the asymptote.
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Student scaling . We pre-train 1.5B student models using the same set of synthetic data methods. At each scale, we measure the loss win of each method over the no synthetic data baseline (purple). We observe a larger loss win at the 1.5B scale than the 300M scale for all methods.
Figure 9: Student and generator scaling laws for continued pre-training. For a small set of 1.3M tokens, we measure QA accuracy when training utilizing Rephrase the Web [ Maini et al., 2024 ] . We find that self-improvement is possible (3B student and generator, 8B student and generator). Additionally, rephrasing only emerges with sufficient scale, stronger students benefit more from synthetic data, and synthetic data improvements eventually stop scaling in generator capability. Figure taken from preliminary experiments in [ Kotha et al., 2025 ] .
Figure 10: Convexity in hyperparameters. For a fixed pool of synthetic data (16 rephrases per document), we show how the optimal loss changes as we vary two hyperparameters: real data epochs and mixing fraction. For our synthetic data scaling experiments, we pick the "locally optimal" solution so that deviating from any of the hyperparameters doesn’t improve loss.
Figure 11: Optimal hyperparameters. We visualize the locally optimal hyperparameters for each of the synthetic data methods we scale in our paper. We observe that megadoc-based methods can epoch the real data more and sustain a higher mixing ratio without overfitting, particularly at the highest synthetic generation counts.
Figure 12: Cross-document attention ablation. In the data-eficient regime with only real data, it is better to mask attention between separate documents in the same context window (purple bars). When also training on rephrased data, the answer changes (orange bars). The benefit of stitching additionally justifies disabling cross-document attention masking.
Figure 13: Mixing ablation. We briefly ablate the importance of mixing in real data. Given a small synthetic data pool (2 rephrases per document), we find that also training on the real data vs. not training on it substantially improves loss from 3.521 to 3.456. Consequently, we always mix real data and search for a locally optimal mixing fraction.
Figure 14: Including real document in synthetic stream. We show how stitching is a benefit over simple rephrasing both when there is no real data in the synthetic stream (striped blue vs striped orange) and when there is real data (solid blue vs solid orange). Since including the real document in the synthetic stream is helpful for both simple rephrasing (solid orange vs striped orange) and stitched rephrasing (solid blue vs striped blue), we opt to include a real document in the synthetic stream for all experiments in the main body.
Figure 15: Additional long-context evaluations. Megadocs achieve lower loss and show less signs of plateauing than simple rephrasing on additional long-context evaluations on Github Python code and Wikipedia articles.
Figure 16: Scaling trends on shorter vs longer documents. We split the validation set in half (based on document length) and evaluate the scaling of our methods on both splits. The short and long splits have max document lengths of 621 and 36319 tokens respectively. The scaling of megadocs vs. simple rephrasing exhibits the same trend on the short and long splits, suggesting that megadocs help more generally with modeling many types of data.
Parameter300M1.5B
Context Length40964096
Hidden Dimension5121536
Intermediate Dimension17925376
Attention Heads816
KV Heads88
Layers636
Appendix
Table 1: Model architecture configurations for different model sizes. We use the 1.5B model for student scaling experiments.
More in this topic

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 440 papers

Explore similar work

CardsList
Apr 15, 2026

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop FinePhrase, a 486-billion-token open dataset of rephrased web text. We show that FinePhrase outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community.

Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9

Hugging Face · University of Sheffield, United Kingdom · National Institute of Informatics, Japan

Synthetic Data PretrainingSynthetic Data Generation for Language ModelsSynthetic Data SelectionPrompt Engineering
AskPDF
Oct 22, 2026

Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

Synthetic rewrites, where a language model rewrites a natural document e.g. into a target style, are now a standard ingredient of pretraining corpora. However, critical aspects of their use remain poorly understood, such as how much compute should be spent rewriting training data and whether a model trained on rewrites can outperform the rewriter. We study these two questions by proposing a rewriting scaling law that predicts loss from model size, training tokens, the fraction of training tokens that are rewrites, and the number of epochs over the natural data, charging the compute spent generating rewrites. We fit this scaling law in two settings. In the compute-limited setting, natural data is plentiful and rewrites come from a third-party rewriter trained with more compute than our largest run. We find that the estimated optimal rewrite fraction increases with compute, from under 10%10\% at 3×10183\times10^{18} FLOPs to 32%32\% at 102210^{22} FLOPs. Our scaling forecast extrapolates 33×33\times in FLOPs to train a 22B-parameter model, whose final loss is within 1.1%1.1\% of our prediction. This model achieves a better BPB than its rewriter on several web domains at one-tenth the rewriter's training compute. In the data-limited setting, we assume we only have a fixed seed set of natural data and that our rewriter is a model of the same size as the target, pretrained on that seed set. Again, we find the optimal rewrite fraction increases with the compute budget. We also demonstrate that a model trained on rewrites of its own pretraining data, generated by a rewriter of the same size, beats the best run on natural data alone at every model size up to 3B. Our results show that synthetic rewriting can be studied through the scaling law lens: that optimal methodological choices are predictable and scaling does not necessarily break down beyond the rewriter's capabilities.

Nicholas Roberts, Ahmed Ahmed, Suhas Kotha +4

Princeton University · Stanford University · Jane Street

Synthetic Data PretrainingLanguage Model Scaling LawsSynthetic Data Generation for Language Models
AskPDF
May 18, 2026

Generating Pretraining Tokens from Organic Data for Data-Bound Scaling

LLM pretraining is shifting from a compute-bound to a data-bound regime, where available human (organic) text falls far short of scaling demands. However, reaching the data-bound regime does not mean the model has fully utilized its organic corpus. In this paper, we introduce SynPro, a synthetic data generation framework that helps LLMs more thoroughly learn from limited organic data. SynPro applies two operations, rephrasing and reformat, that present the same organic source in diverse forms to facilitate deeper learning without introducing external information. Both generators are optimized via reinforcement learning with quality, faithfulness, and data influence rewards, and are continuously updated as pretraining plateaus to target content the model has yet to absorb. We pretrain 400M and 1.1B models with 10% of their Chinchilla-optimal tokens (0.8B and 2.2B) from DCLM-Baseline, reflecting a realistic data-bound regime in frontier pretraining. Our results reveal that organic data is significantly underutilized by standard repetition: SynPro unlocks 3.7-5.2x the effective tokens of repetition, even surpassing the non-data-bound oracle that trains on equivalent unique data at the 1.1B scale. Analyses confirm that faithful, model-aware synthesis sustains data-bound scaling without causing distribution collapse. We open-source our code at https://github.com/cxcscmu/SynPro.

Zichun Yu, Chenyan Xiong

Language Technologies Institute, Carnegie Mellon University · Xlue

Language Model PretrainingSynthetic Data PretrainingSynthetic Data GenerationSynthetic Data Generation for Language Models
AskPDF