cs.LGMar 19, 2026
SaveData-efficient pre-training by scaling synthetic megadocs
Organizations: Stanford University
Abstract
Synthetic data augmentation has emerged as a promising solution when pre-training is constrained by data rather than compute. We study how to design synthetic data algorithms that achieve better loss scaling: not only lowering loss at finite compute but especially as compute approaches infinity. We first show that pre-training on web data mixed with synthetically generated rephrases improves i.i.d. validation loss on the web data, despite the synthetic data coming from an entirely different distribution. With optimal mixing and epoching, loss and benchmark accuracy improve without overfitting as the number of synthetic generations grows, plateauing near data efficiency at 32 rephrases per document. We find even better loss scaling under a new perspective: synthetic generations from the same document can form a single substantially longer megadocument instead of many short documents. We show two ways to construct megadocs: stitching synthetic rephrases from the same web document or stretching a document by inserting rationales. Both methods improve i.i.d. loss, downstream benchmarks, and especially long-context loss relative to simple rephrasing, increasing data efficiency from to at generations per document. Importantly, the improvement of megadocs over simple rephrasing widens as more synthetic data is generated. Our results show how to design synthetic data algorithms that benefit more from increasing compute when data-constrained.
Figures & tables
Figure 1: Synthetic data by scaling generation count and utilizing megadocs. When training on 200M unique real tokens, the best 300M parameter model with tuned epoching and regularization achieves loss ( Kim et al., 2025 ) . By mixing synthetically generated rephrases into pre-training, we can improve i.i.d. loss on the original distribution (orange points), monotonically as the number of generations per document grows. We consider two synthetic data algorithms that leverage multiple generations to produce a single megadoc: stitched rephrasing, which concatenates all generations from the same document (blue points), and latent thoughts, which stretches documents by adding rationales (gray points). Both algorithms improve i.i.d. loss and improve scaling in the number of generations per document. The scaling in synthetic token count is even better as the average latent thought is shorter than the average rephrase.
Figure 2: Scaling synthetic generations. Left: Given 200M DCLM tokens, we tune a 300M parameter baseline utilizing epoching and regularization which achieves loss (purple point). We then measure the benefit of sampling rephrases per real doc and searching for locally optimal learning rate, weight decay, epoch count, and mixing fraction (orange points). We find that loss monotonically improves in the number of rephrases generated, appearing to plateau around generations at loss . Right: The loss improvements and plateau are reflected on downstream benchmarks.
Figure 3: Synthetic data streams for megadocs. We visualize synthetic streams with 2 real docs and 3 generations. Simple rephrasing (Section 3 ) permutes all generations and real docs. Stitched rephrasing concatenates all rephrases from the same real doc and either prepends or appends each real doc. Latent thoughts fixes split points and synthesizes a rationale to derive each suffix from its prefix. Notably, megadocs can exceed the length of the model’s context. When training, masking is always disabled across document and megadocument boundaries.
Figure 4: Different ways to scale document length. We compare synthetic data algorithms (visualized in Figure 3 ) given 8 generations per pre-training doc. We run 3 seeds of each algorithm. Both variants of stitched rephrasing outperform simple rephrasing with real last slightly outperforming real first. Latent thoughts further improves both i.i.d and arxiv loss.
Figure 5: Scaling generation count for stitching and latent thoughts. We find that stitched rephrasing (blue points) and latent thoughts (gray points) improve i.i.d. loss, long-context loss, and downstream benchmark accuracy over simple rephrasing (orange points). The improvements scale in the number of generations per document and show less signs of plateauing.
Figure 6: Megadocs benefit from more steps on real and synthetic data. We attribute the favorable scaling of megadocs to different optimal hyperparameters enabling longer training. For different synthetic data pools, we plot the optimal loss at each real data epoch count (left). We find that more synthetic tokens (higher ) allows for training for more real epochs, and megadocs push this effect further. Controlling for longer training by using the best simple rephrasing hyperparameters (right) shows that megadocs still improve but the loss win doesn’t scale with . The scaling is less clean in the right plot since the hyperparameters are tuned for simple rephrasing.
Figure 7: Rephrasing composes with ensembling. We measure the loss of ensembling synthetically pre-trained models as we increase the member count. For a given synthetic data recipe, we use the asymptote (horizontal dotted line) of a power law to estimate the loss as the member count approaches infinity. Pre-training with self-distillation does not compose with ensembling as ensembling self-distilled models does not improve the asymptote. Meanwhile, simple rephrasing, stitched rephrasing, and latent thoughts all compose with ensembling and improve the asymptote as member count approaches infinity. Note that unlike simple rephrasing, the hyperparameters for stitched rephrasing and latent thoughts have not been tuned to improve the asymptote.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Student scaling . We pre-train 1.5B student models using the same set of synthetic data methods. At each scale, we measure the loss win of each method over the no synthetic data baseline (purple). We observe a larger loss win at the 1.5B scale than the 300M scale for all methods.
Figure 9: Student and generator scaling laws for continued pre-training. For a small set of 1.3M tokens, we measure QA accuracy when training utilizing Rephrase the Web [ Maini et al., 2024 ] . We find that self-improvement is possible (3B student and generator, 8B student and generator). Additionally, rephrasing only emerges with sufficient scale, stronger students benefit more from synthetic data, and synthetic data improvements eventually stop scaling in generator capability. Figure taken from preliminary experiments in [ Kotha et al., 2025 ] .
Figure 10: Convexity in hyperparameters. For a fixed pool of synthetic data (16 rephrases per document), we show how the optimal loss changes as we vary two hyperparameters: real data epochs and mixing fraction. For our synthetic data scaling experiments, we pick the "locally optimal" solution so that deviating from any of the hyperparameters doesn’t improve loss.
Figure 11: Optimal hyperparameters. We visualize the locally optimal hyperparameters for each of the synthetic data methods we scale in our paper. We observe that megadoc-based methods can epoch the real data more and sustain a higher mixing ratio without overfitting, particularly at the highest synthetic generation counts.
Figure 12: Cross-document attention ablation. In the data-eficient regime with only real data, it is better to mask attention between separate documents in the same context window (purple bars). When also training on rephrased data, the answer changes (orange bars). The benefit of stitching additionally justifies disabling cross-document attention masking.
Figure 13: Mixing ablation. We briefly ablate the importance of mixing in real data. Given a small synthetic data pool (2 rephrases per document), we find that also training on the real data vs. not training on it substantially improves loss from 3.521 to 3.456. Consequently, we always mix real data and search for a locally optimal mixing fraction.
Figure 14: Including real document in synthetic stream. We show how stitching is a benefit over simple rephrasing both when there is no real data in the synthetic stream (striped blue vs striped orange) and when there is real data (solid blue vs solid orange). Since including the real document in the synthetic stream is helpful for both simple rephrasing (solid orange vs striped orange) and stitched rephrasing (solid blue vs striped blue), we opt to include a real document in the synthetic stream for all experiments in the main body.
Figure 15: Additional long-context evaluations. Megadocs achieve lower loss and show less signs of plateauing than simple rephrasing on additional long-context evaluations on Github Python code and Wikipedia articles.
Figure 16: Scaling trends on shorter vs longer documents. We split the validation set in half (based on document length) and evaluate the scaling of our methods on both splits. The short and long splits have max document lengths of 621 and 36319 tokens respectively. The scaling of megadocs vs. simple rephrasing exhibits the same trend on the short and long splits, suggesting that megadocs help more generally with modeling many types of data.
| Parameter | 300M | 1.5B |
| Context Length | 4096 | 4096 |
| Hidden Dimension | 512 | 1536 |
| Intermediate Dimension | 1792 | 5376 |
| Attention Heads | 8 | 16 |
| KV Heads | 8 | 8 |
| Layers | 6 | 36 |
Table 1: Model architecture configurations for different model sizes. We use the 1.5B model for student scaling experiments.
More in this topic
Language Model Pretraining
Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.
See all 440 papers