Synthetic rewrites, where a language model rewrites a natural document e.g. into a target style, are now a standard ingredient of pretraining corpora. However, critical aspects of their use remain poorly understood, such as how much compute should be spent rewriting training data and whether a model trained on rewrites can outperform the rewriter. We study these two questions by proposing a rewriting scaling law that predicts loss from model size, training tokens, the fraction of training tokens that are rewrites, and the number of epochs over the natural data, charging the compute spent generating rewrites. We fit this scaling law in two settings. In the compute-limited setting, natural data is plentiful and rewrites come from a third-party rewriter trained with more compute than our largest run. We find that the estimated optimal rewrite fraction increases with compute, from under 10% at 3×1018 FLOPs to 32% at 1022 FLOPs. Our scaling forecast extrapolates 33× in FLOPs to train a 22B-parameter model, whose final loss is within 1.1% of our prediction. This model achieves a better BPB than its rewriter on several web domains at one-tenth the rewriter's training compute. In the data-limited setting, we assume we only have a fixed seed set of natural data and that our rewriter is a model of the same size as the target, pretrained on that seed set. Again, we find the optimal rewrite fraction increases with the compute budget. We also demonstrate that a model trained on rewrites of its own pretraining data, generated by a rewriter of the same size, beats the best run on natural data alone at every model size up to 3B. Our results show that synthetic rewriting can be studied through the scaling law lens: that optimal methodological choices are predictable and scaling does not necessarily break down beyond the rewriter's capabilities.
Figures & tables
Figure 1: (Left; compute-limited) Effective rewriting recipes are predictable via scaling laws: best possible Paloma BPB (lower is better; allowing third-party rewrites) vs FLOPs (accounting for rewrite compute costs) as compared to using natural data only. Unfilled blue points are imputed via linear interpolations of the loss of the closest rewrite fractions that we evaluate, whereas solid points represent observed training runs. (Right; data-limited) The trained model surpasses its rewriter: final Paloma BPB vs FLOP for a 3 B target model trained using only natural data or synthetic rewrites coming from a fixed 5 B-token natural data corpus. Using natural data only, which necessitates epoching, results in worse loss after 3 epochs. The best proportion of rewrites from a bootstrapped rewriter improves over epoching on the natural seed set alone at every budget.
Figure 2: The estimated compute-optimal rewrite fraction increases with compute on every evaluation. The curves show R∗ across evaluations and FLOP budgets. Solid lines indicate that the compute budget is within the evaluated ladder, while dashed lines are extrapolations. Points mark the fit optimum at each ladder budget; filled points land on a tested rewrite fraction and unfilled points fall between two. Stars mark predicted break-even with SmolLM2-Instruct. The vertical line marks the training FLOPs of SmolLM2-Instruct.
Figure 3: The surface fit to budgets through 3×1020 FLOPs forecasts the 1022 runs to within 1.4% . Each panel is one rewrite fraction R . Dots are runs and curves are the fit IsoFLOP slices, colored by training budget. The R=0 ladders reach 3×1020 FLOPs and the R>0 ladders 9×1019 FLOPs. Open stars mark the fit optimum at each budget, filled blue stars mark the compute-optimal runs at 1.8×1020 and 3×1020 FLOPs, and the dashed line extends the compute-optimal forecast to 1022 FLOPs. The orange star is the 1022 run, held out of the fit, labeled with its measured loss relative to the forecast. The R=0.32 run’s forecast error is −1.1% whether the surface is evaluated at R=0.3 or at R=0.32 . Figure 14 shows all eleven rewrite fractions.
Paloma source
R=0
R=0.32
Rewriter
M2D2 Wikipedia
0.726
0.716
0.758
Penn Treebank
0.851
0.843
0.882
WikiText-103
0.682
0.673
0.711
C4 English
0.827
0.828
0.844
Falcon RefinedWeb
0.894
0.887
0.897
Gab
1.719
1.579
1.587
Table 1: Our run at 1022 training FLOPs reaches a lower Paloma BPB than its rewriter on 10 of 16 sources. Per-source Paloma BPB for the natural-only ( R=0 ) and predicted-optimum ( R=0.32 ) runs at 1022 training FLOPs and for SmolLM2-1.7B-Instruct, sorted by the predicted-optimum run’s margin over the rewriter; the rule separates the sources where it is at or below the rewriter. Bold marks the lowest BPB per row. Appendix N.3 reports all four runs at 1022 training FLOPs and the SmolLM2 base model.
Figure 4: The R∗ selected by the scaling law costs at most 1.14% in loss at held-out budgets. The law (Eq. 1 ) is fit per target size to budgets through W=18 ; W=21 and W=24 are held out (shaded, open markers). Top: measured final-checkpoint Paloma macro BPB at R∗ , as a percentage above the loss at the best observed R on the same six-value grid. Bottom: R∗ (blue), the best observed R (orange), and the law’s optimum over continuous R∈[0,1] (dashed).
Figure 5: Final-checkpoint Paloma macro BPB against the epoch budget W for each target size. Solid black: natural-only epoching ( R=0 ) for the full budget of W epochs over the seed set. Dotted black: the best natural-only run at any budget, defined by the loss reached by stopping once further epochs raise it. Blue: best mixture at each W , the run with the lowest final loss at that budget over all rewrite fractions, so it coincides with the natural-only point wherever natural-only is best.
Figure 6: Rewrites beat natural-only epoching on Math and Code at every budget, but on QA only after W=6 . Final-checkpoint BPB for the 3 B target against the epoch budget W on QA, Math, and Code, with the same three series as Figure 5 . Appendix H.1 shows every target size.
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: 1 B and 3 B targets at matched total compute. Each panel pairs the nearest budgets on the grid, with FLOP ratios 0.97 , 0.97 , and 1.09 .
Figure 8: Rewriter capacity. Left: 1 B target loss at W=12 with rewrites from the 550 M, 1 B, and 3 B rewriters. Right: share of the 1 B– 3 B target gap closed by training the 1 B target on 3 B rewrites, at each budget’s best mixture.
Size
SFT learning rate
Epochs
Mean win rate (%)
550M
10−4
15 †
28.3
2×10−4
15
27.7
2×10−4
10
26.6
10−4
10
25.0
2×10−5
15
24.8
2×10−5
10
23.8
Appendix
Table 2: Win rates against the SmolLM2-1.7B FinePhrase rewrite, averaged over both judges. Each row is a rewriter checkpoint. Scores are the Elo-derived win rate against the SmolLM2-1.7B rewrite of the same document and prompt, averaged over GLM-5.2 and MiniMax-M2.7. † marks the checkpoint that generates the seed-set rewrites. Rows are ordered by score within each size. The 1 B results cover seven shortlisted configurations.
Figure 9: Final-checkpoint Paloma macro BPB for the 1 B target trained on rewrites from the SFT and SFT+DPO rewriters at W∈{3,6,9} . Both conditions share the natural-only R=0 runs.
Figure 10: Mean Paloma BPB change per source from replacing the SFT rewriter with the SFT+DPO rewriter across the 16 mixed runs ( 0<R<1 , W≤12 ). Orange bars denote lower BPB under SFT+DPO and blue bars lower BPB under the SFT rewriter, matching the colors of Figure 9 .
Target size
Epoch budget W
3
6
9
12
15
18
21
24
550 M
0.495
0.991
1.486
1.981
2.476
2.972
3.467
3.962
837 M
0.753
1.507
2.260
3.013
3.766
4.520
5.273
6.026
1 B
0.898
1.796
2.695
3.593
4.491
5.389
6.287
7.185
3 B
2.606
5.211
7.817
10.422
13.028
15.633
18.239
20.844
Appendix
Table 3: Total compute, accounting for rewrites, for the complete data-limited grid, in units of 1020 FLOPs. Each compute value has six rewrite-fraction runs.
Figure 11: Final-checkpoint Paloma macro BPB for all 192 data-limited runs. Each panel shows one target size at all eight epoch budgets W and six rewrite fractions R . Open pentagons mark the best observed mixture at each size and budget.
Figure 12: Final-checkpoint BPB against the epoch budget W for every target size (rows) on the QA, Math, and Code suites (columns). Series as in Figure 5 .
Figure 13: Best-checkpoint BPB for the 3 B target by epoch budget W and rewrite fraction R . Hatching marks runs re-evaluated at an earlier checkpoint whose Paloma validation loss is at least 0.01 nats below the final checkpoint. Magenta outlines mark row minima, including ties after rounding to two decimals. Paloma alone selects each checkpoint.
Variant
Prose
List
Table
Header
Code
Natural
84.41
14.32
1.26
0.01
0.00
FAQ
64.69
33.01
0.50
1.74
0.06
Math
69.02
30.06
0.19
0.28
0.45
Table
29.21
10.07
58.99
1.68
0.05
Tutorial
49.16
45.34
0.38
4.31
0.80
Appendix
Table 4: Line composition of 6,000 documents per FinePhrase variant, in percent of classified lines.
Figure 14: The shared L(N,D,R) surface as per- R IsoFLOP slices at all eleven rewrite fractions, in Paloma macro BPB. Dots mark runs, open stars mark fitted frontier optima, and filled stars mark the 1021 and 1022 runs. Labels report measured minus fitted loss as a percentage of fitted loss.
Figure 15: Per- R IsoFLOP optima against the shared frontier. Left: compute-optimal token count DR∗(C) from a separate Approach-2 parabola for each rewrite fraction and budget, colored by R , with the shared frontier D∗(C)∝C0.46 . Right: ratio of each per- R optimum to the shared frontier, by budget.
Figure 16: Whether fitting the frontier per R changes the selected model size. Left: for each sweep with a full rewrite ladder, the grid size nearest the per- R vertex relative to the one nearest the shared N∗(C) . Cells report the BPB cost of the shared choice on that sweep’s own IsoFLOP parabola. Right: the same cost by rewrite fraction, on the parabola (filled) and between the two measured grid runs (open).
Figure 17: Rewrite gains collapse onto one curve across 3,333× compute. IsoFLOP-optimal Paloma BPB after subtracting each budget’s R=0 value. The fitted g~(R) has RMSE 0.0044 BPB over 82 observations from nine budgets.
Figure 18: Natural tokens required to match the estimated floor-optimal mixture, as a function of training compute. The final fit gives 5.01× at 1022 FLOPs and diverges at Cdiv=3.61×1026 FLOPs. Dashed curves exceed the fit range. The 671 -run fit gives 12.17× and 1.75×1024 FLOPs.
Target
RMSE
E0
λn
λs
ρn
ρs
B
β
P
s
550 M
0.0147
1.259
0.0220
0.0045
0.0002
0.0001
1.55×1018
1.939
0.0079
6.990
837 M
0.0085
1.170
0.0458
0.0045
0.0105
0.0001
4.17×108
0.963
0.0096
1.539
1 B
0.0115
1.209
0.0194
0.0000
0.0002
0.0249
1.86×1015
1.636
0.0113
5.075
3 B
0.0150
1.040
0.0823
0.0041
0.0872
0.0001
4.44×105
0.647
0.0174
2.082
Appendix
Table 5: Data-limited fits of Eq. ( 14 ) to final-checkpoint Paloma BPB, one fit per target–rewriter pair, 48 runs per row. RMSE, E0 , λn , λs , and P are in BPB; B is in BPB ⋅ tokens β , with D in tokens as in Eq. ( 14 ); ρn , ρs , β , and s are dimensionless. ρs sits at its 10−4 lower bound except for the 1 B pair.
Figure 19: Estimated optimal rewrite fraction R∗ and best observed R as a function of epoch budget W for each target–rewriter pair. Orange squares mark the best observed R on the tested grid. Dashed blue curves show R∗ from each pair’s scaling-law fit to W≤18 , and the shaded budgets W=21 and W=24 are held out. R∗ increases with budget within every pair. Across pairs, the rewriter changes with the target size.
Target
Held-out
Fit
Held-out
Mean
Per- R
Error at
W
RMSE
RMSE
error
range
best obs. R
550 M
21
0.0114
0.0267
−0.73%
[−3.98,+1.98]%
−0.28%
550 M
24
0.0114
0.0294
−1.39%
[−3.90,+0.85]%
−0.26%
837 M
21
0.0083
0.0060
+0.13%
[−0.29,+0.86]%
−0.27%
837 M
24
0.0083
0.0140
−0.42%
[−1.98,+0.79]%
−0.68%
1 B
21
0.0102
0.0225
−0.50%
[−3.01,+2.05]%
+0.05%
Appendix
Table 6: Forward data-limited forecasts. Each row fits the 36 runs at W≤18 and forecasts the six rewrite fractions at the held-out budget W . RMSE is in BPB.
Parameter
Value
Parameter
Value
E0
0.701293
A
18.3516
λn
0.024066
α
0.204717
ρn
0.024598
B
44.5853
λs
0.011335
β
0.236040
ρs
10−4
H
3.00277
η
0.072262
Appendix
Table 7: Final FineWeb-Edu surface parameters. N and D are absolute parameter and token counts, and C=6ND is in FLOPs. The fitted ρs reaches its lower bound. The fitted floor at R=0 is E(0)=E0−λslogρs=0.805691 (Eq. 10 ).
Test training FLOPs
Largest fit training FLOPs
Extrap.
Mean error
Per- R range
3×1019
1.8×1019
1.7×
−0.50%
[−0.81,−0.16]%
9×1019
3×1019
3.0×
−0.36%
[−0.77,+0.02]%
1.8×1020
9×1019
2.0×
−0.21%
[−0.56,+0.80]%
3×1020
1.8×1020
1.7×
−0.70%
[−1.18,−0.13]%
1021
3×1020
3.3×
−1.32%
[−1.87,−0.73]%
1022
3×1020
33.3×
−1.04%
[−1.40,−0.85]%
Appendix
Table 8: Forward compute-limited forecasts by training budget. The 1022 row fits through 3×1020 FLOPs and predicts the four rewrite fractions trained at the hero-run allocation ( R∈{0,0.321,0.5,1} ).
Penalty
Params
Target
RMSE
R2
Mean error
Error at best obs. R
Rsel
R^24∗
W=21
W=24
W=21
W=24
W=21
W=24
P(1−R)s((1−R)D/T−1)+
9
550 M
0.0147
0.959
−0.73%
−1.39%
−0.28%
−0.26%
0.47
0.71
0.41
(paper)
837 M
0.0085
0.985
+0.13%
−0.42%
−0.27%
−0.68%
0.71
0.71
0.59
1 B
0.0115
0.974
−0.50%
−0.81%
+0.05%
+0.03%
0.71
0.71
0.49
3 B
0.0150
0.968
+0.27%
−0.04%
−0.50%
−0.74%
0.47
0.47
0.57
P(1−R)s((1−R)D/T−1)+δ
10
550 M
0.0145
0.960
−0.73%
−1.39%
−0.28%
−0.26%
0.47
0.71
0.39
Appendix
Table 9: Repetition-penalty form ablation for Eq. ( 14 ), one fit per target–rewriter pair, 48 runs per fit. RMSE is in BPB and R2=1−SSres/SStot over the 48 runs. The held-out columns refit the 36 runs at W≤18 and forecast the six rewrite fractions at W=21 and W=24 : mean error is the mean of (Lmeasured−Lpredicted)/Lpredicted over the six, error at best obs. R is the same quantity at the best observed R ( R=0.71 at every pair and both budgets), and Rsel is the tested fraction with the lowest forecast loss, the selection rule of Figure 4 . R24∗ is the minimizer of the full 48 -run fit over R∈[0,1] at W=24 . The parameter count includes the five floor parameters, B , and β . Fitted parameters are in repetition_penalty/de_penalty_ablation.csv .
Figure 20: Fitted Paloma BPB against rewrite fraction under each repetition penalty (full 48 -run fits), at W=24 (top) and W=3 (bottom), with the measured runs. The four penalties with an additive term are nearly coincident at W=24 between the tested fractions; the two forms without one fall 0.05 – 0.16 BPB below the R=0 run at W=24 and 0.07 – 0.15 BPB above it at W=3 . The no-dilution form places its W=3 minimum below the measured points at small R .
Figure 21: Estimated optimal rewrite fraction R∗(W) under each repetition penalty (full 48 -run fits), with the best observed R at each budget. All forms with the dilution factor (1−R)s give R∗(24) between 0.39 and 0.59 , below the best observed R=0.71 at W≥18 ; the no-dilution form rises faster ( 0.62 – 0.74 ) but at the cost of fit quality and a degenerate floor (Table 9 ). The forms without an additive penalty are flat in W .
Figure 22: Observed against predicted Paloma macro BPB for every repetition penalty (rows, in the order of Table 9 ) and target–rewriter pair (columns), all 48 runs of the full fit, coloured by epoch budget W , with the identity line. Hollow markers are the W=21 and W=24 forecasts of the W≤18 refit. Each panel reports R2 and RMSE of the full fit; axis limits are shared within a column. The three dilution forms and the no-dilution form lie along the identity line; the two forms without an additive penalty flatten to horizontal bands at large W , where the R=0 runs cannot be separated from their neighbours by a term in D alone.
Figure 23: The estimated floor-optimal rewrite fraction R∞∗ against two properties of each Paloma source. Left: R∞∗ has no association with the rewriter’s relative strength on the source (Spearman ρ=0.03 , p=0.91 ). Right: R∞∗ increases with the natural-data model’s BPB on the source (Spearman ρ=0.47 , p=0.07 ).
Figure 24: Estimated compute-optimal rewrite fraction R∗(Ctotal) for each Paloma source. Solid curves cover the measured ladders through ≈2×1020 total FLOPs, and dashed curves extrapolate. The thick curve is Paloma macro BPB. The vertical line marks SmolLM2’s training compute.
Figure 25: Estimated compute-optimal rewrite fraction R∗(Ctotal) for each task. Legends report R∞∗ , and thick curves show the aggregate trajectories of Figure 2 . MMLU, Minerva, and multilingual MBPP average 57 , 7 , and 17 subtasks. TheoremQA switches from R=0 to R=1 because its fitted floor is nearly linear in R ; its fit excludes the two budgets with the fewest model widths.
Figure 26: Knowledge QA is capacity-hungry and Code is data-hungry. Left three panels: natural-only ( R=0 ) IsoFLOP slices from each evaluation’s own surface fit, loss against parameter count, colored by training budget, with dots for runs and stars for the fitted optimum at each budget. The dashed line is the Paloma frontier N∗(C) drawn on that evaluation’s surface. Right: compute-optimal parameter count of each evaluation’s fit against training compute. The QA optimum has more parameters than the Code optimum at every budget, 1.5× at 1022 FLOPs.
Evaluation
C=1019
C=1022
N∗
D∗
D∗/N∗
N∗
D∗
D∗/N∗
Paloma
0.55 B
3.0 B
5.5
23.5 B
70.9 B
3.0
QA
0.66 B
2.5 B
3.8
14.6 B
114.4 B
7.8
Math
0.48 B
3.5 B
7.3
14.2 B
117.2 B
8.2
Code
0.38 B
4.4 B
11.5
9.7 B
171.7 B
17.7
Appendix
Table 10: Compute-optimal allocation from each evaluation’s surface fit at two training budgets. D∗/N∗ is tokens per parameter.
Optimized for
Model
R
N
D
BPB ↓
Acc. ↑
Paloma
QA
Math
Code
QA
–
Natural-only
0
21.9B
76B
1.061
0.731
1.328
1.042
0.619
Paloma
Hero
0.32
21.9B
76B
0.983
0.700
0.842
0.587
0.622
–
Grid
0.5
21.9B
76B
0.988
0.698
0.820
0.568
0.620
–
Grid
1.0
21.9B
76B
1.052
0.719
0.800
0.529
0.594
QA
Grid
0.74
13.8B
120B
0.988
0.696
0.801
0.520
0.620
Appendix
Table 11: The 1022 -FLOP hero run nearly matches SmolLM2-Instruct on Paloma. The Paloma column uses byte-pooled BPB at a 4096 -token context for both model families. The QA, Math, and Code columns retain their matched offline evaluations. The first four rows share the Paloma-optimal (N,D) , hence training compute C≈1022 FLOPs, and differ only in R ; their total compute Ctotal=6ND+4NrRD varies with R . SmolLM2 is trained on 11 T tokens at ≈1.1×1023 FLOPs, 11× the total budget of the hero run.
Domain
1022 training FLOPs, 22 B
SmolLM2-1.7B
R=0
R=0.321
R=0.5
R=1.0
Base
Instruct
4chan
1.162
1.023
1.034
1.143
0.926
0.952
C4 100 Domains
0.790
0.793
0.797
0.844
0.772
0.797
C4 English
0.827
0.828
0.833
0.874
0.817
0.844
Dolma v1.5
0.876
0.804
0.808
0.863
0.763
0.791
Dolma 100 PLs
1.106
0.694
0.680
0.698
0.486
0.511
Appendix
Table 12: The R=0.321 hero run minimizes Paloma macro BPB among the runs at 1022 training FLOPs. The four 22 B runs share (N,D)=(21.9B,76B) ; the best value among them per row is bold. The SmolLM2 columns are evaluated on the same Paloma validation split at a 4096 -token context. The R=0.321 run reaches a lower BPB than SmolLM2-1.7B-Instruct on 10 of the 16 sources.
Task
Natural-only
Grid
QA-optimal
Code-optimal
R=0 , 21.9B
R=0.5 , 21.9B
R=0.74 , 13.8B
R=1.0 , 9.7B
QA (BPB ↓ )
ARC-Easy
0.540
0.490
0.490
0.502
ARC-Challenge
0.736
0.672
0.676
0.686
HellaSwag
0.704
0.704
0.703
0.718
MMLU (57 subjects)
0.850
0.809
0.801
0.818
Appendix
Table 13: Task-level scores behind the QA, Math, and Code aggregates of Table 11 . Columns are the runs at 1022 training FLOPs with downstream evaluations; best value per row in bold. MMLU, Minerva MATH, and multilingual MBPP are macro averages over their subtasks (Table 14 ). SmolLM2 task-level scores are not archived.
Subtask
Natural-only
Grid
QA-optimal
Code-optimal
R=0 , 21.9B
R=0.5 , 21.9B
R=0.74 , 13.8B
R=1.0 , 9.7B
Minerva MATH
Algebra
1.215
0.557
0.524
0.515
Counting and probability
1.078
0.520
0.496
0.492
Geometry
1.132
0.551
0.517
0.509
Intermediate algebra
1.324
0.589
0.544
0.530
Appendix
Table 14: Minerva MATH subjects and multilingual MBPP languages (BPB, lower is better) for the runs of Table 13 ; best value per row in bold.
Metric
Reference
Ctotal
C
N∗
D∗
R^∗
Gen. share
(FLOPs)
(FLOPs)
(B)
(B)
Paloma
0.9716
2.528×1022
2.505×1022
38.960
107.145
0.325
0.94%
QA BPB
0.7090
5.316×1021
4.946×1021
11.254
73.245
0.739
6.96%
Math BPB
0.7074
5.962×1022
5.769×1022
34.113
281.855
1.000
3.23%
Code BPB
0.4256
4.274×1023
4.189×1023
55.993
1246.794
1.000
2.00%
Appendix
Table 15: Fitted break-even allocations against SmolLM2-Instruct. Paloma crosses at 2.53×1022 FLOPs with R∗=0.325 , using matched within-source aggregation and context length. QA crosses at 5.32×1021 FLOPs with R∗=0.739 . The Math and Code crossings lie at the R=1 boundary and beyond the measured ladders.
Model
Parameters
R2
RMSE
BIC
M1: shared center
25
0.98304
0.00858
−5383
M2: center by R
45
0.98579
0.00785
−5359
M3: cell quadratics
165
0.99022
0.00651
−4813
Appendix
Table 16: Surface-model ladder on the same 583 runs. BIC favors the shared-center model M1 although the richer models reduce residual error.
Figure 27: Left: Paloma macro BPB against R on DCLM at two training budgets: 550 M parameters and 2.9 B tokens at 9×1018 FLOPs, and 669 M parameters and 4.7 B tokens at 1.8×1019 FLOPs. Right: final Paloma macro BPB of the observed compute-optimal natural-only ( R=0 ) DCLM and FineWeb-Edu run at each training budget shown. Each point selects the lowest-Paloma run within its corpus and budget.
Figure 28: The seven width-controlled DCLM IsoFLOP ladders. Points are completed runs, curves are per-budget quadratics in log model size, and stars mark their minima. Each panel reports the R2 of its quadratic. Every budget has an interior minimum over model size. The single-run 1022 anchor is omitted.
Figure 29: Final evaluations of the observed compute-optimal natural-only DCLM and FineWeb-Edu runs. At each training budget and corpus, we select the R=0 run with the lowest final Paloma macro BPB and show that run’s Paloma, Math, Code, and QA scores. Lower is better for the three BPB panels and higher for QA accuracy.
Synthetic data generation through document rewriting has emerged as a promising technique for improving language model pretraining, yet most studies focus on English and do not systematically control for the quality of the source data being rewritten. We present a controlled study of how synthetic rewriting interacts with source data quality in the context of Portuguese continued pretraining. Starting from ClassiCC-PT, a Portuguese corpus annotated with STEM and Educational quality scores, we construct two 10B-token subsets at different quality levels and rewrite each into four styles using a 7B instruction-tuned model, producing approximately 40B tokens of synthetic data per condition. We train two English-centric base models (1.1B and 7B parameters) on each condition and evaluate on PoETa V2, a comprehensive 44-task Portuguese benchmark. At the 7B scale, rewriting high-quality data yields a +3.4 NPM gain over the same data unmodified, while rewriting low-quality data provides only +0.5 NPM. At the 1.1B scale, this interaction is weaker, with unmodified low-quality data performing comparably to rewritten high-quality data. Our results demonstrate that synthetic rewriting acts primarily as a quality multiplier rather than a substitute for data curation, and that this effect is scale-dependent.
Institute of Computing (IC) University of Campinas (UNICAMP) · Maritaca AI · School of Electrical and Computer Engineering (FEEC) University of Campinas (UNICAMP)
Synthetic data augmentation has emerged as a promising solution when pre-training is constrained by data rather than compute. We study how to design synthetic data algorithms that achieve better loss scaling: not only lowering loss at finite compute but especially as compute approaches infinity. We first show that pre-training on web data mixed with synthetically generated rephrases improves i.i.d. validation loss on the web data, despite the synthetic data coming from an entirely different distribution. With optimal mixing and epoching, loss and benchmark accuracy improve without overfitting as the number of synthetic generations grows, plateauing near 1.48× data efficiency at 32 rephrases per document. We find even better loss scaling under a new perspective: synthetic generations from the same document can form a single substantially longer megadocument instead of many short documents. We show two ways to construct megadocs: stitching synthetic rephrases from the same web document or stretching a document by inserting rationales. Both methods improve i.i.d. loss, downstream benchmarks, and especially long-context loss relative to simple rephrasing, increasing data efficiency from 1.48× to 1.80× at 32 generations per document. Importantly, the improvement of megadocs over simple rephrasing widens as more synthetic data is generated. Our results show how to design synthetic data algorithms that benefit more from increasing compute when data-constrained.
Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop FinePhrase, a 486-billion-token open dataset of rephrased web text. We show that FinePhrase outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community.
Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9
Hugging Face · University of Sheffield, United Kingdom · National Institute of Informatics, Japan