Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining

Thales Sales Almeida; Rodrigo Nogueira; Hélio Pedrini

cs.CLMar 25, 2026

Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining

Authors: Thales Sales Almeida, Rodrigo Nogueira, Hélio Pedrini

Organizations: Institute of Computing (IC) University of Campinas (UNICAMP) · Maritaca AI · School of Electrical and Computer Engineering (FEEC) University of Campinas (UNICAMP)

Abstract

Synthetic data generation through document rewriting has emerged as a promising technique for improving language model pretraining, yet most studies focus on English and do not systematically control for the quality of the source data being rewritten. We present a controlled study of how synthetic rewriting interacts with source data quality in the context of Portuguese continued pretraining. Starting from ClassiCC-PT, a Portuguese corpus annotated with STEM and Educational quality scores, we construct two 10B-token subsets at different quality levels and rewrite each into four styles using a 7B instruction-tuned model, producing approximately 40B tokens of synthetic data per condition. We train two English-centric base models (1.1B and 7B parameters) on each condition and evaluate on PoETa V2, a comprehensive 44-task Portuguese benchmark. At the 7B scale, rewriting high-quality data yields a +3.4 NPM gain over the same data unmodified, while rewriting low-quality data provides only +0.5 NPM. At the 1.1B scale, this interaction is weaker, with unmodified low-quality data performing comparably to rewritten high-quality data. Our results demonstrate that synthetic rewriting acts primarily as a quality multiplier rather than a substitute for data curation, and that this effect is scale-dependent.

Figures & tables

Figure 1: Average NPM in PoETa V2 for the 7B model across four experimental conditions as a function of training tokens.

Panel of Figure 2 — Figure 2: 7B model: average NPM across task subcategories as a function of training tokens. The nine subcategories shown are those with the largest number of tasks in PoETa V2.

Figure 3: Average NPM in PoETa V2 for the 1.1B model across four experimental conditions as a function of training tokens.

Panel of Figure 4 — Figure 4: 1.1B model: average NPM across task subcategories as a function of training tokens. The nine subcategories shown are those with the largest number of tasks in PoETa V2.

Condition	7B	1.1B
Edu + Rewrites	41.0 (30B)	15.1 (25B)
Edu	38.5 (20B)	13.7 (30B)
Non-edu + Rewrites	35.8 (20B)	13.8 (30B)
Non-edu	35.2 (20B)	15.1 (20B)

Table 1: Peak average NPM on PoETa V2 for each experimental condition at both model scales. Values in parentheses indicate the number of training tokens (in billions) at which the peak was achieved.

Synthetic Data Generation for Language Models

Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.

See all 293 papers

Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints
Oct 22, 2026Nicholas Roberts, Ahmed Ahmed, Suhas Kotha +4Synthetic Data Pretraining Language Model Scaling Laws
Princeton University · Stanford University · Jane Street
How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
Apr 15, 2026Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9Synthetic Data Pretraining Synthetic Data Generation for Language Models
Hugging Face · University of Sheffield, United Kingdom · National Institute of Informatics, Japan
Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
May 11, 2026Xu Guo, Runyu Peng, Jian Tong +4Language Model Pretraining Synthetic Data Pretraining
Shanghai AI Laboratory · Fudan University · Shanghai Innovation Institute

Oct 22, 2026J/K move · Enter open · S save

Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

Nicholas Roberts, Ahmed Ahmed, Suhas Kotha, Percy Liang+3

Princeton University · Stanford University · Jane Street

Synthetic Data Pretraining Language Model Scaling Laws Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Synthetic rewrites, where a language model rewrites a natural document e.g. into a target style, are now a standard ingredient of pretraining corpora. However, critical aspects of their use remain poorly understood, such as how much compute should be spent rewriting training data and whether a model trained on rewrites can outperform the rewriter. We study these two questions by proposing a rewriting scaling law that predicts loss from model size, training tokens, the fraction of training tokens that are rewrites, and the number of epochs over the natural data, charging the compute spent generating rewrites. We fit this scaling law in two settings. In the compute-limited setting, natural data is plentiful and rewrites come from a third-party rewriter trained with more compute than our largest run. We find that the estimated optimal rewrite fraction increases with compute, from under

10\%

3\times10^{18}

FLOPs to

32\%

10^{22}

FLOPs. Our scaling forecast extrapolates

33\times

in FLOPs to train a 22B-parameter model, whose final loss is within

1.1\%

of our prediction. This model achieves a better BPB than its rewriter on several web domains at one-tenth the rewriter's training compute. In the data-limited setting, we assume we only have a fixed seed set of natural data and that our rewriter is a model of the same size as the target, pretrained on that seed set. Again, we find the optimal rewrite fraction increases with the compute budget. We also demonstrate that a model trained on rewrites of its own pretraining data, generated by a rewriter of the same size, beats the best run on natural data alone at every model size up to 3B. Our results show that synthetic rewriting can be studied through the scaling law lens: that optimal methodological choices are predictable and scaling does not necessarily break down beyond the rewriter's capabilities.

Abstract

Synthetic Data Generation for Language Models

Explore similar work

Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data