Rethinking Pre-pretraining

Alexander Stern; Tim Rocktäschel; Acyr Locatelli; Laura Ruis

cs.CLOct 29, 2026

Rethinking Pre-pretraining

Authors: Alexander Stern, Tim Rocktäschel, Acyr Locatelli, Laura Ruis

Organizations: MIT · AI Centre, UCL

Abstract

We report that allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in our compute-matched experiments. This holds across four PPT curricula (k-Dyck, neural cellular automata (NCA), declarative reference corpora, and code) and three training regimes: data-constrained, data-unconstrained, and a high token-to-parameter-ratio regime (881M parameters trained on 50B tokens). When natural language is scarce, repeated epochs over a 100M-token corpus beat every synthetic curriculum on natural-language validation loss, with no consistent downstream accuracy advantage from PPT. We find that PPT gains reported in prior work are sensitive to the weight initialization scheme. In an NCA reproduction, we recover a 5.4% perplexity improvement from PPT under the original initialization scheme. An alternative scheme improves both methods, but the no-PPT baseline more, closing the gap between them. However, when PPT receives additional compute rather than a share of the natural-language budget, gains persist at small natural-language budgets even with improved initialization, but disappear by 1B tokens. Finally, models forget what they learn during PPT in subsequent natural-language training; interleaving the same data throughout training prevents forgetting, but does not significantly improve natural-language performance. Our results suggest limited benefits of synthetic pre-pretraining as a distinct training stage and motivate rethinking how such data are used in language model training.

Figures & tables

Figure 1: Across all regimes, allocating compute to PPT never beats spending it on natural language. (1) Data-constrained: FineWeb validation bits per byte (BPB) against the fraction of a fixed

1{\times}10^{18}

-FLOP budget spent on PPT, with repetition over 100M natural-language tokens. Optimal model sizes range from 561M to 708M parameters. (2) Over-Chinchilla: an 881M-parameter model trained on 50B tokens,

{\sim}12\times

the compute-optimal token count, scored on FineWeb BPB and DCLM CORE. The PPT methods spend 0.5B tokens on PPT and 49.5B on FineWeb. (3) Data-unconstrained: the optimal model size found at each budget varies with the PPT method, from 179M to 186M parameters at

1\times 10^{18}

FLOPs, 252M to 336M at

3\times 10^{18}

, and 438M to 561M at

9\times 10^{18}

Figure 2: Tokenizing synthetic pre-pretraining data. For NCA, each grid of a rollout is split into $2\times 2$ patches that each become one token, and grids are concatenated in time order with start and end tokens. For $k$ -Dyck, the opening and closing bracket types are each their own token. For DeclRef, keywords, scope brackets, names and values are unique tokens. In the example shown, a reference inside the opened scope retrieves the value of a name declared in that scope, and after the scope closes a reference retrieves the value of a name declared before the scope. Generation and tokenization details are given in Section 2 and Appendix A .

Figure 3: Gains from NCA PPT are sensitive to baseline initialization. A 400M-parameter Llama model trained on

{\sim}9

B tokens of OpenWebText, reproducing an NCA PPT experiment of Lee et al. (2026) ; we plot perplexity to match how prior work reports it. The groups differ only in weight initialization. With

\mathcal{N}(0,1)

embeddings and

\mathcal{N}(0,0.02)

elsewhere, NCA PPT improves perplexity over the no-PPT baseline by 5.4%. Initializing the embeddings from

\mathcal{N}(0,0.02)

instead improves the baseline itself by 9.0%, and the two methods become inseparable despite the additional compute the NCA phase receives.

Figure 4: PPT is useful at 50M–100M natural-language tokens, but its gain at 1B disappears under spectral-based initialization. In the additive-compute regime with a 336M-parameter model, NCA and

k

-Dyck, each at its lowest-loss configuration, improve on the optimal no-PPT baseline at 50M and 100M natural-language tokens. Under spectral-based initialization this gain disappears by 1B tokens, but under the other initializations it persists at every budget (Appendix E.1 shows the full breakdown). Because spectral-based initialization yields the lowest loss for every method at this largest budget, we use it in all compute-matched experiments, which run at comparable FLOP budgets.

BLiMP ( $\uparrow$ better)	Data-constrained (100M NL, $1{\times}10^{18}$ )	DeclRef 10%	0.721 $\pm$ 0.004	0.731 $\pm$ 0.005
	Setting	PPT method	Baseline (0% PPT)	PPT score
DCLM CORE ( $\uparrow$ better)	Additive (50M NL)	$k$ -Dyck	0.024 $\pm$ 0.004	0.032 $\pm$ 0.003
	Data-unconstrained ( $1{\times}10^{18}$ )	C4 1%	0.088 $\pm$ 0.006	0.106 $\pm$ 0.004
		Code 1%		0.104 $\pm$ 0.004
		DeclRef 1%		0.097 $\pm$ 0.002
		DeclRef 5%		0.108 $\pm$ 0.011

Table 1: Downstream benchmark improvement. Across the models reported in Figures 1 and 4 , these are the cases whose aggregate BLiMP or DCLM CORE score sits above their baseline. However, C4, the natural-language control, reaches a DCLM CORE score in the same range as the PPT methods. In this table, NL denotes natural-language tokens.

Figure 5: Uniformly distributing the PPT data across training, rather than front-loading it, prevents forgetting on the PPT distribution. In the interleaved runs,

5\%

of every batch consists of NCA,

k

-Dyck, or code throughout training. In the PPT runs, the same

5\%

of compute is instead spent up front, at

2.7\times 10^{19}

FLOPs. Left: FineWeb validation BPB at the end of training. Right: validation BPB on the PPT corpus itself across the whole run, one panel per corpus on its own scale, with the gray dotted line marking

5\%

of total compute. Under the PPT schedule the models lose this ability at the PPT-to-PT boundary; under interleaving they keep it and continue to improve on it. Elsewhere in this work the vocabulary is replaced at the phase boundary for the synthetic PPT distributions, which means the PPT corpus cannot be scored after the PPT phase. Every run here instead shares a single vocabulary, extended to hold the NCA and

k

-Dyck tokens, so the corpus can be tracked throughout training. These runs are not hyperparameter-tuned (configurations in Appendix D.5 , Table 7 ), so the FineWeb panel is included only to show that the interleaved runs sit in a similar range to their PPT counterparts.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The NCA positive control under three baseline weight initializations. A 400M-parameter Llama model trained on 164M NCA tokens, and then

{\sim}9

B tokens of OpenWebText, following Lee et al. (2026) . Validation perplexity for the no-PPT baseline and NCA under three initializations, namely that of the original work (

\mathcal{N}(0,1)

embeddings and

\mathcal{N}(0,0.02)

elsewhere),

\mathcal{N}(0,0.02)

embeddings and head, and the spectral-based initialization of the NanoChat version we use. Under each alternative initialization the baseline improves and the gap between NCA and the baseline narrows.

Figure 7: LLMTuna reaches a lower validation loss in fewer trials than a TPE search. The continuous lines for each method show the best FineWeb validation BPB so far against trial index, while the faint dots are individual trials. LLMTuna reaches

0.9780

at its tenth trial, while TPE reaches

0.9808

at its twenty-second and does not improve over the remaining eight. The axis is broken above

0.992

, where the early exploratory trials of both searches sit.

Setting	Recipe	Tuned hyperparameters
Additive	Unified	Peak matrix learning rate, warmup fraction, warmdown fraction, PT batch size
	Separate	The above, plus PPT peak learning rate, PPT warmup fraction, PPT warmdown fraction, PPT batch size
Compute-matched	Unified	Depth, peak matrix learning rate, warmup fraction, warmdown fraction
	Separate	The above, plus PPT peak learning rate, PPT warmup fraction, PPT warmdown fraction, PPT batch size

Appendix

Table 2: Hyperparameters tuned in each setting. Separate is the standard recipe of Section 3 : the PPT phase gets its own learning rate, and the optimizer state is reset at the phase boundary. Unified covers the no-PPT baseline, the C4 control, and the unified optimizer code variant, which all run both phases under a single schedule with no reset; for C4, restarting the optimizer at the boundary hurt its final loss. In the additive setting the natural-language token count is fixed, so batch size sets the number of optimizer steps; we therefore tune it there, and fix it at 262,144 tokens per step in the compute-matched settings. The PPT-phase optimizer is not searched; in the additive and data-constrained settings we run one search under each of Muon and AdamW and report the lower-BPB result.

Method	$d$	$\eta_{\mathrm{PT}}$	PPT optimizer	$\eta_{\mathrm{PPT}}$	PPT batch
Baseline (0% PPT)	20	3e-2	–	–	–
NCA 1%	20	3e-2	AdamW	3e-5	512k
NCA 5%	20	3e-2	AdamW	3e-5	512k
NCA 10%	22	2e-2	AdamW	3e-6	1M
$k$ -Dyck 1%	20	3e-2	Muon	8e-6	512k
$k$ -Dyck 5%	20	3e-2	Muon	2e-2	512k

Appendix

Table 3: Data-constrained setting: reported configurations. 100M unique natural-language tokens at

1{\times}10^{18}

FLOPs, the experiments of panel (1) of Figure 1 , each at the lower-BPB PPT optimizer. Every PPT run warms down over 0.4 of both its PPT and natural-language phases, and the baseline over 0.2 of training.

d

\eta_{\mathrm{PT}}

\eta_{\mathrm{PPT}}

PT warmdown

PPT warmdown

PPT batch

Method

1{\times}10^{18}

3{\times}10^{18}

9{\times}10^{18}

1{\times}10^{18}

3{\times}10^{18}

9{\times}10^{18}

1{\times}10^{18}

3{\times}10^{18}

9{\times}10^{18}

1{\times}10^{18}

3{\times}10^{18}

9{\times}10^{18}

1{\times}10^{18}

3{\times}10^{18}

9{\times}10^{18}

1{\times}10^{18}

3{\times}10^{18}

9{\times}10^{18}

Baseline (0% PPT)

3e-2

2e-2

1.5e-2

–

0.4

0.5

–

C4 1%

3e-2

1.5e-2

–

0.4

–

C4 5%

3e-2

1.5e-2

–

0.4

–

NCA 1%

2e-2

6e-3

2e-2

1.2e-2

0.4

NCA 5%

3e-2

2e-3

6e-5

0.4

256k

Appendix

Table 4: Data-unconstrained setting: reported configurations. The methods of Table 10 at each budget. Two schedule details have no column:

k

-Dyck 1% warms its PPT phase up over the first 0.1 of the phase at every budget, and the unified code 5% method warms up over 0.02 of training at

3{\times}10^{18}

and

9{\times}10^{18}

; every other experiment has no warmup.

NL tokens	Method	Init	PPT tokens	PT batch	PPT optimizer	$\eta_{\mathrm{PPT}}$	PPT warmdown	PPT batch
50M	Baseline	$\mathcal{N}(0,1)$ emb.	–	64k	–	–	–	–
50M	Baseline	spectral	–	64k	–	–	–	–
50M	NCA	$\mathcal{N}(0,0.02)$	500M	64k	Muon	4e-3	1	256k
50M	$k$ -Dyck	spectral	20M	64k	AdamW	1.2e-4	1	64k
100M	Baseline	$\mathcal{N}(0,1)$ emb.	–	64k	–	–	–	–
100M	Baseline	spectral	–	64k	–	–	–	–

Appendix

Table 5: Additive setting: reported configurations. The twelve points of Figure 4 : per natural-language (NL) budget, the two baselines the figure draws and each curriculum at its lowest-loss initialization, PPT token count and PPT optimizer. Every experiment settled on

\eta_{\mathrm{PT}}=0.02

with a warmdown of 0.4, and the PPT phase runs with no warmup.

Method	$\eta_{\mathrm{PT}}$	PT warmdown	$\eta_{\mathrm{PPT}}$	PPT warmup	PPT warmdown	PPT batch
Baseline (0% PPT)	6e-3	0.5	–	–	–	–
NCA 1%	7.5e-3	0.495	1.3e-3	0	0.6	1M
$k$ -Dyck 1%	6e-3	0.495	1e-2	0.1	0.4	1M

Appendix

Table 6: Over-Chinchilla setting: configurations. Depth 24 (881M parameters) on 50B tokens, the methods of panel (2) of Figure 1 . Hyperparameter search details are given in Appendix D.6 .

Run	$d$	$\eta_{\mathrm{PT}}$	PT warmdown	$\eta_{\mathrm{PPT}}$	PPT warmdown	PPT batch
NCA 5%, interleaved	24	1.5e-2	0.5	–	–	–
$k$ -Dyck 5%, interleaved	24	1.5e-2	0.5	–	–	–
Code 5%, interleaved	24	1.5e-2	0.5	–	–	–
NCA 5%, PPT phase	24	3e-2	0.4	6e-5	0.4	256k
$k$ -Dyck 5%, PPT phase	21	3e-2	0.4	3e-5	0.4	512k
Code 5%, PPT phase	24	1.5e-2	0.5	–	–	–

Appendix

Table 7: Retention experiment: configurations. The runs of Figure 5 at

2.7{\times}10^{19}

FLOPs. The PPT runs use each method’s 5% configuration, extrapolated from the smaller budgets, with a shared vocabulary. The interleaved runs train under one schedule, so they have no PPT-phase entries, and all three share the code 5% configuration so that only the interleaved corpus differs.

Figure 8: Full synthetic PPT token sweep behind Figure 4 . FineWeb validation bits-per-byte against the amount of synthetic PPT added on top of a fixed natural-language token budget. Rows are that budget, columns are the initialization scheme: two as used in the prior literature that introduced these curricula, and the spectral-based initialization. The model size here is 336M parameters. Within each panel the leftmost point is that configuration’s own baseline with no PPT, marked by the gray dashed line. The green dashed line and band mark the lowest-loss no-PPT baseline across the three initialization schemes, providing a reference for comparisons across initializations. Solid lines with filled markers are Muon PPT and dashed lines with open markers are AdamW PPT. AdamW performs better than Muon for the PPT methods at the lower natural-language budgets; at 1B tokens, however, the two are indistinguishable under spectral-based initialization. Importantly, if one is to train with the alternative initializations, the PPT benefit would still exist even at this larger budget.

Initialization	Method	50M	100M	1B
Spectral init	Baseline	0.691 $\pm$ 0.005	0.719 $\pm$ 0.001	0.774 $\pm$ 0.005
	NCA	0.691 $\pm$ 0.002	0.719 $\pm$ 0.008	0.769 $\pm$ 0.007
	$k$ -Dyck	0.688 $\pm$ 0.006	0.723 $\pm$ 0.007	0.778 $\pm$ 0.007
$N(0,1)$ embeddings	Baseline	0.687 $\pm$ 0.003	0.717 $\pm$ 0.004	0.779 $\pm$ 0.003
	NCA	0.683 $\pm$ 0.003	0.717 $\pm$ 0.007	0.776 $\pm$ 0.002
	$k$ -Dyck	0.695 $\pm$ 0.007	0.718 $\pm$ 0.011	0.775 $\pm$ 0.001

Appendix

Table 8: BLiMP in the additive setting. BLiMP accuracy for the additive-compute experiments of Figure 4 : for each natural-language budget and initialization, the baseline and each curriculum at its best (lowest-loss) PPT token count (Table 5 lists the counts). Higher is better, and bold marks values whose one-standard-deviation interval lies entirely above that of the baseline with the same initialization (none are bolded here as all perform similarly to the baseline).

Initialization	Method	50M	100M	1B
Spectral init	Baseline	0.024 $\pm$ 0.004	0.042 $\pm$ 0.011	0.110 $\pm$ 0.009
	NCA	0.032 $\pm$ 0.001	0.046 $\pm$ 0.006	0.120 $\pm$ 0.002
	$k$ -Dyck	0.032 $\pm$ 0.003	0.046 $\pm$ 0.012	0.116 $\pm$ 0.003
$N(0,1)$ embeddings	Baseline	0.030 $\pm$ 0.002	0.040 $\pm$ 0.007	0.107 $\pm$ 0.010
	NCA	0.035 $\pm$ 0.010	0.045 $\pm$ 0.009	0.117 $\pm$ 0.014
	$k$ -Dyck	0.030 $\pm$ 0.005	0.044 $\pm$ 0.006	0.115 $\pm$ 0.015

Appendix

Table 9: DCLM CORE in the additive setting. Centered CORE accuracy for the same experiments as Table 8 : for each natural-language budget and initialization, the baseline and each curriculum at its best (lowest-loss) PPT token count (Table 5 lists the counts). Higher is better, and bold marks values whose one-standard-deviation interval lies entirely above that of the baseline with the same initialization.

Figure 9: No PPT method scales better than the baseline. Here we plot FineWeb validation BPB against training compute for the data-unconstrained experiments. Each line is a separate power law,

\mathrm{BPB}=A\,C^{-\alpha}

( Kaplan et al., 2020 ) , fitted by least squares on

\log\mathrm{BPB}

against

\log C

over that method’s nine runs (three seeds per scale), with

C

in units of

10^{18}

FLOPs, so

A

is the fitted BPB at

10^{18}

FLOPs. The table gives

A

and

\alpha

with their standard errors. The baseline has the lowest

A

, and no method’s exponent is significantly larger than the baseline’s (within standard error). Raising PPT from 1% to 5% raises

A

for every method. However, its effect on the exponent is unclear as at 5% every exponent is within standard error of the baseline’s. Every synthetic PPT method remains above the baseline at every budget, while C4, independent of percentage, is the same as the baseline. Since we only have three budgets per method, and three seeds, we do not fit an irreducible-loss term as in Hoffmann et al. (2022) .

Method	$1{\times}10^{18}$	$3{\times}10^{18}$	$9{\times}10^{18}$
Baseline (0% PPT)	$\mathbf{0.9794}_{\pm 0.0003}$	$\mathbf{0.9234}_{\pm 0.0002}$	$\mathbf{0.8733}_{\pm 0.0002}$
C4 1%	$\mathbf{0.9799}_{\pm 0.0006}$	$\mathbf{0.9232}_{\pm 0.0005}$	$\mathbf{0.8729}_{\pm 0.0004}$
C4 5%	$0.9804_{\pm 0.0004}$	$\mathbf{0.9238}_{\pm 0.0003}$	$\mathbf{0.8735}_{\pm 0.0005}$
NCA 1%	$0.9803_{\pm 0.0005}$	$\mathbf{0.9239}_{\pm 0.0001}$	$\mathbf{0.8751}_{\pm 0.0024}$
NCA 5%	$0.9830_{\pm 0.0002}$	$0.9272_{\pm 0.0002}$	$0.8765_{\pm 0.0002}$
$k$ -Dyck 1%	$0.9802_{\pm 0.0002}$	$0.9248_{\pm 0.0003}$	$0.8765_{\pm 0.0021}$

Appendix

Table 10: Allocating a larger share of a fixed compute budget to synthetic PPT generally increases loss. FineWeb bits-per-byte. Hyperparameters were searched independently for each method at every budget. Code appears under both the separate and the unified optimizer recipe. Bold marks values whose one-standard-deviation interval overlaps that of the column’s lowest.

Method	$1{\times}10^{18}$	$3{\times}10^{18}$	$9{\times}10^{18}$
Baseline (0% PPT)	$\mathbf{1.1461}_{\pm 0.0002}$	$1.0900_{\pm 0.0002}$	$1.0409_{\pm 0.0004}$
C4 1%	$\mathbf{1.1465}_{\pm 0.0007}$	$1.0898_{\pm 0.0007}$	$1.0399_{\pm 0.0005}$
C4 5%	$\mathbf{1.1456}_{\pm 0.0004}$	$\mathbf{1.0885}_{\pm 0.0004}$	$\mathbf{1.0382}_{\pm 0.0008}$
NCA 1%	$1.1471_{\pm 0.0006}$	$1.0905_{\pm 0.0003}$	$1.0424_{\pm 0.0027}$
NCA 5%	$1.1497_{\pm 0.0003}$	$1.0937_{\pm 0.0003}$	$1.0433_{\pm 0.0002}$
$k$ -Dyck 1%	$1.1470_{\pm 0.0002}$	$1.0914_{\pm 0.0003}$	$1.0436_{\pm 0.0021}$

Appendix

Table 11: The data-unconstrained comparison on the held-out C4 split. C4 bits-per-byte for the methods of Table 10 , same checkpoints, same protocol. Bold marks values whose one-standard-deviation interval overlaps that of the column’s lowest.

Method	$1{\times}10^{18}$	$3{\times}10^{18}$	$9{\times}10^{18}$
Baseline (0% PPT)	$\mathbf{1.9435}_{\pm 0.0260}$	$\mathbf{1.7078}_{\pm 0.0785}$	$\mathbf{1.6702}_{\pm 0.0347}$
C4 1%	$\mathbf{1.9738}_{\pm 0.0498}$	$\mathbf{1.7890}_{\pm 0.0528}$	$\mathbf{1.6590}_{\pm 0.0286}$
C4 5%	$\mathbf{1.9012}_{\pm 0.0132}$	$\mathbf{1.7914}_{\pm 0.0732}$	$\mathbf{1.6832}_{\pm 0.0217}$
NCA 1%	$\mathbf{1.9356}_{\pm 0.0883}$	$\mathbf{1.7793}_{\pm 0.0469}$	$\mathbf{1.7353}_{\pm 0.0295}$
NCA 5%	$\mathbf{1.9640}_{\pm 0.0603}$	$\mathbf{1.8043}_{\pm 0.0862}$	$\mathbf{1.7272}_{\pm 0.0227}$
$k$ -Dyck 1%	$\mathbf{1.9867}_{\pm 0.0929}$	$\mathbf{1.7675}_{\pm 0.0580}$	$\mathbf{1.8248}_{\pm 0.1379}$

Appendix

Table 12: The data-unconstrained comparison on the held-out GitHub-code split. GitHub-code bits-per-byte for the methods of Table 10 , same checkpoints, same protocol. Bold marks values whose one-standard-deviation interval overlaps that of the column’s lowest.

Method	$1{\times}10^{18}$	$3{\times}10^{18}$	$9{\times}10^{18}$
Baseline (0% PPT)	0.761 $\pm$ 0.004	0.780 $\pm$ 0.003	0.794 $\pm$ 0.004
C4 1%	0.756 $\pm$ 0.007	0.779 $\pm$ 0.004	0.791 $\pm$ 0.009
C4 5%	0.757 $\pm$ 0.009	0.780 $\pm$ 0.005	0.794 $\pm$ 0.008
NCA 1%	0.753 $\pm$ 0.006	0.781 $\pm$ 0.007	0.792 $\pm$ 0.002
NCA 5%	0.759 $\pm$ 0.006	0.773 $\pm$ 0.010	0.796 $\pm$ 0.002
$k$ -Dyck 1%	0.755 $\pm$ 0.007	0.782 $\pm$ 0.006	0.798 $\pm$ 0.002

Appendix

Table 13: BLiMP across the data-unconstrained sweep. BLiMP accuracy for the data-unconstrained sweep; higher is better, and bold marks values whose one-standard-deviation interval lies entirely above the baseline’s at that budget.

Method	$1{\times}10^{18}$	$3{\times}10^{18}$	$9{\times}10^{18}$
Baseline (0% PPT)	0.088 $\pm$ 0.006	0.121 $\pm$ 0.015	0.165 $\pm$ 0.008
C4 1%	0.106 $\pm$ 0.004	0.131 $\pm$ 0.004	0.156 $\pm$ 0.004
C4 5%	0.101 $\pm$ 0.007	0.137 $\pm$ 0.005	0.154 $\pm$ 0.007
NCA 1%	0.092 $\pm$ 0.005	0.127 $\pm$ 0.011	0.162 $\pm$ 0.007
NCA 5%	0.082 $\pm$ 0.003	0.133 $\pm$ 0.004	0.155 $\pm$ 0.012
$k$ -Dyck 1%	0.096 $\pm$ 0.012	0.135 $\pm$ 0.010	0.153 $\pm$ 0.006

Appendix

Table 14: DCLM CORE across the data-unconstrained sweep. Centered CORE accuracy; higher is better, and bold marks values whose one-standard-deviation interval lies entirely above the baseline’s.

Figure 10: Data-constrained results separated by PPT-phase optimizer. The experiments behind panel (1) of Figure 1 , with Muon PPT drawn as solid lines with filled markers and AdamW PPT as dashed lines with open markers. Each experiment was searched independently under the same search space and stopping rule, so the two differ in the optimizer used during PPT and in the configuration each search found. Neither optimizer is always better than the other: AdamW is lower for NCA and DeclRef at every PPT percentage, while

k

-Dyck is lower under Muon. No experiment of either optimizer reaches the no-PPT baseline.

Method	BLiMP	DCLM CORE
Baseline (0% PPT)	0.721 $\pm$ 0.004	0.068 $\pm$ 0.012
NCA 1%	0.718 $\pm$ 0.003	0.077 $\pm$ 0.007
NCA 5%	0.722 $\pm$ 0.001	0.074 $\pm$ 0.006
NCA 10%	0.730 $\pm$ 0.006	0.069 $\pm$ 0.002
$k$ -Dyck 1%	0.714 $\pm$ 0.007	0.068 $\pm$ 0.009
$k$ -Dyck 5%	0.724 $\pm$ 0.001	0.059 $\pm$ 0.010

Appendix

Table 15: BLiMP and DCLM CORE in the data-constrained regime. 100M unique natural-language tokens at a fixed

1{\times}10^{18}

-FLOP budget, the experiments of panel (1) of Figure 1 , each at the lower-BPB PPT optimizer. Higher is better, and bold marks values whose one-standard-deviation interval lies entirely above the baseline’s.

Method	BLiMP	DCLM CORE
Baseline (0% PPT)	0.809 $\pm$ 0.006	0.263 $\pm$ 0.008
NCA 1%	0.812 $\pm$ 0.005	0.264 $\pm$ 0.009
$k$ -Dyck 1%	0.809 $\pm$ 0.002	0.262 $\pm$ 0.009

Appendix

Table 16: Over-Chinchilla benchmark summary. BLiMP and DCLM CORE for the over-Chinchilla regime: 881M parameters on 50B tokens, NCA and

k

-Dyck at 1%. Higher is better, and bold marks values whose one-standard-deviation interval lies entirely above the baseline’s. No method separates from the baseline on either benchmark.

Figure 11: Breakdown of downstream performance by benchmark sub-tasks. Each row is a PPT model from Figures 1 and 4 , and each column is one of the 13 BLiMP or 22 DCLM CORE tasks. Blue cells are significantly above the baseline, red cells significantly below, and white cells on par. Most cells are white, so on most sub-scores the PPT models perform on par with their baseline. The colored cells show that, even so, the PPT method changes how the model performs on some sub-tasks of these benchmarks.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 448 papers

Sep 30, 2026

Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical prior, i.e., a structural inductive bias learned during PPT that transfers to natural language grammar. However, PPT has only been tested on models of at most 1B parameters and PT budgets below 2B tokens on predominantly web text. It is unknown whether PPT is effective at larger scales and under more realistic PT data mixtures that combine diverse sources (e.g., code and math). We therefore present a comprehensive study on PPT spanning five PPT tasks, four PT data mixtures, four parameter scales (500M to 7B), and PT budgets of up to 100B tokens. Our results demonstrate that the downstream performance and token efficiency gains of PPT persist at scale, e.g., saving at least 21B PT tokens at the 3B scale. However, in contrast to prior work, we find no consistent evidence that these gains stem from a grammatical prior. Downstream performance does not consistently align with grammatical acceptability across model sizes. Instead, we find that downstream gains arise from PPT tasks that improve long-range retrieval. Finally, PPT performance gains are robust to how PT data mixtures are composed and diminish only when web text is absent. Overall, PPT is a low-cost addition to PT, and future PPT task design should target long-range retrieval rather than natural language grammar.

Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus +3

University of Sheffield, UK · Mohamed bin Zayed University of Artificial Intelligence, UAE · Hugging Face +3

Language Model Pretraining Synthetic Data Pretraining Linguistic Bias in Language Models

Ask PDF

May 11, 2026

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data

Large language models (LLMs) rely on web-scale corpora for pre-training. The noise inherent in these datasets tends to obscure meaningful patterns and ultimately degrade model performance. Data curation mitigates but cannot eliminate such noise, so pre-training corpora remain noisy in practice. We therefore study whether a lightweight pre-pre-training (PPT) stage based on synthetic data with learnable temporal structure helps resist noisy data during the pre-training (PT) stage. Across various corruption settings, our method consistently improves robustness to noise during PT, with larger relative gains at higher noise levels. For a 1B-parameter model, a synthetic PPT stage with only 65M tokens achieves the same final loss as the baseline while using up to 49% fewer natural-text PT tokens across different noise levels. Mechanistic analyses suggest PPT does not immediately suppress attention to noisy tokens. Rather, PPT-initialized models gradually downweight attention between corrupted tokens during noisy PT. This indicates that synthetic PPT inhibits noise self-modeling and shapes the subsequent optimization trajectory. Code is available at https://github.com/guox18/formal-language-prepretraining.

Xu Guo, Runyu Peng, Jian Tong +4

Shanghai AI Laboratory · Fudan University · Shanghai Innovation Institute

Language Model Pretraining Synthetic Data Pretraining Neural Network Robustness Noise Robustness Language Model Robustness

Ask PDF

Mar 9, 2026

Training Language Models via Neural Cellular Automata

Pre-training is crucial for large language models (LLMs), as it is when most representations and capabilities are acquired. However, natural language pre-training has problems: high-quality text is finite, it contains human biases, and it entangles knowledge with reasoning. This raises a fundamental question: is natural language the only path to intelligence? We propose using neural cellular automata (NCA) to generate synthetic, non-linguistic data for pre-pre-training LLMs--training on synthetic-then-natural language. NCA data exhibits rich spatiotemporal structure and statistics resembling natural language while being controllable and cheap to generate at scale. We find that pre-pre-training on only 164M NCA tokens improves downstream language modeling by up to 6% and accelerates convergence by up to 1.6x. Surprisingly, this even outperforms pre-pre-training on 1.6B tokens of natural language from Common Crawl with more compute. These gains also transfer to reasoning benchmarks, including GSM8K, HumanEval, and BigBench-Lite. Investigating what drives transfer, we find that attention layers are the most transferable, and that optimal NCA complexity varies by domain: code benefits from simpler dynamics, while math and web text favor more complex ones. These results enable systematic tuning of the synthetic distribution to target domains. More broadly, our work opens a path toward more efficient models with fully synthetic pre-training.

Dan Lee, Seungwook Han, Akarsh Kumar +1

Independent Contributor · MIT · Improbable AI Lab

Language Model Pretraining Synthetic Data Pretraining Synthetic Data Generation for Language Models Neural Cellular Automata

Ask PDF

Abstract

Language Model Pretraining

Explore similar work

Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data

Training Language Models via Neural Cellular Automata