PRECOCIAL: Pretraining with Code Curriculums via Iterative Search for Accelerated Learning

Edward Gu; Ali Payani; Kevin Ellis

cs.LGOct 18, 2026

PRECOCIAL: Pretraining with Code Curriculums via Iterative Search for Accelerated Learning

Authors: Edward Gu, Ali Payani, Kevin Ellis

Organizations: Cornell University · Cisco Research

Abstract

Fully synthetic pretraining data generated by symbolic programs can bootstrap downstream training on naturalistic tasks, such as language modeling, but the generating programs have so far been designed by hand. We introduce PRECOCIAL, a method for automatically optimizing programs that generate synthetic pretraining data. Starting from an initial generator program, an LLM iteratively rewrites the program, and each rewrite is scored by pretraining a randomly-initialized transformer on its data and fine-tuning it on the downstream task. Because every score requires training, PRECOCIAL keeps a single network for the entire search, continuing its pretraining for each accepted program instead of training a new network per candidate, making it significantly more efficient. On four language and code datasets, at a fixed fine-tuning budget, optimized programs reach lower perplexity than hand-designed curricula, published procedural generators, random program mutation and the program the LLM writes in the first iteration of the search, and reach the same perplexity as the hand-designed curricula with 9-33% fewer fine-tuning steps. PRECOCIAL also ends with lower perplexity than FunSearch and AlphaEvolve, which train a new network for every candidate, at the same number of evaluations and a lower cost. We also observe that an optimized program generator from one domain transfers to datasets in other domains. Additionally, optimized programs allow for better downstream performance on four different reasoning and problem-solving tasks.

Figures & tables

Figure 1: PRECOCIAL at a glance. Top : an LLM rewrites a program

\pi

(here, part of the program kept for WikiText-103) whose data pretrains one transformer, continued across rewrites; a fine-tuned copy scores each rewrite. Programs transfer between tasks (§ 4.4 ). Bottom , from the runs of Tables 2 and 5 : WikiText-103 validation perplexity by fine-tuning budget, solve rates, and the Sokoban puzzle of Figure 3 .

Figure 2: The search objective over rounds: validation perplexity

\hat{J}

of each round’s candidate, mean over 3 seeds. PRECOCIAL improves over the rounds and separates from the baselines on three of the four datasets; on CodeSearchNet every method’s trajectory is noisy, and the kept program is the best round, not the last.

Figure 3: One test puzzle per task, and the start, middle and end of the plan each method’s network writes for it, replayed by the task’s exact checker; below each row, the checker’s verdict. Networks trained as in Table 5 with training seed 64. Universal prior : pretrained on the outputs of random programs, as in Grau-Moya et al. (2024) . Green marks a tile, box or block in its goal position; in Sokoban the dot is the player and a red square marks the cell an illegal move tries to enter.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: PRECOCIAL. The LLM rewrites the generator program; the rewrite is scored by continuing to pretrain the one network kept for the whole search, fine-tuning a copy on $\mathcal{D}_{\text{train}}$ and evaluating it on $\mathcal{D}_{\text{val}}$ . An improvement is accepted; otherwise both the program and the network are rolled back to the last accepted state.

Figure 5: Perplexity above PRECOCIAL at each fine-tuning budget, pooled over the four datasets under matched pretraining (best validation perplexity within the budget; mean

\pm

s.e.m. over 36 paired runs; logarithmic axes). PRECOCIAL is ahead of every other curriculum at every budget shown; the lead is largest at small budgets and narrows as fine-tuning proceeds. See also Table 7 .

Figure 6: Head start by model size on WikiText-103: perplexity of a network with no synthetic pretraining minus PRECOCIAL’s, at each fine-tuning budget (best validation perplexity reached within the budget, mean over 3 seeds; above 0: PRECOCIAL is ahead). Larger models reach a larger lead, earlier, and close the gap sooner. The two smallest pretrained networks start behind (at 500 steps, and also at 1k for 0.15M).

Figure 7: Pretraining loss during the search, concatenated over rounds (mean over 3 seeds); the

x

axis is thousands of pretraining steps. From round 1 on, PRECOCIAL sits above the first-iteration and random-mutation baselines (in round 0 all three train on the same first-iteration program) and spikes at each round boundary, where the next candidate changes the data, yet reaches the best downstream perplexity (Table 9 ).

Programs	$n$	Epiplexity $S$	Range of $S$	$H$
Hand-written ( $\pi_{0}$ )	1	1115	1115	0.986
Procedural	1	843	843	0.859
First iteration	3	1427	1293–1590	0.981
Random mutation	3	1380	1285–1557	0.974
PRECOCIAL after round 5	3	1579	1233–1798	0.990
PRECOCIAL	3	1505	1396–1659	0.943

Appendix

Table 15: Epiplexity

S

(nats per token summed over steps) and final held-out loss

H

(nats per token) of each program’s synthetic data, measured with a fresh 0.69M network making one pass over 40,000 fresh streams. Mean and range over the programs of a group; all programs except the last group come from the WikiText-103 searches.

Figure 8: Every matched-pretraining run of Table 5 (dots; bars: means). On Blocksworld the solve rate is nearly all-or-nothing: it is a step function of the test perplexity the fine-tuned model reaches (Table 17 ), so single runs sit at zero or well above it.

Pretraining	8-puzzle	Sokoban	Blocksworld
PRECOCIAL	0.237 $\pm$ .003	0.073 $\pm$ .014	0.137 $\pm$ .237
First iteration	0.143 $\pm$ .120	0.014 $\pm$ .014	0.007 $\pm$ .012
Hand-written ( $\pi_{0}$ )	0.192 $\pm$ .074	0.037 $\pm$ .026	0.000 $\pm$ .000
No pretraining	0.021 $\pm$ .014	0.010 $\pm$ .002	0.000 $\pm$ .000
Universal prior	0.154 $\pm$ .053	0.049 $\pm$ .006	0.000 $\pm$ .000

Appendix

Table 18: Test solve rate of the networks trained for Figure 3 , mean

\pm

SD over training seeds 64, 65 and 66 (the figure shows seed 64).

Figure 9: The 8-puzzle of Figure 3 : six evenly spaced states of each network’s plan.

Figure 10: The Sokoban puzzle of Figure 3 : six evenly spaced states of each network’s plan. The dot is the player; a red square marks the cell the illegal move tries to enter.

Figure 11: The Blocksworld puzzle of Figure 3 (9 blocks): six evenly spaced states of each network’s plan; the block above the stacks is the one in the hand.

Figure 12: Task-matching share by search round, mean

\pm

SD over three searches: planning rules for Blocksworld, the 8-puzzle and Sokoban, arithmetic for Countdown, code syntax for code.

Synthetic Data Pretraining

Synthetic data pretraining uses generated, simulated, or transformed examples to initialize models when real training data are scarce or costly.

See all 119 papers

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
Jan 29, 2026Ajay Patel, Colin Raffel, Chris Callison-BurchLanguage Model Pretraining Synthetic Data Pretraining
University of Pennsylvania · University of Toronto · Vector Institute +1
CodeAlchemy: Synthetic Code Rewriting at Scale
Jun 8, 2026Ankit Gupta, Aditya Prasad, Rameswar PandaSynthetic Data Pretraining Synthetic Data Generation
MIT-IBM Watson AI Lab, IBM Research
Self-Play Pretraining with Zero Data
Sep 24, 2026Aditya Cowsik, Kfir Dolev, Michael Y. Li +4Synthetic Data Pretraining Synthetic Data Generation for Language Models
Independent Researcher · Tel Aviv University · Stanford University +1

Jan 29, 2026J/K move · Enter open · S save

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Ajay Patel, Colin Raffel, Chris Callison-Burch

University of Pennsylvania · University of Toronto · Vector Institute+1

Language Model Pretraining Synthetic Data Pretraining Synthetic Data Generation Synthetic Instruction Generation Instruction Tuning Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .

Abstract

Synthetic Data Pretraining

Explore similar work

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

CodeAlchemy: Synthetic Code Rewriting at Scale

Self-Play Pretraining with Zero Data