cs.LGOct 18, 2026

PRECOCIAL: Pretraining with Code Curriculums via Iterative Search for Accelerated Learning

Authors: Edward Gu, Ali Payani, Kevin Ellis

Organizations: Cornell University · Cisco Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Fully synthetic pretraining data generated by symbolic programs can bootstrap downstream training on naturalistic tasks, such as language modeling, but the generating programs have so far been designed by hand. We introduce PRECOCIAL, a method for automatically optimizing programs that generate synthetic pretraining data. Starting from an initial generator program, an LLM iteratively rewrites the program, and each rewrite is scored by pretraining a randomly-initialized transformer on its data and fine-tuning it on the downstream task. Because every score requires training, PRECOCIAL keeps a single network for the entire search, continuing its pretraining for each accepted program instead of training a new network per candidate, making it significantly more efficient. On four language and code datasets, at a fixed fine-tuning budget, optimized programs reach lower perplexity than hand-designed curricula, published procedural generators, random program mutation and the program the LLM writes in the first iteration of the search, and reach the same perplexity as the hand-designed curricula with 9-33% fewer fine-tuning steps. PRECOCIAL also ends with lower perplexity than FunSearch and AlphaEvolve, which train a new network for every candidate, at the same number of evaluations and a lower cost. We also observe that an optimized program generator from one domain transfers to datasets in other domains. Additionally, optimized programs allow for better downstream performance on four different reasoning and problem-solving tasks.

Figures & tables

Figure 1: PRECOCIAL at a glance. Top : an LLM rewrites a program π\pi (here, part of the program kept for WikiText-103) whose data pretrains one transformer, continued across rewrites; a fine-tuned copy scores each rewrite. Programs transfer between tasks (§ 4.4 ). Bottom , from the runs of Tables 2 and 5 : WikiText-103 validation perplexity by fine-tuning budget, solve rates, and the Sokoban puzzle of Figure 3 .
Figure 2: The search objective over rounds: validation perplexity J^\hat{J} of each round’s candidate, mean over 3 seeds. PRECOCIAL improves over the rounds and separates from the baselines on three of the four datasets; on CodeSearchNet every method’s trajectory is noisy, and the kept program is the best round, not the last.
Figure 3: One test puzzle per task, and the start, middle and end of the plan each method’s network writes for it, replayed by the task’s exact checker; below each row, the checker’s verdict. Networks trained as in Table 5 with training seed 64. Universal prior : pretrained on the outputs of random programs, as in Grau-Moya et al. (2024) . Green marks a tile, box or block in its goal position; in Sokoban the dot is the player and a red square marks the cell an illegal move tries to enter.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: PRECOCIAL. The LLM rewrites the generator program; the rewrite is scored by continuing to pretrain the one network kept for the whole search, fine-tuning a copy on Dtrain\mathcal{D}_{\text{train}} and evaluating it on Dval\mathcal{D}_{\text{val}} . An improvement is accepted; otherwise both the program and the network are rolled back to the last accepted state.
Figure 5: Perplexity above PRECOCIAL at each fine-tuning budget, pooled over the four datasets under matched pretraining (best validation perplexity within the budget; mean ±\pm s.e.m. over 36 paired runs; logarithmic axes). PRECOCIAL is ahead of every other curriculum at every budget shown; the lead is largest at small budgets and narrows as fine-tuning proceeds. See also Table 7 .
Figure 6: Head start by model size on WikiText-103: perplexity of a network with no synthetic pretraining minus PRECOCIAL’s, at each fine-tuning budget (best validation perplexity reached within the budget, mean over 3 seeds; above 0: PRECOCIAL is ahead). Larger models reach a larger lead, earlier, and close the gap sooner. The two smallest pretrained networks start behind (at 500 steps, and also at 1k for 0.15M).
Figure 7: Pretraining loss during the search, concatenated over rounds (mean over 3 seeds); the xx axis is thousands of pretraining steps. From round 1 on, PRECOCIAL sits above the first-iteration and random-mutation baselines (in round 0 all three train on the same first-iteration program) and spikes at each round boundary, where the next candidate changes the data, yet reaches the best downstream perplexity (Table 9 ).
ProgramsnnEpiplexity SSRange of SSHH
Hand-written ( π0\pi_{0} )1111511150.986
Procedural18438430.859
First iteration314271293–15900.981
Random mutation313801285–15570.974
PRECOCIAL after round 5315791233–17980.990
PRECOCIAL315051396–16590.943
Appendix
Table 15: Epiplexity SS (nats per token summed over steps) and final held-out loss HH (nats per token) of each program’s synthetic data, measured with a fresh 0.69M network making one pass over 40,000 fresh streams. Mean and range over the programs of a group; all programs except the last group come from the WikiText-103 searches.
Figure 8: Every matched-pretraining run of Table 5 (dots; bars: means). On Blocksworld the solve rate is nearly all-or-nothing: it is a step function of the test perplexity the fine-tuned model reaches (Table 17 ), so single runs sit at zero or well above it.
Pretraining8-puzzleSokobanBlocksworld
PRECOCIAL0.237 ±\pm .0030.073 ±\pm .0140.137 ±\pm .237
First iteration0.143 ±\pm .1200.014 ±\pm .0140.007 ±\pm .012
Hand-written ( π0\pi_{0} )0.192 ±\pm .0740.037 ±\pm .0260.000 ±\pm .000
No pretraining0.021 ±\pm .0140.010 ±\pm .0020.000 ±\pm .000
Universal prior0.154 ±\pm .0530.049 ±\pm .0060.000 ±\pm .000
Appendix
Table 18: Test solve rate of the networks trained for Figure 3 , mean ±\pm SD over training seeds 64, 65 and 66 (the figure shows seed 64).
Figure 9: The 8-puzzle of Figure 3 : six evenly spaced states of each network’s plan.
Figure 10: The Sokoban puzzle of Figure 3 : six evenly spaced states of each network’s plan. The dot is the player; a red square marks the cell the illegal move tries to enter.
Figure 11: The Blocksworld puzzle of Figure 3 (9 blocks): six evenly spaced states of each network’s plan; the block above the stacks is the one in the hand.
Figure 12: Task-matching share by search round, mean ±\pm SD over three searches: planning rules for Blocksworld, the 8-puzzle and Sokoban, arithmetic for Countdown, code syntax for code.
More in this topic

Synthetic Data Pretraining

Synthetic data pretraining uses generated, simulated, or transformed examples to initialize models when real training data are scarce or costly.

See all 119 papers

Explore similar work

CardsList
Jan 29, 2026

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .

Ajay Patel, Colin Raffel, Chris Callison-Burch

University of Pennsylvania · University of Toronto · Vector Institute +1

Language Model PretrainingSynthetic Data PretrainingSynthetic Data GenerationSynthetic Instruction GenerationInstruction TuningSynthetic Data Generation for Language Models
AskPDF
Jun 8, 2026

CodeAlchemy: Synthetic Code Rewriting at Scale

Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats. While synthetic data has proven transformative for language models, code remains largely unexplored beyond limited quality improvements. We present CodeAlchemy, a synthetic data generation framework that transforms publicly sourced code into semantically-rich training data through 5 strategies: CodeEnhance (quality-aware rewriting), CodeQA (template-based problems), CodeDev (developer tasks), CodeDialogue (multi-turn conversations), and CodeTrace (execution traces). We process 3 corpora across 15 languages to generate 500B+ tokens of synthetic data plus 350B reasoning tokens, orders of magnitude more than prior efforts. CodeTrace instruments and executes 1.3M+ files across 14 languages and 5K libraries, capturing control flow, state tracking, and library knowledge. We introduce DevEval (developer tasks) and TraceEval (execution prediction) benchmarks; frontier models like Claude Sonnet 4.5 achieve only 5.6% exact match on TraceEval, revealing critical gaps in semantic understanding. Our 3B models achieve 83.5% on HumanEval, 63.2% on MBPP, 8.09% win rate on DevEval, and 15.36 ROUGE-2 on TraceEval, outperforming frontier models 10x the size including 27B Gemma-3 and 32B Granite-4.0.

Ankit Gupta, Aditya Prasad, Rameswar Panda

MIT-IBM Watson AI Lab, IBM Research

Synthetic Data PretrainingSynthetic Data GenerationCode Language ModelsCode Generation Benchmarks
AskPDF
Sep 24, 2026

Self-Play Pretraining with Zero Data

Advances in language modeling have been driven by scaling pretraining on ever more data. Yet, the training data is still largely curated on the model's behalf. A more general approach to pretraining would let the model learn to generate the data most useful for its own improvement. This would provide an effectively unbounded source of training data, limited by compute rather than human knowledge. We introduce Self-Play Pretraining with Zero Data, an initial proof-of-concept towards realizing this vision. Our procedure casts synthetic data generation as a search over the space of all computable structure, taking inspiration from Solomonoff induction. Starting from random initialization, two models learn in tandem: a generator proposes programs interpreted by a universal Turing machine, generating byte sequences, while a learner autoregressively predicts these byte sequences. The learner is trained with standard cross-entropy, while the generator is trained with reinforcement learning to produce sequences at the frontier of the learner's capabilities, yielding an adaptive curriculum. A universal Turing machine gives us a search space over all computable data-generating processes, imposing little domain-specific structure, and self-play searches over this space for useful training data. We test whether zero-shot performance on natural data improves predictably with self-play compute; this is a clean test of transfer since neither generator nor learner is trained on natural data. Across several natural datasets, zero-shot loss exhibits predictable scaling in compute. The models also exhibit in-context learning, and discover recognizable mathematical sequences during training.

Aditya Cowsik, Kfir Dolev, Michael Y. Li +4

Independent Researcher · Tel Aviv University · Stanford University +1

Synthetic Data PretrainingSynthetic Data Generation for Language Models
AskPDF