cs.LGMar 9, 2026

Training Language Models via Neural Cellular Automata

Authors: Dan Lee, Seungwook Han, Akarsh Kumar, Pulkit Agrawal

Organizations: Independent Contributor · MIT · Improbable AI Lab

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Pre-training is crucial for large language models (LLMs), as it is when most representations and capabilities are acquired. However, natural language pre-training has problems: high-quality text is finite, it contains human biases, and it entangles knowledge with reasoning. This raises a fundamental question: is natural language the only path to intelligence? We propose using neural cellular automata (NCA) to generate synthetic, non-linguistic data for pre-pre-training LLMs--training on synthetic-then-natural language. NCA data exhibits rich spatiotemporal structure and statistics resembling natural language while being controllable and cheap to generate at scale. We find that pre-pre-training on only 164M NCA tokens improves downstream language modeling by up to 6% and accelerates convergence by up to 1.6x. Surprisingly, this even outperforms pre-pre-training on 1.6B tokens of natural language from Common Crawl with more compute. These gains also transfer to reasoning benchmarks, including GSM8K, HumanEval, and BigBench-Lite. Investigating what drives transfer, we find that attention layers are the most transferable, and that optimal NCA complexity varies by domain: code benefits from simpler dynamics, while math and web text favor more complex ones. These results enable systematic tuning of the synthetic distribution to target domains. More broadly, our work opens a path toward more efficient models with fully synthetic pre-training.

Figures & tables

Figure 1: Overview of NCA Pre-pre-training to Language Pre-training. We pre-pre-train a transformer with next-token prediction on the dynamics of neural cellular automata (NCA) sampled from selected complexity regions. We then conduct standard pre-training on natural language corpora. NCA pre-pre-training improves both validation perplexity and convergence speed on language pre-training. Interestingly, the optimal NCA distribution varies by downstream domain.
Figure 2: NCA pre-pre-training improves and accelerates language model pre-training across diverse domains. We show the validation perplexity during pre-training on ( a ) OpenWebText, ( b ) OpenWebMath, and ( c ) CodeParrot for 1.6B parameter models. Models pre-pre-trained on NCA trajectories consistently outperform the scratch, Dyck pre-pre-training, and surprisingly even C4 pre-pre-training baselines. NCA pre-pre-training achieves 1.4–1.6 ×\times faster convergence to the scratch baseline’s final perplexity while also reaching up to 6% lower final perplexity. We provide a zoomed-in training curve of the last third of training for clarity.
Figure 3: NCA pre-pre-training improves language model training performance across model sizes (Section 5.1 ). We report the final validation perplexity after pre-training on OpenWebText across (400M, 600M, and 1.6B parameter models). At 164M tokens, C4 pre-pre-training likely acquires shallow syntactic patterns that interfere with downstream learning rather than transferable structure. We investigate this further in Figure 4 .
Figure 4: Pre-pre-training on 160M tokens of NCA is better than pre-pre-training on 1.6B tokens of natural language (C4). We report the validation perplexity during pre-training on OpenWebText. Perplexity improvement is calculated relative to the C4 pre-pre-trained model. We add a version where we also preserve the embedding layers from pre-pre-training to pre-training (1.6B tokens w/o embedding reinit). Surprisingly even with the embedding layers, NCA pre-pre-training is better.
Figure 5: Attention weights are most crucial for positive transfer. We report the change in validation perplexity when selectively re-initializing model components after NCA pre-pre-training, relative to full transfer. Higher means the component is more important for transfer. Re-initializing attention causes the largest degradation across both OpenWebText and CodeParrot, while MLP and LayerNorm effects are domain-dependent.
Figure 6: Optimal NCA complexity is domain-dependent. We report the validation perplexity change of models trained with different NCA complexities from the scratch model. OpenWebText benefits from higher-complexity data (50%+), while CodeParrot peaks at intermediate complexity (30–40%). This suggests that matching synthetic data complexity to the target domain is necessary to maximize transfer.
Figure 7: NCA alphabet size changes scaling behavior. We report the perplexity change relative to the scratch model (higher is better) on OpenWebText across different alphabet sizes nn and NCA pre-pre-training token budgets. NCA improves perplexity at all token budgets. The smaller alphabet scales better, and the improvements degrade with scale for larger alphabets.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: NCA data exhibits a similar Zipfian or power-law structure to natural language. We compare the relative token frequency distribution for each of the natural language corpora and NCA data. Natural language from different domains has different average complexity as measured by gzip compressibility (see legend).
Figure 9: Different NCA alphabet sizes, n=2,10,15n=2,10,15 naturally yield different complexity distributions of the data. Increasing nn inherently increases the complexity of the data.
HyperparameterPre-pre-trainingPre-training
Effective batch size16512
Sequence length1024 tokens1024 tokens
Learning rate1×10−41\times 10^{-4}5×10−45\times 10^{-4} (Math/Text), 2×10−42\times 10^{-4} (Code)
LR scheduleCosine w/ warmupCosine w/ warmup
Warmup steps (% total)10%10%
Weight decayNone1×10−41\times 10^{-4}
Appendix
Table 2: Hyperparameters for pre-pre-training and pre-training experiments.
More in this topic

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 440 papers

Explore similar work

CardsList
May 11, 2026

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data

Large language models (LLMs) rely on web-scale corpora for pre-training. The noise inherent in these datasets tends to obscure meaningful patterns and ultimately degrade model performance. Data curation mitigates but cannot eliminate such noise, so pre-training corpora remain noisy in practice. We therefore study whether a lightweight pre-pre-training (PPT) stage based on synthetic data with learnable temporal structure helps resist noisy data during the pre-training (PT) stage. Across various corruption settings, our method consistently improves robustness to noise during PT, with larger relative gains at higher noise levels. For a 1B-parameter model, a synthetic PPT stage with only 65M tokens achieves the same final loss as the baseline while using up to 49% fewer natural-text PT tokens across different noise levels. Mechanistic analyses suggest PPT does not immediately suppress attention to noisy tokens. Rather, PPT-initialized models gradually downweight attention between corrupted tokens during noisy PT. This indicates that synthetic PPT inhibits noise self-modeling and shapes the subsequent optimization trajectory. Code is available at https://github.com/guox18/formal-language-prepretraining.

Xu Guo, Runyu Peng, Jian Tong +4

Shanghai AI Laboratory · Fudan University · Shanghai Innovation Institute

Language Model PretrainingSynthetic Data PretrainingNeural Network RobustnessNoise RobustnessLanguage Model Robustness
AskPDF
Aug 9, 2026

Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages

Pretraining LLMs on artificial languages ("pre-pretraining") is a technique that could reportedly increase token efficiency by 33%, i.e., save up to 33% of training tokens needed to reach a certain performance. We validate this prior result for English on a larger set of natural languages across four language families, using two different tokenizers and varying model sizes. We also relate the observed gains (or losses) in token efficiency to quantified linguistic properties of the languages, such as sentence length, morphological richness, and features of dependency syntactic trees (tree depth, number of children, number of crossing dependencies). Our empirical results indicate that the reported gains depend heavily on the experiment setup and the choice of random seed, although we can confirm the trend of stable gains with 128-Dyck pretraining of small models with the Llama tokenizer for most of the examined languages. On a general note, we argue that multiple training runs should be carried out at least for a subset of experiments to avoid the community adopting unstable approaches.

Sofiia Riazhskykh, Nam Luu, Ondřej Bojar

Charles University, Faculty of Mathematics and Physics

Language Model PretrainingLinguistic TypologyLLM Training
AskPDF
Sep 30, 2026

Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical prior, i.e., a structural inductive bias learned during PPT that transfers to natural language grammar. However, PPT has only been tested on models of at most 1B parameters and PT budgets below 2B tokens on predominantly web text. It is unknown whether PPT is effective at larger scales and under more realistic PT data mixtures that combine diverse sources (e.g., code and math). We therefore present a comprehensive study on PPT spanning five PPT tasks, four PT data mixtures, four parameter scales (500M to 7B), and PT budgets of up to 100B tokens. Our results demonstrate that the downstream performance and token efficiency gains of PPT persist at scale, e.g., saving at least 21B PT tokens at the 3B scale. However, in contrast to prior work, we find no consistent evidence that these gains stem from a grammatical prior. Downstream performance does not consistently align with grammatical acceptability across model sizes. Instead, we find that downstream gains arise from PPT tasks that improve long-range retrieval. Finally, PPT performance gains are robust to how PT data mixtures are composed and diminish only when web text is absent. Overall, PPT is a low-cost addition to PT, and future PPT task design should target long-range retrieval rather than natural language grammar.

Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus +3

University of Sheffield, UK · Mohamed bin Zayed University of Artificial Intelligence, UAE · Hugging Face +3

Language Model PretrainingSynthetic Data PretrainingLinguistic Bias in Language Models
AskPDF