cs.LGMar 9, 2026

Training Language Models via Neural Cellular Automata

Authors: Dan Lee, Seungwook Han, Akarsh Kumar, Pulkit Agrawal

Organizations: Independent Contributor · MIT · Improbable AI Lab

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Pre-training is crucial for large language models (LLMs), as it is when most representations and capabilities are acquired. However, natural language pre-training has problems: high-quality text is finite, it contains human biases, and it entangles knowledge with reasoning. This raises a fundamental question: is natural language the only path to intelligence? We propose using neural cellular automata (NCA) to generate synthetic, non-linguistic data for pre-pre-training LLMs--training on synthetic-then-natural language. NCA data exhibits rich spatiotemporal structure and statistics resembling natural language while being controllable and cheap to generate at scale. We find that pre-pre-training on only 164M NCA tokens improves downstream language modeling by up to 6% and accelerates convergence by up to 1.6x. Surprisingly, this even outperforms pre-pre-training on 1.6B tokens of natural language from Common Crawl with more compute. These gains also transfer to reasoning benchmarks, including GSM8K, HumanEval, and BigBench-Lite. Investigating what drives transfer, we find that attention layers are the most transferable, and that optimal NCA complexity varies by domain: code benefits from simpler dynamics, while math and web text favor more complex ones. These results enable systematic tuning of the synthetic distribution to target domains. More broadly, our work opens a path toward more efficient models with fully synthetic pre-training.

Figures & tables

Figure 1: Overview of NCA Pre-pre-training to Language Pre-training. We pre-pre-train a transformer with next-token prediction on the dynamics of neural cellular automata (NCA) sampled from selected complexity regions. We then conduct standard pre-training on natural language corpora. NCA pre-pre-training improves both validation perplexity and convergence speed on language pre-training. Interestingly, the optimal NCA distribution varies by downstream domain.
Figure 2: NCA pre-pre-training improves and accelerates language model pre-training across diverse domains. We show the validation perplexity during pre-training on ( a ) OpenWebText, ( b ) OpenWebMath, and ( c ) CodeParrot for 1.6B parameter models. Models pre-pre-trained on NCA trajectories consistently outperform the scratch, Dyck pre-pre-training, and surprisingly even C4 pre-pre-training baselines. NCA pre-pre-training achieves 1.4–1.6 ×\times faster convergence to the scratch baseline’s final perplexity while also reaching up to 6% lower final perplexity. We provide a zoomed-in training curve of the last third of training for clarity.
Figure 3: NCA pre-pre-training improves language model training performance across model sizes (Section 5.1 ). We report the final validation perplexity after pre-training on OpenWebText across (400M, 600M, and 1.6B parameter models). At 164M tokens, C4 pre-pre-training likely acquires shallow syntactic patterns that interfere with downstream learning rather than transferable structure. We investigate this further in Figure 4 .
Figure 4: Pre-pre-training on 160M tokens of NCA is better than pre-pre-training on 1.6B tokens of natural language (C4). We report the validation perplexity during pre-training on OpenWebText. Perplexity improvement is calculated relative to the C4 pre-pre-trained model. We add a version where we also preserve the embedding layers from pre-pre-training to pre-training (1.6B tokens w/o embedding reinit). Surprisingly even with the embedding layers, NCA pre-pre-training is better.
Figure 5: Attention weights are most crucial for positive transfer. We report the change in validation perplexity when selectively re-initializing model components after NCA pre-pre-training, relative to full transfer. Higher means the component is more important for transfer. Re-initializing attention causes the largest degradation across both OpenWebText and CodeParrot, while MLP and LayerNorm effects are domain-dependent.
Figure 6: Optimal NCA complexity is domain-dependent. We report the validation perplexity change of models trained with different NCA complexities from the scratch model. OpenWebText benefits from higher-complexity data (50%+), while CodeParrot peaks at intermediate complexity (30–40%). This suggests that matching synthetic data complexity to the target domain is necessary to maximize transfer.
Figure 7: NCA alphabet size changes scaling behavior. We report the perplexity change relative to the scratch model (higher is better) on OpenWebText across different alphabet sizes nn and NCA pre-pre-training token budgets. NCA improves perplexity at all token budgets. The smaller alphabet scales better, and the improvements degrade with scale for larger alphabets.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: NCA data exhibits a similar Zipfian or power-law structure to natural language. We compare the relative token frequency distribution for each of the natural language corpora and NCA data. Natural language from different domains has different average complexity as measured by gzip compressibility (see legend).
Figure 9: Different NCA alphabet sizes, n=2,10,15n=2,10,15 naturally yield different complexity distributions of the data. Increasing nn inherently increases the complexity of the data.
HyperparameterPre-pre-trainingPre-training
Effective batch size16512
Sequence length1024 tokens1024 tokens
Learning rate1×10−41\times 10^{-4}5×10−45\times 10^{-4} (Math/Text), 2×10−42\times 10^{-4} (Code)
LR scheduleCosine w/ warmupCosine w/ warmup
Warmup steps (% total)10%10%
Weight decayNone1×10−41\times 10^{-4}
Appendix
Table 2: Hyperparameters for pre-pre-training and pre-training experiments.
More in this topic

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 441 papers

Explore similar work

CardsList

  1. Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data

    May 11, 2026Xu Guo, Runyu Peng, Jian Tong +4Language Model PretrainingSynthetic Data Pretraining

    Shanghai AI Laboratory · Fudan University · Shanghai Innovation Institute

  2. Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages

    Aug 9, 2026Sofiia Riazhskykh, Nam Luu, Ondřej BojarLanguage Model PretrainingLinguistic Typology

    Charles University, Faculty of Mathematics and Physics

  3. Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

    Sep 30, 2026Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus +3Language Model PretrainingSynthetic Data Pretraining

    University of Sheffield, UK · Mohamed bin Zayed University of Artificial Intelligence, UAE · Hugging Face +3