Language, Not Code Abstraction: Natural-language Training Shapes Brain-aligned Geometry
Organizations: Department of Linguistics, University of California, Berkeley, USA
Abstract
Code-trained language models produce positive brain scores, which raises the question of whether abstract structure learned from programming languages can substitute for natural-language training. We trained 25 GPT-2-small models on TinyStories, Python, C functions, x86-64 assembly, and the corresponding machine code, and scored them using a shared tokenizer and encoding procedure against matched random-weight models. On Pereira-243 and Tuckute2024, TinyStories exceeds the mean of the four code models by and (Holm-corrected ), while code training leaves alignment near the random-weight level. The language advantage is benchmark-specific: Blank2014 shows no corrected language-code difference, and Fedorenko2016 shows no language benefit. Geometry tracks the encoding result on Pereira-243: TinyStories seeds converge on the most similar representations, and its fit-free correspondence with neural dissimilarity peaks at , twice the code and random-weight levels. Binary-trained models have about three effective dimensions, so seed agreement alone does not imply neural correspondence. Updating only the embedding matrix on English improves mean-over-depth alignment for every code model more than an equal update on its own corpus. Within this architecture and training setup, natural-language training produces a distinct representational geometry with closer neural correspondence on Pereira-243
Figures & tables
| Benchmark | Random | Code | TinyStories | TS rand. | Code rand. | TS code |
|---|---|---|---|---|---|---|
| Pereira-243 | 0.064 | 0.069 | 0.122 | +0.057 | +0.004 | +0.053 |
| Tuckute2024 | 0.274 | 0.266 | 0.325 | +0.051 | 0.007 | +0.059 |
| Blank2014 | 0.040 | 0.049 | 0.054 | +0.014 | +0.009 | +0.005 |
| Fedorenko2016 | 0.175 | 0.170 | 0.170 | 0.004 | 0.005 | +0.000 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Corpus | Unit | Stored representation | Rows | % unique | % val. in train | % truncated |
|---|---|---|---|---|---|---|
| TinyStories | story | English prose | 2.1M | 85 | 27.0 | 0.0 |
| Python | file | Python, AST-normalized | 165k | 36 | 72.8 | 24.1 |
| C | C function | C source | 2.8M | 100 | 0.0 | 2.4 |
| Assembly | function | x86-64 AT&T assembly | 2.8M | 96 | 6.7 | 20.7 |
| Binary (hex) | function | hex text of machine code | 2.8M | 73 | 32.1 | 8.1 |
| Corpus | Documents | % truncated | Tokens kept | Train sequences | Logged runs | Passes (logged) |
|---|---|---|---|---|---|---|
| TinyStories | 2.1M | 0.0 | 463M | 406.1-408.0k | 4/5 | 1.18 |
| Python | 165k | 24.1 | 80.4M | 70.6-70.8k | 4/5 | 6.78-6.79 |
| C | 2.8M | 2.4 | 547M | 479.6-482.2k | 4/5 | 1.00 |
| Assembly | 2.8M | 20.7 | 1.54B | 1350.8-1353.4k | 4/5 | 0.35 |
| Binary (hex) | 2.8M | 8.1 | 900M | 789.5-792.2k | 4/5 | 0.61 |
| Corpus | All tokens | Train split | Passes at 30k | Type cov. | Tok./byte | Peak | BLiMP |
|---|---|---|---|---|---|---|---|
| TinyStories | 463M | 416M | 1.18 | 0.99 | 0.24 | 0.154 | 0.639 |
| Python | 175M | 72.3M | 6.78-6.80 | 0.98 | 0.32 | 0.098 | 0.533 |
| C | 588M | 491M | 1.00 | 0.99 | 0.36 | 0.095 | 0.554 |
| Assembly | 2.22B | 1.38B | 0.35-0.36 | 0.98 | 0.52 | 0.081 | 0.537 |
| Binary (hex) | 1.10B | 808M | 0.61 | 0.01 | 0.44 | 0.085 | 0.528 |
| Benchmark | Modality, task | Unit | Units | Targets | Subj. | Model context |
|---|---|---|---|---|---|---|
| Pereira-243 | fMRI, sentence reading | sentence | 243 | 8,031 | 6 | topic-cumulative |
| Pereira-384 | fMRI, sentence reading | sentence | 384 | 12,155 | 9 | passage-cumulative |
| Tuckute2024 | fMRI, sentence reading | sentence | 1,000 | 1 | 5 (avg.) | isolated sentence |
| Blank2014 | fMRI, story listening | fMRI time point | 1,317 | 60 | 5 | story-cumulative |
| Fedorenko2016 | ECoG, sentence reading | word | 416 | 97 | 5 | sentence-cumulative |
| Benchmark | Contrast | Estimate | 95% CI | df | Holm | |
|---|---|---|---|---|---|---|
| Pereira-243 | L-R | +0.057 | [0.022, 0.092] | 2.3 | 0.018 | 0.145 |
| C-R | +0.004 | [ 0.032, 0.040] | 2.2 | 0.675 | 1.000 | |
| L-C | +0.053 | [0.046, 0.060] | 9.9 | |||
| Tuckute2024 | L-R | +0.051 | [0.017, 0.085] | 3.4 | 0.016 | 0.145 |
| C-R | 0.007 | [ 0.046, 0.031] | 2.4 | 0.534 | 1.000 | |
| L-C | +0.059 | [0.043, 0.074] | 6.2 |
| Receiver | Peak | Own | English | Depth | Own | English |
|---|---|---|---|---|---|---|
| Python | 0.098 | +0.000 | +0.014 | 0.066 | +0.002 | +0.031 |
| C | 0.095 | +0.005 | +0.018 | 0.072 | +0.008 | +0.017 |
| Assembly | 0.081 | 0.010 | +0.025 | 0.057 | 0.004 | +0.032 |
| Binary (hex) | 0.085 | +0.001 | +0.033 | 0.063 | +0.001 | +0.031 |
| Mean | 0.001 | +0.022 | +0.002 | +0.028 | ||
| Paired | 4.06, | 4.31, |
| Model | Seed CKA | Peak RSA | Block | PR b0 | PR max | PR b11 | |
|---|---|---|---|---|---|---|---|
| TinyStories | 5 | 0.899 | 0.139 | 6 | 1.7 | 26.0 | 23.9 |
| Python | 5 | 0.714 | 0.064 | 1 | 3.6 | 40.9 | 40.6 |
| C | 5 | 0.811 | 0.070 | 2 | 1.8 | 30.0 | 28.6 |
| Assembly | 5 | 0.687 | 0.065 | 1 | 2.1 | 19.2 | 18.5 |
| Binary (hex) | 5 | 0.859 | 0.057 | 0 | 2.0 | 3.3 | 2.9 |
| Random weights | 2 | 0.674 | 0.072 | 3 | 83.2 | 83.2 | 61.8 |
| Block | 0 | 2 | 4 | 8 | 11 |
|---|---|---|---|---|---|
| TinyStories within TinyStories-to-code | 0.055 | 0.130 | 0.394 | 0.410 | 0.385 |
| All corpora within between | 0.060 | 0.118 | 0.317 | 0.337 | 0.333 |
| Model | None removed | 1 PC | 2 PCs | 3 PCs |
|---|---|---|---|---|
| TinyStories | 0.075 | 0.061 | 0.057 | 0.042 |
| Python | 0.049 | 0.039 | 0.033 | 0.030 |
| C | 0.047 | 0.028 | 0.030 | 0.029 |
| Assembly | 0.048 | 0.030 | 0.027 | 0.023 |
| Binary (hex) | 0.048 | 0.040 | 0.022 | 0.024 |
| Random weights | 0.042 | 0.025 | 0.024 | 0.026 |
LLM Training
Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.
See all 376 papers