Code-trained language models produce positive brain scores, which raises the question of whether abstract structure learned from programming languages can substitute for natural-language training. We trained 25 GPT-2-small models on TinyStories, Python, C functions, x86-64 assembly, and the corresponding machine code, and scored them using a shared tokenizer and encoding procedure against matched random-weight models. On Pereira-243 and Tuckute2024, TinyStories exceeds the mean of the four code models by Δr=0.053 and 0.059 (Holm-corrected p<0.001), while code training leaves alignment near the random-weight level. The language advantage is benchmark-specific: Blank2014 shows no corrected language-code difference, and Fedorenko2016 shows no language benefit. Geometry tracks the encoding result on Pereira-243: TinyStories seeds converge on the most similar representations, and its fit-free correspondence with neural dissimilarity peaks at ρ=0.14, twice the code and random-weight levels. Binary-trained models have about three effective dimensions, so seed agreement alone does not imply neural correspondence. Updating only the embedding matrix on English improves mean-over-depth alignment for every code model more than an equal update on its own corpus. Within this architecture and training setup, natural-language training produces a distinct representational geometry with closer neural correspondence on Pereira-243
Figures & tables
Figure 1: Design. Five corpora share one tokenizer and one GPT-2-small recipe, yielding 25 trained models (five seeds per corpus); C, assembly, and binary share the same functions across three representations. The matched reference comprises three random-weight models for encoding and two for geometry. Every model is read at all twelve blocks on four benchmark families, with Pereira contributing two stimulus sets.
Benchmark
Random
Code
TinyStories
TS − rand.
Code − rand.
TS − code
Pereira-243
0.064
0.069
0.122
+0.057
+0.004
+0.053
Tuckute2024
0.274
0.266
0.325
+0.051
− 0.007
+0.059
Blank2014
0.040
0.049
0.054
+0.014
+0.009
+0.005
Fedorenko2016
0.175
0.170
0.170
− 0.004
− 0.005
+0.000
Table 1: Mean-over-depth encoding scores (raw Pearson r averaged over the twelve blocks, five seeds per trained condition, three random initializations) and the three prespecified contrasts. “Code” is the equally weighted mean of Python, C, assembly, and binary. Bold contrasts survive Holm correction over all twelve tests, with random-initialization variance included. Intervals and p -values are in Table 6 .
Figure 2: Pereira-243. (a) Encoding score at each block: mean of five seeds per corpus, with ±1 SD across seeds for TinyStories; the gray band spans the three random initializations and the dotted line is their mean. (b) Mean over twelve blocks for each model (points; bars are condition means). The dotted line is the mean. Seeds are the replicates; folds do not enter the uncertainty.
Figure 3: Representational geometry on Pereira-243 stimuli (cumulative topic context, final token). (a) Mean linear CKA between seeds of the same corpus. (b) Participation ratio, log scale; bands span the seeds. (c) Brain-model RSA (Spearman ρ between dissimilarity matrices); lines are seed means and bands span the seeds. Five seeds per corpus and two random initializations; all panels are descriptive.
Figure 4: Interventions on Pereira-243, one donor (TinyStories, seed 42) and one receiver checkpoint per code corpus. (a) Encoding r at block 9 after replacing the receiver’s first k blocks with the donor’s; k=0 is the untouched receiver and k=12 also copies the final normalization. (b) The same block transplants, but also copies the donor’s token and position embeddings; k=12 is the donor. (c) Change in mean-over-depth r after updating only the tied embedding matrix on the receiver’s own corpus or on TinyStories (equal budgets).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Corpus
Unit
Stored representation
Rows
% unique
% val. in train
% truncated
TinyStories
story
English prose
2.1M
85
27.0
0.0
Python
file
Python, AST-normalized
165k
36
72.8
24.1
C
C function
C source
2.8M
100
0.0
2.4
Assembly
function
x86-64 AT&T assembly
2.8M
96
6.7
20.7
Binary (hex)
function
hex text of machine code
2.8M
73
32.1
8.1
Appendix
Table 2: Corpus provenance measured on dataset revision cc83566 . “% val. in train”: share of validation documents (seed-43 split) whose exact text also appears in the training documents. “% truncated”: documents longer than the 1,024-token limit, which training cuts to their first 1,024 tokens.
Corpus
Documents
% truncated
Tokens kept
Train sequences
Logged runs
Passes (logged)
TinyStories
2.1M
0.0
463M
406.1-408.0k
4/5
1.18
Python
165k
24.1
80.4M
70.6-70.8k
4/5
6.78-6.79
C
2.8M
2.4
547M
479.6-482.2k
4/5
1.00
Assembly
2.8M
20.7
1.54B
1350.8-1353.4k
4/5
0.35
Binary (hex)
2.8M
8.1
900M
789.5-792.2k
4/5
0.61
Appendix
Table 3: Realized training exposure. Tokens kept: corpus size after per-document truncation. Training sequences: range over the five seeds. Passes: processed tokens at the evaluated checkpoint divided by the training-split size, for the four logged runs. The full-split token counts in Table 4 overstate the available data for every corpus except TinyStories.
Corpus
All tokens
Train split
Passes at 30k
Type cov.
Tok./byte
Peak r
BLiMP
TinyStories
463M
416M
1.18
0.99
0.24
0.154
0.639
Python
175M
72.3M
6.78-6.80
0.98
0.32
0.098
0.533
C
588M
491M
1.00
0.99
0.36
0.095
0.554
Assembly
2.22B
1.38B
0.35-0.36
0.98
0.52
0.081
0.537
Binary (hex)
1.10B
808M
0.61
0.01
0.44
0.085
0.528
Appendix
Table 4: Corpus diagnostics. All tokens: the full public split before truncation. Train split: training-split tokens after truncation and packing (smallest seed). Passes at 30k: pass equivalents at update 30,000 over the five seeds. Type cov.: share of Pereira stimulus word types present. Peak r : maximum over blocks of the seed-42 model on Pereira-243 under the benchmark’s own aggregation (median over neuroids), a selected descriptive value. BLiMP: accuracy of the seed-42 model (chance 0.5).
Benchmark
Modality, task
Unit
Units
Targets
Subj.
Model context
Pereira-243
fMRI, sentence reading
sentence
243
8,031
6
topic-cumulative
Pereira-384
fMRI, sentence reading
sentence
384
12,155
9
passage-cumulative
Tuckute2024
fMRI, sentence reading
sentence
1,000
1
5 (avg.)
isolated sentence
Blank2014
fMRI, story listening
fMRI time point
1,317
60
5
story-cumulative
Fedorenko2016
ECoG, sentence reading
word
416
97
5
sentence-cumulative
Appendix
Table 5: Benchmarks as loaded by the pinned implementation. Units: presentations scored (stimuli or time points). Targets: response dimensions (voxels, functional regions, electrodes, or one averaged response). Subj.: participants contributing to the loaded data (averaged for Tuckute2024).
Benchmark
Contrast
Estimate
95% CI
df
p
Holm p
Pereira-243
L-R
+0.057
[0.022, 0.092]
2.3
0.018
0.145
C-R
+0.004
[ − 0.032, 0.040]
2.2
0.675
1.000
L-C
+0.053
[0.046, 0.060]
9.9
1.89×10−8
2.26×10−7
Tuckute2024
L-R
+0.051
[0.017, 0.085]
3.4
0.016
0.145
C-R
− 0.007
[ − 0.046, 0.031]
2.4
0.534
1.000
L-C
+0.059
[0.043, 0.074]
6.2
7.51×10−5
8.26×10−4
Appendix
Table 6: The twelve primary contrasts on mean-over-depth scores. L: TinyStories; C: code mean; R: random weights. Intervals are unadjusted 95% Welch intervals; Holm correction covers all twelve rows, and bold marks Holm p<0.05 .
Figure 5: The twelve primary contrasts (Table 6 ). Lines are unadjusted 95% intervals; filled markers survive Holm correction over the twelve tests, so an interval excluding zero does not by itself imply a corrected decision. Zero means no change relative to the comparison condition, not zero alignment.
Figure 6: Depth-resolved replication. (a) TinyStories minus the code mean at each block on both Pereira sets; filled markers are significant after Holm correction over blocks (Welch t , 5 TinyStories versus 20 code models). (b, c) Code corpora: mean ±1 SD over five seeds; asterisks mark blocks with significant heterogeneity among the four code corpora (one-way ANOVA, Holm over blocks).
Figure 7: Change relative to the random-weight mean at each block on the four benchmark families (raw r ; mean of five seeds per corpus). The gray band spans the three random initializations around their mean. Vertical scales differ between benchmarks.
Figure 8: Corpus diagnostics against the Pereira-243 mean over blocks (points: mean of five seeds; error bars: range over seeds). (a) Share of stimulus word types present in the corpus. (b) Mean corpus count per stimulus token (log scale). (c) BLiMP accuracy of the seed-42 model; the dotted vertical line is chance. The dashed line is the matched random-weight reference (mean of three initializations).
Figure 9: Text manipulations, GPT-2 small, Pereira-243, mean of blocks 0, 6 and 11. (a) Intact versus pseudoword-relabeled TinyStories during training; small markers are seeds (offset horizontally for visibility), lines join seed means. (b) Endpoint at 500M tokens for the four factorial cells and the form control; points are seeds and bars are means. Three seeds per condition; descriptive.
Figure 10: Single-part transplants from the TinyStories donor into each code receiver (Pereira-243). Cells show the change in r for each readout block (rows) after replacing the embeddings (E) or a single block (columns); gray cells appear after the readout and cannot be changed. Outlined cells differ from the receiver after Holm correction over patch positions, using fold-paired tests that are descriptive rather than seed-level inference.
Receiver
Peak r
Own Δ
English Δ
Depth r
Own Δ
English Δ
Python
0.098
+0.000
+0.014
0.066
+0.002
+0.031
C
0.095
+0.005
+0.018
0.072
+0.008
+0.017
Assembly
0.081
− 0.010
+0.025
0.057
− 0.004
+0.032
Binary (hex)
0.085
+0.001
+0.033
0.063
+0.001
+0.031
Mean
− 0.001
+0.022
+0.002
+0.028
Paired t(3)
4.06, p=0.027
4.31, p=0.023
Appendix
Table 7: Embedding-only adaptation. Peak: each model’s maximum over blocks under the benchmark’s aggregation (selected, descriptive; the receiver’s and the adapted model’s peaks can occur at different blocks). Depth: mean over twelve blocks. Δ : change from the untouched receiver. Paired t tests compare English and own-corpus changes across the four receivers.
Model
n
Seed CKA
Peak RSA
Block
PR b0
PR max
PR b11
TinyStories
5
0.899
0.139
6
1.7
26.0
23.9
Python
5
0.714
0.064
1
3.6
40.9
40.6
C
5
0.811
0.070
2
1.8
30.0
28.6
Assembly
5
0.687
0.065
1
2.1
19.2
18.5
Binary (hex)
5
0.859
0.057
0
2.0
3.3
2.9
Random weights
2
0.674
0.072
3
83.2
83.2
61.8
Appendix
Table 8: Geometry on Pereira-243. Seed CKA: mean CKA between seeds of a corpus over blocks 4-11. Peak RSA: maximum seed-mean Spearman ρ over blocks, and its block. PR: participation ratio (seed mean) at block 0, its maximum over blocks, and block 11.
Block
0
2
4
8
11
TinyStories within − TinyStories-to-code
0.055
0.130
0.394
0.410
0.385
All corpora within − between
0.060
0.118
0.317
0.337
0.333
Appendix
Table 9: Corpus separation in CKA. First row: mean CKA among TinyStories seeds minus mean CKA between TinyStories and code models. Second row: mean within-corpus CKA minus mean between-corpus CKA over all five corpora.
Model
None removed
1 PC
2 PCs
3 PCs
TinyStories
0.075
0.061
0.057
0.042
Python
0.049
0.039
0.033
0.030
C
0.047
0.028
0.030
0.029
Assembly
0.048
0.030
0.027
0.023
Binary (hex)
0.048
0.040
0.022
0.024
Random weights
0.042
0.025
0.024
0.026
Appendix
Table 10: RSA after removing leading principal components (Pereira-243). Representations are column-centered before removing the components, which lowers the unablated values relative to Table 8 ; each value is the mean over seeds of each model’s maximum over blocks.
Figure 11: One-checkpoint spectrum audit (seed 42, isolated sentences, five grouped outer folds). (a) Out-of-fold r through depth; lines are fold means and bands span the five folds (descriptive, not a confidence interval). (b) Cumulative variance explained at block 11. (c) Spearman correlation across blocks between four spectral summaries and alignment.