Small language models are often trained under tight data budgets, where the choice of pretraining corpus may matter as much as model size or training recipe. We compare two contrasting corpora at matched 10-million- and 37-million-word budgets: the 1911 Encyclopaedia Britannica and the BabyLM 2026 strict pretraining corpus. For each corpus, we train teacher models, distilled 58-million-parameter Baby-Llama students, and no-distillation student ablations, using the same model families, training recipes, and evaluation protocol across corpora. Across BLiMP, BLiMP Supplement, EWoK, Entity Tracking, and COMPS, the corpora produce different skill profiles rather than a single quality ranking. BabyLM 2026 is consistently stronger on linguistic acceptability benchmarks, while Britannica is often stronger on Entity Tracking, a synthetic state-tracking task. The contrast appears across teacher and student models and at both budgets, with the cleanest cross-seed replication in the distilled student comparison. Targeted controls show that the central contrast persists after matching training exposure and at a longer diagnostic context; direct teacher ensembling also does not reproduce the 10M Britannica student's Entity Tracking behavior. Entity Tracking magnitudes nevertheless remain sensitive to training dynamics. For this corpus pair, corpus choice can therefore produce different relative skill profiles at small scale, motivating multi-benchmark evaluation when comparing pretraining data.
Figures & tables
Statistic
Brit. 10M
Brit. 37M
BabyLM 10M
BabyLM 37M
Train documents
9,181
34,295
1.17M
4.30M
Mean document words
1,089
1,079
8.6
8.6
Median document words
318
318
5
5
Mean sentence words
20.3
20.2
7.5
7.5
Word types
239k
532k
170k
344k
Tokenizer tokens / word
1.42
1.42
1.95
1.95
Table 1: Corpus properties for the matched 10M- and 37M-word training splits. Britannica is a single-source formal encyclopedia corpus; BabyLM 2026 is a heterogeneous mixture of child-directed speech, dialogue, simple Wikipedia, transcribed speech, and other text. Sentences are segmented on terminal punctuation. Token counts use each track’s corpus-specific tokenizer and include document-boundary tokens.
Llama
GPT-2
GPT-J
Layers
24
24
24
Attention heads
8
16
16
Hidden dimension
1024
1536
1536
Intermediate dimension
3072
6144
6144
Parameters
360M
705M
705M
Epochs
4
6
4
Table 2: Teacher architectures and training recipes. All teachers share a cosine learning-rate schedule with 300 warm-up steps. GPT-J is trained only at the 10M budget.
Figure 1: Skill profiles of the seed-2023 experimental sweep on five zero-shot metrics. Each panel shows one metric; bars give the mean accuracy per (corpus, budget) combination, and circles mark individual models (3 teachers + 3 students at 10M; 2 teachers + 2 students at 37M). The Britannica corpus produces higher Entity Tracking in the main matched comparisons; the BabyLM 2026 corpus produces higher BLiMP, BLiMP Supplement, and (slightly) COMPS; EWoK is essentially tied. Table 3 reports the paired effects for both training seeds.
Δ BLiMP
Δ Supp
Δ Entity
Budget
Matched model family
s2023
s43
s2023
s43
s2023
s43
10M
Llama-360M teacher
+ 8.19
+ 7.97
+ 7.65
+ 10.46
− 8.38
− 8.05
10M
GPT-2-705M teacher
+ 7.73
+ 8.19
+ 7.78
+ 5.21
− 8.02
− 16.49
10M
GPT-J-705M teacher
+ 5.99
+ 7.22
+ 11.96
+ 5.53
− 12.29
− 4.59
10M
Baby-Llama-58M student (KD, Llama+GPT-2)
+ 7.07
+ 8.19
+ 10.23
+ 11.44
− 22.27
− 17.00
10M
Baby-Llama-58M student (KD, Llama+GPT-J)
+ 6.91
+ 6.41
+ 10.15
+ 11.75
− 22.58
− 5.49
Table 3: Paired corpus effects under both training seeds. Deltas are BabyLM 2026 minus Britannica for the same model family and source-word budget. Positive BLiMP/Supplement values favor BabyLM 2026; negative Entity Tracking values favor Britannica.
Model
BLiMP
Supp
EWoK
Entity
COMPS
Britannica 10M
Baby-Llama-58M (KD, Llama+GPT-2)
64.69
46.60
49.81
37.10
51.23
Baby-Llama-58M (KD, Llama+GPT-J)
64.19
46.27
51.02
37.26
51.21
Baby-Llama-58M (no KD)
63.01
47.45
50.84
26.11
50.63
BabyLM 2026 10M
Baby-Llama-58M (KD, Llama+GPT-2)
71.76
56.83
50.08
14.83
52.00
Table 4: Seed-2023 zero-shot evaluation of 58M students at matched 10M-word budget on the Britannica and BabyLM 2026 corpora. KD labels name the two-teacher ensemble used for distillation; “no KD” is the matched non-distilled baseline. Bold marks the best entry per metric across both corpora.
Corpus
Model
BLiMP
Supp
EWoK
Entity
COMPS
Britannica 37M
KD, Llama+GPT-2
65.95
51.08
50.36
21.80
53.42
Britannica 37M
no KD
64.66
47.87
50.58
22.22
53.27
BabyLM 2026 37M
KD, Llama+GPT-2
76.27
56.82
51.24
17.10
54.66
BabyLM 2026 37M
no KD
74.57
56.47
50.93
16.73
54.18
Table 5: Seed-2023 zero-shot evaluation of 58M Baby-Llama students at the larger 37M-word budget. KD rows use the Llama+GPT-2 two-teacher distillation ensemble; no-KD rows use the same architecture and training schedule without teacher logits. Bold marks the best entry per metric across both corpora.
Corpus
Δ BLiMP
Δ Supp
Δ EWoK
Δ Entity
Δ COMPS
Britannica 10M
+1.68
-0.85
-1.03
+10.99
+0.60
BabyLM 2026 10M
+1.87
-0.57
-0.89
-8.31
+0.32
Britannica 37M
+1.29
+3.21
-0.22
-0.42
+0.15
BabyLM 2026 37M
+1.70
+0.35
+0.31
+0.37
+0.48
Table 6: Effect of knowledge distillation on the 58M student, expressed as KD score minus no-KD score on the same corpus and word budget. Positive values indicate that distillation helps.
Seed
Budget
Teacher pair / context
Teacher ensemble
KD student
Student − Ensemble
2023
10M
Llama+GPT-2 @ 128 tokens
25.15
37.10
+ 11.95
43
10M
Llama+GPT-2 @ 128 tokens
29.15
35.58
+ 6.43
2023
10M
Llama+GPT-2 @ 256 tokens
15.83
38.96
+ 23.13
2023
10M
Llama+GPT-J @ 128 tokens
31.29
37.26
+ 5.97
2023
10M
Llama+GPT-J @ 256 tokens
26.94
37.71
+ 10.77
2023
37M
Llama+GPT-2 @ 128 tokens
23.85
21.80
− 2.05
Table 7: Direct ensemble comparison for the Britannica Entity Tracking models. All entries are macro averages: accuracy is first computed for each benchmark subtask (UID group) and then averaged. For the ensemble, each candidate completion is scored by the per-token logit mean of the two Britannica-trained teachers used during KD training. Main rows use seed 2023; the seed-43 row repeats the default 10M Llama+GPT-2 check. At 10M, the student exceeds the ensemble in every configuration; at 37M, the ensemble exceeds the student at both context lengths.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Corpus
Model
BLiMP
Supp
EWoK
Entity
COMPS
Eye
SPR
Britannica 10M
Llama-360M teacher
61.45
48.81
50.35
26.13
50.69
0.35
0.01
Britannica 10M
GPT-2-705M teacher
62.61
46.83
51.04
25.38
51.06
0.12
0.01
Britannica 10M
GPT-J-705M teacher
62.43
44.93
50.38
36.54
50.78
0.25
0.03
Britannica 10M
Baby-Llama-58M student (KD, Llama+GPT-2)
64.69
46.60
49.81
37.10
51.23
0.13
0.01
Britannica 10M
Baby-Llama-58M student (KD, Llama+GPT-J)
64.19
46.27
51.02
37.26
51.21
0.17
0.00
Britannica 10M
Baby-Llama-58M student (no KD)
63.01
47.45
50.84
26.11
50.63
0.26
0.07
Appendix
Table 8: Full zero-shot results for the matched 10M-word corpus comparisons. KD student labels name the two-teacher ensemble used for distillation.
Corpus
Model
BLiMP
Supp
EWoK
Entity
COMPS
Eye
SPR
Britannica 37M
Llama-360M teacher
65.52
51.22
50.80
19.34
53.11
0.41
0.02
Britannica 37M
GPT-2-705M teacher
64.86
50.32
51.07
30.06
52.85
0.39
0.01
Britannica 37M
Baby-Llama-58M student (KD, Llama+GPT-2)
65.95
51.08
50.36
21.80
53.42
0.31
0.02
Britannica 37M
Baby-Llama-58M student (no KD)
64.66
47.87
50.58
22.22
53.27
–
–
BabyLM 2026 37M
Llama-360M teacher
74.28
54.93
50.78
17.00
53.86
0.24
0.01
BabyLM 2026 37M
GPT-2-705M teacher
75.02
59.19
51.93
16.36
54.82
0.19
0.02
Appendix
Table 9: Full zero-shot results for the matched 37M-word corpus comparisons. Reading scores for the 37M no-KD runs were not collected; these models were evaluated only on the five core metrics.
Corpus
Model
ET @ 128
ET @ 256
Δ
Britannica 10M
Llama-360M teacher
26.13
23.00
− 3.13
Britannica 10M
GPT-2-705M teacher
25.38
13.48
− 11.90
Britannica 10M
GPT-J-705M teacher
36.54
32.08
− 4.46
Britannica 10M
Baby-Llama-58M (Llama+GPT-2)
37.10
38.96
+ 1.86
Britannica 10M
Baby-Llama-58M (Llama+GPT-J)
37.26
37.71
+ 0.45
Britannica 10M
Baby-Llama-58M (no KD)
26.11
23.34
− 2.77
Appendix
Table 10: Entity Tracking accuracy at the 128-token (tokenizer-declared) and 256-token (position-embedding maximum) context lengths. The 256-token setting is a diagnostic rather than the main protocol because all models were trained on 128-token slices. Most student rows move modestly, and the Britannica-vs-BabyLM ranking holds in every matched row at both context lengths. Some teacher rows, especially the 10M GPT-2 rows with learned absolute position embeddings, are more sensitive to the diagnostic setting.
Corpus
Budget
Source words
Boundary markers
BPE train tokens
Tokens / word
Britannica
10M
10,000,066
18,362
14,216,855
1.42
BabyLM 2026
10M
10,000,092
2,331,378
19,518,278
1.95
Britannica
37M
37,002,217
68,590
52,559,454
1.42
BabyLM 2026
37M
37,002,217
8,605,126
72,172,588
1.95
Appendix
Table 11: Training-token counts after corpus-specific BPE tokenization. The matched-word design follows the BabyLM convention, but it does not match the number of BPE tokens seen per epoch. In these splits, BabyLM 2026 yields about 1.37 × as many training tokens as Britannica at both budgets, reflecting both corpus-specific BPE segmentation and BabyLM’s many short source segments, which introduce far more document-boundary markers.
BabyLM 2026 model
Steps
Eff. ep.
BLiMP
Supp
EWoK
Entity
COMPS
Llama-360M teacher
3,472
2.91
68.91
57.59
50.60
20.52
51.51
GPT-2-705M teacher
2,604
4.37
69.14
53.66
50.12
14.24
51.65
Baby-Llama-58M student (KD)
20,826
4.37
70.22
56.39
50.95
16.40
51.66
Baby-Llama-58M student (no KD)
20,826
4.37
68.82
56.36
50.40
27.39
51.64
Appendix
Table 12: Seed-2023 zero-shot results for the BabyLM 2026 10M matched-exposure control. Total optimizer steps and approximately total 128-token sequence presentations match the corresponding Britannica models.
Split
Train words
Train docs
Val words
Val docs
Source rows consumed
BabyLM 2026 10M
10,000,092
1,165,689
2,027,481
236,073
1,401,762
BabyLM 2026 37M
37,002,217
4,302,563
1,003,428
116,690
4,419,253
Appendix
Table 13: BabyLM 2026 strict split manifests. Both splits are deterministic random document-level cuts from the same shuffled Strict source pool using seed 42.
Budget
Training seed
Δ BLiMP [95% CI]
Δ Supp [95% CI]
Δ Entity [95% CI]
10M
2023
+7.07 [ +3.01 , +11.11 ]
+10.23 [ +3.54 , +16.49 ]
−22.27 [ −24.96 , −19.22 ]
10M
43
+8.19 [ +4.05 , +12.36 ]
+11.44 [ +3.32 , +19.58 ]
−17.00 [ −19.77 , −13.96 ]
37M
2023
+10.32 [ +7.21 , +13.75 ]
+5.74 [ +0.47 , +11.67 ]
−4.70 [ −7.07 , −2.36 ]
37M
43
+10.79 [ +7.46 , +14.53 ]
+8.42 [ +2.29 , +14.24 ]
−5.06 [ −7.36 , −2.89 ]
Appendix
Table 14: Paired corpus effects and percentile 95% hierarchical-bootstrap confidence intervals for the default KD students. Positive values favor BabyLM 2026; negative Entity Tracking values favor Britannica. All intervals exclude zero. Bootstrap seed: 2026.