Small language models are often trained under tight data budgets, where the choice of pretraining corpus may matter as much as model size or training recipe. We compare two contrasting corpora at matched 10-million- and 37-million-word budgets: the 1911 Encyclopaedia Britannica and the BabyLM 2026 strict pretraining corpus. For each corpus, we train teacher models, distilled 58-million-parameter Baby-Llama students, and no-distillation student ablations, using the same model families, training recipes, and evaluation protocol across corpora. Across BLiMP, BLiMP Supplement, EWoK, Entity Tracking, and COMPS, the corpora produce different skill profiles rather than a single quality ranking. BabyLM 2026 is consistently stronger on linguistic acceptability benchmarks, while Britannica is often stronger on Entity Tracking, a synthetic state-tracking task. The contrast appears across teacher and student models and at both budgets, with the cleanest cross-seed replication in the distilled student comparison. Targeted controls show that the central contrast persists after matching training exposure and at a longer diagnostic context; direct teacher ensembling also does not reproduce the 10M Britannica student's Entity Tracking behavior. Entity Tracking magnitudes nevertheless remain sensitive to training dynamics. For this corpus pair, corpus choice can therefore produce different relative skill profiles at small scale, motivating multi-benchmark evaluation when comparing pretraining data.
Figures & tables
Statistic
Brit. 10M
Brit. 37M
BabyLM 10M
BabyLM 37M
Train documents
9,181
34,295
1.17M
4.30M
Mean document words
1,089
1,079
8.6
8.6
Median document words
318
318
5
5
Mean sentence words
20.3
20.2
7.5
7.5
Word types
239k
532k
170k
344k
Tokenizer tokens / word
1.42
1.42
1.95
1.95
Table 1: Corpus properties for the matched 10M- and 37M-word training splits. Britannica is a single-source formal encyclopedia corpus; BabyLM 2026 is a heterogeneous mixture of child-directed speech, dialogue, simple Wikipedia, transcribed speech, and other text. Sentences are segmented on terminal punctuation. Token counts use each track’s corpus-specific tokenizer and include document-boundary tokens.
Llama
GPT-2
GPT-J
Layers
24
24
24
Attention heads
8
16
16
Hidden dimension
1024
1536
1536
Intermediate dimension
3072
6144
6144
Parameters
360M
705M
705M
Epochs
4
6
4
Table 2: Teacher architectures and training recipes. All teachers share a cosine learning-rate schedule with 300 warm-up steps. GPT-J is trained only at the 10M budget.
Figure 1: Skill profiles of the seed-2023 experimental sweep on five zero-shot metrics. Each panel shows one metric; bars give the mean accuracy per (corpus, budget) combination, and circles mark individual models (3 teachers + 3 students at 10M; 2 teachers + 2 students at 37M). The Britannica corpus produces higher Entity Tracking in the main matched comparisons; the BabyLM 2026 corpus produces higher BLiMP, BLiMP Supplement, and (slightly) COMPS; EWoK is essentially tied. Table 3 reports the paired effects for both training seeds.
Δ BLiMP
Δ Supp
Δ Entity
Budget
Matched model family
s2023
s43
s2023
s43
s2023
s43
10M
Llama-360M teacher
+ 8.19
+ 7.97
+ 7.65
+ 10.46
− 8.38
− 8.05
10M
GPT-2-705M teacher
+ 7.73
+ 8.19
+ 7.78
+ 5.21
− 8.02
− 16.49
10M
GPT-J-705M teacher
+ 5.99
+ 7.22
+ 11.96
+ 5.53
− 12.29
− 4.59
10M
Baby-Llama-58M student (KD, Llama+GPT-2)
+ 7.07
+ 8.19
+ 10.23
+ 11.44
− 22.27
− 17.00
10M
Baby-Llama-58M student (KD, Llama+GPT-J)
+ 6.91
+ 6.41
+ 10.15
+ 11.75
− 22.58
− 5.49
Table 3: Paired corpus effects under both training seeds. Deltas are BabyLM 2026 minus Britannica for the same model family and source-word budget. Positive BLiMP/Supplement values favor BabyLM 2026; negative Entity Tracking values favor Britannica.
Model
BLiMP
Supp
EWoK
Entity
COMPS
Britannica 10M
Baby-Llama-58M (KD, Llama+GPT-2)
64.69
46.60
49.81
37.10
51.23
Baby-Llama-58M (KD, Llama+GPT-J)
64.19
46.27
51.02
37.26
51.21
Baby-Llama-58M (no KD)
63.01
47.45
50.84
26.11
50.63
BabyLM 2026 10M
Baby-Llama-58M (KD, Llama+GPT-2)
71.76
56.83
50.08
14.83
52.00
Table 4: Seed-2023 zero-shot evaluation of 58M students at matched 10M-word budget on the Britannica and BabyLM 2026 corpora. KD labels name the two-teacher ensemble used for distillation; “no KD” is the matched non-distilled baseline. Bold marks the best entry per metric across both corpora.
Corpus
Model
BLiMP
Supp
EWoK
Entity
COMPS
Britannica 37M
KD, Llama+GPT-2
65.95
51.08
50.36
21.80
53.42
Britannica 37M
no KD
64.66
47.87
50.58
22.22
53.27
BabyLM 2026 37M
KD, Llama+GPT-2
76.27
56.82
51.24
17.10
54.66
BabyLM 2026 37M
no KD
74.57
56.47
50.93
16.73
54.18
Table 5: Seed-2023 zero-shot evaluation of 58M Baby-Llama students at the larger 37M-word budget. KD rows use the Llama+GPT-2 two-teacher distillation ensemble; no-KD rows use the same architecture and training schedule without teacher logits. Bold marks the best entry per metric across both corpora.
Corpus
Δ BLiMP
Δ Supp
Δ EWoK
Δ Entity
Δ COMPS
Britannica 10M
+1.68
-0.85
-1.03
+10.99
+0.60
BabyLM 2026 10M
+1.87
-0.57
-0.89
-8.31
+0.32
Britannica 37M
+1.29
+3.21
-0.22
-0.42
+0.15
BabyLM 2026 37M
+1.70
+0.35
+0.31
+0.37
+0.48
Table 6: Effect of knowledge distillation on the 58M student, expressed as KD score minus no-KD score on the same corpus and word budget. Positive values indicate that distillation helps.
Seed
Budget
Teacher pair / context
Teacher ensemble
KD student
Student − Ensemble
2023
10M
Llama+GPT-2 @ 128 tokens
25.15
37.10
+ 11.95
43
10M
Llama+GPT-2 @ 128 tokens
29.15
35.58
+ 6.43
2023
10M
Llama+GPT-2 @ 256 tokens
15.83
38.96
+ 23.13
2023
10M
Llama+GPT-J @ 128 tokens
31.29
37.26
+ 5.97
2023
10M
Llama+GPT-J @ 256 tokens
26.94
37.71
+ 10.77
2023
37M
Llama+GPT-2 @ 128 tokens
23.85
21.80
− 2.05
Table 7: Direct ensemble comparison for the Britannica Entity Tracking models. All entries are macro averages: accuracy is first computed for each benchmark subtask (UID group) and then averaged. For the ensemble, each candidate completion is scored by the per-token logit mean of the two Britannica-trained teachers used during KD training. Main rows use seed 2023; the seed-43 row repeats the default 10M Llama+GPT-2 check. At 10M, the student exceeds the ensemble in every configuration; at 37M, the ensemble exceeds the student at both context lengths.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Corpus
Model
BLiMP
Supp
EWoK
Entity
COMPS
Eye
SPR
Britannica 10M
Llama-360M teacher
61.45
48.81
50.35
26.13
50.69
0.35
0.01
Britannica 10M
GPT-2-705M teacher
62.61
46.83
51.04
25.38
51.06
0.12
0.01
Britannica 10M
GPT-J-705M teacher
62.43
44.93
50.38
36.54
50.78
0.25
0.03
Britannica 10M
Baby-Llama-58M student (KD, Llama+GPT-2)
64.69
46.60
49.81
37.10
51.23
0.13
0.01
Britannica 10M
Baby-Llama-58M student (KD, Llama+GPT-J)
64.19
46.27
51.02
37.26
51.21
0.17
0.00
Britannica 10M
Baby-Llama-58M student (no KD)
63.01
47.45
50.84
26.11
50.63
0.26
0.07
Appendix
Table 8: Full zero-shot results for the matched 10M-word corpus comparisons. KD student labels name the two-teacher ensemble used for distillation.
Corpus
Model
BLiMP
Supp
EWoK
Entity
COMPS
Eye
SPR
Britannica 37M
Llama-360M teacher
65.52
51.22
50.80
19.34
53.11
0.41
0.02
Britannica 37M
GPT-2-705M teacher
64.86
50.32
51.07
30.06
52.85
0.39
0.01
Britannica 37M
Baby-Llama-58M student (KD, Llama+GPT-2)
65.95
51.08
50.36
21.80
53.42
0.31
0.02
Britannica 37M
Baby-Llama-58M student (no KD)
64.66
47.87
50.58
22.22
53.27
–
–
BabyLM 2026 37M
Llama-360M teacher
74.28
54.93
50.78
17.00
53.86
0.24
0.01
BabyLM 2026 37M
GPT-2-705M teacher
75.02
59.19
51.93
16.36
54.82
0.19
0.02
Appendix
Table 9: Full zero-shot results for the matched 37M-word corpus comparisons. Reading scores for the 37M no-KD runs were not collected; these models were evaluated only on the five core metrics.
Corpus
Model
ET @ 128
ET @ 256
Δ
Britannica 10M
Llama-360M teacher
26.13
23.00
− 3.13
Britannica 10M
GPT-2-705M teacher
25.38
13.48
− 11.90
Britannica 10M
GPT-J-705M teacher
36.54
32.08
− 4.46
Britannica 10M
Baby-Llama-58M (Llama+GPT-2)
37.10
38.96
+ 1.86
Britannica 10M
Baby-Llama-58M (Llama+GPT-J)
37.26
37.71
+ 0.45
Britannica 10M
Baby-Llama-58M (no KD)
26.11
23.34
− 2.77
Appendix
Table 10: Entity Tracking accuracy at the 128-token (tokenizer-declared) and 256-token (position-embedding maximum) context lengths. The 256-token setting is a diagnostic rather than the main protocol because all models were trained on 128-token slices. Most student rows move modestly, and the Britannica-vs-BabyLM ranking holds in every matched row at both context lengths. Some teacher rows, especially the 10M GPT-2 rows with learned absolute position embeddings, are more sensitive to the diagnostic setting.
Corpus
Budget
Source words
Boundary markers
BPE train tokens
Tokens / word
Britannica
10M
10,000,066
18,362
14,216,855
1.42
BabyLM 2026
10M
10,000,092
2,331,378
19,518,278
1.95
Britannica
37M
37,002,217
68,590
52,559,454
1.42
BabyLM 2026
37M
37,002,217
8,605,126
72,172,588
1.95
Appendix
Table 11: Training-token counts after corpus-specific BPE tokenization. The matched-word design follows the BabyLM convention, but it does not match the number of BPE tokens seen per epoch. In these splits, BabyLM 2026 yields about 1.37 × as many training tokens as Britannica at both budgets, reflecting both corpus-specific BPE segmentation and BabyLM’s many short source segments, which introduce far more document-boundary markers.
BabyLM 2026 model
Steps
Eff. ep.
BLiMP
Supp
EWoK
Entity
COMPS
Llama-360M teacher
3,472
2.91
68.91
57.59
50.60
20.52
51.51
GPT-2-705M teacher
2,604
4.37
69.14
53.66
50.12
14.24
51.65
Baby-Llama-58M student (KD)
20,826
4.37
70.22
56.39
50.95
16.40
51.66
Baby-Llama-58M student (no KD)
20,826
4.37
68.82
56.36
50.40
27.39
51.64
Appendix
Table 12: Seed-2023 zero-shot results for the BabyLM 2026 10M matched-exposure control. Total optimizer steps and approximately total 128-token sequence presentations match the corresponding Britannica models.
Split
Train words
Train docs
Val words
Val docs
Source rows consumed
BabyLM 2026 10M
10,000,092
1,165,689
2,027,481
236,073
1,401,762
BabyLM 2026 37M
37,002,217
4,302,563
1,003,428
116,690
4,419,253
Appendix
Table 13: BabyLM 2026 strict split manifests. Both splits are deterministic random document-level cuts from the same shuffled Strict source pool using seed 42.
Budget
Training seed
Δ BLiMP [95% CI]
Δ Supp [95% CI]
Δ Entity [95% CI]
10M
2023
+7.07 [ +3.01 , +11.11 ]
+10.23 [ +3.54 , +16.49 ]
−22.27 [ −24.96 , −19.22 ]
10M
43
+8.19 [ +4.05 , +12.36 ]
+11.44 [ +3.32 , +19.58 ]
−17.00 [ −19.77 , −13.96 ]
37M
2023
+10.32 [ +7.21 , +13.75 ]
+5.74 [ +0.47 , +11.67 ]
−4.70 [ −7.07 , −2.36 ]
37M
43
+10.79 [ +7.46 , +14.53 ]
+8.42 [ +2.29 , +14.24 ]
−5.06 [ −7.36 , −2.89 ]
Appendix
Table 14: Paired corpus effects and percentile 95% hierarchical-bootstrap confidence intervals for the default KD students. Positive values favor BabyLM 2026; negative Entity Tracking values favor Britannica. All intervals exclude zero. Bootstrap seed: 2026.
Evaluating grammatical knowledge in language models requires benchmarks that cover diverse linguistic phenomena and are validated against human judgments. We introduce BLiMP-RO, a Romanian minimal-pair benchmark covering 50 grammatical phenomena across ten domains. It comprises 493 linguist-constructed sentence pairs, evaluated by native speakers in five experiments, and 5,942 additional semi-automatically generated, human-validated pairs. We further investigate grammatical generalization under limited linguistic input by training GPT-2 and GPT-BERT models from scratch on Romanian BabyBabelLM data and a newly collected corpus of 10 million words from child-targeted video captions. Increasing caption training data yields approximately 9 percentage points of accuracy improvement per tenfold increase in corpus size. At matched data sizes, BabyBabelLM generally outperforms video captions, while GPT-BERT consistently outperforms GPT-2. Our findings highlight how training data quantity, composition, and model architecture shape grammatical generalization in Romanian.
How does data shape language model (LM) behavior throughout pretraining? We investigate this question through a case study on entity comparison, e.g., Between France and Brazil, which country is larger?, across nine comparison tasks spanning five entity types. We begin with controlled experiments in which we train small LMs (163M parameters) on mixtures of natural text from pretraining corpora and synthetic task data. Our analysis identifies three distinct phases of learning: (1) an early phase where the LM selects entities by frequency, (2) a middle phase where the LM selects entities by position in the prompt (first vs. last mentioned entity), and (3) a late phase where the LM selects the entity that is the correct answer. We show that the emergence of these three phases is causally determined by statistical properties of the training data. With small amounts of task-specific synthetic data, we observe only the first two phases and the model fails to learn the task; with large amounts, the model jumps directly from the frequency-based heuristic to solving the task correctly. Moreover, changing the entity frequency distribution from a naturally occurring Zipfian distribution (a small number of entities are very common and the vast majority are rare) to a uniform one weakens the frequency phase and speeds up learning of the task. Lastly, we show that the three phases of training dynamics from our controlled experiments appear in the open source model OLMo-2 32B. Together, our findings demonstrate that properties of pretraining data are causal drivers of heuristic learning and show that small-scale synthetic experiments can predict training dynamics at larger scales.
Yik Siu Chan, Jing Huang, Yanai Elazar +1
University of Maryland · Pr(Ai)2R Group · Stanford University +2
Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LittleCurriculum, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LittleCurriculum yields LittleLearner, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LittleCurriculum and LittleLearner as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox's utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LittleLearner better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.
Fanfei Li, Jana Zeller, Manuel Prada-Corral +4
MPI-IS, Ellis Institute · MPI-IS, Ellis Institute, ETHZ · ETHZ