Bilingual language models (LMs) offer a controlled setting for studying how training conditions shape second-language (L2) behaviour, but prior work typically varies exposure structure, scale, and architecture at once, making it difficult to attribute effects to any single factor. We introduce Beetle, a controlled language model pretraining framework in which tokeniser, target language, training budget, and exposure structure are each independently manipulable, enabling systematic and comparable experimentation of training conditions. Using Beetle, we train and release 285 bilingual and 45 monolingual open-source LMs with rich checkpoints across a range of exposure schedules, data scales and first languages (L1s) to study multilingual pretraining and computational modelling of bilingualism and second language learning. Evaluating models on human bilingual and second language reading-time prediction and grammaticality judgement tasks, we find that staged and temporally structured curricula consistently improve alignment with language learner reading time compared to balanced bilingual training, with the largest gains at smaller data scales and for typologically closer language pairs. The Beetle models are well suited tools to help move computational psycholinguistics beyond its prevailing monolingual, English-centric focus toward models of human bilingual processing, to study cross-lingual learning dynamics, while supporting community-based development of controlled model families.
Figures & tables
Figure 1: Beetle is an open-source language model pretraining framework. We use it to study bilingual and multilingual language acquisition in language models under tightly controlled experimental conditions. Each facet shows different structured exposure curriculum ( B1--B5 ) used to pretrain bilingual models that vary in the relative per-language data distribution across training steps.
Figure 2: Beetle English BLiMP Accuracy for 100M FineWeb models across 15 L1s (Dutch, German, Italian, Russian, Basque, Hindi, Turkish, Polish, Danish, Greek, Estonian, Finnish, Icelandic and Spanish) and 5 exposure curricula. Full Monolingual and Multi-BLiMP accuracies for the Beetle models are in Appendix G .
Figure 3: FLORES Sentence NLL. Each facet is one model; curves report mean sentence-level NLL in bits on 1,012 parallel FLORES-200 devtest sentences (lower is better), with solid lines for L1 and dashed lines for English. The vertical dashed line shows L2 English introduction.
Figure 4: Comparison of L2 Reading Time Alignment (MECO, ( Siegelman et al., 2025 ; Kuperman, 2025 ) ) of Beetle and multilingual LLMs : (a) B1-B5 Dutch/Chinese/German-English Beetle (FineWeb 100M, 2B, 24B ( Penedo et al., 2025 ) , BabyBabelLM ( Jumelet et al., 2025a ) ) and (b) B1 - B3 Basque/Italian/Russian/Turkish-English Beetle (100M FineWeb)
Figure 5: BLiSS: Beetle vs. multilingual LLM performance. The best Beetle models (zho–eng HumanScale and nld–eng FineWeb) are compared with the same baselines and a Dutch monolingual anchor (chance = 50). Large multilingual LLMs reach a higher ceiling (RP @0≈66 – 71 ). Unlike on MECO, Beetle does not outperform the multilingual baselines on BLiSS, indicating a dissociation between the two measures. Detailed results are reported in subsection F.1 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
100M
2B
24B
Architecture
PicoDecoder
PicoDecoder
PicoDecoder
Parameters
125M
125M
125M
Layers
14
14
14
dmodel
768
768
768
Heads (Q/KV)
12 / 1
12 / 1
12 / 1
FFN size
3072
3072
3072
Appendix
Table 2: Training hyperparameters across data scales.
Family
Strategy
Mechanism
Key hyperparameters
Parameter regularisation
EWC ( Kirkpatrick et al., 2017 )
Diagonal-Fisher quadratic penalty at phase change
λ∈[0,40] , KFisher=10
SI
Online importance accumulation
c , damping
MAS
Gradient-magnitude importance
λ
Replay
LAMOL ( Sun et al., 2019 )
Pseudo-sample generation, nucleus sampling
λreplay∈[0,0.5] , top- p
Appendix
Table 3: The continual-learning strategies implemented in the Beetle training library. The experiments reported here use EWC, LAMOL, and exposure-LR plasticity overlays.
B1
B2
B3
B4
B5
L2 onset
0
50%
50%
0
80%
Final L2
50%
50%
67%
20%
50%
Transition
none
sig.
sig.
bursts
sig.
Bursting
no
no
no
yes
no
Episodic
no
no
no
yes
no
CL overlay
none
none
none
none
none
Appendix
Table 4: Controlled curriculum factors for bilingual conditions B1–B5. B4 uses bursty episodic exposure with the same mean L2 ratio as B2 and no EWC.
Figure 6: Overall BLiSS learner-minimal-pair accuracy for Beetle models against baseline-matched multilingual LLMs, aggregated across the matched-L1 cohorts. Curriculum differences on BLiSS are smaller than the corresponding differences in MECO reading-time alignment.
Figure 7: Performance of Beetle 125M models across pedagogical evaluations (CLOTH and Cambridge cloze, JFLEG grammatical correction, false-friend disambiguation, and the CEFR perplexity gradient), by curriculum and training scale. Balanced exposure (B1) is consistently strongest on these fluency- and correction-based measures.
BLiSS (matched-L1 cohort)
Pair
Data
Curr.
RP@0
RP@ τ
NGS
CPS
deu--eng
Human-scale
Mono
51.0
38.3
51.3
29.5
Human-scale
B1 balanced
57.2
48.2
55.5
46.8
Human-scale
B2 simult.
54.8
44.5
54.9
35.1
Human-scale
B3 sequential
55.5
47.0
55.5
36.7
Human-scale
B4 classroom
54.8
44.0
54.7
39.3
Appendix
Table 5: BLiSS learner-minimal-pair scores (%, 0–100) for Beetle models on the matched-L1 cohort , grouped by L1 then token budget. Cell shading encodes score on a fixed scale (darker green = higher). Per L1, the highest value in each metric column is shown in bold . Metrics: RP@0 / RP@ τ (random-over-human preference, with and without a margin), NGS (normalised gap score), CPS (correct-preferred sanity check).
BLiSS (matched-L1 cohort)
Model
Params
L1
RP@0
RP@ τ
NGS
CPS
BLOOM-560M
0.56B
Dutch
59.0
52.7
55.4
55.8
German
58.9
50.7
54.9
52.3
Chinese
70.2
62.1
65.0
56.1
BLOOM-1.1B
1.1B
Dutch
59.7
51.6
55.8
56.6
German
57.6
49.4
54.7
53.1
Appendix
Table 6: BLiSS learner-minimal-pair scores (%, 0–100) for frontier multilingual-LLM baselines, scored per L1 cohort. Params gives the parameter count and L1 the cohort. Cell shading encodes score on a fixed scale (darker green = higher). Per model, the highest value in each metric column is in bold . Metrics: RP@0 / RP@ τ (random-over-human preference, with and without a margin), NGS (normalised gap score), CPS (correct-preferred sanity check).
Model
Family
CLOTH
Camb. OC
JFLEG
FF
A1 BPT
C2 BPT
A1 → C2
B1 nld–eng
HS
0.380
0.644
0.949
0.585
6.572
8.818
2.247
B2 nld–eng
HS
0.302
0.596
0.750
0.156
8.051
10.213
2.163
B3-33 nld–eng
HS
0.310
0.587
0.751
0.174
8.106
10.275
2.169
B4 nld–eng
HS
0.304
0.582
0.801
− 0.055
7.719
9.755
2.036
B5 nld–eng
HS
0.302
0.584
0.716
0.238
8.127
10.193
2.066
B1 nld–eng
FW-2B
0.369
0.638
0.833
0.457
8.851
8.614
− 0.237
Appendix
Table 7: Downstream pedagogical performance of Beetle models. CLOTH closed-cloze accuracy, Cambridge open-cloze accuracy, JFLEG learner-fluency margin (bits/token; positive = model prefers corrected over learner-error sentence), false-friend disambiguation margin (bits/token), CEFR-A1 / CEFR-C2 BPT, and CEFR gradient. Bold = best per column within the matched-L1 nld–eng family.
Figure 8: Per-phenomenon BLiMP accuracy for the 24B-token Beetle models, broken down by curriculum and L1. At this scale, differences between balanced and staged curricula are localised to specific phenomena rather than uniform across the benchmark, with the balanced and simultaneous curricula dominating on most syntactic and agreement phenomena.
Figure 9: Per-phenomenon BLiMP accuracy for the 2B-token Beetle models, broken down by curriculum and L1. The overall curriculum ordering mirrors the 24B grid in Figure 8 , but accuracies are lower and the per-phenomenon gaps between curricula are wider, indicating that the additional data at 24B mainly narrows the harder phenomena rather than reshuffling which curriculum wins.
MultiBLiMP
MonoBLiMP
Pair
Data
Curr.
L1
Eng
L1
Eng
deu--eng
Human-scale
Mono
89.9
64.7
-
51.2
Human-scale
B1 balanced
86.9
80.5
-
65.5
Human-scale
B2 simult.
89.5
70.9
-
56.1
Human-scale
B3 sequential
89.9
71.3
-
57.4
Human-scale
B4 classroom
89.7
72.2
-
57.0
Appendix
Table 8: Accuracy of Beetle models across curricula and L1s and data exposure (100, 2B, 24B) on L1 and L2 MultiBLiMP ( Jumelet et al., 2025b ) and Monolingual BLiMPs ( English: BLiMP ( Warstadt et al., 2020 ) ); Dutch: BLiMP-NL ( Suijkerbuijk et al., 2025a ) ; Chinese: ZhoBLiMP ( Liu et al., 2025 )
MultiBLiMP
MonoBLiMP
Model
Par.
Cohort
L1
Eng
L1
Eng
BLOOM-560M
0.56B
Dutch
59.8
81.7
52.6
66.7
German
65.9
81.7
66.7
Chinese
81.7
72.5
66.7
BLOOM-1.1B
1.1B
Dutch
63.0
92.5
48.7
76.7
German
76.5
92.5
76.7
Appendix
Table 9: Frontier multilingual-LLM baselines. Par. is the parameter count; Cohort is the evaluation language.
Multilingualism is incredibly common around the world, leading to many important theoretical and practical questions about how children learn multiple languages at once. For example, does multilingual acquisition lead to delays in learning? Are there better and worse ways to structure multilingual input? Many correlational studies address these questions, but it is surprisingly difficult to get definitive answers because children cannot be randomly assigned to be multilingual and data are typically not matched between languages. We use language model training as a method for simulating a variety of highly controlled exposure conditions, and create matched 100M-word mono- and bilingual datasets using synthetic data and machine translation. We train GPT-2 models on monolingual and bilingual data organized to reflect a range of exposure regimes, and evaluate their performance on perplexity, grammaticality, and semantic knowledge. Across model scales and measures, bilingual models perform similarly to monolingual models in one language, but show strong performance in the second language as well. These results suggest that there are no strong differences between different bilingual exposure regimes, and that bilingual input poses no in-principle challenges for agnostic statistical learners.
We investigate two aspects of multilingual pre-training with limited data, namely how language exposure is distributed across English, Dutch, and Chinese, and how that exposure is ordered during training. In particular, we compare balanced multilingual exposure to language-dominant exposure, where one language is exposed with more training data than the others. We also compare mixed training, where multilingual training data is used throughout training, with curriculum training, where monolingual exposure of the dominant language is followed by balanced multilingual exposure. Our results show that multilingual BabyLM performance varies with both the amount and order of language exposure. Mixed training outperforms curriculum learning across all 15 matched configurations, indicating that simultaneous multilingual exposure is more effective than staged exposure in our setting. Several language-dominant mixed training configurations achieve higher scores than the balanced configuration, suggesting that a balanced distribution is not necessarily the highest-performing configuration in our setting. Interestingly, the Dutch-primary configurations achieve the best overall performance.
JiWoo Hwang, Lukas Edman, Alexander Fraser
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
A concept can carry different associations across languages, while modern language models learn English alongside many other languages during pretraining. Yet comparisons among existing models cannot easily isolate how any one language changes the way these models represent English concepts because their training corpora, compute, architectures, and random seeds all differ. We study this question through a controlled experiment with 40 matched 310M-parameter decoder-only models that share an architecture, tokenizer, training recipe, and English data source. Each bilingual condition adds one of eight languages, while four experimental comparisons separately account for English exposure, total training, and English-document overlap. We align each model pair using 3,000 common English words, then measure where 1,000 held-out English concepts fall along 50 fixed semantic contrasts, such as red versus white. Across 32 experimental comparisons, English concept positions differ more between bilingual and English-only conditions than between English-only runs with different random seeds. These differences are larger in contextual states than in token embeddings and peak in middle layers. The language learned alongside English can therefore change how a model represents English concepts even when its English input representations are explicitly aligned.