Beetle: A Bilingual Model Suite for Modelling Second-Language Processing

Suchir Salhan; Catherine Arnett; James Michaelov; Paula Buttery

cs.CLSep 18, 2026

Beetle: A Bilingual Model Suite for Modelling Second-Language Processing

Authors: Suchir Salhan, Catherine Arnett, James Michaelov, Paula Buttery

Organizations: University of Cambridge · EleutherAI · MIT

Abstract

Bilingual language models (LMs) offer a controlled setting for studying how training conditions shape second-language (L2) behaviour, but prior work typically varies exposure structure, scale, and architecture at once, making it difficult to attribute effects to any single factor. We introduce Beetle, a controlled language model pretraining framework in which tokeniser, target language, training budget, and exposure structure are each independently manipulable, enabling systematic and comparable experimentation of training conditions. Using Beetle, we train and release 285 bilingual and 45 monolingual open-source LMs with rich checkpoints across a range of exposure schedules, data scales and first languages (L1s) to study multilingual pretraining and computational modelling of bilingualism and second language learning. Evaluating models on human bilingual and second language reading-time prediction and grammaticality judgement tasks, we find that staged and temporally structured curricula consistently improve alignment with language learner reading time compared to balanced bilingual training, with the largest gains at smaller data scales and for typologically closer language pairs. The Beetle models are well suited tools to help move computational psycholinguistics beyond its prevailing monolingual, English-centric focus toward models of human bilingual processing, to study cross-lingual learning dynamics, while supporting community-based development of controlled model families.

Figures & tables

Figure 1: Beetle is an open-source language model pretraining framework. We use it to study bilingual and multilingual language acquisition in language models under tightly controlled experimental conditions. Each facet shows different structured exposure curriculum ( B1--B5 ) used to pretrain bilingual models that vary in the relative per-language data distribution across training steps.

Figure 2: Beetle English BLiMP Accuracy for 100M FineWeb models across 15 L1s (Dutch, German, Italian, Russian, Basque, Hindi, Turkish, Polish, Danish, Greek, Estonian, Finnish, Icelandic and Spanish) and 5 exposure curricula. Full Monolingual and Multi-BLiMP accuracies for the Beetle models are in Appendix G .

Figure 3: FLORES Sentence NLL. Each facet is one model; curves report mean sentence-level NLL in bits on 1,012 parallel FLORES-200 devtest sentences (lower is better), with solid lines for L1 and dashed lines for English. The vertical dashed line shows L2 English introduction.

Figure 4: Comparison of L2 Reading Time Alignment (MECO, ( Siegelman et al., 2025 ; Kuperman, 2025 ) ) of Beetle and multilingual LLMs : (a) B1-B5 Dutch/Chinese/German-English Beetle (FineWeb 100M, 2B, 24B ( Penedo et al., 2025 ) , BabyBabelLM ( Jumelet et al., 2025a ) ) and (b) B1 - B3 Basque/Italian/Russian/Turkish-English Beetle (100M FineWeb)

Figure 5: BLiSS: Beetle vs. multilingual LLM performance. The best Beetle models (zho–eng HumanScale and nld–eng FineWeb) are compared with the same baselines and a Dutch monolingual anchor (chance = 50). Large multilingual LLMs reach a higher ceiling (RP $@0\approx 66$ – $71$ ). Unlike on MECO, Beetle does not outperform the multilingual baselines on BLiSS, indicating a dissociation between the two measures. Detailed results are reported in subsection F.1 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	100M	2B	24B
Architecture	PicoDecoder	PicoDecoder	PicoDecoder
Parameters	125M	125M	125M
Layers	14	14	14
$d_{\mathrm{model}}$	768	768	768
Heads (Q/KV)	12 / 1	12 / 1	12 / 1
FFN size	3072	3072	3072

Appendix

Table 2: Training hyperparameters across data scales.

Family	Strategy	Mechanism	Key hyperparameters
Parameter regularisation
	EWC ( Kirkpatrick et al., 2017 )	Diagonal-Fisher quadratic penalty at phase change	$\lambda\in[0,40]$ , $K_{\text{Fisher}}=10$
	SI	Online importance accumulation	$c$ , damping
	MAS	Gradient-magnitude importance	$\lambda$
Replay
	LAMOL ( Sun et al., 2019 )	Pseudo-sample generation, nucleus sampling	$\lambda_{\text{replay}}\in[0,0.5]$ , top- $p$

Appendix

Table 3: The continual-learning strategies implemented in the Beetle training library. The experiments reported here use EWC, LAMOL, and exposure-LR plasticity overlays.

	B1	B2	B3	B4	B5
L2 onset	0	50%	50%	0	80%
Final L2	50%	50%	67%	20%	50%
Transition	none	sig.	sig.	bursts	sig.
Bursting	no	no	no	yes	no
Episodic	no	no	no	yes	no
CL overlay	none	none	none	none	none

Appendix

Table 4: Controlled curriculum factors for bilingual conditions B1–B5. B4 uses bursty episodic exposure with the same mean L2 ratio as B2 and no EWC.

Figure 6: Overall BLiSS learner-minimal-pair accuracy for Beetle models against baseline-matched multilingual LLMs, aggregated across the matched-L1 cohorts. Curriculum differences on BLiSS are smaller than the corresponding differences in MECO reading-time alignment.

Figure 7: Performance of Beetle 125M models across pedagogical evaluations (CLOTH and Cambridge cloze, JFLEG grammatical correction, false-friend disambiguation, and the CEFR perplexity gradient), by curriculum and training scale. Balanced exposure (B1) is consistently strongest on these fluency- and correction-based measures.

			BLiSS (matched-L1 cohort)
Pair	Data	Curr.	RP@0	RP@ $\tau$	NGS	CPS
deu--eng	Human-scale	Mono	51.0	38.3	51.3	29.5
	Human-scale	B1 balanced	57.2	48.2	55.5	46.8
	Human-scale	B2 simult.	54.8	44.5	54.9	35.1
	Human-scale	B3 sequential	55.5	47.0	55.5	36.7
	Human-scale	B4 classroom	54.8	44.0	54.7	39.3

Appendix

Table 5: BLiSS learner-minimal-pair scores (%, 0–100) for Beetle models on the matched-L1 cohort , grouped by L1 then token budget. Cell shading encodes score on a fixed scale (darker green = higher). Per L1, the highest value in each metric column is shown in bold . Metrics: RP@0 / RP@

\tau

(random-over-human preference, with and without a margin), NGS (normalised gap score), CPS (correct-preferred sanity check).

			BLiSS (matched-L1 cohort)
Model	Params	L1	RP@0	RP@ $\tau$	NGS	CPS
BLOOM-560M	0.56B	Dutch	59.0	52.7	55.4	55.8
		German	58.9	50.7	54.9	52.3
		Chinese	70.2	62.1	65.0	56.1
BLOOM-1.1B	1.1B	Dutch	59.7	51.6	55.8	56.6
		German	57.6	49.4	54.7	53.1

Appendix

Table 6: BLiSS learner-minimal-pair scores (%, 0–100) for frontier multilingual-LLM baselines, scored per L1 cohort. Params gives the parameter count and L1 the cohort. Cell shading encodes score on a fixed scale (darker green = higher). Per model, the highest value in each metric column is in bold . Metrics: RP@0 / RP@

\tau

(random-over-human preference, with and without a margin), NGS (normalised gap score), CPS (correct-preferred sanity check).

Model	Family	CLOTH	Camb. OC	JFLEG	FF	A1 BPT	C2 BPT	A1 $\to$ C2
B1 nld–eng	HS	0.380	0.644	0.949	0.585	6.572	8.818	2.247
B2 nld–eng	HS	0.302	0.596	0.750	0.156	8.051	10.213	2.163
B3-33 nld–eng	HS	0.310	0.587	0.751	0.174	8.106	10.275	2.169
B4 nld–eng	HS	0.304	0.582	0.801	$-$ 0.055	7.719	9.755	2.036
B5 nld–eng	HS	0.302	0.584	0.716	0.238	8.127	10.193	2.066
B1 nld–eng	FW-2B	0.369	0.638	0.833	0.457	8.851	8.614	$-$ 0.237

Appendix

Table 7: Downstream pedagogical performance of Beetle models. CLOTH closed-cloze accuracy, Cambridge open-cloze accuracy, JFLEG learner-fluency margin (bits/token; positive = model prefers corrected over learner-error sentence), false-friend disambiguation margin (bits/token), CEFR-A1 / CEFR-C2 BPT, and CEFR gradient. Bold = best per column within the matched-L1 nld–eng family.

Figure 8: Per-phenomenon BLiMP accuracy for the 24B-token Beetle models, broken down by curriculum and L1. At this scale, differences between balanced and staged curricula are localised to specific phenomena rather than uniform across the benchmark, with the balanced and simultaneous curricula dominating on most syntactic and agreement phenomena.

Figure 9: Per-phenomenon BLiMP accuracy for the 2B-token Beetle models, broken down by curriculum and L1. The overall curriculum ordering mirrors the 24B grid in Figure 8 , but accuracies are lower and the per-phenomenon gaps between curricula are wider, indicating that the additional data at 24B mainly narrows the harder phenomena rather than reshuffling which curriculum wins.

			MultiBLiMP		MonoBLiMP
Pair	Data	Curr.	L1	Eng	L1	Eng
deu--eng	Human-scale	Mono	89.9	64.7	-	51.2
	Human-scale	B1 balanced	86.9	80.5	-	65.5
	Human-scale	B2 simult.	89.5	70.9	-	56.1
	Human-scale	B3 sequential	89.9	71.3	-	57.4
	Human-scale	B4 classroom	89.7	72.2	-	57.0

Appendix

Table 8: Accuracy of Beetle models across curricula and L1s and data exposure (100, 2B, 24B) on L1 and L2 MultiBLiMP ( Jumelet et al., 2025b ) and Monolingual BLiMPs ( English: BLiMP ( Warstadt et al., 2020 ) ); Dutch: BLiMP-NL ( Suijkerbuijk et al., 2025a ) ; Chinese: ZhoBLiMP ( Liu et al., 2025 )

			MultiBLiMP		MonoBLiMP
Model	Par.	Cohort	L1	Eng	L1	Eng
BLOOM-560M	0.56B	Dutch	59.8	81.7	52.6	66.7
		German	65.9	81.7		66.7
		Chinese		81.7	72.5	66.7
BLOOM-1.1B	1.1B	Dutch	63.0	92.5	48.7	76.7
		German	76.5	92.5		76.7

Appendix

Table 9: Frontier multilingual-LLM baselines. Par. is the parameter count; Cohort is the evaluation language.

Explore similar work

Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models
Mar 31, 2026Linda Zeng, Steven Y. Feng, Michael C. FrankMultilingual Language Models Multilingual Language Model Evaluation
Stanford University
Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training
Oct 14, 2026JiWoo Hwang, Lukas Edman, Alexander FraserLanguage Model Pretraining Multilingual Language Models
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations
Aug 27, 2026Anjishnu Mukherjee, Ziwei Zhu, Antonios AnastasopoulosLanguage Model Pretraining Multilingual Language Models
George Mason University

Mar 31, 2026J/K move · Enter open · S save

Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models

Linda Zeng, Steven Y. Feng, Michael C. Frank

Stanford University

Multilingual Language Models Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Multilingualism is incredibly common around the world, leading to many important theoretical and practical questions about how children learn multiple languages at once. For example, does multilingual acquisition lead to delays in learning? Are there better and worse ways to structure multilingual input? Many correlational studies address these questions, but it is surprisingly difficult to get definitive answers because children cannot be randomly assigned to be multilingual and data are typically not matched between languages. We use language model training as a method for simulating a variety of highly controlled exposure conditions, and create matched 100M-word mono- and bilingual datasets using synthetic data and machine translation. We train GPT-2 models on monolingual and bilingual data organized to reflect a range of exposure regimes, and evaluate their performance on perplexity, grammaticality, and semantic knowledge. Across model scales and measures, bilingual models perform similarly to monolingual models in one language, but show strong performance in the second language as well. These results suggest that there are no strong differences between different bilingual exposure regimes, and that bilingual input poses no in-principle challenges for agnostic statistical learners.

Abstract

Explore similar work

Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models

Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training

Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations