Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training

JiWoo Hwang; Lukas Edman; Alexander Fraser

cs.CLOct 14, 2026

Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training

Authors: JiWoo Hwang, Lukas Edman, Alexander Fraser

Organizations: School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute

Abstract

We investigate two aspects of multilingual pre-training with limited data, namely how language exposure is distributed across English, Dutch, and Chinese, and how that exposure is ordered during training. In particular, we compare balanced multilingual exposure to language-dominant exposure, where one language is exposed with more training data than the others. We also compare mixed training, where multilingual training data is used throughout training, with curriculum training, where monolingual exposure of the dominant language is followed by balanced multilingual exposure. Our results show that multilingual BabyLM performance varies with both the amount and order of language exposure. Mixed training outperforms curriculum learning across all 15 matched configurations, indicating that simultaneous multilingual exposure is more effective than staged exposure in our setting. Several language-dominant mixed training configurations achieve higher scores than the balanced configuration, suggesting that a balanced distribution is not necessarily the highest-performing configuration in our setting. Interestingly, the Dutch-primary configurations achieve the best overall performance.

Figures & tables

Figure 1: Mixed and curriculum training for the Dutch-primary 60/20/20 configuration. Mixed training shuffles all 100M tokens in a single stage, whereas curriculum training first uses 40M Dutch tokens and then a balanced multilingual stage containing 20M tokens from each language. For this configuration, mixed training achieves a higher multilingual average than curriculum training and produces the best overall result in our experiments.

Configuration	Primary	Other 1	Other 2
Balanced	33%	33%	33%
50/25/25	50%	25%	25%
60/20/20	60%	20%	20%
70/15/15	70%	15%	15%
80/10/10	80%	10%	10%
90/5/5	90%	5%	5%

Table 1: Data proportion configurations. Each language-dominant configuration is instantiated with English, Dutch, and Chinese as the primary language respectively.

Figure 2: Multilingual average performance across primary-language data proportions, grouped by the language designated as primary and by training strategy. Each point averages performance across English, Dutch, and Chinese, while the dashed horizontal line marks the balanced 33/33/33 baseline.

Figure 3: Relationship between language-specific data proportion and evaluation performance. Columns show English, Dutch, and Chinese data proportions, while rows show zero-shot, finetuned, and overall average performance for each language. Dashed lines indicate linear trends across all models, with $R^{2}$ reported in each panel. The black square represents the balanced 33/33/33 baseline.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Model size	$\sim$ 110M parameters
Tokenizer	Byte-level BPE
Vocabulary size	32,000
Context length	1024
Optimizer	AdamW
Learning rate	$6\times 10^{-4}$

Appendix

Table 2: Training setup used for all models.

Balanced baseline
Model		English			Dutch			Chinese			Multi
Config	Strategy	ZS	FT	Avg	ZS	FT	Avg	ZS	FT	Avg	Avg
33/33/33	Mixed	55.48	45.52	47.06	54.99	42.70	46.61	48.73	43.78	45.03	46.23
English-primary
50/25/25	Mixed	56.50	45.94	48.15	54.77	41.80	45.89	48.96	43.25	44.50	46.18
	Curr	54.39	42.79	45.91	53.05	43.47	46.39	48.13	44.97	45.87	46.06

Appendix

Table 3: Full results for all 31 models. ZS = zero-shot average, FT = finetune average, Avg = combined average for that language, Multi = multilingual average across all three languages. Bold indicates the best score in each column.

Explore similar work

BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop
Feb 23, 2026Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul +7Language Model Pretraining Small Language Models
IBM Research, MIT · ETH Zürich · Cornell University +7
Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence
Sep 24, 2026Dries Rooryck, Alex Cai, Yonatan Belinkov +2Multilingual Language Models Language Model Pretraining
Kempner Institute, Harvard University, Cambridge, MA, USA · Technion – Israel Institute of Technology, Haifa, Israel
Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models
Mar 31, 2026Linda Zeng, Steven Y. Feng, Michael C. FrankMultilingual Language Models Multilingual Language Model Evaluation
Stanford University

Feb 23, 2026J/K move · Enter open · S save

BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop

Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Jaap Jumelet+6

IBM Research, MIT · ETH Zürich · Cornell University+7

Language Model Pretraining Small Language Models Multilingual Language Models

Open paper AskPDF arXiv

The goal of the BabyLM is to stimulate new research connections between cognitive modeling and language model pretraining. We invite contributions in this vein to the BabyLM Workshop, which will also include the 4th iteration of the BabyLM Challenge. As in previous years, the challenge features two ``standard'' tracks (Strict and Strict-Small), in which participants must train language models on under 100M or 10M words of data, respectively. This year, we move beyond our previous English-only pretraining datasets with a new Multilingual track, focusing on English, Dutch, and Chinese. For the workshop, we call for papers related to the overall theme of BabyLM, which includes training efficiency, small-scale training datasets, cognitive modeling, model evaluation, and architecture innovation.

Abstract

Explore similar work

BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop

Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence

Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models