Multilingualism is incredibly common around the world, leading to many important theoretical and practical questions about how children learn multiple languages at once. For example, does multilingual acquisition lead to delays in learning? Are there better and worse ways to structure multilingual input? Many correlational studies address these questions, but it is surprisingly difficult to get definitive answers because children cannot be randomly assigned to be multilingual and data are typically not matched between languages. We use language model training as a method for simulating a variety of highly controlled exposure conditions, and create matched 100M-word mono- and bilingual datasets using synthetic data and machine translation. We train GPT-2 models on monolingual and bilingual data organized to reflect a range of exposure regimes, and evaluate their performance on perplexity, grammaticality, and semantic knowledge. Across model scales and measures, bilingual models perform similarly to monolingual models in one language, but show strong performance in the second language as well. These results suggest that there are no strong differences between different bilingual exposure regimes, and that bilingual input poses no in-principle challenges for agnostic statistical learners.
Figures & tables
Figure 1: General motivation and condition structure for main experiments. Note ”by-speaker” refers to a one speaker–one language policy (shown in the example), while “random” allows either speaker to produce either language (randomly assigned per dialogue).
Training Condition
PPL (EN)
PPL (SP)
WS (EN)
Zorro (%)
English Topline
2.43 ± 0.00
> 100000
0.58 ± 0.01
77.91 ± 1.09
Spanish Topline
> 1000
2.64 ± 0.01
0.08 ± 0.02
49.49 ± 2.12
English Baseline (random)
2.69 ± 0.01
> 100000
0.56 ± 0.02
75.55 ± 1.23
English Baseline (by-speaker)
4.63 ± 0.18
> 100000
0.55 ± 0.01
75.43 ± 0.55
Multilingual (random)
2.61 ± 0.02
2.82 ± 0.02
0.54 ± 0.02
74.51 ± 2.20
Multilingual (by-speaker)
3.04 ± 0.01
3.37 ± 0.02
0.54 ± 0.01
75.71 ± 0.89
Table 1: Evaluation results for GPT-2 (124M) across training conditions. We report mean ± std (across seeds) on perplexity (PPL), English word similarity (WS), and Zorro accuracy.
Figure 2: Multilingual and cross-lingual word similarity (X-WS) by training condition for GPT-2 models (124M). Dashes indicate the mean score across three training seeds.
Figure 3: Evaluation results for GPT-2 (124M) trained with varying language proportions in the Multilingual (random) condition. Markers denote the means over three seeds.
Reduced Model Size
Reduced Data Size
Condition
WS (EN)
Zorro (%)
WS (EN)
Zorro (%)
English Topline
0.58±0.01
77.11±0.61
0.47±0.03
73.86±1.92
Spanish Topline
0.08±0.00
50.21±1.24
0.06±0.01
47.57±2.12
English Baseline (random)
0.55±0.01
75.25±1.11
0.33±0.02
72.09±0.75
English Baseline (by-speaker)
0.55±0.01
75.73±1.02
0.35±0.01
69.65±0.57
Multilingual (random)
0.55±0.01
74.97±2.03
0.33±0.02
70.82±1.06
Table 2: Reduced size evaluation results on English WS and Zorro benchmarks. Left columns show GPT-2 Mini (39M) trained on 100M words, while right columns show GPT-2 Small (124M) trained on 20M words instead of 100M. Values show mean ± std across three seeds. Additional results on X-WS and PPL are shown in Appendix F .
Figure 4: Two-dimensional visualizations of the trained GPT-2 (124M) token embedding space, with English, Spanish, and shared tokens in blue, red, and purple respectively.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Language
Example
English
Mom: Hey, can you please clean up your art supplies before snack time? I don’t want to nag, but it’s getting a bit messy over there.\n\n Child: But I just started drawing! I want to finish this picture first. It’s going to be a present for Grandma!\n\n Mom: I know, sweetie, and I love that idea! But if you clean up now, you’ll have extra time to work on it later without any distractions.\n\n Child: Okay, but can I have a cookie after I clean up? Just one, please!\n\n Mom: Of course! One cookie after you tidy up. That way, you can enjoy your snack while you finish your drawing. Deal?
Spanish
Mom: Oye, ¿puedes por favor recoger tus materiales de arte antes de la hora del bocadillo? No quiero ser pesada, pero se está volviendo un poco desordenado allí.\n\n Child: ¡Pero acabo de empezar a dibujar! Quiero terminar esta imagen primero. ¡Va a ser un regalo para la abuela!\n\n Mom: Lo sé, cariño, y me encanta esa idea. Pero si limpias ahora, tendrás tiempo extra para trabajar en ello más tarde sin distracciones.\n\n Child: Está bien, pero ¿puedo comer una galleta después de limpiar? ¡Solo una, por favor!\n\n Mom: ¡Por supuesto! Una galleta después de que ordenes. Así podrás disfrutar de tu bocadillo mientras terminas tu dibujo. ¿Trato hecho?
Code-switching (sentence-level)
Mom: Hey, can you please clean up your art supplies before snack time? I don’t want to nag, but it’s getting a bit messy over there.\n\n Niño: ¡Pero acabo de empezar a dibujar! Quiero terminar esta imagen primero. ¡Va a ser un regalo para la abuela!\n\n Mom: I know, sweetie, and I love that idea! But if you clean up now, you’ll have extra time to work on it later without any distractions.\n\n Niño: Está bien, pero ¿puedo comer una galleta después de limpiar? ¡Solo una, por favor!\n\n Mamá: ¡Por supuesto! One cookie after you tidy up. That way, you can enjoy your snack while you finish your drawing. Deal?
Code-switching (word-level)
Mom: Hey, ¿puedes please clean up tus materiales de arte before snack time? No quiero nag, but it’s getting un poco messy over there.\n\n Child: But I just started dibujando! Quiero finish this picture first. It’s going to be un regalo para Grandma!\n\n Mom: I know, sweetie, y me encanta that idea! But if you limpias now, tendrás extra time to work on it later without distracciones.\n\n Child: Okay, but ¿puedo have una cookie after I clean up? Just una, please!\n\n Mom: Of course! Una cookie after you tidy up. Así you can enjoy your snack mientras you finish your drawing. Deal?
Appendix
Table 3: Example of the same generated dialogue (Mom, home, age 10, 5 turns, functional type) across the four language conditions of data collection after preprocessing.
Conversation Type
Explanation
Explanatory
It should involve explaining something(s) and potentially answering question(s).
Functional
It should involve attempting to get something(s) done or accomplishing particular goal(s).
Narrative
It should involve telling a story (real or fictional) or sharing/recounting an experience.
Argumentative
It should involve conflict(s) or disagreement(s) that lead to an argument.
In most cases, the argument should be resolved, resulting in the {child, toddler, teenager} learning.
Appendix
Table 4: The four conversation types along with their explanations.
Language
Total Words
Unique (Regex)
Unique (NLTK)
English
112,350,555
31,725
45,839
Spanish
103,559,408
79,637
111,127
Code-switching (sentence-level)
102,029,061
92,301
123,314
Code-switching (word-level)
109,510,491
87,392
118,219
Appendix
Table 5: Total and unique words for each generated language dataset after preprocessing and before creating conditions and splitting into train and validation data.
Scale
Condition
Total
Train
Val
100M
Full
799,885
759,893
39,992
100M
Baseline
399,949
379,953
19,996
20M
Full
152,000
144,400
7,600
20M
Baseline
76,000
72,200
3,800
Appendix
Table 6: Dialogue counts across dataset scales and conditions after preprocessing, creating conditions, and splitting into training and validation sets. The 100M setting corresponds to the main experiment, while the 20M setting represents the reduced-data condition. Corresponding baselines contain half as many dialogues (and words) as the full datasets. All datasets are split into 95% training and 5% validation.
Scale
Evaluation
Kept (Ratio)
100M
WR (EN)
7,207 / 7,917 (0.910)
Zorro (%)
33,494 / 46,000 (0.728)
X-WS (EN)
261 / 500 (0.522)
X-WS (SP)
259 / 500 (0.518)
X-WS (EN–SP)
511 / 978 (0.523)
20M
WR (EN)
6,907 / 7,917 (0.872)
Appendix
Table 7: Evaluation filtering statistics across tasks and dataset scales. “Kept” indicates the number of evaluation examples retained after vocabulary filtering, along with the corresponding proportion.
Training Condition
WS (EN)
Zorro (%)
English Topline
0.59
75.70
Spanish Topline
0.07
53.07
English Baseline (random)
0.56
74.01
Multilingual (random)
0.56
77.50
Code-switching (sentence-level)
0.55
78.07
Code-switching (word-level)
0.52
73.94
Appendix
Table 8: Evaluation results for GPT-BERT across training conditions.
Name
Zorro
Zorro (Mom)
English Topline
77.91 ± 1.09
79.06 ± 0.88
Spanish Topline
49.49 ± 2.12
47.89 ± 2.64
English Baseline (random)
75.55 ± 1.23
76.79 ± 0.88
English Baseline (Mom)
74.90 ± 1.89
76.07 ± 0.71
English Baseline (Dad)
75.96 ± 1.90
74.41 ± 0.54
Multilingual (random)
74.51 ± 2.20
75.77 ± 1.94
Appendix
Table 9: Results evaluating the effect of including a speaker label during Zorro evaluation across all language conditions. For multilingual conditions, the label in parentheses (e.g., Multilingual (Mom)) indicates that Mom is assigned to the English-speaking role. Results are reported as mean ± std across runs.
Training Condition
X-WS (EN)
X-WS (SP)
X-WS (EN-SP)
PPL (EN)
PPL (SP)
English Topline
0.61 ± 0.01
0.05 ± 0.01
0.01 ± 0.01
2.43 ± 0.00
> 100000
Spanish Topline
0.03 ± 0.01
0.55 ± 0.04
0.11 ± 0.01
> 1000
2.64 ± 0.00
English Baseline (random)
0.56 ± 0.01
0.09 ± 0.01
0.01 ± 0.00
2.70 ± 0.01
> 100000
English Baseline (by-speaker)
0.55 ± 0.01
0.11 ± 0.01
0.01 ± 0.01
4.73 ± 0.17
> 100000
Multilingual (random)
0.50 ± 0.02
0.43 ± 0.03
0.46 ± 0.03
2.62 ± 0.01
2.83 ± 0.01
Multilingual (by-speaker)
0.50 ± 0.01
0.44 ± 0.02
0.45 ± 0.02
3.05 ± 0.02
3.34 ± 0.02
Appendix
Table 10: Reduced model size evaluation results on multilingual X-WS and perplexity (PPL) for GPT-2 Mini (39M) trained on 100M words. Values show mean ± std across three seeds. X-WS (EN), X-WS (SP), and X-WS (EN–SP) refer to English–English, Spanish–Spanish, and cross-lingual pairs, respectively.
Training Condition
X-WS (EN)
X-WS (SP)
X-WS (EN-SP)
PPL (EN)
PPL (SP)
English Topline
0.51 ± 0.04
0.18 ± 0.02
0.02 ± 0.02
3.11 ± 0.02
> 100000
Spanish Topline
0.06 ± 0.01
0.44 ± 0.02
0.09 ± 0.03
> 1000
3.50 ± 0.01
English Baseline (random)
0.34 ± 0.02
0.12 ± 0.01
0.03 ± 0.01
3.70 ± 0.01
> 100000
English Baseline (by-speaker)
0.36 ± 0.02
0.13 ± 0.01
0.02 ± 0.03
6.52 ± 0.08
> 100000
Multilingual (random)
0.34 ± 0.03
0.21 ± 0.02
0.17 ± 0.04
3.58 ± 0.00
3.95 ± 0.01
Multilingual (by-speaker)
0.34 ± 0.01
0.24 ± 0.01
0.21 ± 0.01
4.09 ± 0.03
4.53 ± 0.02
Appendix
Table 11: Reduced data size evaluation results on multilingual X-WS and perplexity (PPL) for GPT-2 Small (124M) trained on 20M words instead of 100M. Values show mean ± std across three seeds. X-WS (EN), X-WS (SP), and X-WS (EN–SP) refer to English–English, Spanish–Spanish, and cross-lingual pairs, respectively.
We investigate two aspects of multilingual pre-training with limited data, namely how language exposure is distributed across English, Dutch, and Chinese, and how that exposure is ordered during training. In particular, we compare balanced multilingual exposure to language-dominant exposure, where one language is exposed with more training data than the others. We also compare mixed training, where multilingual training data is used throughout training, with curriculum training, where monolingual exposure of the dominant language is followed by balanced multilingual exposure. Our results show that multilingual BabyLM performance varies with both the amount and order of language exposure. Mixed training outperforms curriculum learning across all 15 matched configurations, indicating that simultaneous multilingual exposure is more effective than staged exposure in our setting. Several language-dominant mixed training configurations achieve higher scores than the balanced configuration, suggesting that a balanced distribution is not necessarily the highest-performing configuration in our setting. Interestingly, the Dutch-primary configurations achieve the best overall performance.
JiWoo Hwang, Lukas Edman, Alexander Fraser
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
Cross-linguistic effects are a central topic in bilingual first-language acquisition. Artificial learners can help investigate L1-L2 interactions by enabling controlled comparisons across language combinations and learning conditions. Recent work explores this direction by training bilingual language models under developmentally plausible constraints. However, human and model learners still diverge in fundamental ways, with one major difference being input modality: children learn primarily from spoken input, whereas language models are typically trained on orthographic text. To reduce this gap, researchers have trained models on phonemic representations of speech. In this work, we combine these research directions to train bilingual BabyLMs with phonemic input. We keep English fixed as the L2 and vary the L1 across German, Swedish, Persian, and Basque, selected to represent contrasting combinations of syntactic and phoneme-inventory distance from English. Our results show stronger L1-related variation in grammatical learning trajectories under phonemic than orthographic input, while early lexical differences align with phoneme-inventory similarity.
Nikitas Theodoropoulos, Maria Lymperaiou, Giorgos Filandrianos
Independent Researcher · National Technical University of Athens · Instituto de Telecomunicações, Portugal
Research on developmentally plausible language models has largely focused on English, leaving open questions about multilingual settings. We present a systematic study of compact language models by extending BabyBERTa to English-French scenarios under strictly size-matched data conditions, covering monolingual, bilingual, and cross-lingual settings. Our design contrasts two types of training corpora: (i) child-directed speech (about 2.5M tokens), following BabyBERTa and related work, and (ii) multi-domain corpora (about 10M tokens), extending the BabyLM framework to French. To enable fair evaluation, we also introduce new resources, including French versions of QAMR and QASRL, as well as English and French multi-domain corpora. We evaluate the models on both syntactic and semantic tasks and compare them with models trained on Wikipedia-only data. The results reveal context-dependent effects: training on Wikipedia consistently benefits semantic tasks, whereas child-directed speech improves grammatical judgments in monolingual settings. Bilingual pretraining yields notable gains for textual entailment, with particularly strong improvements for French. Importantly, similar patterns emerge across BabyBERTa, RoBERTa, and LTG-BERT, suggesting consistent trends across architectures.
Liel Binyamin, Elior Sulem
Faculty of Computer and Information Science, Institute for Applied AI Research Data Science Research Center Ben-Gurion University of the Negev