cs.CLMar 31, 2026
SaveBringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models
Organizations: Stanford University
Abstract
Multilingualism is incredibly common around the world, leading to many important theoretical and practical questions about how children learn multiple languages at once. For example, does multilingual acquisition lead to delays in learning? Are there better and worse ways to structure multilingual input? Many correlational studies address these questions, but it is surprisingly difficult to get definitive answers because children cannot be randomly assigned to be multilingual and data are typically not matched between languages. We use language model training as a method for simulating a variety of highly controlled exposure conditions, and create matched 100M-word mono- and bilingual datasets using synthetic data and machine translation. We train GPT-2 models on monolingual and bilingual data organized to reflect a range of exposure regimes, and evaluate their performance on perplexity, grammaticality, and semantic knowledge. Across model scales and measures, bilingual models perform similarly to monolingual models in one language, but show strong performance in the second language as well. These results suggest that there are no strong differences between different bilingual exposure regimes, and that bilingual input poses no in-principle challenges for agnostic statistical learners.
Figures & tables
Figure 1: General motivation and condition structure for main experiments. Note ”by-speaker” refers to a one speaker–one language policy (shown in the example), while “random” allows either speaker to produce either language (randomly assigned per dialogue).
| Training Condition | PPL (EN) | PPL (SP) | WS (EN) | Zorro (%) |
|---|---|---|---|---|
| English Topline | 2.43 0.00 | 100000 | 0.58 0.01 | 77.91 1.09 |
| Spanish Topline | 1000 | 2.64 0.01 | 0.08 0.02 | 49.49 2.12 |
| English Baseline (random) | 2.69 0.01 | 100000 | 0.56 0.02 | 75.55 1.23 |
| English Baseline (by-speaker) | 4.63 0.18 | 100000 | 0.55 0.01 | 75.43 0.55 |
| Multilingual (random) | 2.61 0.02 | 2.82 0.02 | 0.54 0.02 | 74.51 2.20 |
| Multilingual (by-speaker) | 3.04 0.01 | 3.37 0.02 | 0.54 0.01 | 75.71 0.89 |
Table 1: Evaluation results for GPT-2 (124M) across training conditions. We report mean ± std (across seeds) on perplexity (PPL), English word similarity (WS), and Zorro accuracy.
Figure 2: Multilingual and cross-lingual word similarity (X-WS) by training condition for GPT-2 models (124M). Dashes indicate the mean score across three training seeds.
Figure 3: Evaluation results for GPT-2 (124M) trained with varying language proportions in the Multilingual (random) condition. Markers denote the means over three seeds.
| Reduced Model Size | Reduced Data Size | |||
|---|---|---|---|---|
| Condition | WS (EN) | Zorro (%) | WS (EN) | Zorro (%) |
| English Topline | ||||
| Spanish Topline | ||||
| English Baseline (random) | ||||
| English Baseline (by-speaker) | ||||
| Multilingual (random) | ||||
Table 2: Reduced size evaluation results on English WS and Zorro benchmarks. Left columns show GPT-2 Mini (39M) trained on 100M words, while right columns show GPT-2 Small (124M) trained on 20M words instead of 100M. Values show mean std across three seeds. Additional results on X-WS and PPL are shown in Appendix F .
Figure 4: Two-dimensional visualizations of the trained GPT-2 (124M) token embedding space, with English, Spanish, and shared tokens in blue, red, and purple respectively.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Language | Example |
|---|---|
| English | Mom: Hey, can you please clean up your art supplies before snack time? I don’t want to nag, but it’s getting a bit messy over there.\n\n Child: But I just started drawing! I want to finish this picture first. It’s going to be a present for Grandma!\n\n Mom: I know, sweetie, and I love that idea! But if you clean up now, you’ll have extra time to work on it later without any distractions.\n\n Child: Okay, but can I have a cookie after I clean up? Just one, please!\n\n Mom: Of course! One cookie after you tidy up. That way, you can enjoy your snack while you finish your drawing. Deal? |
| Spanish | Mom: Oye, ¿puedes por favor recoger tus materiales de arte antes de la hora del bocadillo? No quiero ser pesada, pero se está volviendo un poco desordenado allí.\n\n Child: ¡Pero acabo de empezar a dibujar! Quiero terminar esta imagen primero. ¡Va a ser un regalo para la abuela!\n\n Mom: Lo sé, cariño, y me encanta esa idea. Pero si limpias ahora, tendrás tiempo extra para trabajar en ello más tarde sin distracciones.\n\n Child: Está bien, pero ¿puedo comer una galleta después de limpiar? ¡Solo una, por favor!\n\n Mom: ¡Por supuesto! Una galleta después de que ordenes. Así podrás disfrutar de tu bocadillo mientras terminas tu dibujo. ¿Trato hecho? |
| Code-switching (sentence-level) | Mom: Hey, can you please clean up your art supplies before snack time? I don’t want to nag, but it’s getting a bit messy over there.\n\n Niño: ¡Pero acabo de empezar a dibujar! Quiero terminar esta imagen primero. ¡Va a ser un regalo para la abuela!\n\n Mom: I know, sweetie, and I love that idea! But if you clean up now, you’ll have extra time to work on it later without any distractions.\n\n Niño: Está bien, pero ¿puedo comer una galleta después de limpiar? ¡Solo una, por favor!\n\n Mamá: ¡Por supuesto! One cookie after you tidy up. That way, you can enjoy your snack while you finish your drawing. Deal? |
| Code-switching (word-level) | Mom: Hey, ¿puedes please clean up tus materiales de arte before snack time? No quiero nag, but it’s getting un poco messy over there.\n\n Child: But I just started dibujando! Quiero finish this picture first. It’s going to be un regalo para Grandma!\n\n Mom: I know, sweetie, y me encanta that idea! But if you limpias now, tendrás extra time to work on it later without distracciones.\n\n Child: Okay, but ¿puedo have una cookie after I clean up? Just una, please!\n\n Mom: Of course! Una cookie after you tidy up. Así you can enjoy your snack mientras you finish your drawing. Deal? |
Table 3: Example of the same generated dialogue (Mom, home, age 10, 5 turns, functional type) across the four language conditions of data collection after preprocessing.
| Conversation Type | Explanation |
|---|---|
| Explanatory | It should involve explaining something(s) and potentially answering question(s). |
| Functional | It should involve attempting to get something(s) done or accomplishing particular goal(s). |
| Narrative | It should involve telling a story (real or fictional) or sharing/recounting an experience. |
| Argumentative | It should involve conflict(s) or disagreement(s) that lead to an argument. |
| In most cases, the argument should be resolved, resulting in the {child, toddler, teenager} learning. |
Table 4: The four conversation types along with their explanations.
| Language | Total Words | Unique (Regex) | Unique (NLTK) |
|---|---|---|---|
| English | 112,350,555 | 31,725 | 45,839 |
| Spanish | 103,559,408 | 79,637 | 111,127 |
| Code-switching (sentence-level) | 102,029,061 | 92,301 | 123,314 |
| Code-switching (word-level) | 109,510,491 | 87,392 | 118,219 |
Table 5: Total and unique words for each generated language dataset after preprocessing and before creating conditions and splitting into train and validation data.
| Scale | Condition | Total | Train | Val |
|---|---|---|---|---|
| 100M | Full | 799,885 | 759,893 | 39,992 |
| 100M | Baseline | 399,949 | 379,953 | 19,996 |
| 20M | Full | 152,000 | 144,400 | 7,600 |
| 20M | Baseline | 76,000 | 72,200 | 3,800 |
Table 6: Dialogue counts across dataset scales and conditions after preprocessing, creating conditions, and splitting into training and validation sets. The 100M setting corresponds to the main experiment, while the 20M setting represents the reduced-data condition. Corresponding baselines contain half as many dialogues (and words) as the full datasets. All datasets are split into 95% training and 5% validation.
| Scale | Evaluation | Kept (Ratio) |
|---|---|---|
| 100M | WR (EN) | 7,207 / 7,917 (0.910) |
| Zorro (%) | 33,494 / 46,000 (0.728) | |
| X-WS (EN) | 261 / 500 (0.522) | |
| X-WS (SP) | 259 / 500 (0.518) | |
| X-WS (EN–SP) | 511 / 978 (0.523) | |
| 20M | WR (EN) | 6,907 / 7,917 (0.872) |
Table 7: Evaluation filtering statistics across tasks and dataset scales. “Kept” indicates the number of evaluation examples retained after vocabulary filtering, along with the corresponding proportion.
| Training Condition | WS (EN) | Zorro (%) |
|---|---|---|
| English Topline | 0.59 | 75.70 |
| Spanish Topline | 0.07 | 53.07 |
| English Baseline (random) | 0.56 | 74.01 |
| Multilingual (random) | 0.56 | 77.50 |
| Code-switching (sentence-level) | 0.55 | 78.07 |
| Code-switching (word-level) | 0.52 | 73.94 |
Table 8: Evaluation results for GPT-BERT across training conditions.
| Name | Zorro | Zorro (Mom) |
|---|---|---|
| English Topline | 77.91 1.09 | 79.06 0.88 |
| Spanish Topline | 49.49 2.12 | 47.89 2.64 |
| English Baseline (random) | 75.55 1.23 | 76.79 0.88 |
| English Baseline (Mom) | 74.90 1.89 | 76.07 0.71 |
| English Baseline (Dad) | 75.96 1.90 | 74.41 0.54 |
| Multilingual (random) | 74.51 2.20 | 75.77 1.94 |
Table 9: Results evaluating the effect of including a speaker label during Zorro evaluation across all language conditions. For multilingual conditions, the label in parentheses (e.g., Multilingual (Mom)) indicates that Mom is assigned to the English-speaking role. Results are reported as mean std across runs.
| Training Condition | X-WS (EN) | X-WS (SP) | X-WS (EN-SP) | PPL (EN) | PPL (SP) |
|---|---|---|---|---|---|
| English Topline | 0.61 0.01 | 0.05 0.01 | 0.01 0.01 | 2.43 0.00 | 100000 |
| Spanish Topline | 0.03 0.01 | 0.55 0.04 | 0.11 0.01 | 1000 | 2.64 0.00 |
| English Baseline (random) | 0.56 0.01 | 0.09 0.01 | 0.01 0.00 | 2.70 0.01 | 100000 |
| English Baseline (by-speaker) | 0.55 0.01 | 0.11 0.01 | 0.01 0.01 | 4.73 0.17 | 100000 |
| Multilingual (random) | 0.50 0.02 | 0.43 0.03 | 0.46 0.03 | 2.62 0.01 | 2.83 0.01 |
| Multilingual (by-speaker) | 0.50 0.01 | 0.44 0.02 | 0.45 0.02 | 3.05 0.02 | 3.34 0.02 |
Table 10: Reduced model size evaluation results on multilingual X-WS and perplexity (PPL) for GPT-2 Mini (39M) trained on 100M words. Values show mean ± std across three seeds. X-WS (EN), X-WS (SP), and X-WS (EN–SP) refer to English–English, Spanish–Spanish, and cross-lingual pairs, respectively.
| Training Condition | X-WS (EN) | X-WS (SP) | X-WS (EN-SP) | PPL (EN) | PPL (SP) |
|---|---|---|---|---|---|
| English Topline | 0.51 0.04 | 0.18 0.02 | 0.02 0.02 | 3.11 0.02 | 100000 |
| Spanish Topline | 0.06 0.01 | 0.44 0.02 | 0.09 0.03 | 1000 | 3.50 0.01 |
| English Baseline (random) | 0.34 0.02 | 0.12 0.01 | 0.03 0.01 | 3.70 0.01 | 100000 |
| English Baseline (by-speaker) | 0.36 0.02 | 0.13 0.01 | 0.02 0.03 | 6.52 0.08 | 100000 |
| Multilingual (random) | 0.34 0.03 | 0.21 0.02 | 0.17 0.04 | 3.58 0.00 | 3.95 0.01 |
| Multilingual (by-speaker) | 0.34 0.01 | 0.24 0.01 | 0.21 0.01 | 4.09 0.03 | 4.53 0.02 |
Table 11: Reduced data size evaluation results on multilingual X-WS and perplexity (PPL) for GPT-2 Small (124M) trained on 20M words instead of 100M. Values show mean ± std across three seeds. X-WS (EN), X-WS (SP), and X-WS (EN–SP) refer to English–English, Spanish–Spanish, and cross-lingual pairs, respectively.