We investigate two aspects of multilingual pre-training with limited data, namely how language exposure is distributed across English, Dutch, and Chinese, and how that exposure is ordered during training. In particular, we compare balanced multilingual exposure to language-dominant exposure, where one language is exposed with more training data than the others. We also compare mixed training, where multilingual training data is used throughout training, with curriculum training, where monolingual exposure of the dominant language is followed by balanced multilingual exposure. Our results show that multilingual BabyLM performance varies with both the amount and order of language exposure. Mixed training outperforms curriculum learning across all 15 matched configurations, indicating that simultaneous multilingual exposure is more effective than staged exposure in our setting. Several language-dominant mixed training configurations achieve higher scores than the balanced configuration, suggesting that a balanced distribution is not necessarily the highest-performing configuration in our setting. Interestingly, the Dutch-primary configurations achieve the best overall performance.
Figures & tables
Figure 1: Mixed and curriculum training for the Dutch-primary 60/20/20 configuration. Mixed training shuffles all 100M tokens in a single stage, whereas curriculum training first uses 40M Dutch tokens and then a balanced multilingual stage containing 20M tokens from each language. For this configuration, mixed training achieves a higher multilingual average than curriculum training and produces the best overall result in our experiments.
Configuration
Primary
Other 1
Other 2
Balanced
33%
33%
33%
50/25/25
50%
25%
25%
60/20/20
60%
20%
20%
70/15/15
70%
15%
15%
80/10/10
80%
10%
10%
90/5/5
90%
5%
5%
Table 1: Data proportion configurations. Each language-dominant configuration is instantiated with English, Dutch, and Chinese as the primary language respectively.
Figure 2: Multilingual average performance across primary-language data proportions, grouped by the language designated as primary and by training strategy. Each point averages performance across English, Dutch, and Chinese, while the dashed horizontal line marks the balanced 33/33/33 baseline.
Figure 3: Relationship between language-specific data proportion and evaluation performance. Columns show English, Dutch, and Chinese data proportions, while rows show zero-shot, finetuned, and overall average performance for each language. Dashed lines indicate linear trends across all models, with R2 reported in each panel. The black square represents the balanced 33/33/33 baseline.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Model size
∼ 110M parameters
Tokenizer
Byte-level BPE
Vocabulary size
32,000
Context length
1024
Optimizer
AdamW
Learning rate
6×10−4
Appendix
Table 2: Training setup used for all models.
Model
English
Dutch
Chinese
Multi
Config
Strategy
ZS
FT
Avg
ZS
FT
Avg
ZS
FT
Avg
Avg
Balanced baseline
33/33/33
Mixed
55.48
45.52
47.06
54.99
42.70
46.61
48.73
43.78
45.03
46.23
English-primary
50/25/25
Mixed
56.50
45.94
48.15
54.77
41.80
45.89
48.96
43.25
44.50
46.18
Curr
54.39
42.79
45.91
53.05
43.47
46.39
48.13
44.97
45.87
46.06
Appendix
Table 3: Full results for all 31 models. ZS = zero-shot average, FT = finetune average, Avg = combined average for that language, Multi = multilingual average across all three languages. Bold indicates the best score in each column.
The goal of the BabyLM is to stimulate new research connections between cognitive modeling and language model pretraining. We invite contributions in this vein to the BabyLM Workshop, which will also include the 4th iteration of the BabyLM Challenge. As in previous years, the challenge features two ``standard'' tracks (Strict and Strict-Small), in which participants must train language models on under 100M or 10M words of data, respectively. This year, we move beyond our previous English-only pretraining datasets with a new Multilingual track, focusing on English, Dutch, and Chinese. For the workshop, we call for papers related to the overall theme of BabyLM, which includes training efficiency, small-scale training datasets, cognitive modeling, model evaluation, and architecture innovation.
Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul +7
IBM Research, MIT · ETH Zürich · Cornell University +7
Children in multilingual communities often code-switch, using multiple languages in a single utterance. Can we induce cross-lingual alignment in language models by training on code-switched text? We pretrain small decoder-only transformers on two 100M-word multilingual corpora: a base corpus formed by mixing the English, Dutch, and Chinese BabyBabelLM datasets, and a corpus generated from it by inserting word- and sentence-level code-switching using an LLM. We find that training on code-switched data aligns the representations of parallel text, particularly across different scripts, and that this alignment persists through training on monolingual documents. Under a learning curriculum that progresses from word-level code-switching, to sentence-level code-switching, to monolingual documents, models trained on code-switched data outperform baselines trained without it on the BabyLM evaluation suite. Our work characterizes code-switching curriculum learning as an effective data augmentation method for multilingual pretraining. We release our code, data, and models at https://github.com/drooryck/multilingual-macaroni.
Dries Rooryck, Alex Cai, Yonatan Belinkov +2
Kempner Institute, Harvard University, Cambridge, MA, USA · Technion – Israel Institute of Technology, Haifa, Israel
Multilingualism is incredibly common around the world, leading to many important theoretical and practical questions about how children learn multiple languages at once. For example, does multilingual acquisition lead to delays in learning? Are there better and worse ways to structure multilingual input? Many correlational studies address these questions, but it is surprisingly difficult to get definitive answers because children cannot be randomly assigned to be multilingual and data are typically not matched between languages. We use language model training as a method for simulating a variety of highly controlled exposure conditions, and create matched 100M-word mono- and bilingual datasets using synthetic data and machine translation. We train GPT-2 models on monolingual and bilingual data organized to reflect a range of exposure regimes, and evaluate their performance on perplexity, grammaticality, and semantic knowledge. Across model scales and measures, bilingual models perform similarly to monolingual models in one language, but show strong performance in the second language as well. These results suggest that there are no strong differences between different bilingual exposure regimes, and that bilingual input poses no in-principle challenges for agnostic statistical learners.