BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop
Authors: Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Jaap Jumelet, Tal Linzen, Aaron Mueller, Suchir Salhan, Raj Sanjay Shah, +2 more
Organizations: IBM Research, MIT · ETH Zürich · Cornell University · University of Groningen · NYU · Boston University · University of Cambridge · Georgia Tech · UC San Diego · Georgetown University
The goal of the BabyLM is to stimulate new research connections between cognitive modeling and language model pretraining. We invite contributions in this vein to the BabyLM Workshop, which will also include the 4th iteration of the BabyLM Challenge. As in previous years, the challenge features two ``standard'' tracks (Strict and Strict-Small), in which participants must train language models on under 100M or 10M words of data, respectively. This year, we move beyond our previous English-only pretraining datasets with a new Multilingual track, focusing on English, Dutch, and Chinese. For the workshop, we call for papers related to the overall theme of BabyLM, which includes training efficiency, small-scale training datasets, cognitive modeling, model evaluation, and architecture innovation.
Figures & tables
Dataset
Description
# Words (multimodal data)
# Words (strict data)
# Images
Localized Narratives Pont-Tuset et al. (2020)
Image Caption
27M
–
0.6M
Conceptual Captions 3M Sharma et al. (2018)
Image Caption
23M
–
2.3M
CHILDES ( MacWhinney, 2000 )
Child-directed speech
15M
28.4M
–
British National Corpus (BNC), dialogue portion
Dialogue
4M
7.6M
–
Project Gutenberg (children’s stories) ( Gerlach and Font-Clos, 2018 )
Written English
13M
25.5M
–
OpenSubtitles ( Lison and Tiedemann, 2016 )
Movie subtitles
10M
21.8M
–
Table 1: Datasets for the strict track of the BabyLM competition. We release a text-only and multimodal dataset; either is allowed, as are outside corpora, so long as the total word count of the training corpus is less than 100M. Datasets have been detoxified following the procedure of Trhlik et al. (2026) (§ 4.3 ).
This paper describes the first ChineseBabyLM challenge, which will be held in the 2026 NLPCC conference. The challenge calls for researchers to train language models from scratch with 100 million Chinese tokens and evaluates the models on 3 tracks of tasks: NLU, cognitive alignment and Hanzi knowledge. There is no restriction on tokenizer, model architecture and the number of training epochs. Details of the challenge can be found in https://chinese-babylm.github.io/.
Siyuan Song, Zhiheng Qian, Yunhao Zhang +11
Princeton University · Shanghai Jiao Tong University · Chinese Academy of Sciences +5
We investigate two aspects of multilingual pre-training with limited data, namely how language exposure is distributed across English, Dutch, and Chinese, and how that exposure is ordered during training. In particular, we compare balanced multilingual exposure to language-dominant exposure, where one language is exposed with more training data than the others. We also compare mixed training, where multilingual training data is used throughout training, with curriculum training, where monolingual exposure of the dominant language is followed by balanced multilingual exposure. Our results show that multilingual BabyLM performance varies with both the amount and order of language exposure. Mixed training outperforms curriculum learning across all 15 matched configurations, indicating that simultaneous multilingual exposure is more effective than staged exposure in our setting. Several language-dominant mixed training configurations achieve higher scores than the balanced configuration, suggesting that a balanced distribution is not necessarily the highest-performing configuration in our setting. Interestingly, the Dutch-primary configurations achieve the best overall performance.
JiWoo Hwang, Lukas Edman, Alexander Fraser
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
This work concerns our submission for the Multilingual track of the 2026 BabyLM Challenge. We improve on our baseline using data that has been summarized and translated by an LLM, showing that higher-quality data can come from synthetic sources. We further improve with 2 simple changes: using a larger model, and using a character-level vocabulary. Both of these changes require LoRA-style fine-tuning for realizing their full potential. We dub our final Qwen3-based model Qwenfant, which, at the time of submission, ranks 1st in the Multilingual track.
Lukas Edman, Alexander Fraser
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute