Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence

Dries Rooryck; Alex Cai; Yonatan Belinkov; David Alvarez-Melis; Kianté Brantley

cs.CLSep 24, 2026

Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence

Authors: Dries Rooryck, Alex Cai, Yonatan Belinkov, David Alvarez-Melis, Kianté Brantley

Organizations: Kempner Institute, Harvard University, Cambridge, MA, USA · Technion – Israel Institute of Technology, Haifa, Israel

Abstract

Children in multilingual communities often code-switch, using multiple languages in a single utterance. Can we induce cross-lingual alignment in language models by training on code-switched text? We pretrain small decoder-only transformers on two 100M-word multilingual corpora: a base corpus formed by mixing the English, Dutch, and Chinese BabyBabelLM datasets, and a corpus generated from it by inserting word- and sentence-level code-switching using an LLM. We find that training on code-switched data aligns the representations of parallel text, particularly across different scripts, and that this alignment persists through training on monolingual documents. Under a learning curriculum that progresses from word-level code-switching, to sentence-level code-switching, to monolingual documents, models trained on code-switched data outperform baselines trained without it on the BabyLM evaluation suite. Our work characterizes code-switching curriculum learning as an effective data augmentation method for multilingual pretraining. We release our code, data, and models at https://github.com/drooryck/multilingual-macaroni.

Figures & tables

Panel of Figure 1 — Figure 1: Composition of our synthetic code-switched corpus. A : We synthesize code-switched (CS) data by prompting an LLM to translate words or sentences from documents in the monolingual competition datasets. Examples shown are from our data. B : We train models on two corpora. Both are one-third English, Dutch, and Chinese. The non-CS corpus comprises monolingual source documents, while the CS corpus splits each language’s subset equally into word-level CS, sentence-level CS, and monolingual documents, shown left to right in curriculum training order.

matrix	emb.	Word	Sent.	Non-CS	Total
English	$\to$ NL	5.2	6.0	11.1	33.4
English	$\to$ ZH	5.9	5.1	11.1	33.4
Dutch	$\to$ EN	5.8	6.3	11.7	35.1
Dutch	$\to$ ZH	5.9	5.4	11.7	35.1
Chinese	$\to$ EN	5.7	5.8	10.4	31.2
Chinese	$\to$ NL	4.7	4.6	10.4	31.2

Table 1: Composition of the CS corpus by matrix language (the source document language) and embedded language (the language of the inserted words or translated sentences) in millions of byte-premium-adjusted words (see Appendix A ). Cells may not sum exactly to totals due to rounding.

					Selected tasks (accuracy, avg. over languages)
Ordering	Corpus	Leaderboard	Fine-tune avg.	Zero-shot avg.	SIB-200	INCLUDE	Global PIQA	BMLAMA	MultiBLiMP
Shuffled	non-CS	46.44 $\pm$ 0.25	42.27 $\pm$ 0.48	51.77 $\pm$ 0.17	71.85 $\pm$ 1.64	29.02 $\pm$ 2.61	36.84 $\pm$ 1.24	21.60 $\pm$ 2.56	86.91 $\pm$ 0.72
Shuffled	CS	46.55 $\pm$ 0.47	42.33 $\pm$ 0.79	51.95 $\pm$ 0.14	73.33 $\pm$ 1.23	28.96 $\pm$ 2.13	35.86 $\pm$ 0.69	19.87 $\pm$ 1.97	88.05 $\pm$ 0.87
Curriculum	non-CS	46.36 $\pm$ 0.37	42.26 $\pm$ 0.74	51.60 $\pm$ 0.22	72.33 $\pm$ 1.83	27.79 $\pm$ 3.33	36.85 $\pm$ 0.60	21.11 $\pm$ 1.56	86.52 $\pm$ 0.46
Curriculum	CS	46.72 $\pm$ 0.25	42.81 $\pm$ 0.42	51.72 $\pm$ 0.13	74.27 $\pm$ 1.20	29.13 $\pm$ 1.56	36.66 $\pm$ 0.78	20.31 $\pm$ 2.42	87.17 $\pm$ 0.50
Shuffled (controls)	word-level only	46.59 $\pm$ 0.24	42.60 $\pm$ 0.38	51.68 $\pm$ 0.25	73.44 $\pm$ 1.70	28.63 $\pm$ 2.54	35.79 $\pm$ 1.08	20.96 $\pm$ 1.80	87.02 $\pm$ 0.73

Table 2: Accuracy on the BabyLM evaluation suite (mean

\pm

SD) across eight seeds per condition. We select five tasks: the two language-averaged tasks with the largest positive gap between the curriculum CS and non-CS models (SIB-200

+1.94

, INCLUDE

+1.34

), the two with the most negative (BMLAMA

-0.80

, Global PIQA

-0.20

), and MultiBLiMP as a grammatical reference task. Bold cells mark statistical significance on a two-sided paired

t

-test between the CS and non-CS models (uncorrected for multiple hypotheses). Full per-task results are in Table 3 (Appendix C ).

Figure 2: Layerwise bitext retrieval precision@1 (§ 6 ) on

997

parallel sentences from FLORES+ across English, Dutch, and Chinese. We compare CS models against non-CS models (ignoring data ordering) and plot the mean

\pm 1

SD across the

16

runs per group. Retrieval in non-CS models fails for language pairs that differ in script. See Figure 6 (Appendix D ) for the median percentile rank of the translation out of the candidate set. Shading marks layers

6

–

11

, which the reported P@1 averages in other experiments are taken over.

Figure 3: Sentence-level alignment across the learning curriculum. Alignment improves throughout CS training and persists through training on monolingual documents. The color of the CS model’s line indicates the CS type of the current stage.

Figure 4: Median percentile rank of a word’s correct translation among a set of target-language words matched by part of speech and frequency (Appendix E ). The number of word pairs for each language pair is shown in parentheses.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Word-level CS	Sentence-level CS	Non-CS
English	EN $\leftarrow$ NL Rondo is a dorp in de VS staat Arkansas.	EN $\leftrightarrow$ ZH Landaff is a district in Wales. \cjkfont 它是英国作家罗尔德·达尔的出生地。	Rondo is a town in the US state of Arkansas.
Dutch	NL $\leftarrow$ ZH ‘Ga \cjkfont 走 , voordat het te laat \cjkfont 太晚 is!’ snauwde opa \cjkfont 爷爷 .	NL $\leftrightarrow$ EN Iemand voor schut zetten. To put someone to shame is to insult someone.	‘Ga, voordat het te laat is!’ snauwde opa.
Chinese	ZH $\leftarrow$ EN \cjkfont 但是我不会 swim \cjkfont 呀,你那 swimming posture \cjkfont 也分解不对呀。	ZH $\leftrightarrow$ NL \cjkfont 我知道了。 Dan moeten we morgen zeker ijs eten.	\cjkfont 但是我不会游泳呀,你那游泳姿势也分解不对呀。

Appendix

Figure 5: Verbatim corpus examples by matrix language (rows) and switch type (columns). Embedded-language spans are colored by language ( English , Dutch , Chinese ). The non-CS cell in each row is the true source document of that row’s word-level example.

English
Task	Chance	Shuf	CS/Shuf	Curr	CS/Curr	Word	Sent	Par	Salad
BLiMP	50.0	67.00 $\pm$ 0.97	68.27 $\pm$ 0.56	66.17 $\pm$ 0.90	67.18 $\pm$ 0.98	67.95 $\pm$ 0.45	68.68 $\pm$ 0.62	68.03 $\pm$ 0.73	65.88 $\pm$ 0.78
MultiBLiMP	50.0	83.75 $\pm$ 1.23	85.70 $\pm$ 1.55	83.64 $\pm$ 0.84	84.74 $\pm$ 0.85	83.85 $\pm$ 1.27	85.55 $\pm$ 0.89	85.50 $\pm$ 1.22	82.48 $\pm$ 1.66
HellaSwag	25.0	26.67 $\pm$ 0.14	26.48 $\pm$ 0.14	26.60 $\pm$ 0.10	26.56 $\pm$ 0.26	26.63 $\pm$ 0.14	26.59 $\pm$ 0.24	26.59 $\pm$ 0.17	26.52 $\pm$ 0.16
Winogrande	50.0	50.37 $\pm$ 0.66	50.06 $\pm$ 1.08	50.30 $\pm$ 1.06	50.60 $\pm$ 0.75	49.97 $\pm$ 0.63	50.01 $\pm$ 0.62	50.13 $\pm$ 0.80	49.89 $\pm$ 0.51
XStoryCloze	50.0	50.60 $\pm$ 0.60	50.22 $\pm$ 0.71	50.24 $\pm$ 0.48	50.24 $\pm$ 0.37	50.18 $\pm$ 0.57	49.63 $\pm$ 0.72	50.74 $\pm$ 0.62	50.07 $\pm$ 0.76

Appendix

Table 3: Accuracy on the BabyLM evaluation suite per task and per language. We report (%) mean

\pm

SD across eight seeds per condition. Chance performance is random-guess accuracy (

1/\#

options) where applicable. Word/Sent/Par/Salad are the shuffled-ordering controls (§ 3 ). Leaderboard is the official multilingual-average score as in Table 2 .

Condition	English	Dutch	Chinese
Shuf	1.60 $\pm$ 0.05	1.56 $\pm$ 0.04	1.78 $\pm$ 0.03
CS/Shuf	1.45 $\pm$ 0.01	1.45 $\pm$ 0.01	1.75 $\pm$ 0.01
Curr	1.56 $\pm$ 0.02	1.53 $\pm$ 0.01	1.78 $\pm$ 0.01
CS/Curr	1.56 $\pm$ 0.03	1.56 $\pm$ 0.03	1.80 $\pm$ 0.01
Word-level CS only	1.52 $\pm$ 0.04	1.53 $\pm$ 0.03	1.78 $\pm$ 0.02
Sentence-level CS only	1.42 $\pm$ 0.02	1.42 $\pm$ 0.02	1.73 $\pm$ 0.02

Appendix

Table 4: Bits per UTF-8 byte on

1{,}200

held-out BabyBabelLM documents per language (lower is better). The first

512

tokens of each document are scored.

Figure 6: In addition to sentence-level bitext retrieval precision@1 (Figure 2 ), here we show the per-layer median percentile rank of the correct translation. Averaged over layers 6–11, the non-CS models place the correct translation at median rank

105

and

121

out of

997

for English-Chinese and Dutch-Chinese respectively, far above chance at

499

. This suggests that code-switching does not create alignment “from scratch”, but rather “sharpens” a general degree of alignment found even in the non-CS models.

Pair	P@1	CKA	cos-gap	centroid $\downarrow$
En–Nl	91/84	0.69/0.67	0.11/0.09	0.36/0.33
En–Zh	68/7	0.63/0.56	0.10/0.03	0.48/0.53
Nl–Zh	58/6	0.60/0.54	0.10/0.03	0.47/0.53
MEXA Dutch	0.87/0.77
MEXA Chinese	0.61/0.07

Appendix

Table 5: Cross-lingual alignment measures beyond retrieval (curriculum ordering). Each cell shows the mean CS/non-CS values across seeds at the layer of peak CSLS retrieval in the CS model; We measure linear CKA ( Kornblith et al., 2019 ) , the cosine gap (mean cosine over the

N

true translation pairs minus that over a fixed random derangement of the pairs), and the per-language centroid distance ( Libovický et al., 2020 , lower is better;) . The last two rows are per-language MEXA ( Kargaran et al., 2025 ) (max over layers vs. the English pivot).

Condition	En–Nl	En–Zh	Nl–Zh	Leaderboard
	Bitext retrieval P@1 (%)
Shuf	78.5 $\pm$ 0.7	5.4 $\pm$ 0.8	4.5 $\pm$ 0.8	46.44
CS/Shuf	88.8 $\pm$ 0.5	65.5 $\pm$ 1.1	54.5 $\pm$ 1.0	46.55
Curr	79.0 $\pm$ 0.8	5.5 $\pm$ 0.6	4.5 $\pm$ 0.5	46.36
CS/Curr	87.5 $\pm$ 0.7	61.4 $\pm$ 0.8	52.5 $\pm$ 0.6	46.72
Word	85.5 $\pm$ 0.8	48.1 $\pm$ 1.5	38.5 $\pm$ 1.4	46.59

Appendix

Table 6: Cross-lingual alignment of all eight conditions at their final checkpoints; alignment columns are

8

-seed mean

\pm

SD. Left: FLORES+ bitext retrieval P@1 (CSLS,

k{=}10

) averaged over layers 6–11, per language pair. Right: the leaderboard score of Table 2 . Column abbreviations as in Table 3 .

Figure 7: Per-layer bitext retrieval P@1 for all eight experimental conditions (§ 4 ), overlaid in each language-pair panel. We use neutral colors for the four main conditions, each pairing of corpus (CS/non-CS) and ordering (shuffled/curriculum), and color for the four shuffled-ordering controls. Lines are

8

-seed means, bands

\pm 1

SD, dotted line chance. Shading marks layers

6

–

11

, the range averaged over in Figure 3 .

Figure 8: Layerwise cosine similarity between the CS and non-CS models’ word representations (§ 7 ), shown separately for the embedded words and their never-embedded controls (

8

-seed mean

\pm 1

SD).

Explore similar work

Introducing Code-Switched Contexts to Cognitively-Inspired Bilingual Model Training
Oct 5, 2026Zhuojing Huang, Luise Pohlmann, Lisa BeinbornLanguage Model Pretraining Multilingual Language Models
University of Göttingen, Germany
Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training
Oct 14, 2026JiWoo Hwang, Lukas Edman, Alexander FraserLanguage Model Pretraining Multilingual Language Models
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning
May 28, 2026Shunta Asano, Jeonghun Baek, Toshihiko YamasakiMultilingual Language Models Code-Switching
The University of Tokyo

Oct 5, 2026J/K move · Enter open · S save

Introducing Code-Switched Contexts to Cognitively-Inspired Bilingual Model Training

Zhuojing Huang, Luise Pohlmann, Lisa Beinborn

University of Göttingen, Germany

Language Model Pretraining Multilingual Language Models Cross-Lingual Representation Alignment Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

During language acquisition, bilingual children are regularly exposed to code-switched input and use it as a cognitive scaffold to accelerate vocabulary growth and cross-linguistic syntactic mapping. In contrast, computational bilingual models are conventionally pretrained on interleaved monolingual corpora. While introducing synthetic code-switching during pretraining has become a promising strategy to enhance cross-lingual alignment and downstream performance, the structural and developmental parameters governing the success remain poorly understood. In this work, we investigate the efficiency of training with synthetic code-switched data across two typologically distinct language pairs by controlling two key variables: the structural location of code-switches and the dynamic switching rate across training stages. Our results show that training with code-switched data improves cross-lingual alignment for typologically close languages.

Abstract

Explore similar work

Introducing Code-Switched Contexts to Cognitively-Inspired Bilingual Model Training

Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training

Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning