Children in multilingual communities often code-switch, using multiple languages in a single utterance. Can we induce cross-lingual alignment in language models by training on code-switched text? We pretrain small decoder-only transformers on two 100M-word multilingual corpora: a base corpus formed by mixing the English, Dutch, and Chinese BabyBabelLM datasets, and a corpus generated from it by inserting word- and sentence-level code-switching using an LLM. We find that training on code-switched data aligns the representations of parallel text, particularly across different scripts, and that this alignment persists through training on monolingual documents. Under a learning curriculum that progresses from word-level code-switching, to sentence-level code-switching, to monolingual documents, models trained on code-switched data outperform baselines trained without it on the BabyLM evaluation suite. Our work characterizes code-switching curriculum learning as an effective data augmentation method for multilingual pretraining. We release our code, data, and models at https://github.com/drooryck/multilingual-macaroni.
Figures & tables
Figure 1: Composition of our synthetic code-switched corpus. A : We synthesize code-switched (CS) data by prompting an LLM to translate words or sentences from documents in the monolingual competition datasets. Examples shown are from our data. B : We train models on two corpora. Both are one-third English, Dutch, and Chinese. The non-CS corpus comprises monolingual source documents, while the CS corpus splits each language’s subset equally into word-level CS, sentence-level CS, and monolingual documents, shown left to right in curriculum training order.
matrix
emb.
Word
Sent.
Non-CS
Total
English
→ NL
5.2
6.0
11.1
33.4
→ ZH
5.9
5.1
Dutch
→ EN
5.8
6.3
11.7
35.1
→ ZH
5.9
5.4
Chinese
→ EN
5.7
5.8
10.4
31.2
→ NL
4.7
4.6
Table 1: Composition of the CS corpus by matrix language (the source document language) and embedded language (the language of the inserted words or translated sentences) in millions of byte-premium-adjusted words (see Appendix A ). Cells may not sum exactly to totals due to rounding.
Selected tasks (accuracy, avg. over languages)
Ordering
Corpus
Leaderboard
Fine-tune avg.
Zero-shot avg.
SIB-200
INCLUDE
Global PIQA
BMLAMA
MultiBLiMP
Shuffled
non-CS
46.44 ± 0.25
42.27 ± 0.48
51.77 ± 0.17
71.85 ± 1.64
29.02 ± 2.61
36.84 ± 1.24
21.60 ± 2.56
86.91 ± 0.72
CS
46.55 ± 0.47
42.33 ± 0.79
51.95 ± 0.14
73.33 ± 1.23
28.96 ± 2.13
35.86 ± 0.69
19.87 ± 1.97
88.05 ± 0.87
Curriculum
non-CS
46.36 ± 0.37
42.26 ± 0.74
51.60 ± 0.22
72.33 ± 1.83
27.79 ± 3.33
36.85 ± 0.60
21.11 ± 1.56
86.52 ± 0.46
CS
46.72 ± 0.25
42.81 ± 0.42
51.72 ± 0.13
74.27 ± 1.20
29.13 ± 1.56
36.66 ± 0.78
20.31 ± 2.42
87.17 ± 0.50
Shuffled (controls)
word-level only
46.59 ± 0.24
42.60 ± 0.38
51.68 ± 0.25
73.44 ± 1.70
28.63 ± 2.54
35.79 ± 1.08
20.96 ± 1.80
87.02 ± 0.73
Table 2: Accuracy on the BabyLM evaluation suite (mean ± SD) across eight seeds per condition. We select five tasks: the two language-averaged tasks with the largest positive gap between the curriculum CS and non-CS models (SIB-200 +1.94 , INCLUDE +1.34 ), the two with the most negative (BMLAMA −0.80 , Global PIQA −0.20 ), and MultiBLiMP as a grammatical reference task. Bold cells mark statistical significance on a two-sided paired t -test between the CS and non-CS models (uncorrected for multiple hypotheses). Full per-task results are in Table 3 (Appendix C ).
Figure 2: Layerwise bitext retrieval precision@1 (§ 6 ) on 997 parallel sentences from FLORES+ across English, Dutch, and Chinese. We compare CS models against non-CS models (ignoring data ordering) and plot the mean ±1 SD across the 16 runs per group. Retrieval in non-CS models fails for language pairs that differ in script. See Figure 6 (Appendix D ) for the median percentile rank of the translation out of the candidate set. Shading marks layers 6 – 11 , which the reported P@1 averages in other experiments are taken over.
Figure 3: Sentence-level alignment across the learning curriculum. Alignment improves throughout CS training and persists through training on monolingual documents. The color of the CS model’s line indicates the CS type of the current stage.
Figure 4: Median percentile rank of a word’s correct translation among a set of target-language words matched by part of speech and frequency (Appendix E ). The number of word pairs for each language pair is shown in parentheses.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Word-level CS
Sentence-level CS
Non-CS
English
EN ← NL Rondo is a dorp in de VS staat Arkansas.
EN ↔ ZH Landaff is a district in Wales. \cjkfont 它是英国作家罗尔德·达尔的出生地。
Rondo is a town in the US state of Arkansas.
Dutch
NL ← ZH ‘Ga \cjkfont 走 , voordat het te laat \cjkfont 太晚 is!’ snauwde opa \cjkfont 爷爷 .
NL ↔ EN Iemand voor schut zetten. To put someone to shame is to insult someone.
ZH ↔ NL \cjkfont 我知道了。 Dan moeten we morgen zeker ijs eten.
\cjkfont 但是我不会游泳呀,你那游泳姿势也分解不对呀。
Appendix
Figure 5: Verbatim corpus examples by matrix language (rows) and switch type (columns). Embedded-language spans are colored by language ( English , Dutch , Chinese ). The non-CS cell in each row is the true source document of that row’s word-level example.
Task
Chance
Shuf
CS/Shuf
Curr
CS/Curr
Word
Sent
Par
Salad
English
BLiMP
50.0
67.00 ± 0.97
68.27 ± 0.56
66.17 ± 0.90
67.18 ± 0.98
67.95 ± 0.45
68.68 ± 0.62
68.03 ± 0.73
65.88 ± 0.78
MultiBLiMP
50.0
83.75 ± 1.23
85.70 ± 1.55
83.64 ± 0.84
84.74 ± 0.85
83.85 ± 1.27
85.55 ± 0.89
85.50 ± 1.22
82.48 ± 1.66
HellaSwag
25.0
26.67 ± 0.14
26.48 ± 0.14
26.60 ± 0.10
26.56 ± 0.26
26.63 ± 0.14
26.59 ± 0.24
26.59 ± 0.17
26.52 ± 0.16
Winogrande
50.0
50.37 ± 0.66
50.06 ± 1.08
50.30 ± 1.06
50.60 ± 0.75
49.97 ± 0.63
50.01 ± 0.62
50.13 ± 0.80
49.89 ± 0.51
XStoryCloze
50.0
50.60 ± 0.60
50.22 ± 0.71
50.24 ± 0.48
50.24 ± 0.37
50.18 ± 0.57
49.63 ± 0.72
50.74 ± 0.62
50.07 ± 0.76
Appendix
Table 3: Accuracy on the BabyLM evaluation suite per task and per language. We report (%) mean ± SD across eight seeds per condition. Chance performance is random-guess accuracy ( 1/# options) where applicable. Word/Sent/Par/Salad are the shuffled-ordering controls (§ 3 ). Leaderboard is the official multilingual-average score as in Table 2 .
Condition
English
Dutch
Chinese
Shuf
1.60 ± 0.05
1.56 ± 0.04
1.78 ± 0.03
CS/Shuf
1.45 ± 0.01
1.45 ± 0.01
1.75 ± 0.01
Curr
1.56 ± 0.02
1.53 ± 0.01
1.78 ± 0.01
CS/Curr
1.56 ± 0.03
1.56 ± 0.03
1.80 ± 0.01
Word-level CS only
1.52 ± 0.04
1.53 ± 0.03
1.78 ± 0.02
Sentence-level CS only
1.42 ± 0.02
1.42 ± 0.02
1.73 ± 0.02
Appendix
Table 4: Bits per UTF-8 byte on 1,200 held-out BabyBabelLM documents per language (lower is better). The first 512 tokens of each document are scored.
Figure 6: In addition to sentence-level bitext retrieval precision@1 (Figure 2 ), here we show the per-layer median percentile rank of the correct translation. Averaged over layers 6–11, the non-CS models place the correct translation at median rank 105 and 121 out of 997 for English-Chinese and Dutch-Chinese respectively, far above chance at 499 . This suggests that code-switching does not create alignment “from scratch”, but rather “sharpens” a general degree of alignment found even in the non-CS models.
Pair
P@1
CKA
cos-gap
centroid ↓
En–Nl
91/84
0.69/0.67
0.11/0.09
0.36/0.33
En–Zh
68/7
0.63/0.56
0.10/0.03
0.48/0.53
Nl–Zh
58/6
0.60/0.54
0.10/0.03
0.47/0.53
MEXA Dutch
0.87/0.77
MEXA Chinese
0.61/0.07
Appendix
Table 5: Cross-lingual alignment measures beyond retrieval (curriculum ordering). Each cell shows the mean CS/non-CS values across seeds at the layer of peak CSLS retrieval in the CS model; We measure linear CKA ( Kornblith et al., 2019 ) , the cosine gap (mean cosine over the N true translation pairs minus that over a fixed random derangement of the pairs), and the per-language centroid distance ( Libovický et al., 2020 , lower is better;) . The last two rows are per-language MEXA ( Kargaran et al., 2025 ) (max over layers vs. the English pivot).
Bitext retrieval P@1 (%)
Condition
En–Nl
En–Zh
Nl–Zh
Leaderboard
Shuf
78.5 ± 0.7
5.4 ± 0.8
4.5 ± 0.8
46.44
CS/Shuf
88.8 ± 0.5
65.5 ± 1.1
54.5 ± 1.0
46.55
Curr
79.0 ± 0.8
5.5 ± 0.6
4.5 ± 0.5
46.36
CS/Curr
87.5 ± 0.7
61.4 ± 0.8
52.5 ± 0.6
46.72
Word
85.5 ± 0.8
48.1 ± 1.5
38.5 ± 1.4
46.59
Appendix
Table 6: Cross-lingual alignment of all eight conditions at their final checkpoints; alignment columns are 8 -seed mean ± SD. Left: FLORES+ bitext retrieval P@1 (CSLS, k=10 ) averaged over layers 6–11, per language pair. Right: the leaderboard score of Table 2 . Column abbreviations as in Table 3 .
Figure 7: Per-layer bitext retrieval P@1 for all eight experimental conditions (§ 4 ), overlaid in each language-pair panel. We use neutral colors for the four main conditions, each pairing of corpus (CS/non-CS) and ordering (shuffled/curriculum), and color for the four shuffled-ordering controls. Lines are 8 -seed means, bands ±1 SD, dotted line chance. Shading marks layers 6 – 11 , the range averaged over in Figure 3 .
Figure 8: Layerwise cosine similarity between the CS and non-CS models’ word representations (§ 7 ), shown separately for the embedded words and their never-embedded controls ( 8 -seed mean ±1 SD).
During language acquisition, bilingual children are regularly exposed to code-switched input and use it as a cognitive scaffold to accelerate vocabulary growth and cross-linguistic syntactic mapping. In contrast, computational bilingual models are conventionally pretrained on interleaved monolingual corpora. While introducing synthetic code-switching during pretraining has become a promising strategy to enhance cross-lingual alignment and downstream performance, the structural and developmental parameters governing the success remain poorly understood. In this work, we investigate the efficiency of training with synthetic code-switched data across two typologically distinct language pairs by controlling two key variables: the structural location of code-switches and the dynamic switching rate across training stages. Our results show that training with code-switched data improves cross-lingual alignment for typologically close languages.
We investigate two aspects of multilingual pre-training with limited data, namely how language exposure is distributed across English, Dutch, and Chinese, and how that exposure is ordered during training. In particular, we compare balanced multilingual exposure to language-dominant exposure, where one language is exposed with more training data than the others. We also compare mixed training, where multilingual training data is used throughout training, with curriculum training, where monolingual exposure of the dominant language is followed by balanced multilingual exposure. Our results show that multilingual BabyLM performance varies with both the amount and order of language exposure. Mixed training outperforms curriculum learning across all 15 matched configurations, indicating that simultaneous multilingual exposure is more effective than staged exposure in our setting. Several language-dominant mixed training configurations achieve higher scores than the balanced configuration, suggesting that a balanced distribution is not necessarily the highest-performing configuration in our setting. Interestingly, the Dutch-primary configurations achieve the best overall performance.
JiWoo Hwang, Lukas Edman, Alexander Fraser
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
Recent studies have shown that code-switching data (CSD), in which multiple languages are mixed within the same context, can improve cross-lingual transfer and multilingual alignment in large language models (LLMs). However, existing studies primarily focus on bilingual transfer between English and a target language, leaving multilingual settings involving three or more languages largely unexplored. In this work, we investigate multilingual code-switching instruction tuning across four languages: English, Japanese, Korean, and Chinese. We evaluate multilingual understanding on Belebele. Our experiments show that simple sentence-level multilingual CSD consistently improves average multilingual performance across all four languages, indicating that multilingual code-switching can be effective beyond bilingual transfer settings.