When the Weakest Language Keeps Moving: ARMOR-DP for Multilingual BabyLM

Hanyu Liao; Yijing Chen; Tao Ji

cs.CLOct 24, 2026

When the Weakest Language Keeps Moving: ARMOR-DP for Multilingual BabyLM

Authors: Hanyu Liao, Yijing Chen, Tao Ji

Organizations: East China Normal University · Fudan University

Abstract

Multilingual BabyLM systems usually fix language sampling proportions even though the weakest language can change during training. We present ARMOR-DP, a modular framework that diagnoses imbalance arising from representation, exposure, capacity, and objective pressure, and uses per-language validation loss to update sampling. Controlled experiments show that equal character budgets create a severe representation artifact for Chinese; byte-level encoding removes much of it, while static Chinese protection can subsequently make English the weakest language. The dynamic controller tracks these transitions and achieves a better score--balance tradeoff than fixed protection. On official evaluation, curriculum-dynamic checkpoints outperform a matched static Chinese-protective baseline while using fewer Chinese epochs; scaling capacity yields the strongest internal ARMOR-DP system. These results support a simple conclusion: multilingual weakness is a training-time state rather than a fixed property of a language. ARMOR-DP therefore separates diagnosis from intervention, allowing protection to follow the current bottleneck instead of a predetermined language label.

Figures & tables

Figure 1: Central hypothesis of ARMOR-DP. Panel A shows that equal character budgets do not create equal representation burden across scripts; Panel B shows that the bottleneck can move after protection is added. The takeaway is that weakest-language protection should be measured during training rather than fixed to one language in advance.

System family	Representation	Capacity	Exposure	Controller	Objective
Balanced adapters	UTF-8 bytes	Adapters	Balanced	None	Token mean
Static ZH50	UTF-8 bytes	Adapters	25/25/50	None	Token mean
Curriculum-dynamic	UTF-8 bytes	Adapters	Balanced $\rightarrow$ adaptive	Loss-based	Token mean
Dynamic + pressure	UTF-8 bytes	Adapters	Adaptive	Loss-based	Pressure-aware
Curr.-dyn. + pressure	UTF-8 bytes	Adapters	Balanced $\rightarrow$ adaptive	Loss-based	Pressure-aware

Table 1: Component map for the checkpoint families reported in Sections 4–7. The d256, d512, and d768 final systems are capacity-scaled instances of curriculum-dynamic.

Condition	Mean loss	Gap	EN	NL	ZH
Char balanced baseline	3.066 $\pm$ 0.020	1.858 $\pm$ 0.051	2.509	2.415	4.273
Byte balanced baseline	2.362 $\pm$ 0.007	0.185 $\pm$ 0.063	2.455	2.361	2.269
Byte + ZH50 baseline	2.317 $\pm$ 0.012	0.548 $\pm$ 0.042	2.531	2.435	1.983
Byte + ZH50 adapters	2.235 $\pm$ 0.007	0.511 $\pm$ 0.028	2.436	2.344	1.926
Byte balanced $\rightarrow$ ZH50 adapters	2.246 $\pm$ 0.010	0.408 $\pm$ 0.022	2.417	2.312	2.008
Byte + ZH50 routing gate	2.272 $\pm$ 0.014	0.459 $\pm$ 0.051	2.457	2.361	1.998

Table 2: Controlled ARMOR-DP probe on official BabyBabelLM rows. Representation armor removes the Chinese collapse; exposure and capacity improve mean loss but can transfer worst-language status to English.

Figure 2: End-of-run score/balance tradeoff across diagnostic systems. The axes are final mean validation loss and final language gap (not training steps); lower is better on both axes. Byte representation removes the first collapse, while ZH-protective adapters improve mean loss but widen the language gap, making English the weakest language.

Condition	Mean loss	Gap
Balanced adapters	2.230	0.179
Dynamic-loss adapters	2.229	0.199
Dynamic-loss routing gate	2.264	0.168
Dynamic + script prior adapters	2.217	0.290
Curriculum adapters	2.216	0.353
Static ZH50 adapters	2.198	0.489

Table 3: Dynamic protection probe over three seeds. Dynamic loss control is most useful as a balance-preserving controller; static ZH50 still optimizes mean loss at the cost of English.

Figure 3: How the dynamic controller changes exposure. Each curve is the sampling probability for one language, and the x-axis is the logged controller-update index from a dynamic probe (unlike the end-of-run axes in Figure 2 ). The controller first raises Chinese exposure, then returns probability mass to English and Dutch as Chinese becomes easier, showing why fixed ZH protection can become stale.

Figure 4: Weakest-language identity over training. The highlighted language at each step is the current highest-loss language. The bottleneck moves from Dutch to Chinese and then to English, supporting the claim that weakness is a training-time state rather than a fixed language label.

Condition	Mean loss	Gap	Worst
Adaptive exp.	2.344	0.118	EN
Balanced	2.346	0.119	EN
Adaptive exp. + pressure	2.360	0.158	EN
ZH50 + pressure	2.354	0.232	EN
ZH50	2.320	0.348	EN
ZH50 + rare token	2.359	0.481	EN

Table 4: Objective-layer probe over three seeds. Pressure-aware weighting is substantially less damaging than rare-token weighting and reduces the over-protection gap under ZH50, but adaptive exposure remains the better balance-preserving controller.

System	Overall	Worst	Gap	Var.	EN	NL	ZH
Balanced adapters	0.4633	NL 0.4465	0.0334	0.00019	0.4798	0.4465	0.4635
Static ZH50	0.4574	NL 0.4413	0.0379	0.00025	0.4791	0.4413	0.4519
Curriculum-dynamic	0.4634	NL 0.4439	0.0359	0.00022	0.4798	0.4439	0.4665
Curr.-dyn. + token pressure	0.4612	NL 0.4443	0.0330	0.00018	0.4772	0.4443	0.4620
Dynamic + pressure	0.4541	NL 0.4443	0.0181	0.00006	0.4623	0.4443	0.4558
Curriculum-dynamic + pressure	0.4554	NL 0.4438	0.0292	0.00016	0.4730	0.4438	0.4495

Table 5: Full public official zero-shot evaluation for 1M-token checkpoints. Overall is the mean over English, Dutch, and Chinese task-group means. Gap and variance are computed over language means, so lower is more balanced.

System	Fine	Combined	Worst	Gap	Var.	EN	NL	ZH
Balanced adapters	0.2326	0.3479	ZH 0.2099	0.0485	0.00040	0.2584	0.2295	0.2099
Static ZH50	0.2254	0.3414	ZH 0.2072	0.0391	0.00026	0.2464	0.2226	0.2072
Curriculum-dynamic	0.2395	0.3514	NL 0.2180	0.0388	0.00026	0.2567	0.2180	0.2437
Dynamic + pressure	0.2228	0.3385	ZH 0.2094	0.0337	0.00021	0.2159	0.2431	0.2094
Curriculum-dynamic + pressure	0.2364	0.3459	NL 0.2304	0.0143	0.00004	0.2447	0.2304	0.2342

Table 6: Full public official fine-tuning evaluation. Fine is the mean over language-level task means. Combined is the unweighted average of the zero-shot overall score in Table 5 and the fine-tuning mean here. Gap and variance are computed over fine-tuning language means.

System	Mean	Gap	Worst	EN loss	NL loss	ZH loss	Max ep.
Balanced adapters	2.258	0.226	EN	2.372	2.258	2.145	4.410
Static ZH50	2.248	0.389	EN	2.414	2.305	2.025	6.555
Curriculum-dynamic	2.264	0.253	EN	2.389	2.266	2.136	4.523
Dynamic + pressure	2.295	0.337	EN	2.438	2.345	2.101	4.387
Curriculum-dynamic + pressure	2.279	0.273	EN	2.407	2.298	2.133	4.505

Table 7: Focused 1M-token grid on cached official rows. The epoch column is the maximum actual consumed epoch over English, Dutch, and Chinese, recorded in each checkpoint’s training metadata.

System	Non-pad tok.	Zero	Fine	Comb.	Worst comb.	Gap
1M curr.-dyn.	1.0M	0.4634	0.2395	0.3514	NL 0.3309	0.0373
100M d64 curr.-dyn.	87.3M	0.4692	0.1976	0.3334	NL 0.3202	0.0248
100M d256 curr.-dyn.	87.3M	0.4842	0.2587	0.3714	NL 0.3663	0.0118
100M d512 curr.-dyn.	87.0M	0.4954	0.2659	0.3807	NL 0.3667	0.0256
100M d768 curr.-dyn.	87.0M	0.5032	0.2699	0.3866	NL 0.3734	0.0200
100M d512 static ZH50	89.0M	0.4845	0.2330	0.3587	EN 0.3538	0.0095

Table 8: Public official capacity frontier. “Non-pad tok.“ counts actual predicted non-padding tokens; all ARMOR-DP rows remain under the 10-epoch competition cap. The d512 curr.-dyn. and d512 static rows are matched-capacity systems. The GPT-2 row is the released equal-trilingual baseline from the official evaluation README; its subword tokens are not content-equivalent to ARMOR-DP’s byte tokens.

Figure 5: Internal ARMOR-DP public score/balance frontier for the 100M line. The released GPT-2 equal-trilingual baseline is higher (0.4097 combined) and is reported in Table 8 , but is not drawn because it is an external organizer baseline rather than an ARMOR-DP variant. Within the internal frontier, capacity scaling turns curriculum-dynamic protection into the best-performing ARMOR-DP scorer, while the matched static ZH50 baseline sits lower despite a small public gap.

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 264 papers

Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training
Oct 14, 2026JiWoo Hwang, Lukas Edman, Alexander FraserLanguage Model Pretraining Multilingual Language Models
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop
Feb 23, 2026Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul +7Language Model Pretraining Small Language Models
IBM Research, MIT · ETH Zürich · Cornell University +7
Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence
Sep 24, 2026Dries Rooryck, Alex Cai, Yonatan Belinkov +2Multilingual Language Models Language Model Pretraining
Kempner Institute, Harvard University, Cambridge, MA, USA · Technion – Israel Institute of Technology, Haifa, Israel

Oct 14, 2026J/K move · Enter open · S save

Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training

JiWoo Hwang, Lukas Edman, Alexander Fraser

School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute

Language Model Pretraining Multilingual Language Models Curriculum Learning Multilingual Language Model Evaluation

Open paper AskPDF arXiv

We investigate two aspects of multilingual pre-training with limited data, namely how language exposure is distributed across English, Dutch, and Chinese, and how that exposure is ordered during training. In particular, we compare balanced multilingual exposure to language-dominant exposure, where one language is exposed with more training data than the others. We also compare mixed training, where multilingual training data is used throughout training, with curriculum training, where monolingual exposure of the dominant language is followed by balanced multilingual exposure. Our results show that multilingual BabyLM performance varies with both the amount and order of language exposure. Mixed training outperforms curriculum learning across all 15 matched configurations, indicating that simultaneous multilingual exposure is more effective than staged exposure in our setting. Several language-dominant mixed training configurations achieve higher scores than the balanced configuration, suggesting that a balanced distribution is not necessarily the highest-performing configuration in our setting. Interestingly, the Dutch-primary configurations achieve the best overall performance.

Abstract

Multilingual Language Models

Explore similar work

Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training

BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop

Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence