In multilingual pretraining, the test loss of a pretrained model is heavily influenced by the proportion of each language in the pretraining data, namely the language mixture ratios. Multilingual scaling laws can predict the test loss under different language mixture ratios and can therefore be used to estimate the optimal ratios. However, the current approaches to multilingual scaling laws do not measure the cross-lingual transfer effect, resulting in suboptimal mixture ratios. In this paper, we consider multilingual pretraining as a cooperative game in which each language acts as a player that jointly contributes to pretraining, gaining the resulting reduction in test loss as the payoff. Consequently, from the perspective of cooperative game theory, we quantify the cross-lingual transfer from each language by its contribution in the game, and propose a game-theoretic multilingual scaling law called ShapleyLaw. Our experiments show that ShapleyLaw outperforms baseline methods in model performance prediction and language mixture optimization.
Figures & tables
Figure 1: (a) Cross-lingual transfer effects quantified by normalized SVs estimated on a small-scale configuration ( N=50 M, D=50 B), shown as an asymmetric matrix. In contrast, the FamilyLaw He et al. (2025) baseline yields zero off-diagonal entries. (b) Across diverse pre-training mixtures and model scales N∈{50M,243M,684M,1460M} with total data size D=100 B, the proposed ShapleyLaw Lj(N,D,Θja) accurately fits and predicts Japanese performance.
Figure 2: Consider pretraining two models for a Chinese task with training data mixtures in Chinese (zh), Japanese (ja), and Spanish (es), which belong to different language families. We first pretrain Model 1 by sampling Chinese, Spanish, and Japanese data with ratios [0.500,0.495,0.005] , respectively. We then swap the mixture ratios of Japanese and Spanish to pretrain Model 2. Although Models 1 and 2 share the same model size N =684M and data size D =100B, they exhibit significantly different losses on the Chinese task.
Figure 3: Comparison of Shapley values across model sizes. Left : normalized Shapley values estimated with N=50 M and D=50 B. Right : normalized Shapley values estimated with N=243 M and D=50 B. The two heatmaps exhibit highly consistent cross-lingual transfer patterns across model sizes.
Figure 4: Fitting ShapleyLaw across multilingual mixtures and target languages. Top row : fits with respect to aggregate cross-lingual transfer Θ on curated multilingual mixtures for Chinese (ZH), Spanish (ES), and Japanese (JA) across four model scales {50M,243M,684M,1460M} ; circles denote fitting points and yellow diamonds denote held-out evaluation points. Middle row : fits under randomly sampled mixtures at model scale N=243 M; circles indicate fitting points, yellow diamonds indicate held-out points, and orange crosses denote removed outliers. Bottom row : fits with respect to aggregate cross-lingual transfer Θ on curated multilingual mixtures for low-resource languages Finnish (FI), Croatian (HR), and Standard Malay (MS).
Figure 5: Controlled comparison between FamilyLaw and ShapleyLaw on a Chinese evaluation task. We pretrain multilingual models with N=50 M and D=100 B using Chinese, Japanese, and Spanish corpora. Starting from the mixture [0.05,0.55,0.40] , we vary the Spanish–Japanese proportions up to [0.05,0.945,0.05] while keeping the Chinese ratio fixed. Performance is evaluated on the Chinese test set. Left: FamilyLaw, where x-axis stands for language ratio, yields a poor fit. Right: ShapleyLaw, where the x-axis represents different Chinese aggregate transfer Θ , fits accurately.
Table 1: Comparison of mixture optimization strategies and their CE loss under the Unweighted Sum ( Left ) and Normalized Sum ( Right ) settings.
Figure 6: Fitting Relationship between Downstream Performance and Test CE Loss. Experiments are conducted on six downstream tasks for English, German, French, Spanish, Croatian, and Japanese. Across all tasks, we observe strong negative correlations ( ∣r∣>0.90 , with an average R2=0.933 ), indicating that lower test loss consistently corresponds to higher downstream performance.
Task
de
en
fr
es
ja
hr
HellaSwag
0.918
0.996
0.864
0.897
–
0.992
XWinograd
–
0.980
0.975
–
0.675
–
XStorycloze
–
0.947
–
0.905
–
–
Table 2: Corresponding R2 values for the fitted relationship between downstream performance and test CE loss for each task–language pair. English achieves the highest R2 , while Japanese is the lowest. The results consistently show a negative correlation between downstream performance and test CE loss.
Strategy
Language Mixture Ratio (%)
Overall Acc. ↑
en
de
fr
es
hr
HellaSwag
Uniform
20.0
20.0
20.0
20.0
20.0
0.343
α=0.5
25.2
22.9
21.9
23.6
6.4
0.344
Common Crawl
73.9
10.0
7.7
7.8
0.6
0.354
FineWeb-2
90.3
3.7
2.5
3.0
0.5
0.355
FamilyLaw
31.7
31.7
4.1
4.1
28.4
0.346
Table 3: Comparison of mixture optimization strategies and average HellaSwag downstream-task accuracies across five languages under the unweighted-sum setting. ShapleyLaw achieves the best performance, with an average accuracy of 0.356.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Params
Layers
Hidden
Heads
Context
Embedding
Non-embedding
size
length
parameters
parameters
150M (50M)
12
512
8
4,096
101,874,688
50,344,448
440M (243M)
16
1,024
8
4,096
203,749,376
243,303,424
980M (684M)
20
1,536
8
4,096
305,624,064
684,258,816
1.8B (1460M)
24
2,048
16
4,096
407,498,752
1,459,718,144
3.7B (3171M)
28
3,072
24
4,096
611,248,128
3,171,068,928
Appendix
Table 4: Model configurations and parameter breakdown.
Family
Languages
Tokens (B)
Germanic
English / German
373 / 450
Romance
French / Spanish
340 / 397
Sino-Tibetan
Chinese
788
Japonic
Japanese
281
Korean
Korean
52
Uralic
Finnish
54
Appendix
Table 5: Corpora used in our experiments. The study covers ten languages from eight language families, the available token counts are listed for each language.
Dataset / Language
en
de
fr
es
zh
ja
ko
fi
hr
ms
Common Crawl
61.4%
8.3%
6.4%
6.4%
7.2%
8.2%
1.2%
0.5%
0.3%
0.1%
FineWeb-2
80.6%
3.3%
2.3%
2.7%
6.7%
3.1%
0.5%
0.4%
0.3%
0.1%
Appendix
Table 6: Normalized statistics across languages in open-source corpora. The statistics are derived from the official Common Crawl statistics and, for FineWeb-2, estimated based on the reported disk sizes Penedo et al. (2025) . Specifically, the disk sizes of the FineWeb-2 subsets for {en, de, fr, es, zh, ja, ko, fi, hr, ms} are {33 TB, 1.51 TB, 1.11 TB, 1.32 TB, 2.42 TB, 1.50 TB, 213.43 GB, 143.03 GB, 35.91 GB, 31.94 GB}, respectively.
Figure 7: Additional fitting results for Shapley-based scaling laws on <EN,FR>. Top row : Fits on curated multilingual mixtures for EN , and FR across four model scales {50M,243M,684M,1460M} ; gray circles denote training points and yellow diamonds denote held-out points. Bottom row : Fits under randomly sampled mixtures under model scale N =243M. For both languages, the fitted curves remain smooth and monotonic with respect to Θ , and held-out points closely follow predicted trends, demonstrating strong generalization to unseen mixtures.
Figure 8: Additional fitting results for Shapley-based scaling laws on <DE,KO>. Top row : Fits on curated multilingual mixtures for DE and KO across the same four model scales. Bottom row : Fits under randomly sampled mixtures with model scale N =243M. For both languages, the fitted curves remain smooth and monotonic with respect to aggregate cross-lingual transfer Θ , and held-out points closely follow predicted trends, demonstrating strong generalization to unseen mixtures.
Task
ℓ
−ℓ
logℓ
exp(−ℓ)
1/ℓ
HellaSwag
0.994
0.994
0.995
0.996
0.996
XWinograd
0.971
0.971
0.975
0.980
0.978
XStorycloze
0.947
0.947
0.943
0.936
0.939
Appendix
Table 7: Model Selection. Model comparison for s=a⋅g(ℓ)+b using different transformations of loss. Numbers denote R2 . Across tasks except XStorycloze, g(ℓ)=exp(−ℓ) consistently achieves the highest explanatory power, suggesting that accuracy scales approximately linearly with token-level probability p=exp(−ℓ) .
Task
Pearson r
p -value
Best g(ℓ)
R2
a
b
HellaSwag
-0.997
8.15×10−7
exp(−ℓ)
0.996
4.030
0.098
XWinograd
-0.985
4.92×10−5
exp(−ℓ)
0.980
4.824
0.382
XStorycloze
-0.973
2.25×10−4
exp(−ℓ)
0.980
3.034
0.433
Appendix
Table 8: Fitted Results (English Downstream Tasks). Pearson correlation between pretraining loss ℓ and few-shot accuracy. All tasks exhibit strong negative correlation ( ∣r∣>0.90 ), indicating that lower loss consistently corresponds to higher downstream performance. Correlation between pretraining loss ℓ and few-shot accuracy, and best-fit results for s^=ag(ℓ)+b . Across all tasks, exp(−ℓ) consistently achieves the highest R2 .
Figure 9: Left. Downstream performance versus test cross-entropy (CE) loss on Chinese tasks ( XCOPA_zh and XStoryCloze_zh ). Middle. Downstream performance versus test CE loss on the Japanese task ( XWinograd ). Right. Downstream performance for English versus test CE loss across 50 models ( N=243 M, D=100 B) trained with randomly sampled data mixtures.
Comparison
Cosine Similarity
Pearson
Exact SV vs. IRDS SV
0.9584
0.9390
Appendix
Table 9: Similarity comparison between exact SV and IRDS SV.
Figure 10: Fitting results with exact and IRDS-approximated SVs. To reduce the workload of SV computation, we use the IRDS approximation by pretraining a small-scale model ( N=50 M, D=50 B) with a uniform mixture ratio over seven languages. The resulting IRDS SVs produce fitting curves that closely match those obtained with exact SVs across target languages.
Target
de
en
fr
es
ja
ko
zh
Exact
0.015
0.004
0.015
0.014
0.006
0.009
0.015
IRDS
0.015
0.003
0.036
0.017
0.008
0.023
0.013
Appendix
Table 10: Comparison of corresponding PEfit values for the fitted results with exact and IRDS SVs.
Figure 11: Additional fitting results for ShapleyLaw at a larger model scale. We extend the evaluation to N=3171 M and D=100 B. Fits are shown for DE , EN , FR , ES , JA , and KO across model scales {50M,243M,684M,1460M,3171M} . Gray circles denote fitting points and yellow diamonds denote held-out points.
Method
de
en
fr
es
ja
ko
zh
Signed Normalization
0.039
0.004
0.037
0.023
0.029
0.042
0.037
Min–Max Normalization
0.039
0.006
0.039
0.024
0.026
0.035
0.035
Ours
0.016
0.004
0.016
0.011
0.005
0.009
0.013
Appendix
Table 11: Comparison of Shapley-value normalization schemes. Prediction errors obtained with Signed Normalization, Min–Max Normalization, and the softmax-like normalization adopted in ShapleyLaw across different test languages. Lower values indicate better prediction performance. The proposed normalization achieves the lowest or tied-lowest prediction error for all evaluated languages.
de
en
fr
es
ja
ko
zh
IRDS Error ( ℓ1 distance)
10.605
9.903
9.922
9.916
9.892
10.603
9.888
ΔPEfit
0.000
-0.001
0.021
0.003
0.002
0.014
-0.002
Appendix
Table 12: IRDS approximation error and the corresponding change in fitting performance across test languages. The first row reports the ℓ1 distance between the IRDS-approximated and reference Shapley values, while the second row reports the change in fitting error after using the IRDS approximation. Positive values of ΔPEfit indicate increased fitting error, whereas negative values indicate a slight improvement.
Excluded Language
Germanic/EN
Germanic/DE
Romance/FR
Romance/ES
PEfit
0.006
0.004
0.006
0.009
PEtest
0.009
0.004
0.012
0.005
Appendix
Table 13: Leave-one-language-out evaluation within language families. Experiments are conducted with N=50 M and D=50 B. For each setting, all fitting mixtures containing the designated language are excluded, and the fitted ShapleyLaw is evaluated on the corresponding held-out within-family mixture.
Endowing models with consistent multilingual performance can be achieved by mixing pre-training data, or post-training approaches such as language-specific model merging. In this work, we test whether merging can be applied to monolingually pre-trained models. We conduct a controlled study on the efficacy of mixed, merged, and monolingual pre-training setups. We find that while monolingual pre-training results in strong in-language performance, merging any combination of monolingual models leads to performance collapse due to interference. Our analysis suggests representational similarity is a prerequisite for model merging. We therefore conclude that the flexibility of merging in fine-tuning does not extend trivially to language-specific pre-training.
Large language models exhibit impressive cross-lingual capabilities. However, prior work analyzes this phenomenon through isolated factors and at sparse points during training, limiting our understanding of how cross-lingual generalization emerges--particularly in the early phases of learning. To study the early trajectory of linguistic and translation capabilities, we pretrain a multilingual 1.7B model on nine diverse languages, capturing checkpoints at a much finer granularity. We use word-level translation as a testbed, introducing a novel dataset to trace how translation develops over training through behavioral analyses, model-component analysis, and parameter-based ablations. We find that the model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which more generalizing translation mechanisms are developed while copying is refined. Together, these findings provide a fine-grained view of how cross-lingual generalization develops during multilingual pretraining.
Felicia Körner, Maria Matveev, Florian Eichin +3
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML) · Department of Mathematics, LMU Munich, Germany +2
Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across languages. A further challenge is the performance interference that can arise from joint multilingual training, commonly referred to as the "curse of multilinguality". We study multilingual data curation across thirteen languages and find that many reported regressions are not inherent to multilingual scaling but instead stem from correctable deficiencies in data quality and composition rather than fundamental capacity limits. In controlled bilingual experiments, improving data quality for any single language benefits others: curating English improves non-English performance in 12 of 13 languages, while curating non-English yields reciprocal improvements in English. Bespoke per-language curation produces substantially larger within-language improvements. Extending these findings to large-scale general-purpose training mixtures, we show that curated multilingual allocations comprising under 8% of total tokens remain remarkably effective. We operationalize this approach within an effort that produced a 20T-token pretraining corpus derived entirely from public sources. Models with 3B and 8B parameters trained on a 1T-token random subset achieve competitive multilingual accuracy with 4-10x fewer training FLOPs than strong public baselines, establishing a new Pareto frontier in multilingual performance versus compute. Moreover, these benefits extend to frontier model scale: the 20T-token corpus served as part of the pretraining dataset for Trinity Large (400B/A13B), which exhibits strong multilingual performance relative to its training FLOPs. These results show that targeted, per-language data curation mitigates multilingual interference and enables compute-efficient multilingual scaling.