ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws

Xuyang Cao; Qianying Liu; Chuan Xiao; Yusuke Oda; Jiayi Wang; Pontus Stenetorp; Daisuke Kawahara; Makoto Onizuka

cs.CLMar 18, 2026

ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws

Authors: Xuyang Cao, Qianying Liu, Chuan Xiao, Yusuke Oda, Jiayi Wang, Pontus Stenetorp, Daisuke Kawahara, Makoto Onizuka, +2 more

Organizations: NII LLMC · The University of Osaka · Nara Institute of Science and Technology · University College London · Waseda University

Abstract

In multilingual pretraining, the test loss of a pretrained model is heavily influenced by the proportion of each language in the pretraining data, namely the language mixture ratios. Multilingual scaling laws can predict the test loss under different language mixture ratios and can therefore be used to estimate the optimal ratios. However, the current approaches to multilingual scaling laws do not measure the cross-lingual transfer effect, resulting in suboptimal mixture ratios. In this paper, we consider multilingual pretraining as a cooperative game in which each language acts as a player that jointly contributes to pretraining, gaining the resulting reduction in test loss as the payoff. Consequently, from the perspective of cooperative game theory, we quantify the cross-lingual transfer from each language by its contribution in the game, and propose a game-theoretic multilingual scaling law called ShapleyLaw. Our experiments show that ShapleyLaw outperforms baseline methods in model performance prediction and language mixture optimization.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Cross-lingual transfer effects quantified by normalized SVs estimated on a small-scale configuration ( $N=50$ M, $D=50$ B), shown as an asymmetric matrix. In contrast, the FamilyLaw He et al. (2025) baseline yields zero off-diagonal entries. (b) Across diverse pre-training mixtures and model scales $N\in\{50\text{M},243\text{M},684\text{M},1460\text{M}\}$ with total data size $D=100$ B, the proposed ShapleyLaw $\mathcal{L}_{j}(N,D,\Theta_{ja})$ accurately fits and predicts Japanese performance.

Figure 2: Consider pretraining two models for a Chinese task with training data mixtures in Chinese (zh), Japanese (ja), and Spanish (es), which belong to different language families. We first pretrain Model 1 by sampling Chinese, Spanish, and Japanese data with ratios

[0.500,0.495,0.005]

, respectively. We then swap the mixture ratios of Japanese and Spanish to pretrain Model 2. Although Models 1 and 2 share the same model size

N

=684M and data size

D

=100B, they exhibit significantly different losses on the Chinese task.

Panel of Figure 3 — Figure 3: Comparison of Shapley values across model sizes. Left : normalized Shapley values estimated with $N=50$ M and $D=50$ B. Right : normalized Shapley values estimated with $N=243$ M and $D=50$ B. The two heatmaps exhibit highly consistent cross-lingual transfer patterns across model sizes.

Panel of Figure 4 — Figure 4: Fitting ShapleyLaw across multilingual mixtures and target languages. Top row : fits with respect to aggregate cross-lingual transfer $\Theta$ on curated multilingual mixtures for Chinese (ZH), Spanish (ES), and Japanese (JA) across four model scales $\{50\text{M},243\text{M},684\text{M},1460\text{M}\}$ ; circles denote fitting points and yellow diamonds denote held-out evaluation points. Middle row : fits under randomly sampled mixtures at model scale $N=243$ M; circles indicate fitting points, yellow diamonds indicate held-out points, and orange crosses denote removed outliers. Bottom row : fits with respect to aggregate cross-lingual transfer $\Theta$ on curated multilingual mixtures for low-resource languages Finnish (FI), Croatian (HR), and Standard Malay (MS).

Figure 5: Controlled comparison between FamilyLaw and ShapleyLaw on a Chinese evaluation task. We pretrain multilingual models with

N=50

M and

D=100

B using Chinese, Japanese, and Spanish corpora. Starting from the mixture

[0.05,0.55,0.40]

, we vary the Spanish–Japanese proportions up to

[0.05,0.945,0.05]

while keeping the Chinese ratio fixed. Performance is evaluated on the Chinese test set. Left: FamilyLaw, where x-axis stands for language ratio, yields a poor fit. Right: ShapleyLaw, where the x-axis represents different Chinese aggregate transfer

\Theta

, fits accurately.

Table 1: Comparison of mixture optimization strategies and their CE loss under the Unweighted Sum ( Left ) and Normalized Sum ( Right ) settings.

Figure 6: Fitting Relationship between Downstream Performance and Test CE Loss. Experiments are conducted on six downstream tasks for English, German, French, Spanish, Croatian, and Japanese. Across all tasks, we observe strong negative correlations (

|r|>0.90

, with an average

R^{2}=0.933

), indicating that lower test loss consistently corresponds to higher downstream performance.

Task	de	en	fr	es	ja	hr
HellaSwag	0.918	0.996	0.864	0.897	–	0.992
XWinograd	–	0.980	0.975	–	0.675	–
XStorycloze	–	0.947	–	0.905	–	–

Table 2: Corresponding

R^{2}

values for the fitted relationship between downstream performance and test CE loss for each task–language pair. English achieves the highest

R^{2}

, while Japanese is the lowest. The results consistently show a negative correlation between downstream performance and test CE loss.

Strategy	Language Mixture Ratio (%)					Overall Acc. $\uparrow$
Strategy	en	de	fr	es	hr	HellaSwag
Uniform	$20.0$	$20.0$	$20.0$	$20.0$	$20.0$	$0.343$
$\alpha=0.5$	$25.2$	$22.9$	$21.9$	$23.6$	$6.4$	$0.344$
Common Crawl	$73.9$	$10.0$	$7.7$	$7.8$	$0.6$	$0.354$
FineWeb-2	$90.3$	$3.7$	$2.5$	$3.0$	$0.5$	$0.355$
FamilyLaw	$31.7$	$31.7$	$4.1$	$4.1$	$28.4$	$0.346$

Table 3: Comparison of mixture optimization strategies and average HellaSwag downstream-task accuracies across five languages under the unweighted-sum setting. ShapleyLaw achieves the best performance, with an average accuracy of 0.356.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Params	Layers	Hidden	Heads	Context	Embedding	Non-embedding
		size		length	parameters	parameters
150M (50M)	12	512	8	4,096	101,874,688	50,344,448
440M (243M)	16	1,024	8	4,096	203,749,376	243,303,424
980M (684M)	20	1,536	8	4,096	305,624,064	684,258,816
1.8B (1460M)	24	2,048	16	4,096	407,498,752	1,459,718,144
3.7B (3171M)	28	3,072	24	4,096	611,248,128	3,171,068,928

Appendix

Table 4: Model configurations and parameter breakdown.

Family	Languages	Tokens (B)
Germanic	English / German	373 / 450
Romance	French / Spanish	340 / 397
Sino-Tibetan	Chinese	788
Japonic	Japanese	281
Korean	Korean	52
Uralic	Finnish	54

Appendix

Table 5: Corpora used in our experiments. The study covers ten languages from eight language families, the available token counts are listed for each language.

Dataset / Language	en	de	fr	es	zh	ja	ko	fi	hr	ms
Common Crawl	61.4%	8.3%	6.4%	6.4%	7.2%	8.2%	1.2%	0.5%	0.3%	0.1%
FineWeb-2	80.6%	3.3%	2.3%	2.7%	6.7%	3.1%	0.5%	0.4%	0.3%	0.1%

Appendix

Table 6: Normalized statistics across languages in open-source corpora. The statistics are derived from the official Common Crawl statistics and, for FineWeb-2, estimated based on the reported disk sizes Penedo et al. (2025) . Specifically, the disk sizes of the FineWeb-2 subsets for {en, de, fr, es, zh, ja, ko, fi, hr, ms} are {33 TB, 1.51 TB, 1.11 TB, 1.32 TB, 2.42 TB, 1.50 TB, 213.43 GB, 143.03 GB, 35.91 GB, 31.94 GB}, respectively.

Panel of Figure 7 — Figure 7: Additional fitting results for Shapley-based scaling laws on <EN,FR>. Top row : Fits on curated multilingual mixtures for EN , and FR across four model scales $\{50\text{M},243\text{M},684\text{M},1460\text{M}\}$ ; gray circles denote training points and yellow diamonds denote held-out points. Bottom row : Fits under randomly sampled mixtures under model scale $N$ =243M. For both languages, the fitted curves remain smooth and monotonic with respect to $\Theta$ , and held-out points closely follow predicted trends, demonstrating strong generalization to unseen mixtures.

Panel of Figure 8 — Figure 8: Additional fitting results for Shapley-based scaling laws on <DE,KO>. Top row : Fits on curated multilingual mixtures for DE and KO across the same four model scales. Bottom row : Fits under randomly sampled mixtures with model scale $N$ =243M. For both languages, the fitted curves remain smooth and monotonic with respect to aggregate cross-lingual transfer $\Theta$ , and held-out points closely follow predicted trends, demonstrating strong generalization to unseen mixtures.

Task	$\ell$	$-\ell$	$\log\ell$	$\exp(-\ell)$	$1/\ell$
HellaSwag	0.994	0.994	0.995	0.996	0.996
XWinograd	0.971	0.971	0.975	0.980	0.978
XStorycloze	0.947	0.947	0.943	0.936	0.939

Appendix

Table 7: Model Selection. Model comparison for

s=a\cdot g(\ell)+b

using different transformations of loss. Numbers denote

R^{2}

. Across tasks except XStorycloze,

g(\ell)=\exp(-\ell)

consistently achieves the highest explanatory power, suggesting that accuracy scales approximately linearly with token-level probability

p=\exp(-\ell)

Task	Pearson $r$	$p$ -value	Best $g(\ell)$	$R^{2}$	$a$	$b$
HellaSwag	-0.997	$8.15\times 10^{-7}$	$\exp(-\ell)$	0.996	4.030	0.098
XWinograd	-0.985	$4.92\times 10^{-5}$	$\exp(-\ell)$	0.980	4.824	0.382
XStorycloze	-0.973	$2.25\times 10^{-4}$	$\exp(-\ell)$	0.980	3.034	0.433

Appendix

Table 8: Fitted Results (English Downstream Tasks). Pearson correlation between pretraining loss

\ell

and few-shot accuracy. All tasks exhibit strong negative correlation (

|r|>0.90

), indicating that lower loss consistently corresponds to higher downstream performance. Correlation between pretraining loss

\ell

and few-shot accuracy, and best-fit results for

\hat{s}=ag(\ell)+b

. Across all tasks,

\exp(-\ell)

consistently achieves the highest

R^{2}

Figure 9: Left. Downstream performance versus test cross-entropy (CE) loss on Chinese tasks ( XCOPA_zh and XStoryCloze_zh ). Middle. Downstream performance versus test CE loss on the Japanese task ( XWinograd ). Right. Downstream performance for English versus test CE loss across 50 models (

N=243

D=100

B) trained with randomly sampled data mixtures.

Comparison	Cosine Similarity	Pearson
Exact SV vs. IRDS SV	0.9584	0.9390

Appendix

Table 9: Similarity comparison between exact SV and IRDS SV.

Figure 10: Fitting results with exact and IRDS-approximated SVs. To reduce the workload of SV computation, we use the IRDS approximation by pretraining a small-scale model (

N=50

D=50

B) with a uniform mixture ratio over seven languages. The resulting IRDS SVs produce fitting curves that closely match those obtained with exact SVs across target languages.

Target	de	en	fr	es	ja	ko	zh
Exact	0.015	0.004	0.015	0.014	0.006	0.009	0.015
IRDS	0.015	0.003	0.036	0.017	0.008	0.023	0.013

Appendix

Table 10: Comparison of corresponding

\mathrm{PE}_{fit}

values for the fitted results with exact and IRDS SVs.

Panel of Figure 11 — Figure 11: Additional fitting results for ShapleyLaw at a larger model scale. We extend the evaluation to $N=3171$ M and $D=100$ B. Fits are shown for DE , EN , FR , ES , JA , and KO across model scales $\{50\text{M},243\text{M},684\text{M},1460\text{M},3171\text{M}\}$ . Gray circles denote fitting points and yellow diamonds denote held-out points.

Method	de	en	fr	es	ja	ko	zh
Signed Normalization	0.039	0.004	0.037	0.023	0.029	0.042	0.037
Min–Max Normalization	0.039	0.006	0.039	0.024	0.026	0.035	0.035
Ours	0.016	0.004	0.016	0.011	0.005	0.009	0.013

Appendix

Table 11: Comparison of Shapley-value normalization schemes. Prediction errors obtained with Signed Normalization, Min–Max Normalization, and the softmax-like normalization adopted in ShapleyLaw across different test languages. Lower values indicate better prediction performance. The proposed normalization achieves the lowest or tied-lowest prediction error for all evaluated languages.

	de	en	fr	es	ja	ko	zh
IRDS Error ( $\ell_{1}$ distance)	10.605	9.903	9.922	9.916	9.892	10.603	9.888
$\Delta PE_{\mathrm{fit}}$	0.000	-0.001	0.021	0.003	0.002	0.014	-0.002

Appendix

Table 12: IRDS approximation error and the corresponding change in fitting performance across test languages. The first row reports the

\ell_{1}

distance between the IRDS-approximated and reference Shapley values, while the second row reports the change in fitting error after using the IRDS approximation. Positive values of

\Delta PE_{\mathrm{fit}}

indicate increased fitting error, whereas negative values indicate a slight improvement.

Excluded Language	Germanic/EN	Germanic/DE	Romance/FR	Romance/ES
$PE_{\mathrm{fit}}$	0.006	0.004	0.006	0.009
$PE_{\mathrm{test}}$	0.009	0.004	0.012	0.005

Appendix

Table 13: Leave-one-language-out evaluation within language families. Experiments are conducted with

N=50

M and

D=50

B. For each setting, all fitting mixtures containing the designated language are excluded, and the fitted ShapleyLaw is evaluated on the corresponding held-out within-family mixture.

Explore similar work

Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation
Oct 14, 2026Joel Dag, Nikit Srivastava, Hamada M. Zahera +1Multilingual Language Models Mixture-of-Experts Language Models
Data Science Group, Heinz Nixdorf Institute, Paderborn University, Germany
On the Limits of Model Merging for Multilinguality in Pre-Training
May 25, 2026Seth Aycock, Fedor Vitiugin, Aleksandr Umnov +2Language Model Pretraining Multilingual Language Models
University of Amsterdam
Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
Apr 19, 2026Felicia Körner, Maria Matveev, Florian Eichin +3Language Model Pretraining Multilingual Language Models
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML) · Department of Mathematics, LMU Munich, Germany +2

Oct 14, 2026J/K move · Enter open · S save

Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation

Joel Dag, Nikit Srivastava, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo

Data Science Group, Heinz Nixdorf Institute, Paderborn University, Germany

Multilingual Language Models Mixture-of-Experts Language Models Language Model Scaling Laws

Open paper AskPDF arXiv

Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this capacity with little additional per-token computation. Given this extra capacity, does adding experts reduce what each additional language costs? To answer this question, we propose a repetition-aware joint scaling law relating number of active parameters, training tokens per language, number of experts, and language coverage. Fitted on 26,300 per-language loss observations from 64 pretrained configurations, our law reduces prediction error on held-out configurations by 28% relative to the strongest evaluated baseline. At matched number of active parameters and training tokens per language, 32 experts reduce the loss penalty of expanding from 12 to 200 languages by 83% compared with a dense model. Our law further shows that broader language coverage favors more experts across a wider range of compute budgets, guiding capacity allocation as multilingual models scale.

Abstract

Explore similar work

Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation

On the Limits of Model Merging for Multilinguality in Pre-Training

Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining