Skill Issue: Are Skills Language-Invariant in LLMs?

Bobby Cheng; Adam Gaber; Zhengyuan Liu; Catherine Arnett; Omer Goldman; Cheston Tan; Leshem Choshen

cs.CLAug 26, 2026

Skill Issue: Are Skills Language-Invariant in LLMs?

Authors: Bobby Cheng, Adam Gaber, Zhengyuan Liu, Catherine Arnett, Omer Goldman, Cheston Tan, Leshem Choshen

Organizations: NA · Weizmann Institute of Science · QMIT-IBM Watson AI Lab · EleutherAI ·

Abstract

Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work quantifies cross-lingual skill inconsistency orthogonally from knowledge and general benchmark performance. We do this via multilingual self-play: two instances of the same model compete in a text-based game, each interacting through a different language interface. Since the model, opponent, rules, state space, and available actions remain fixed, this setting isolates the effect of language on the model's realized behavior. We build a multilingual extension to TextArena and evaluate three open-weight models across eight languages and six games covering spatial reasoning, imperfect information, resource allocation, and repeated interaction. We find that the same model can exhibit markedly different playing strength across languages, with systematic variation in win--loss margins, invalid actions, and strategic tendencies. Detailed analyses reveal language-specific failures in spatial reasoning, card-conditioned decisions, and optimal move selection. In some settings, changing only the intermediate reasoning language recovers much of the lost performance, suggesting that language can affect different stages of the decision process. These results show that skill discrepancies are a measurable major roadblock in the development of truly multilingual models. Better understanding these discrepancies can help us design models that perform more equitably across languages.

Figures & tables

Figure 1: The six game environments and the primary skill each probes; full rules in App. D .

Figure 2: Overall, wins of each language against others, per model. Each heatmap shows the role-pooled win–loss margin $\Delta(A>B)=(W_{A}-L_{A})/N$ aggregated over all games, where positive values mean row language $A$ outperforms column language $B$ . The rightmost column reports each language’s mean margin $\bar{\Delta}_{A}$ . Diagonal is same language games, with randomly assigned sides for comparison. Across models, English is consistently strong and Hebrew weak, with Qwen3-4B showing the sharpest hierarchy.

Figure 3: Illustration of Gemma-4-E4B-it playing TicTacToe with itself in German and English respectively.

Tier	Resource class							Total
	5	4	3	2	1	0	–
A	6	0	0	2	0	0	0	8
B	1	17	18	2	4	0	0	42
C	7	1	6	14	76	15	5	124
E	0	0	0	0	10	5	3	18
Total	14	18	24	18	90	20	8	192

Table 1: Languages by verification tier and resource class. We focus on the eight Tier-A languages.

Model	Blotto	Nim	TTT	Tak	IPD	Kuhn	Avg.
Gemma	0.44	0.02	0.47	0.43	0.25	0.05	0.28
Qwen	1.48	0.85	0.32	0.44	0.03	0.13	0.54
Ministral	1.28	0.49	0.36	0.28	0.74	0.20	0.56
Avg.	1.07	0.45	0.38	0.38	0.34	0.13	0.46

Table 2: Language sensitivity by game, measured as language gap,

\max_{\ell}\mu_{\ell}-\min_{\ell}\mu_{\ell}

. Bold marks most language-sensitive model for each game. Bottom row shows the average across models.

Game	Floor	Reasoning language	Ceiling
	$\mu_{\text{weak}}$	$\mu$ / recovery	$\mu_{\text{best}}$
Kuhn	$-0.03$ zh/zh	$-0.01$ zh/es / $37.6\%$ $-0.01$ zh/fr / $49.3\%$	$+0.02$ es/es
ST	$-0.21$ de/de	$+0.05$ de/en / $60.5\%$ $-0.16$ de/es / $11.6\%$	$+0.22$ en/en
TTT	$-0.22$ de/de	$+0.20$ de/en / $89.4\%$ $-0.14$ de/es / $17.0\%$	$+0.25$ en/en

Table 3: Role-corrected strength

\mu

under each interface/reasoning language pair. The middle column reports the strongest reasoning language, followed by the second strongest, while holding the weak interface language fixed. Recovery is

(\mu-\mu_{\text{weak}})/(\mu_{\text{best}}-\mu_{\text{weak}})

. Kuhn, ST and TTT denote Kuhn Poker, SimpleTak and TicTacToe.

Panel of Figure 4 — Figure 4: Relationship between within-model language strength and external measures of language capability and data availability. Each point represents one of eight languages for a given model, lines show per-model least-squares fits, and error bars show standard errors across the language’s seven pairwise margins. In 4(a) , Pearson $r$ is reported in the legend ( $n=8$ ). In 4(b) , highlighted points mark the largest deviations from the fitted trends. Margins are comparable across languages within, but not across, models.

Lang.	Gemma-4 E4B-it	Qwen3-4B	Ministral3-3B
Belebele
Arabic	78.0	75.0	73.7
German	79.1	75.3	82.8
English	82.4	84.1	83.8
Spanish	77.8	75.7	79.8
French	79.2	80.8	76.0

Table 4: Accuracy (%) on Belebele and 5-shot Global MMLU. Bold indicates the best-performing models for each language and benchmark average.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Language	ISO	Script	Translator	Rev.	Tier	Class	Speakers
Chinese	zho	Hans	GPT-5.2 / Opus	native	A	5	1299.9M
English	eng	Latn	GPT-5.2 / Opus	native	A	5	753.4M
Spanish	spa	Latn	GPT-5.2 / Opus	native	A	5	485.0M
Arabic	ara	Arab	GPT-5.2 / Opus	native	A	5	315.4M
French	fra	Latn	GPT-5.2 / Opus	native	A	5	208.2M
Japanese	jpn	Jpan	Qwen2.5-72B	LLM	B	5	128.0M

Appendix

Table E.1 : Track A languages (higher/mid-resource): 8 core experiment languages plus 42 expansion languages, ordered higher- to lower-resource. Translator is the model that produced the localization; Rev. distinguishes native human review ( native , core set) from LLM-only verification ( LLM , expansion). Class : Joshi et al. ( Joshi et al., 2020 ) resource class; Speakers : Wikidata P1098 ( Vrandečić and Krötzsch, 2014 ) .

Language	ISO	Script	Translator	Tier	Class	Speakers	Fid.	Cov.
North Levantine Arabic	apc	Arab	NLLB-200	C	5	44.0M	98	91
Moroccan Arabic	ary	Arab	NLLB-200	C	5	27.5M	100	91
Mesopotamian Arabic	acm	Latn	NLLB-200	C	5	15.7M	95	92
South Levantine Arabic	ajp	Latn	NLLB-200	C	5	11.6M	98	92
Tunisian Arabic	aeb	Arab	NLLB-200	C	5	11.6M	90	90
Taizzi-Adeni Arabic	acq	Latn	NLLB-200	C	5	10.5M	95	92

Appendix

Table E.2 : Track B languages (low-resource tier), ordered higher- to lower-resource. Translator is the forward MT model: NLLB-200 ( Costa-Jussà et al., 2022 ) for all except Chokwe (Toucan ( Elmadany et al., 2024 ) ); meaning fidelity was then verified by a two-model LLM judge (Llama-3.1-405B ( Grattafiori et al., 2024 ) + Qwen2.5-72B ( Qwen Team, 2024 ) ), not by a native reviewer. Tier : C = certified-flagged (fidelity

\geq

85%), E = experimental. Class : Joshi et al. ( Joshi et al., 2020 ) ; Speakers : Wikidata P1098 ( Vrandečić and Krötzsch, 2014 ) . Fid. : measured meaning-fidelity %; Cov. : target-language coverage %.

Language	ISO	Script	Translator	Tier	Class	Speakers	Fid.	Cov.
Northern Kurdish	kmr	Latn	NLLB-200	C	1	14.6M	98	87
Tajik	tgk	Cyrl	NLLB-200	C	1	14.0M	100	90
South Azerbaijani	azb	Latn	NLLB-200	C	1	13.8M	95	82
Tsonga	tso	Latn	NLLB-200	C	1	13.0M	95	88
Kinyarwanda	kin	Latn	NLLB-200	C	1	12.1M	92	86
Nyanja	nya	Latn	NLLB-200	E	1	12.0M	82	90

Appendix

Table E.3 : Track B languages (low-resource tier), continued .

Language	ISO	Script	Translator	Tier	Class	Speakers	Fid.	Cov.
Welsh	cym	Latn	NLLB-200	C	1	724k	95	93
Crimean Tatar	crh	Cyrl	NLLB-200	C	1	553k	100	90
Occitan	oci	Latn	NLLB-200	C	1	542k	100	91
Ligurian	lij	Latn	NLLB-200	C	1	500k	100	87
Silesian	szl	Latn	NLLB-200	C	1	458k	98	91
Asturian	ast	Latn	NLLB-200	C	1	450k	98	84

Appendix

Table E.4 : Track B languages (low-resource tier), continued .

Figure F.1 : Per-game language strength profiles for Gemma-4-E4B-it, Ministral-3-3B-Instruct, and Qwen3-4B. Each cell reports the mean language margin $\mu_{m,g}(A)$ for model $m$ , game $g$ , and language interface $A$ , obtained by averaging the role-pooled pairwise margins of $A$ against all other evaluated languages. Positive values indicate stronger average performance through language $A$ , while negative values indicate weaker average performance.

Language	Row	Column	Diagonal
Gemma-4-E4B-it
English	28.5%	34.8%	36.7%
Chinese	17.5%	28.0%	54.5%
Spanish	18.5%	34.4%	47.0%
French	22.6%	35.6%	41.7%
German	19.8%	37.5%	42.8%

Appendix

Table G.1 : Distribution of defeat types in TicTacToe by language and model. Each value is the percentage of defeats in which the opponent completed a row, column, or diagonal. These distributions indicate which spatial relationships each model fails to track under different language interfaces.

Language	Row	Column
English	53.2%	46.8%
Chinese	52.3%	47.7%
Spanish	51.5%	48.5%
French	46.0%	54.0%
German	39.7%	60.3%
Hebrew	42.8%	57.2%

Appendix

Table G.2 : Distribution of loss types in SimpleTak by language for Gemma4-E4B-it: percentage of losses where the opponent completed a row or column.

Lang.	Inv.	Bluff J	Bet Q	Value K	Call K	Fold K
Gemma-4-E4B-it
English	2.5%	1.3%	42.9%	92.2%	90.7%	2.0%
Chinese	2.4%	2.4%	57.0%	92.7%	93.7%	0.5%
Spanish	0.7%	0.9%	60.3%	98.2%	98.3%	0.1%
French	0.5%	1.3%	59.0%	98.3%	99.3%	0.3%
German	0.9%	1.9%	62.3%	96.8%	97.8%	0.4%

Appendix

Table G.3 : Card-conditioned behavior in KuhnPoker across languages and models. Inv. is the fraction of invalid actions. Bluff J , Bet Q , and Value K denote the probabilities of betting with

J

Q

, and

K

, respectively, when [check] and [bet] are available. Call K and Fold K denote the probabilities of calling and folding with

K

when facing a bet. Higher Value K and Call K indicate more reliable play with the strongest card, whereas a high Fold K indicates a severe strategic error. Mean and SD are computed across languages.

Language	Win %	Opt. Strategy Mentions	Opt. Move
Gemma-4-E4B-it
English	50.2%	43,927	99.9%
Chinese	50.1%	32,207	99.8%
Spanish	50.3%	85,103	99.7%
French	50.1%	71,607	99.2%
German	50.3%	54,801	99.9%

Appendix

Table G.4 : Cross-lingual Nim performance across three models. We report the overall win rate, the number of optimal strategy mentions, and the % of successful first move optimal plays (Opt. Move). Qwen3-4B and Ministral3-3B show rough alignment between mention count and win%, while interestingly mention count and execution % aren’t necessarily aligned.

Cross-Lingual Consistency in Language Models

Cross-lingual consistency in language models studies whether equivalent inputs across languages produce comparable knowledge, reasoning, and safety behavior.

See all 72 papers

May 27, 2026

DEPART: DEcomposing PARiTy across Multilingual LLMs

Multilingual Large Language Models (mLLMs) leaderboards report per-language accuracy but rarely explain why disparities emerge, leaving systemic biases unattributed and offering practitioners no actionable levers. We first establish that these gaps are systematic rather than artifacts of sampling noise via distribution-free Friedman and Kruskal--Wallis tests, then introduce a two-step Bayesian hierarchical framework that decomposes multilingual performance variance into interpretable components. First, isolating the variance attributable to language identity, we show that observable language features (script, family, typological distance) explain

R^2_{\text{ling}} = 79\%

of this variance on understanding tasks and

92\%

on reasoning, with a model's internal representational similarity to English emerging as the dominant predictor across both task buckets. Second, decomposing the full (model

\times

benchmark

\times

language) cube, we find that NLU and reasoning have fundamentally divergent variance profiles: model identity dominates understanding (

66.7\%

of variance), whereas the benchmark

\times

model interaction dominates reasoning (

46.3\%

). Together these results recast multilingual evaluation from passive performance mapping into an explainable, diagnostic framework with concrete levers for targeting the root drivers of language disparity.

Manan Uppadhyay, Prashant Kodali, Pranjal Chitale +3

Microsoft Research India · IIT Gandhinagar

Multilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models

Ask PDF

Feb 23, 2026

Multilingual Large Language Models do not comprehend all natural languages to equal degrees

Large Language Models (LLMs) play a critical role in how humans access information. While their core use relies on comprehending written requests, our understanding of this ability is currently limited, because most benchmarks evaluate LLMs in high-resource languages predominantly spoken by Western, Educated, Industrialised, Rich, and Democratic (WEIRD) communities. The default assumption is that English is the best-performing language for LLMs, while smaller, low-resource languages are linked to less reliable outputs, even in multilingual, state-of-the-art models. To track variation in the comprehension abilities of LLMs, we prompt 3 popular models on a language comprehension task across 12 languages, representing the Indo-European, Afro-Asiatic, Turkic, Sino-Tibetan, and Japonic language families. Our results suggest that the models exhibit remarkable linguistic accuracy across typologically diverse languages, yet they fall behind human baselines in all of them, albeit to different degrees. Contrary to what was expected, English is not the best-performing language, as it was systematically outperformed by several Romance languages, even lower-resource ones. We frame the results by discussing the role of several factors that drive LLM performance, such as tokenization, language distance from Spanish and English, size of training data, and data origin in high- vs. low-resource languages and WEIRD vs. non-WEIRD communities.

Natalia Moskvina, Raquel Montero, Masaya Yoshida +8

Universitat Autònoma de Barcelona · Institució Catalana de Recerca i Estudis Avançats (ICREA) · Radboud University Nijmegen +1

Multilingual Language Models Multilingual Language Model Evaluation

Ask PDF

Aug 2, 2026

Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+

We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others. Unlike static or preference-based evaluation, this paradigm is multi-turn, reference-free and programmatically scored, and because the game mechanics are language-agnostic it extends to a new language by localising a fixed set of prompt and word-list files. Evaluating nine open-weight and commercial LLMs, we find that no open-weight model covers the EU-24 well: in every official language both commercial systems outscore every open-weight model, and the two weakest average below 40 points across the EU-24. The commercial systems stay ahead even in languages with four orders of magnitude less public web text, showing that linguistic parity is achievable, but not from public crawls alone. A model's home region lifts it without closing the gap: Chinese is the strongest of all 30 languages for two Chinese-developed models, yet the best Chinese score of any model belongs to a US commercial system. Coverage is also not parity of service. Pooled over models and languages, the median non-English language costs 31% more to run than English, and scores 10% lower.

Sherzod Hakimov, Karl Osswald, Jelle Psurek +3

Computational Linguistics, Department of Linguistics University of Potsdam, Germany · 1German Research Center for Artificial Intelligence (DFKI), Berlin, Germany

Multilingual Language Model Evaluation LLM Evaluation Open-Weight Language Models Cross-Lingual Consistency in Language Models

Ask PDF

Abstract

Cross-Lingual Consistency in Language Models

Explore similar work

DEPART: DEcomposing PARiTy across Multilingual LLMs

Multilingual Large Language Models do not comprehend all natural languages to equal degrees

Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+