Organizations: Leipzig University & ScaDS.AI Dresden/Leipzig · The Hong Kong Polytechnic University · CENTAL, UCLouvain & FNRS · Uppsala University · DZSKJ, University Medical Center Hamburg-Eppendorf · Georgetown University · Federal University of Juiz de Fora, FrameNet Brasil, CNPq
Languages use different morphosyntactic strategies to express the same meaning. Speakers of multiple languages are able to apply strategies from one language to another, either by mistake or intentionally, to produce a literal translation, or calque. Considering that large language models (LLMs) have been shown to generalize crosslingually, despite being predominantly trained on English, we investigate whether they overgeneralize constructional strategies from English to other languages. To test this, we create a dataset of 5583 sentence pairs across seven languages, matching literal translations of strategies from English into other languages, and literal translations of the other languages' strategies into English. We then quantify constructional bleeding by measuring how often strategies from English are assigned a higher probability than native strategies. We find that models often fail to assign higher probabilities to native strategies and tend to prefer English-derived strategies even when evaluating sentences in other languages. Our findings highlight the limitations of the current English-dominant training paradigm.
Figures & tables
Figure 1: English and French speakers describe playing the cello with distinct morphosyntactic strategies. We measure bleeding of English morphosyntax into French ( Δlogp(⋅) between French sentences) and additionally conduct a control experiment on English ( Δlogp(⋅) between English sentences).
Strategy Source Language
Target language
English
German
Das Wasser fühlt sich kalt an.
The water feels itself cold.
English
Das Wasser fühlt kalt.
The water feels cold.
Swedish
Jag bäddade sängen.
I bedded the bed.
English
Jag gjorde sängen.
I made the bed.
French
Je joue du violoncelle.
I play of the cello.
English
Je joue le violoncelle.
I play the cello.
Table 1: Examples of sentence pairs in each language.
Figure 2: Distribution of strategy preferences across languages for sentence pairs with strong native-speaker preferences ( ≫ ), measured by Δ SLOR. Following Zhong et al. (2025) , continually pretrained models (CPMs) are adapted from English-centric pretrained models through additional pretraining on target-language data, whereas balanced language models (BLMs) are pretrained from scratch with an emphasis on English and their respective focus languages. Positive values indicate a preference for the native strategy. Numbers in parentheses indicate sample sizes.
Figure 3: Failure rates (%) across languages, defined as the percentage of sentence pairs for which a model assigns a higher SLOR to the transferred strategy than to the native strategy.
Figure 4: Comparison of model preferences for native Japanese strategies over strategies transferred from English and other languages, measured by Δ SLOR. All comparisons are evaluated in Japanese, with matched semantic content and restricted to sentence pairs with strong preference on the native Japanese strategy ( ≫ ) over both the English-transferred and the corresponding non-English-transferred strategies. Positive values indicate a preference for the native Japanese strategy. Numbers in parentheses indicate sample sizes.
Figure 5: Standardized regression coefficients from multivariable linear regression models predicting Δ SLOR. Predictors include human preference annotations, model size, instruction tuning, and cross-linguistic structural features. Positive coefficients indicate a positive association with preference for the native strategy.
Figure 6: Pairwise Spearman rank correlations of Δ SLOR across language models, computed over matched sentence pairs. Higher correlations indicate greater similarity in strategy preference distributions between models.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Strategy
Body Part
Everyday Activities
Sensory Experience
Native
Transferred
≫
>
≫
>
≫
>
en
de
162
7
4
36
66
1
se
78
13
39
22
70
1
fr
188
3
182
1
39
14
pt
68
4
18
48
85
6
zh
60
5
122
7
15
11
Appendix
Table 2: Distribution of human preferences for native over transferred strategies across three construction groups. ≫ and > indicate strong and weak preferences for the native strategy, respectively. Japanese-control comparisons use Japanese as the native strategy and a non-English language as the source of the transferred strategy.
Category
Value
Count
Temporal
Past
236
Ongoing
133
Recurrent
121
Current state
77
Anterior
54
Generic
48
Appendix
Table 3: Dataset composition by temporal interpretation, subject type, and construction group. The distribution is identical across all evaluated languages.
Family
Model
Citation
Aya Expanse
CohereLabs/aya-expanse-8b
Dang et al. (2024)
CohereLabs/aya-expanse-32b
Tiny Aya
CohereLabs/tiny-aya-base
Salamanca et al. (2026)
CohereLabs/tiny-aya-global
CohereLabs/tiny-aya-earth
CohereLabs/tiny-aya-fire
Appendix
Table 4: Models evaluated, grouped by model family.
Figure 7: Distribution of strategy preferences across languages for sentence pairs with strong native-speaker preferences ( ≫ ), measured by Δ MeanLP. Positive values indicate a preference for the native strategy.
Figure 8: Comparison of model preferences for native Japanese strategies over strategies transferred from English and other languages, measured by Δ MeanLP. All comparisons are evaluated in Japanese, with matched semantic content and restricted to sentence pairs with a strong preference for the native Japanese strategy ( ≫ ) over both the English-transferred and the corresponding non-English-transferred strategies. Positive values indicate a preference for the native Japanese strategy.
Language
Candidate pairs
Groups
Threshold
German
4,879
157
31.07
French
5,674
148
38.34
Japanese
9,632
181
53.22
Portuguese
551
102
5.40
Swedish
575
116
4.96
Chinese
8,429
188
44.84
Appendix
Table 5: Candidate statistics after filtering. The threshold corresponds to the mean number of candidate pairs per group within each language.
≫ vs others
( ≫ or > ) vs others
Lang.
N
Agr.
κ
Agr.
κ
en
50
0.80
0.567
0.78
0.556
de
50
0.82
0.613
0.78
0.551
se
50
0.78
0.428
0.94
0.849
fr
50
0.82
0.526
0.98
0.929
pt
50
0.76
0.525
0.86
0.451
Appendix
Table 6: Inter-annotator agreement by target language (Lang). Strong preference distinguishes ≫ from all other labels, whereas preference direction distinguishes ≫ or > from all other labels. Agreement (Agr) denotes the proportion of matching annotations, and κ denotes Cohen’s kappa.
Lang.
Native strategy
A1
A2
Transferred strategy
Failed
de
Ich habe Zähne geputzt.
>
>
Ich habe meine Zähne geputzt.
27/28
Ich bin umgeknickt.
≫
≫
Ich habe meinen Knöchel umgeknickt.
26/28
Ich habe mir das Gesicht gewaschen.
>
>
Ich habe mein Gesicht gewaschen.
26/28
sv
Axeln gick ur led på mig.
≫
>
Jag dislokerade min axel.
28/28
Maten luktade vitlök.
>
>
Måltiden luktade som vitlök.
27/28
Jag borstade tänderna.
>
>
Jag borstade mina tänder.
25/28
Appendix
Table 7: High-failure-rate canonical sentence examples across 28 English-centric models in six languages. For each language, the three sentence pairs are shown, for which both annotators preferred the native strategy ( > or ≫ ). The A1 and A2 columns report the respective native-speaker preference annotations. The Failed column indicates the number of models assigning a lower SLOR to the native strategy than to the transferred strategy, out of 28 evaluated models.
Model
IT
EN
DE
SV
FR
PT
JA
ZH
Multilingual models
Tiny Aya
Base
6.6
35.5
44.3
22.0
59.6
52.6
31.8
Earth
✓
9.5
48.2
46.6
22.9
66.8
49.7
40.9
Fire
✓
8.1
44.2
48.9
23.5
67.3
50.9
40.3
Global
✓
10.2
48.7
46.1
21.8
66.6
49.7
41.2
Water
✓
10.0
47.5
47.3
22.2
66.6
53.2
40.3
Appendix
Table 8: Failure rates (%) across languages, defined as the percentage of sentence pairs for which a model assigns a higher SLOR to the transferred strategy than to the native strategy. Darker red cells indicate higher failure rates. ✓ in the IT column marks instruction-tuned models.
Constructed languages (conlangs) are intentionally created human languages with a rich tradition of linguistic creativity. Despite their potential for studying language learning in large language models (LLMs), existing conlangs remain largely underexplored in LLM research. We present ConlangBench, the first large-scale benchmark for evaluating and training LLMs on 21 existing conlangs. We collect over 21M conlang-English parallel sentence pairs (including 430K pairs across the 20 non-Esperanto conlangs) and 321K vocabulary entries. In bidirectional translation experiments, we find that models perform better on a posteriori conlangs, whose vocabularies are derived from natural languages, reflecting the design characteristics of conlangs. Training on ConlangBench also shows that models can learn all eight conlangs for which sufficient parallel corpora are available, while their learning curves vary depending on how the conlangs were created. Our findings suggest that conlangs provide a unique testbed for investigating how LLMs acquire low-resource languages.
Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work quantifies cross-lingual skill inconsistency orthogonally from knowledge and general benchmark performance. We do this via multilingual self-play: two instances of the same model compete in a text-based game, each interacting through a different language interface. Since the model, opponent, rules, state space, and available actions remain fixed, this setting isolates the effect of language on the model's realized behavior. We build a multilingual extension to TextArena and evaluate three open-weight models across eight languages and six games covering spatial reasoning, imperfect information, resource allocation, and repeated interaction. We find that the same model can exhibit markedly different playing strength across languages, with systematic variation in win--loss margins, invalid actions, and strategic tendencies. Detailed analyses reveal language-specific failures in spatial reasoning, card-conditioned decisions, and optimal move selection. In some settings, changing only the intermediate reasoning language recovers much of the lost performance, suggesting that language can affect different stages of the decision process. These results show that skill discrepancies are a measurable major roadblock in the development of truly multilingual models. Better understanding these discrepancies can help us design models that perform more equitably across languages.
Bobby Cheng, Adam Gaber, Zhengyuan Liu +4
NA · Weizmann Institute of Science · QMIT-IBM Watson AI Lab +3
Despite the widespread multilingual deployment of large language models, post-training pipelines remain predominantly English-centric, contributing to performance disparities across languages. We present a systematic, controlled study of the interplay between training language coverage, model scale, and task domain, based on 220 supervised fine-tuning runs on parallel translated multilingual data mixtures spanning mathematical reasoning and API calling tasks, with models up to 8B parameters. We find that English-only post-training is typically suboptimal: incorporating even a single non-English language improves both English performance and cross-lingual generalization. Increasing language diversity during post-training generally yields further gains, particularly for low-resource languages, while performance on high-resource languages tends to plateau rather than degrade. Moreover, greater language diversity enables strong zero-shot transfer to unseen languages, reducing the need for direct inclusion, though gains remain limited for typologically distant, low-resource languages.