A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMs

Anna Feldman; Libby Barak; Jing Peng

cs.CLFeb 28, 2026

A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMs

Authors: Anna Feldman, Libby Barak, Jing Peng

Organizations: Montclair State University New Jersey, USA

Abstract

We introduce a typology-aware diagnostic for multilingual masked language models that tests reliance on word order versus inflectional form. Using Universal Dependencies, we apply inference-time perturbations: full token scrambling, content-word scrambling with function words fixed, dependency-based head--dependent swaps, and sentence-level lemma substitution (+L), which lemmatizes both the context and the masked target label. We evaluate mBERT and XLM-R on English, Chinese, German, Spanish, and Russian. Full scrambling drives word-level reconstruction accuracy near zero in all languages; partial and head--dependent perturbations cause smaller but still large drops. +L has little effect in Chinese but substantially lowers accuracy in German/Spanish/Russian, and it does not mitigate the impact of scrambling. Top-5 word accuracy shows the same pattern: under full scrambling, the gold word rarely appears among the five highest-ranked reconstructions. We release code, sampling scripts, and balanced evaluation subsets; Turkish results under strict reconstruction are reported in the appendix.

Figures & tables

Condition	Input string (EN)	Gold
Orig	the[1] scientist[2] [MASK] the[3] books[4] yesterday[5].	analyzed
Orig+L	the[1] scientist[2] [MASK] the[3] book[4] yesterday[5].	analyze
Full	yesterday[5] the[3] the[1] scientist[2] books[4] [MASK] .	analyzed
Full+L	yesterday[5] the[3] the[1] scientist[2] book[4] [MASK] .	analyze
Part	the[1] books[4] scientist[2] the[3] yesterday[5] [MASK] .	analyzed
Part+L	the[1] book[4] scientist[2] the[3] yesterday[5] [MASK] .	analyze

Table 1: Illustrative English examples for each condition. Bracketed indices [i] mark token identity for expository purposes only. “+L” applies sentence-level UD lemma substitution; the gold label is the lemma of the masked target.

Lang	+L tok. chg	Full pos. chg	Part pos. chg	Head pos. chg
DE	0.344	0.910	0.173	0.339
EN	0.191	0.906	0.160	0.455
ES	0.350	0.945	0.195	0.452
RU	0.555	0.935	0.268	0.570
ZH	0.007	0.950	0.272	0.478

Table 2: Perturbation magnitude on evaluated UD sentences (non-punctuation tokens). +L tok. chg: fraction of tokens with FORM

\neq

LEMMA. Pos. chg: fraction of tokens whose absolute position differs from Orig.

Lang	$N$ (mBERT)	$N$ (XLM-R)
DE	300	300
EN	290	304
ES	300	304
RU	286	304
ZH	300	304

Table 3: Balanced per-condition counts (

N

) for five languages.

mBERT
Lang	Orig	Full	Part	Head	Orig+L	Full+L	Part+L
DE	0.143	0.013	0.075	0.078	0.060	0.003	0.033
EN	0.219	0.016	0.123	0.060	0.167	0.019	0.090
ES	0.251	0.013	0.143	0.063	0.143	0.003	0.085
RU	0.276	0.008	0.119	0.063	0.119	0.008	0.061
ZH	0.443	0.023	0.223	0.133	0.448	0.023	0.230

Table 4: Balanced word-level accuracy for mBERT. “+L” = sentence-level UD lemma substitution (context and label).

XLM-R
Lang	Orig	Full	Part	Head	Orig+L	Full+L	Part+L
DE	0.338	0.013	0.195	0.168	0.150	0.008	0.078
EN	0.303	0.006	0.166	0.079	0.218	0.003	0.123
ES	0.338	0.013	0.175	0.100	0.181	0.010	0.095
RU	0.293	0.013	0.147	0.078	0.126	0.015	0.068
ZH	0.325	0.025	0.150	0.110	0.333	0.025	0.150

Table 5: Balanced word-level accuracy for XLM-R. “+L” = sentence-level UD lemma substitution (context and label).

mBERT
Lang	$S_{\text{full}}$	$S_{\text{part}}$	$S_{\text{head}}$	$S_{\text{+L}}$	$I_{\text{full}}$
DE	0.912	0.474	0.456	0.579	0.073
EN	0.925	0.438	0.725	0.238	0.055
ES	0.950	0.430	0.750	0.430	0.098
RU	0.972	0.569	0.771	0.569	0.157
ZH	0.949	0.497	0.701	$-0.011$	$-0.005$

Table 6: Balanced sensitivities and interaction for mBERT computed in accuracy space.

XLM-R
Lang	$S_{\text{full}}$	$S_{\text{part}}$	$S_{\text{head}}$	$S_{\text{+L}}$	$I_{\text{full}}$
DE	0.963	0.422	0.504	0.556	0.183
EN	0.982	0.450	0.739	0.279	0.082
ES	0.963	0.481	0.704	0.467	0.155
RU	0.957	0.500	0.733	0.569	0.169
ZH	0.923	0.538	0.662	$-0.023$	$-0.008$

Table 7: Balanced sensitivities and interaction for XLM-R in accuracy space.

Figure 1: Accuracy across perturbation conditions with Wilson 95% CIs.

Figure 2: Relative-drop sensitivities $S$ (95% CIs).

Figure 3: Interaction effects $I_{\text{full}}$ (95% CIs).

Figure 4: Head vs. Part ( $\Delta$ accuracy = Part $-$ Head).

	$N$ (mBERT)	$N$ (XLM-R)
TR (balanced per condition)	108	152

Table 8: Balanced per-condition counts (

N

) for Turkish (TR).

TR: Balanced word-level accuracy
Model	Orig	Full	Part	Head	Orig+L	Full+L	Part+L
mBERT	0.000	0.000	0.019	0.006	0.013	0.013	0.019
XLM-R	0.063	0.000	0.057	0.013	0.006	0.006	0.019

Table 9: Balanced word-level accuracies for Turkish (TR). Interpret with caution (floor, selection, reconstruction effects).

mBERT
Lang	$S_{\text{full}}$	$S_{\text{part}}$	$S_{\text{head}}$	$S_{\text{morph}}$	$I_{\text{full}}$	$I_{\text{part}}$
DE	0.980	0.418	0.438	0.559	0.081	0.036
EN	0.954	0.457	0.706	0.323	0.073	0.026
ES	0.964	0.420	0.712	0.407	0.094	0.042
RU	0.982	0.533	0.724	0.629	0.166	0.096
TR	0.719	0.450	0.465	0.143	0.004	0.000

Table 10: Sensitivities and interaction in accuracy space on unbalanced sets (all languages shown for cross-reference).

Figure 5: Turkish (TR) only: Head vs. Part ( $\Delta$ accuracy = Part $-$ Head).

mBERT (Top-5)
Lang	Orig	Full	Part	Head	Orig+L	Full+L	Part+L
DE	0.282	0.021	0.185	0.153	0.166	0.013	0.102
EN	0.400	0.031	0.236	0.140	0.312	0.029	0.191
ES	0.423	0.033	0.264	0.170	0.266	0.037	0.175
RU	0.403	0.014	0.201	0.127	0.172	0.017	0.105
ZH	0.588	0.053	0.331	0.247	0.583	0.055	0.328

Table 11: Balanced top-5 word accuracy for mBERT.

XLM-R (Top-5)
Lang	Orig	Full	Part	Head	Orig+L	Full+L	Part+L
DE	0.503	0.030	0.311	0.289	0.281	0.018	0.171
EN	0.485	0.031	0.288	0.184	0.390	0.022	0.252
ES	0.496	0.030	0.294	0.233	0.300	0.022	0.193
RU	0.439	0.022	0.262	0.184	0.224	0.025	0.153
ZH	0.464	0.067	0.263	0.237	0.466	0.066	0.263

Table 12: Balanced top-5 word accuracy for XLM-R.

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 485 papers

Vocabulary shapes cross-lingual variation of word-order learnability in language models
Mar 19, 2026Jonas Mayer Martins, Jaap Jumelet, Viola Priesemann +1Linguistic Typology
University of Göttingen, Germany · University of Groningen, Netherlands · MPI for Dynamics and Self-Organization, Germany
Tiny-Scale Chinese BERT Pretraining: A Controlled Comparison of MLM, WWM, and MacBERT Strategies
Oct 6, 2026Yiping BaiLanguage Model Pretraining Language Modeling
Guangdong Haiqixing Marine Technology Co., Ltd.
Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations
Sep 3, 2026Karthika Nhayakkat, Rajat Verma, Maharaj Brahma +4Numerical Reasoning in Language Models Multilingual Language Model Evaluation
Department of CSE, Indian Institute of Technology Bombay · SCEE, Indian Institute of Technology Mandi · Department of CSE, Indian Institute of Technology Hyderabad

Mar 19, 2026J/K move · Enter open · S save

Vocabulary shapes cross-lingual variation of word-order learnability in language models

Jonas Mayer Martins, Jaap Jumelet, Viola Priesemann, Lisa Beinborn

University of Göttingen, Germany · University of Groningen, Netherlands · MPI for Dynamics and Self-Organization, Germany

Linguistic Typology

Open paper AskPDF arXiv

Why do some languages like Czech permit free word order, while others like English do not? We address this question by pretraining transformer language models on a spectrum of synthetic word-order variants of natural languages. We observe that greater word-order irregularity consistently raises model surprisal, indicating reduced learnability. Sentence reversal, however, affects learnability only weakly. A coarse distinction of free- (e.g., Czech and Finnish) and fixed-word-order languages (e.g., English and French) does not explain cross-lingual variation. Instead, the structure of the word and subword vocabulary strongly predicts the model surprisal. Overall, vocabulary structure emerges as a key driver of computational word-order learnability across languages.

Abstract

Multilingual Language Model Evaluation

Explore similar work

Vocabulary shapes cross-lingual variation of word-order learnability in language models

Tiny-Scale Chinese BERT Pretraining: A Controlled Comparison of MLM, WWM, and MacBERT Strategies

Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations