Why do some languages like Czech permit free word order, while others like English do not? We address this question by pretraining transformer language models on a spectrum of synthetic word-order variants of natural languages. We observe that greater word-order irregularity consistently raises model surprisal, indicating reduced learnability. Sentence reversal, however, affects learnability only weakly. A coarse distinction of free- (e.g., Czech and Finnish) and fixed-word-order languages (e.g., English and French) does not explain cross-lingual variation. Instead, the structure of the word and subword vocabulary strongly predicts the model surprisal. Overall, vocabulary structure emerges as a key driver of computational word-order learnability across languages.
Figures & tables
Figure 1: We create a spectrum of synthetic language variants by deterministically permuting words within each sentence. For each sentence length, a permutation is sampled from the Mallows permutation model, where the order parameter θ controls preference for the original word order. As an example, we show the probability distribution of a word originally at position 16 in a 20 -word sentence.
Figure 2: (a) Surprisal change ΔS due to word-order perturbations with order θ for each language (named in panel b). Color shades encode word order: fixed as solid red and free as dashed blue. (b) Zoom-in of surprisal change ΔSirreg at irregular order θ=0 against the original surprisal Sorig . Red and blue markers are projections onto the axes that indicate a separation of free- and fixed-word-order languages in Sorig but an overlap in ΔSirreg . Transparent bands in panel (a) and error bars in (b) show the 25th to 75th percentile over seeds; the lines and points are the median seed, respectively.
Figure 3: Surprisal difference between word- and subword-level shuffling at irregular word order, with median seed and interquartile ranges.
Figure 4: Percentage of (a) words and (b) subwords in the corpus accounted for by the most frequent vocabulary items. This coverage increases more slowly for languages with freer word order compared to languages with relatively fixed word order (shades of blue and red, respectively).
Figure 5: The absolute surprisal S(θ)=Sorig+ΔS(θ) per language modeled through a set of vocabulary statistics, encompassing coverage, sentence length, and proxies for morphological complexity. The predictions are cross-validated through leave-one-language-out: Each language is predicted solely on the basis of its own vocabulary statistics by a model trained on the surprisal of the other languages and their predictors.
Figure 6: (a) The cross-validated explained variance per slice of θ of only the vocabulary component (green) with mean Rˉ2=0.65 , ranging from 0.26 to 0.76 , and of both components (green-yellow) with mean Rˉ2=0.79 , ranging from 0.66 to 0.86 . (b) PLS scores of the main component (vocabulary) and the secondary component (morphological complexity).
Figure 7: Effect of the vocabulary size on (a) the original surprisal Sorig and (b) the surprisal change at irregular word order ΔSirreg . (c) Subword coverage per vocabulary size, grouped by free and fixed word order, where line transparency and the vertical gray lines encode the vocabulary size. Colored lines show medians and shaded regions show interquartile ranges, computed across random seeds for panels (a) and (b) and across languages for panel (c).
Language
ISO
Branch
Family
Flexibility
Morphology
Reference
English
en
Germanic
IE
fixed
analytic
Aarts (2011)
Danish
da
analytic
Lundskaer-Nielsen and Holmes (2015)
Swedish
sv
analytic
Holmes and Hinchliffe (2013)
Portuguese
pt
Romance
fusional
Kabatek (2022) ; Harris and Vincent (2012)
French
fr
fusional
Harris and Vincent (2012)
Latvian
lv
Baltic
IE
free
fusional
Praulinš (2012)
Table 1: Language selection with name and ISO-code abbreviation, grouped by branch and family (IE: Indo-European). We list word-order flexibility and morphology.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Normalized analytical mean and variance of Kendall’s τ with Mallows shuffling over the order θ for different sentence lengths n .
Parameter
Value
Architecture
Transformer decoder
Total parameters
50.5 M
Layers
12
Embedding size
384
Attention heads
12
Attention KV heads
4
Appendix
Table 2: Hyperparameters for the PicoLM models used in our experiments.
Coverage
Sentence length
Morph. complexity
Lang.
Order
Cw,100
Cs,100
C simil.
∫Cw
Subw./sent.
Words/sent.
Fertility
Word len.
Types
fr
Fixed
52.7
49.0
0.974
61.0
28.1
24.4
1.15
6.02
96 727
pt
51.0
47.9
0.979
60.6
28.1
24.2
1.16
6.03
108 442
en
55.4
52.6
0.976
62.6
26.4
23.8
1.11
5.70
70 536
sv
52.6
47.5
0.971
60.1
24.4
20.4
1.20
6.22
177 002
da
54.2
49.6
0.971
60.8
25.7
21.7
1.19
6.09
179 915
Appendix
Table 3: Vocabulary statistics for each language: Word and subword coverage at rank 100, coverage similarity and integral; sentence length in subwords and words; morphological complexity measured by fertility, word length in characters, and word types.
Figure 9: Coverage measures: the (a) word coverage integral and (b) word-subword-coverage similarity.
Figure 10: Correlation matrix of the predictors, grouped by coverage (C), sentence length (S), and proxies of morphological complexity (M).
Figure 11: (a) Predictor and (b) response loadings of the vocabulary and morphological complexity component.
We systematically compare word order preferences in decoder-only language models across 192 artificial languages and typologically diverse natural languages. On artificial languages, models exhibit a left-branching preference that aligns with neither natural language universals nor human word order learning biases. On natural languages, monolingual models show no clear base word order bias at small scales, but as data grows, a preference for right-branching subject-verb-object (SVO) languages emerges while SOV falls behind despite being the most frequent order cross-linguistically. This SVO advantage extends to multilingual models and correlates with language resource level and data quality rather than word order. Thus, the same architecture exhibits opposite preferences on artificial and natural languages, establishing that word order biases observed in practice are data-driven. Since highly-resourced languages are overwhelmingly SVO, these biases risk gradually reducing word order diversity, particularly in languages that productively use multiple word orders, with the widespread adoption of LLMs.
Varvara Arzt, Allan Hanbury, Terra Blevins
Faculty of Informatics, TU Wien · Khoury College of Computer Sciences, Northeastern University
A central question in language acquisition is whether linguistic biases can emerge from general learning mechanisms operating over underdetermined input. Artificial Language Learning (ALL) studies have shown that human learners reliably generalize beyond the evidence provided, including by preferring scope-homomorphic noun phrase modifier orders. In this work, we investigate whether language models exhibit the same bias under similar conditions. We create a controlled learning environment in which models are trained on a corpus where all noun phrases containing multiple modifiers have been removed, eliminating direct evidence about modifier ordering, and are then evaluated on multiple modifier sentences. Across three model sizes, we find that they consistently prefer scope-homomorphic orders despite never observing them during training. These preferences vary in strength by modifier type. To investigate the source of these preferences, we examine noun-modifier association strength using pointwise mutual information (PMI). While PMI reflects known modifier-ordering patterns, it does not explain the models' ordering preferences. These findings demonstrate that LMs can recover human-like linguistic generalizations from impoverished input and provide a controlled framework for investigating the mechanisms underlying such biases.
Amanda Popadich, Shane Steinert-Threlkeld
Department of Linguistics, University of Washington
We introduce a typology-aware diagnostic for multilingual masked language models that tests reliance on word order versus inflectional form. Using Universal Dependencies, we apply inference-time perturbations: full token scrambling, content-word scrambling with function words fixed, dependency-based head--dependent swaps, and sentence-level lemma substitution (+L), which lemmatizes both the context and the masked target label. We evaluate mBERT and XLM-R on English, Chinese, German, Spanish, and Russian. Full scrambling drives word-level reconstruction accuracy near zero in all languages; partial and head--dependent perturbations cause smaller but still large drops. +L has little effect in Chinese but substantially lowers accuracy in German/Spanish/Russian, and it does not mitigate the impact of scrambling. Top-5 word accuracy shows the same pattern: under full scrambling, the gold word rarely appears among the five highest-ranked reconstructions. We release code, sampling scripts, and balanced evaluation subsets; Turkish results under strict reconstruction are reported in the appendix.