cs.CLMar 19, 2026
SaveVocabulary shapes cross-lingual variation of word-order learnability in language models
Organizations: University of Göttingen, Germany · University of Groningen, Netherlands · MPI for Dynamics and Self-Organization, Germany
Abstract
Why do some languages like Czech permit free word order, while others like English do not? We address this question by pretraining transformer language models on a spectrum of synthetic word-order variants of natural languages. We observe that greater word-order irregularity consistently raises model surprisal, indicating reduced learnability. Sentence reversal, however, affects learnability only weakly. A coarse distinction of free- (e.g., Czech and Finnish) and fixed-word-order languages (e.g., English and French) does not explain cross-lingual variation. Instead, the structure of the word and subword vocabulary strongly predicts the model surprisal. Overall, vocabulary structure emerges as a key driver of computational word-order learnability across languages.
Figures & tables
Figure 1: We create a spectrum of synthetic language variants by deterministically permuting words within each sentence. For each sentence length, a permutation is sampled from the Mallows permutation model, where the order parameter controls preference for the original word order. As an example, we show the probability distribution of a word originally at position 16 in a -word sentence.
Figure 2: (a) Surprisal change due to word-order perturbations with order for each language (named in panel b). Color shades encode word order: fixed as solid red and free as dashed blue. (b) Zoom-in of surprisal change at irregular order against the original surprisal . Red and blue markers are projections onto the axes that indicate a separation of free- and fixed-word-order languages in but an overlap in . Transparent bands in panel (a) and error bars in (b) show the 25th to 75th percentile over seeds; the lines and points are the median seed, respectively.
Figure 3: Surprisal difference between word- and subword-level shuffling at irregular word order, with median seed and interquartile ranges.
Figure 4: Percentage of (a) words and (b) subwords in the corpus accounted for by the most frequent vocabulary items. This coverage increases more slowly for languages with freer word order compared to languages with relatively fixed word order (shades of blue and red, respectively).
Figure 5: The absolute surprisal per language modeled through a set of vocabulary statistics, encompassing coverage, sentence length, and proxies for morphological complexity. The predictions are cross-validated through leave-one-language-out: Each language is predicted solely on the basis of its own vocabulary statistics by a model trained on the surprisal of the other languages and their predictors.
Figure 6: (a) The cross-validated explained variance per slice of of only the vocabulary component (green) with mean , ranging from to , and of both components (green-yellow) with mean , ranging from to . (b) PLS scores of the main component (vocabulary) and the secondary component (morphological complexity).
Figure 7: Effect of the vocabulary size on (a) the original surprisal and (b) the surprisal change at irregular word order . (c) Subword coverage per vocabulary size, grouped by free and fixed word order, where line transparency and the vertical gray lines encode the vocabulary size. Colored lines show medians and shaded regions show interquartile ranges, computed across random seeds for panels (a) and (b) and across languages for panel (c).
| Language | ISO | Branch | Family | Flexibility | Morphology | Reference |
|---|---|---|---|---|---|---|
| English | en | Germanic | IE | fixed | analytic | Aarts (2011) |
| Danish | da | analytic | Lundskaer-Nielsen and Holmes (2015) | |||
| Swedish | sv | analytic | Holmes and Hinchliffe (2013) | |||
| Portuguese | pt | Romance | fusional | Kabatek (2022) ; Harris and Vincent (2012) | ||
| French | fr | fusional | Harris and Vincent (2012) | |||
| Latvian | lv | Baltic | IE | free | fusional | Praulinš (2012) |
Table 1: Language selection with name and ISO-code abbreviation, grouped by branch and family (IE: Indo-European). We list word-order flexibility and morphology.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Normalized analytical mean and variance of Kendall’s with Mallows shuffling over the order for different sentence lengths .
| Parameter | Value |
|---|---|
| Architecture | Transformer decoder |
| Total parameters | 50.5 M |
| Layers | 12 |
| Embedding size | 384 |
| Attention heads | 12 |
| Attention KV heads | 4 |
Table 2: Hyperparameters for the PicoLM models used in our experiments.
| Coverage | Sentence length | Morph. complexity | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Lang. | Order | simil. | Subw./sent. | Words/sent. | Fertility | Word len. | Types | |||
| fr | Fixed | 52.7 | 49.0 | 0.974 | 61.0 | 28.1 | 24.4 | 1.15 | 6.02 | 96 727 |
| pt | 51.0 | 47.9 | 0.979 | 60.6 | 28.1 | 24.2 | 1.16 | 6.03 | 108 442 | |
| en | 55.4 | 52.6 | 0.976 | 62.6 | 26.4 | 23.8 | 1.11 | 5.70 | 70 536 | |
| sv | 52.6 | 47.5 | 0.971 | 60.1 | 24.4 | 20.4 | 1.20 | 6.22 | 177 002 | |
| da | 54.2 | 49.6 | 0.971 | 60.8 | 25.7 | 21.7 | 1.19 | 6.09 | 179 915 | |
Table 3: Vocabulary statistics for each language: Word and subword coverage at rank 100, coverage similarity and integral; sentence length in subwords and words; morphological complexity measured by fertility, word length in characters, and word types.
Figure 9: Coverage measures: the (a) word coverage integral and (b) word-subword-coverage similarity.
Figure 10: Correlation matrix of the predictors, grouped by coverage (C), sentence length (S), and proxies of morphological complexity (M).
Figure 11: (a) Predictor and (b) response loadings of the vocabulary and morphological complexity component.
More in this topic
Linguistic Typology
Linguistic typology compares languages to identify patterns in their sounds, grammar, meaning, and history.
See all 26 papers