cs.CLOct 1, 2026

Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languages

Authors: Nadine El-Naggar, Tatsuki Kuribayashi, Ted Briscoe

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Tohoku University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Some properties of languages, e.g., subject-object-verb (SOV) word order, are more prevalent than others among the thousands of attested natural languages (NLs). Such typological commonality is often attributed to learning biases. Computational simulations, recently with language models (LMs), have facilitated the exploration of this theory. In this paper, we extend existing analyses of the relationship between LMs' learning biases and typological commonality on both data and model sides, focusing on: (i) cross-serial dependencies, the upper limit of attested syntactic complexity, and (ii) stack-based LMs (SLMs), potentially facilitating learning of hierarchical patterns. We first evaluate generalization of SLMs on cross-serial dependencies across diverse artificial languages and confirm that they struggle with such constructions. However, SLMs with limited working memory generalize better suggesting a possible basis for such inductive bias and thus the typological commonality of some word order configurations.

Figures & tables

GCG Lexical Syntactic Category El-Naggar et al. (2025a) ; El-Naggar et al. (2025b)Example
Noun Phrase (NP) – NPTom ga chased Jerry o
Subject Marker – NP SUBJ{}_{\text{SUBJ}} \\backslash NPTom ga chased Jerry o
Object Marker – NP OBJ{}_{\text{OBJ}} \\backslash NPTom ga chased Jerry o
Transitive Verb (VT) – (S ∣1\mid_{1} NP SUBJ{}_{\text{SUBJ}} ) ∣1\mid_{1} NP OBJ{}_{\text{OBJ}}Tom ga chased Jerry o
Intransitive Verb (VI) – S ∣1\mid_{1} NP SUBJ{}_{\text{SUBJ}}grey cat ga fell
Verb with Complement (VCOMP) – (S ∣1\mid_{1} NP SUBJ{}_{\text{SUBJ}} ) ∣1\mid_{1} SCOMPKim ga believed that Sandy o lied
Table 1: Lexical syntactic categories, their derivations, and their examples, where a word corresponding to the category is in bold. The top half shows the categories defined by El-Naggar et al. (2025a) ; El-Naggar et al. (2025b) . The bottom half shows the new categories that we added to extend the grammar. The vertical bars “ ∣\mid ” in the GCG lexical syntactic categories represent either forward or backward slashes, and those with the same index are controlled by the same word order parameter (see Table 2 ).
Digit01
1 (Base)SOVVOS
2 (COMP)Preposed complementizerPostposed complementizer
3 (PP)PostpositionPreposition
4 (ADJ)Prenominal adjectivePostnominal adjective
5 (REL)Preposed relativizerPostposed relativizer
Table 2: Word order parameters and the constructions introduced by their assignment (0/1). The exact implementation is shown in Table 1 .
Figure 1: Partial dependency representations of CSDs with embedded relative clauses. The examples are for SOV with prenominal relative clauses ( 0XXX0 ); the pattern will be, for example, fully mirrored for VOS with postnominal relative clauses ( 1XXX1 ). Colored arcs indicate cross-serial noun–verb dependencies. Black arcs indicate relative-clause attachment. Relative clauses are bracketed.
Figure 2: PPL distribution on the Medium test set (average ±\pm standard deviation). The bar shows the standard deviation among five runs with different random seeds. Note that all training was conducted and stopped under the same early-stopping setting, and the high variance indicates training instability across different random seeds (see section 6 ).
ModelShortMediumLongRelConj-1Conj-2NestedAvg.
RNN33.40 ±\pm 0.14100.69 ±\pm 13.91345.68 ±\pm 510.02376.50 ±\pm 223.62552.94 ±\pm 483.43731.62 ±\pm 1033.28538.97 ±\pm 574.27382.83
RNN SUP{}_{\text{SUP}}33.32 ±\pm 0.1888.29 ±\pm 16.40249.02 ±\pm 75.80299.37 ±\pm 97.15404.95 ±\pm 126.49485.74 ±\pm 160.56437.99 ±\pm 171.39285.53
RNN ND{}_{\text{ND}}33.43 ±\pm 0.1394.98 ±\pm 16.35333.45 ±\pm 134.35328.11 ±\pm 101.49497.27 ±\pm 165.78615.10 ±\pm 241.20504.19 ±\pm 245.97343.79
LSTM33.49 ±\pm 0.2272.72 ±\pm 13.04313.71 ±\pm 86.15277.15 ±\pm 100.83501.66 ±\pm 208.33688.17 ±\pm 305.38522.85 ±\pm 257.08344.25
LSTM SUP{}_{\text{SUP}}33.49 ±\pm 0.2073.80 ±\pm 11.75415.24 ±\pm 190.50364.23 ±\pm 154.01661.19 ±\pm 298.721057.02 ±\pm 601.37772.05 ±\pm 377.53482.43
LSTM ND{}_{\text{ND}}34.52 ±\pm 7.5188.03 ±\pm 24.14516.53 ±\pm 226.21422.71 ±\pm 188.28773.97 ±\pm 388.831159.86 ±\pm 664.59882.36 ±\pm 526.66554.00
Table 3: PPL for each model and evaluation data (average ±\pm standard deviation). The standard deviation here indicates how the average PPL varies among 32 languages. PPL values obtained from SLMs, which are better than those of the corresponding base model, are shown in bold . The best PPLs among the LMs with the same base architecture are shown with an underline .
ModelShortMed.LongRelConj1Conj2Nest
RNN0.57−- 0.66 †−- 0.57 †−- 0.31 †−- 0.48 †−- 0.59 †−- 0.25
RNN SUP{}_{\text{SUP}}−- 0.24−- 0.58 †−- 0.43 †−- 0.60 †−- 0.62 †−- 0.45 †−- 0.39 †
RNN ND{}_{\text{ND}}0.36−- 0.40 †−- 0.24−- 0.61 †−- 0.26−- 0.12−- 0.36 †
LSTM0.43−- 0.56 †−- 0.69 †−- 0.57 †−- 0.26−- 0.28−- 0.38 †
LSTM SUP{}_{\text{SUP}}0.54−- 0.60 †−- 0.66 †−- 0.49 †−- 0.48 †−- 0.56 †−- 0.55 †
LSTM ND{}_{\text{ND}}0.59−- 0.250.17−- 0.240.100.200.12
Table 4: Spearman Correlation to Word Order Frequency by Model and Test/Evaluation Type. †\dagger indicates a significantly negative correlation (p < 0.05).
FeatureCoefficientSignificance
β0\beta_{0}0.0371 
βF\beta_{F}−- 0.0651
βM(LSTM)\beta_{M(\text{LSTM})}−- 0.5231
βM(TF)\beta_{M(\text{TF})}−- 0.0628 
βS(Sup)\beta_{S(\text{Sup})}−- 0.0213 
βS(Nd)\beta_{S(\text{Nd})}−- 0.0790 
Table 5: Fixed effects from the regression model predicting perplexity. The word order digit effects ( βd\bm{\beta}_{d} , βd:M\bm{\beta}_{d:M} , βd:S\bm{\beta}_{d:S} ) are excluded from the Table. Coefficients for each parameter βeffect\beta_{\text{effect}} are shown. Significance markers denote: *** p<0.001, ** p<0.01, * p<0.05.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

ArtifactLicenseUsage
NLTK Bird et al. (2009)Apache License 2.0to create ALs and targeted evaluation data
Docker Merkel and others (2014)Apache 2.0to train SLMs
El-Naggar et al. (2025b) DatasetsCreative Commons CC-BY 4.0training and testing SLMs
Codes from DuSell and Cotterell (2025) ( https://github.com/bdusell/bearing-syntactic-fruit/commit/ebe4ee8 )MIT Licenseto implement models
Rau ( https://github.com/bdusell/rau )MIT Licenceto implement models
WALS Dryer and Haspelmath (2013)Creative Commons CC-BY 4.0to find word order statistics in NLs
Appendix
Table 6: Artifacts used in this paper.
S82A Order of Subject and Verb Dryer (2013e)
VP83A Order of Object and Verb Dryer (2013c)
O81A Order of Subject, Object and Verb Dryer (2013f)
COMPFeature GB421: Is there a preposed complementizer in complements of verbs of thinking and/or knowing? Skirgård et al. (2023)
Feature GB422: Is there a postposed complementizer in complements of verbs of thinking and/or knowing? Skirgård et al. (2023)
PP85A Order of Adposition and Noun Phrase Dryer (2013b)
ADJ87A Order of Adjective and Noun Dryer (2013a)
Appendix
Table 7: WALS and Grambank chapters used in this paper.
ModelShortMediumLongRelConj-1Conj-2NestedAvg.
RNN33.40 ±\pm 0.14100.69 ±\pm 13.91345.68 ±\pm 510.02376.50 ±\pm 223.62552.94 ±\pm 483.43731.62 ±\pm 1033.28538.97 ±\pm 574.27382.83
RNN SUP{}_{\text{SUP}} (w/o SC)33.44 ±\pm 0.18100.43 ±\pm 13.04292.64 ±\pm 88.90352.21 ±\pm 127.53506.62 ±\pm 189.70623.84 ±\pm 258.64533.75 ±\pm 249.39348.99
RNN ND{}_{\text{ND}} (w/o SC)33.45 ±\pm 0.14100.40 ±\pm 12.66271.09 ±\pm 86.73324.98 ±\pm 89.09428.46 ±\pm 117.36492.40 ±\pm 150.42447.41 ±\pm 147.36299.74
LSTM33.49 ±\pm 0.2272.72 ±\pm 13.04313.71 ±\pm 86.15277.15 ±\pm 100.83501.66 ±\pm 208.33688.17 ±\pm 305.38522.85 ±\pm 257.08344.25
LSTM SUP{}_{\text{SUP}} (w/o SC)33.45 ±\pm 0.1875.78 ±\pm 12.06374.29 ±\pm 133.49301.13 ±\pm 112.04574.42 ±\pm 283.83850.53 ±\pm 507.61615.52 ±\pm 351.33403.59
LSTM ND{}_{\text{ND}} (w/o SC)33.48 ±\pm 0.2179.94 ±\pm 14.51359.74 ±\pm 105.87307.96 ±\pm 106.65537.33 ±\pm 202.30753.00 ±\pm 314.25581.48 ±\pm 258.67378.99
Appendix
Table 8: PPL for each dataset and evaluation data (average ±\pm standard deviation). The standard deviation here indicates how the average PPL varies among 32 languages. “SC” denotes the shortcut connection. PPL values obtained from SLMs, which are better than those of the corresponding base model, are shown in bold . The best PPLs among the LMs with the same base architecture are shown with an underline .
ModelShortMediumLongRelConj-1Conj-2Nested
RNN0.57−- 0.66 †−- 0.57 †−- 0.31 †−- 0.48 †−- 0.59 †−- 0.25
RNN SUP{}_{\text{SUP}} (w/o shortcut)0.520.04−- 0.070.150.300.390.29
RNN ND{}_{\text{ND}} (w/o shortcut)0.360.18−- 0.31 †−- 0.050.030.220.20
LSTM0.43−- 0.56 †−- 0.69 †−- 0.57 †−- 0.26−- 0.28−- 0.38 †
LSTM SUP{}_{\text{SUP}} (w/o shortcut)0.55−- 0.65 †−- 0.56 †−- 0.68 †−- 0.38 †−- 0.34 †−- 0.48 †
LSTM ND{}_{\text{ND}} (w/o shortcut)0.61−- 0.53 †−- 0.58 †−- 0.67 †−- 0.39 †−- 0.33 †−- 0.44 †
Appendix
Table 9: Spearman Correlation to Word Order Frequency by Model and Test/Evaluation Type. †\dagger indicates significantly negative correlation (p < 0.05).
ModelShortMediumLongRelConj-1Conj-2Nested
RNN0.0030.0890.4420.2580.3790.4890.325
RNN SUP (w/o shortcut)0.0030.0890.2160.1530.1760.2030.172
RNN SUP0.0040.0880.1880.1840.1970.2170.205
RNN ND (w/o shortcut)0.0030.0890.1900.1720.1910.2200.173
RNN ND0.0030.1140.2760.2390.2780.3180.316
LSTM0.0040.0890.1780.2310.2490.2760.281
Appendix
Table 10: Coefficient of variations of PPLs among five different random seeds for each model and evaluation data. The value is averaged over 32 languages in each setting.

Explore similar work

CardsList

  1. What Kind of Language is Easy to Language-Model Under Curriculum Learning?

    Apr 29, 2026Nadine El-Naggar, Tatsuki Kuribayashi, Ted BriscoeLanguage AcquisitionTypology

    Mohamed bin Zayed University of Artificial Intelligence

  2. Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models

    Date pendingVarvara Arzt, Allan Hanbury, Terra BlevinsLanguage ModelingLanguage Pairs

    Faculty of Informatics, TU Wien · Khoury College of Computer Sciences, Northeastern University

  3. Language Models Generalize to Human-like Word Order Preferences

    Aug 5, 2026Amanda Popadich, Shane Steinert-ThrelkeldLanguage ModelingLarge Language Model Bias

    Department of Linguistics, University of Washington