Recent work has shown that language models (LMs) trained on synthetic corpora can exhibit typological preferences that resemble cross-linguistic regularities in human languages, particularly for syntactic phenomena such as word order. In this paper, we extend this paradigm to differential argument marking (DAM), a semantic licensing system in which morphological marking depends on semantic prominence. Using a controlled synthetic learning method, we train GPT-2 models on 18 corpora implementing distinct DAM systems and evaluate their generalization using minimal pairs. Our results reveal a dissociation between two typological dimensions of DAM. Models reliably exhibit human-like preferences for natural markedness direction, favoring systems in which overt marking targets semantically atypical arguments. In contrast, models do not reproduce the strong object preference in human languages, in which overt marking in DAM more often targets objects rather than subjects. These findings suggest that different typological tendencies may arise from distinct underlying sources.
Figures & tables
Figure 1: Overview of the DAM rule injection and the overall results for markedness and argument preference averaged over all DAM rule mastery accuracies.
Local: Hebrew (definiteness-based)
Global: Malimasa (animacy-based)
(1)
dani pagaš et ha-student.
(2)
\textipa th \textturnm 33 n \textturnm 21 \textipa N \textscripta 33 g \textipa 7 45 xi 45 ga 21 .
Dani meet. 3.sg.m acc the-student
3 sg - agt 1 sg - pat deceive
‘Dani met the student.’
‘He deceived me.’
(3)
dani pagaš student.
(4)
\textipa N \textscripta 33 tha 33 \textipa l \textschwa 33 \textipa lo 21 .
Dani meet. 3.sg.m student
1 sg book read
‘Dani met a student.’
‘I read the book.’
Table 1: Illustrative examples of differential argument marking in human languages.
Table 2: Representative examples of DAM rule injection. See Section 3 for formal definitions of each rule.
Rule
Trigger
Dependency
Direction
Target
SVO%
ALL%
Baseline
—
—
—
—
0.00
0.00
Full
—
—
—
A+P
100.00
4.91
L-P-Ani
Animacy
Local
Natural
P
10.61
0.52
L-P-Ani-inv
Animacy
Local
Inverse
P
89.39
4.39
L-P-Def
Definiteness
Local
Natural
P
30.65
1.50
L-P-Def-inv
Definiteness
Local
Inverse
P
69.35
3.41
Table 3: Overview of experimental conditions and the proportion of sentences perturbed by each injected rule. SVO% indicates the proportion of SVO clauses in the corpus that license overt marking under each rule, and ALL% indicates the proportion of all sentences in the training corpus that are affected by the corresponding rule. (Abbreviations: L=Local, G=Global, P=Object-targeting, A=Subject-targeting, Ani=Animacy, Def=Definiteness, Pro=Pronominality, inv=inverse.)
Figure 2: Rule mastery accuracy over training steps for DAM rules.
Figure 3: Rule mastery accuracy of the best checkpoint across DAM conditions. We select models based on best validation perplexity.
Figure 4: Scatter plot of the perturbation ratio in the training corpus (ALL%) versus best rule mastery accuracy across DAM rules. The dashed line shows the linear regression fit.
Figure 5: Marker placement accuracy of the best checkpoint across DAM conditions. We select models based on best validation perplexity.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Animacy pairs
Definiteness pairs
Pronominality pairs
Pair
Count
Pair
Count
Pair
Count
animate–inanimate
858,545
definite–indefinite
669,965
pronoun–common
701,852
animate–animate
103,571
definite–definite
288,003
common–common
205,098
inanimate–inanimate
62,858
indefinite–indefinite
44,862
pronoun–pronoun
98,329
inanimate–animate
5,783
indefinite–definite
27,927
common–pronoun
25,478
Appendix
Table 4: Distribution of subject–object pairings for three semantic triggers.
Animacy
Definiteness
Pronominality
Category
Subject
Object
Category
Subject
Object
Category
Subject
Object
animate
962,116
109,354
definite
957,968
315,930
pronoun
906,950
230,576
inanimate
68,641
921,403
indefinite
72,789
714,827
common
123,807
800,181
Appendix
Table 5: Marginal distributions for subjects and objects across the three semantic triggers.
Rule
Set
N
Correct
Accuracy
L-P-Def
affected
50
47
94.00%
L-P-Def
unaffected
50
49
98.00%
L-A-Pro-inv
affected
50
48
96.00%
L-A-Pro-inv
unaffected
50
50
100.00%
G-Ani
affected
50
49
98.00%
G-Ani
unaffected
50
50
100.00%
Appendix
Table 6: Results of human inspection for representative DAM rules and the Invalid set.
Hyperparameter
Value
Model
GPT-2-small
Context length
1024
Training steps
15k
Epochs
∼ 8.0--8.2 3 3 3 Models are trained for a fixed number of optimization steps. The effective number of epochs varies slightly across DAM rules due to differences in marker density.
Optimizer
AdamW
Learning rate
3×10−4
Appendix
Table 7: Training hyperparameters shared across all LM runs.
Figure 6: Semantic probing accuracy at the best training checkpoint across DAM conditions, evaluated separately for subject and object representations.
Figure 7: Full BLiMP evaluation results across all DAM rules and the baseline. Column abbreviations: A-Pass = Animate Subject Passive; A-Trans = Animate Subject Transitive; Det-N Agr. = Determiner–Noun Agreement; Exist. Quant. = Existential Quantifiers; Intr. = Intransitive; NPI = Negative Polarity Item Licensing; S–V Agr. = Subject–Verb Agreement; Trans. = Transitive.
Figure 8: BLiMP accuracies across eight sub-tasks. Boxplots show the distribution across DAM rules; black "×" markers denote the baseline model.
Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403