Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects

Titus von der Malsburg; Sebastian Padó

cs.CLMar 17, 2026

Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects

Authors: Titus von der Malsburg, Sebastian Padó

Organizations: University of Stuttgart, Germany

Abstract

Transformers underlie almost all state-of-the-art language models in computational linguistics, yet their cognitive adequacy as models of human sentence processing remains disputed. In this work, we use a surprisal-based linking mechanism to systematically evaluate eleven autoregressive transformers of varying sizes and architectures on a more comprehensive set of English agreement attraction configurations than prior work. Our experiments yield mixed results: While transformer predictions generally align with human reading time data for prepositional phrase configurations, performance degrades significantly on object-extracted relative clause configurations. In the latter case, predictions also diverge markedly across models, and no model successfully replicates the asymmetric interference patterns observed in humans. We conclude that current transformer models do not explain human morphosyntactic processing, and that evaluations of transformers as cognitive models must adopt rigorous, comprehensive experimental designs to avoid spurious generalizations from isolated syntactic configurations or individual models.

Figures & tables

	Subject
	Number	Grammatical?	Example
(a/b)	singular	grammatical	The slogan on the poster(s) unsurprisingly was designed …
(c/d)	singular	ungrammatical	*The slogan on the poster(s) unsurprisingly were designed …
(e/f)	plural	grammatical	The slogans on the poster(s) unsurprisingly was designed …
(g/h)	plural	ungrammatical	*The slogans on the poster(s) unsurprisingly were designed …

Table 1: The eight conditions of Experiment 1: agreement attraction in prepositional modifier constructions. Attraction is theoretically possible when the number of the distractor (here “poster(s)”) differs from subject number. In humans, attraction is observed only in (c/d).

Figure 1: Surprisal predictions across models and conditions for Experiment 1. Top left panel: Schematic human performance. Each sub-panel shows the effect of attraction in one of the four relevant configurations (Table 1 ): Singular subjects (sg) in sub-panels 1–2; plural (pl) in sub-panels 3–4. Geometric means, confidence intervals 95%.

		Reference		GPT			Bloom			XGLM			Gemma-3
Subject	Gramm.	Human	Theor. poss.	120M	2 (127M)	2 (744M)	560M	1.7B	3B	564M	1.7B	2.9B	1B	4B
singular	gramm.		$\nearrow$	$\searrow$						$\searrow$
singular	ungramm.	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$
plural	gramm.		$\nearrow$
plural	ungramm.		$\searrow$	$\searrow$	$\searrow$			$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$

Table 2: Experiment 1: Observed and predicted agreement attraction effects for prepositional modifiers. Gramm. : (un)grammatical condition. Arrows indicate significant effects: upward = inhibitory, downward = facilitatory. Theor. poss. : effects that are explainable in terms of agreement attraction.

	RC subject number	Grammatical?	Example
(a/b)	singular	grammatical	The marine(s) who the officer wants to promote …
(c/d)	singular	ungrammatical	*The marine(s) who the officer want to promote …
(e/f)	plural	grammatical	The marine(s) who the officers want to promote …
(g/h)	plural	ungrammatical	*The marine(s) who the officers wants to promote …

Table 3: The eight conditions of Experiment 2: agreement attraction in object-extracted relative clause constructions. Attraction is theoretically possible when the number of the distractor (here “marine(s)”) differs from subject number. In humans, attraction is observed only in (c/d).

Figure 2: Surprisal predictions across models and conditions for Experiment 2. Top left panel: Schematic human performance. Sub-panels show attraction effects in one of the four relevant configurations (able 3 ): Singular RC-subjects (sg) in sub-panels 1–2; plural (pl) in sub-panels 3–4. Geometric means, confidence intervals 95%.

		Reference		GPT			Bloom			XGLM			Gemma-3
RC subject	Gramm.	Human	Theor. poss.	120M	2 (127M)	2 (744M)	560M	1.7B	3B	564M	1.7B	2.9B	1B	4B
singular	gramm.		$\nearrow$				$\searrow$			$\searrow$			$\nearrow$	$\nearrow$
singular	ungramm.	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$				$\searrow$	$\searrow$
plural	gramm.		$\nearrow$	$\searrow$	$\searrow$	$\searrow$		$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$	$\searrow$
plural	ungramm.		$\searrow$	$\searrow$	$\searrow$		$\searrow$	$\searrow$		$\searrow$	$\searrow$	$\searrow$	$\nearrow$

Table 4: Experiment 2: Observed and predicted agreement attraction effects for relative clauses. Gramm. : (un)grammatical condition. Arrows indicate significant effects: upward = inhibitory, downward = facilitatory. Theor. poss. : effects that are explainable by agreement attraction. NB. All predicted effects in the plural/grammatical configuration are in the direction that cannot be explained.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Fixed effects	Estimate	Std. Error	$t$ value
(Intercept)	5.51182	0.35612	15.478
pl	0.04180	0.07201	0.580
attr	-0.11356	0.07201	-1.577
pl $\times$ attr	0.08413	0.10184	0.826

Appendix

Table 5: Regression results for prepositional modifier construction, grammatical sentences. Observations: 1056. Random effects: by-item and by-model random intercepts (item: 24 levels; model: 11 levels); random slopes were omitted because models with a maximal random-effects structure did not reliably converge.

Fixed effects	Estimate	Std. Error	$t$ value
(Intercept)	12.7320	0.4395	28.97
pl	-1.5696	0.1036	-15.14
attr	-2.3755	0.1036	-22.92
pl $\times$ attr	2.0008	0.1466	13.65

Appendix

Table 6: Regression results for prepositional modifier construction, ungrammatical sentences. Observations: 1056. Random effects: by-item and by-model random intercepts (item: 24 levels; model: 11 levels); random slopes were omitted because models with a maximal random-effects structure did not reliably converge.

Fixed effects	Estimate	Std. Error	$t$ value
(Intercept)	11.8113	0.4490	26.306
pl	-0.8533	0.1128	-7.563
attr	0.1168	0.1128	1.035
pl $\times$ attr	-0.9307	0.1596	-5.833

Appendix

Table 7: Regression results for object-extracted relative clauses, grammatical sentences. Observations: 2112. Random effects: by-item and by-model random intercepts (item: 48 levels; model: 11 levels); random slopes were omitted because models with a maximal random-effects structure did not reliably converge.

Fixed effects	Estimate	Std. Error	$t$ value
(Intercept)	15.4838	0.4439	34.880
pl	2.7572	0.1388	19.859
attr	-0.9144	0.1388	-6.586
pl $\times$ attr	0.3678	0.1963	1.873

Appendix

Table 8: Regression results for object-extracted relative clauses, ungrammatical sentences. Observations: 2112. Random effects: by-item and by-model random intercepts (item: 48 levels; model: 11 levels); random slopes were omitted because models with a maximal random-effects structure did not reliably converge.

Fixed effects	Estimate	Std. Error	$t$ value
(Intercept)	5.5118	0.3783	14.570
ungram	7.2202	0.1076	67.082
attr	-0.1136	0.1076	-1.055
ungram $\times$ attr	-2.2620	0.1522	-14.860

Appendix

Table 9: Regression results for prepositional modifier construction, singular subject. Observations: 1056. Random effects: by-item and by-model random intercepts (item: 24 levels; model: 11 levels); random slopes were omitted because models with a maximal random-effects structure did not reliably converge.

Fixed effects	Estimate	Std. Error	$t$ value
(Intercept)	5.55362	0.36811	15.087
ungram	5.60875	0.08834	63.491
attr	-0.02943	0.08834	-0.333
ungram $\times$ attr	-0.34526	0.12493	-2.764

Appendix

Table 10: Regression results for prepositional modifier construction, plural subject. Observations: 1056. Random effects: by-item and by-model random intercepts (item: 24 levels; model: 11 levels); random slopes were omitted because models with a maximal random-effects structure did not reliably converge.

Fixed effects	Estimate	Std. Error	$t$ value
(Intercept)	11.8113	0.4759	24.818
ungram	3.6725	0.1384	26.531
attr	0.1168	0.1384	0.844
ungram $\times$ attr	-1.0312	0.1958	-5.268

Appendix

Table 11: Regression results for object-extracted relative clauses, singular subject. Observations: 2112. Random effects: by-item and by-model random intercepts (item: 48 levels; model: 11 levels); random slopes were omitted because models with a maximal random-effects structure did not reliably converge.

Fixed effects	Estimate	Std. Error	$t$ value
(Intercept)	10.9580	0.3971	27.593
ungram	7.2830	0.1140	63.869
attr	-0.8139	0.1140	-7.137
ungram $\times$ attr	0.2673	0.1613	1.657

Appendix

Table 12: Regression results for object-extracted relative clauses, plural subject. Observations: 2112. Random effects: by-item and by-model random intercepts (item: 48 levels; model: 11 levels); random slopes were omitted because models with a maximal random-effects structure did not reliably converge.

Explore similar work

Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?
Aug 24, 2026Lanni Bu, Xiulin Yang, Christian Clark +2Transformer Attention Transformer
Georgetown University · The Ohio State University · UC San Diego
Energy-Based Transformers as Predictors of Reading Difficulty
Jun 22, 2026Jakub Dotlacil, Ece TakmazEnergy-Based Models Computational Psycholinguistics
Utrecht University
The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models
Jan 9, 2026Nora Graichen, Iria de-Dios-Flores, Gemma BoledaTransformer Interpretability LLM Evaluation
Universitat Pompeu Fabra1 · ICREA2

Aug 24, 2026J/K move · Enter open · S save

Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?

Lanni Bu, Xiulin Yang, Christian Clark, Alex Warstadt+1

Georgetown University · The Ohio State University · UC San Diego

Transformer Attention Transformer Computational Psycholinguistics

Open paper AskPDF arXiv

Transformer-based language models are widely used as models of human language processing, yet their attention mechanisms allow lossless access to the full preceding context, unlike the limited memory systems of humans. We hypothesize that installing memory constraints into transformers' attention mechanisms can improve their fit to human behavioral data. While previous work has explored individual constraints in isolation, we conduct a systematic comparison of multiple attention-based memory mechanisms across different model sizes and training corpora, evaluating both psychometric predictive power for human reading times and grammatical competence. We additionally compare static constraints, in which the constraint strength is fixed throughout training, to dynamic memory curricula. We find that constraints that are sensitive to the content of intervening tokens consistently achieve the highest alignment with human reading times, outperforming distance-based constraints. We observe a dissociation between psychometric fit and grammatical competence under dynamic memory curricula, suggesting that Transformers cannot serve as a one-size-fits-all cognitive model.

Abstract

Explore similar work

Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?

Energy-Based Transformers as Predictors of Reading Difficulty

The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models