We show how causal interventions in Transformer models provide insights into English syntax by focusing on a long-standing challenge for syntactic theory: syntactic islands. Extraction from coordinated verb phrases is often degraded, yet acceptability varies gradiently with lexical content (e.g., "I know what he hates art and loves" vs. "I know what he looked down and saw"). We show that modern Transformer language models replicate human judgments across this gradient. Using causal interventions that isolate functionally relevant subspaces in Transformer blocks, attention modules, and MLPs, we demonstrate that extraction from coordination islands engages the same filler-gap mechanisms as canonical wh-dependencies, but that these mechanisms are selectively blocked to varying degrees. By projecting a large corpus of unrelated text onto these causally identified subspaces, we derive a novel linguistic hypothesis: the conjunction "and" is represented differently in extractable versus non-extractable constructions, corresponding to expressions encoding relational dependencies versus purely conjunctive uses. These results illustrate how mechanistic interpretability can inform syntax, generating new hypotheses about linguistic representation and processing.
Figures & tables
Figure 1: 1. LM judgments of gradiently acceptable conjuncts correlate with human judgments. 2. These constructions rely on a more general-purpose filler-gap mechanism, blocked in the island case. 3. We identify the relevant blocking subspaces, showing they correlate with model acceptability, and 4. pick out linguistically meaningful structures in a corpus.
Figure 2: Exemplar wh -licensing calculations for extractable (top) and unextractable (bottom) conjuncts respectively.
Figure 3: LM behavioral metrics.
Figure 4: We use DAS to find the mechanisms to process embedded wh -questions before measuring the generalization of said mechanisms to our conjunct stimuli.
Figure 5: We use DAS to find the causal drawbridges responsible for stranding and unstranding coordination islands.
Position
Layer
Δ Odds
∣r∣
‘Extractable’ Examples
‘Unextractable’ Examples
and
6
5.03
0.67
…his life he had crept up and stolen…
…you men, we offer you meat and bread…
…valleys, so that men look up and see…
…the groom’s books and papers…
…I think we’re in for a…
…offers on the best terms for cash or approved…
…it in motion, she started up and said…
…chief took a spoonful of meat and liquor…
…ieved in and preached what he had just heard…
…as pigs, fowl, dogs , cattle…
…ourd Wood, we sat down to make…
…a. I hab no wife nor child…
Table 1: Exemplar chunks activated near ‘extractable’ and ‘unextractable’ conjuncts in the Transformer Block Output of gpt2 . Full lists of top 75 such chunks at this site are in appendix G .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Correlation between LM mean licensing interaction and human acceptability judgments across embedded wh -questions with verb-phrase conjuncts. Across a broad class of LM families and sizes, we largely find agreement (high pearson r correlation coefficient) between the two measures.
Verb 1
Complement
And
Verb 2
Human Acceptability
gpt2 Mean Wh -Licensing
pythia-410m-ded. Mean Wh -Licensing
Qwen2-0.5b Mean Wh -Licensing
Class
looked
down
and
saw
6.29
6.75
5.68
4.90
Acceptable
went
home
and
got
6.26
4.70
5.58
5.04
Acceptable
woke
up
and
smelled
6.16
5.76
5.85
7.13
Acceptable
came
back
and
talked about
6.17
5.17
5.29
5.89
Acceptable
ran
out
and
got
5.89
2.99
5.48
4.69
Acceptable
woke
up
and
thought of
5.79
6.35
6.86
7.06
Acceptable
Appendix
Table 2: Left Block: The coordinated clause of the coordination-islands used to train DAS on the blocking mechanism. The top items are ‘acceptable’ coordination-islands (strong preference for extraction), and the bottom four ‘unacceptable’ (strong lack of preference for extraction). Right Block : Human Acceptability scores along a Likert scale and mean licensing-scores for each LM we mechanistically evaluate. We use conjuncts which a) broadly demonstrate calibration between human and LM judgments and b) have length-matched tokens.
Figure 7: Absolute correlation between each conjunct’s average position along the learned subspaces (on which an intervention is performed) and human judgments. We see strong correlation between position on subspace and human judgments.
Top ‘Extractable’ Examples
Top ‘Unextractable’ Examples
1
\nhis life he had crept up and stolen
you men, we offer you meat and bread
2
orsyth and Ronald were walking back and forth
partly to discover the places where gold and \r
3
her room she saw him pacing back and forth
the eating of animal food, wine and marriage
4
quilt. Her fingers went in and out
\nyoung men how he threw spears and har
5
\nUneasy wanders to and fro
the groom’s books and papers
6
\r\n\r\nSo far, so clear
the most nutritive of all meat and \r
Appendix
Table 3: Top 75 Activating ‘Extractable’ and ‘Unextractable’ Chunks for GPT2 on the Subspace Learned at the Transformer Block Output of Layer 6. The relevant token is highlighted.
Transformer language models develop attention heads that specialize in syntactic relations, for example, attending from verbs to their subjects. Do these heads play a causal role in the model's syntactic predictions? Evidence for such a role is mixed to negative: even when we force these heads to attend to a distractor noun other than the subject, the model's syntactic predictions are still more consistent with the subject than with the distractor. Interventions on specialized syntax heads had a significantly stronger effect on syntactic predictions than interventions on random heads, but only in one of the two measures we propose.
We present a systematic review of 337 articles evaluating the syntactic abilities of Transformer-based language models (TLMs), reporting on over 3,000 datapoints spanning a wide range of syntactic phenomena, languages, models, and methods. We take the data to collectively show that TLMs encode a non-trivial amount of syntactic knowledge. Behavioral evidence shows strong performance on formal syntactic phenomena, but weaker and more variable performance on phenomena at the syntax-semantics interface. Performance is also consistently lower for languages with less digital support. Probing and mechanistic studies further support the presence of syntactic knowledge in TLMs. Yet, because most work remains observational and methodologically heterogeneous, insight into the detailed computational mechanisms underlying syntactic processing remains limited. At the same time, the literature remains heavily concentrated on English and BERT-like models. We discuss the implications of our results and provide recommendations for future research.
Transformers underlie almost all state-of-the-art language models in computational linguistics, yet their cognitive adequacy as models of human sentence processing remains disputed. In this work, we use a surprisal-based linking mechanism to systematically evaluate eleven autoregressive transformers of varying sizes and architectures on a more comprehensive set of English agreement attraction configurations than prior work. Our experiments yield mixed results: While transformer predictions generally align with human reading time data for prepositional phrase configurations, performance degrades significantly on object-extracted relative clause configurations. In the latter case, predictions also diverge markedly across models, and no model successfully replicates the asymmetric interference patterns observed in humans. We conclude that current transformer models do not explain human morphosyntactic processing, and that evaluations of transformers as cognitive models must adopt rigorous, comprehensive experimental designs to avoid spurious generalizations from isolated syntactic configurations or individual models.