The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

Nora Graichen; Iria de-Dios-Flores; Gemma Boleda

cs.CLJan 9, 2026

The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

Authors: Nora Graichen, Iria de-Dios-Flores, Gemma Boleda

Organizations: Universitat Pompeu Fabra1 · ICREA2

Abstract

We present a systematic review of 337 articles evaluating the syntactic abilities of Transformer-based language models (TLMs), reporting on over 3,000 datapoints spanning a wide range of syntactic phenomena, languages, models, and methods. We take the data to collectively show that TLMs encode a non-trivial amount of syntactic knowledge. Behavioral evidence shows strong performance on formal syntactic phenomena, but weaker and more variable performance on phenomena at the syntax-semantics interface. Performance is also consistently lower for languages with less digital support. Probing and mechanistic studies further support the presence of syntactic knowledge in TLMs. Yet, because most work remains observational and methodologically heterogeneous, insight into the detailed computational mechanisms underlying syntactic processing remains limited. At the same time, the literature remains heavily concentrated on English and BERT-like models. We discuss the implications of our results and provide recommendations for future research.

Explore similar work

Late Transformer Layers Recode Syntax Canonically: Evidence from Greek Scrambling and Cross-Layer Generalisation
Aug 31, 2026Christos Nikolaos Zacharopoulos, Revekka Kyriakoglou, Chara Tsoukala +1Transformer Layers Syntactic Structure
Implicit Representations of Grammaticality in Language Models
May 6, 2026Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu +2Grammaticality Text Corpora
Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs
Sep 1, 2026Mikhail Sonkin, Tanja Baeumel, Daniil Gurgurov +2Multilingual Large Language Models Machine Translation

Aug 31, 2026 · cs.CLJ/K move · Enter open · S save

Late Transformer Layers Recode Syntax Canonically: Evidence from Greek Scrambling and Cross-Layer Generalisation

Christos Nikolaos Zacharopoulos, Revekka Kyriakoglou, Chara Tsoukala, Théo Desbordes

Probing studies have established that syntactic information is decodable in early and middle transformer layers, but what happens to that information in later layers remains poorly understood. We apply a cross-layer generalisation analysis to three Greek-tuned large language models evaluated on tightly controlled minimal pairs: object-relative constructions in Modern Greek, where canonical (Subject-Verb-Object; SVO) and non-canonical (Verb-Subject-Object; VSO) orders differ only in within-clause word order, while preserving propositional meaning. When a probe trained on late layers (20-31) is tested on each early layer individually, it produces below-chance transfer (cluster-corrected, p<0.01), classifying 99.3% of non-canonical sentences as canonical. Probe coefficients reverse sign around layer 22, indicating a directional recoding toward the canonical form rather than simple information loss. These findings characterise a representational format change in late transformer layers that goes beyond the well-established decline in syntactic decodability, and they generate a directly testable prediction for human EEG and MEG decoding studies using the same stimuli. Code and stimuli are publicly available on OSF.

Transformer Layers Syntactic Structure Ancient Greek Cross-Layer Interactions Pronoun Electroencephalography Spatial Super-Resolution

Open paper Ask about it PDF

Abstract

Explore similar work

Late Transformer Layers Recode Syntax Canonically: Evidence from Greek Scrambling and Cross-Layer Generalisation

Implicit Representations of Grammaticality in Language Models

Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs