Syntactic Generalization in Language Models

Follow this topic Ask about this topic

Latest papers

27 papers7 papers in the last four weeks, up 75% on the four weeks before. 0.0% of all new papers.

Encoding and Propagation of Grammatical Number in Large Language Models
Nov 5, 2026Jacob Solinsky, Serguei PakhomovLLM Interpretability Syntactic Generalization in Language Models
University of Minnesota
Language Models Keep Abandoned Analyses: Residue and Adaptation after Garden-Path Sentences
Oct 27, 2026Xiang DingSyntactic Generalization in Language Models
Nagoya University
Specialization, Diversity, and Scaling in Child-Scale Language Models
Oct 27, 2026Steven Y. Feng, Michael C. FrankSmall Language Models LLM Training
Stanford University
Examining form-based linguistic productivity in people and large language models
Oct 24, 2026Michael Wilson, Robert FrankSyntactic Generalization in Language Models Computational Psycholinguistics
Department of English, University of Nevada, Reno, 1664 N. Virginia Street, Reno, NV, 89557, USA. · Department of Linguistics, Yale University, 37 Hillhouse Avenue, New Haven, CT, 06511, USA.
The Emergence and Sparsification of a Syntactic Circuit
Oct 23, 2026Qing Yao, Sasha Boguraev, Tiago Pimentel +1Circuits in Language Models Syntactic Generalization in Language Models
The University of Texas at Austin · ETH Zurich
Once a Tree, Always a Tree? Cross-lingual Transfer of Hierarchical Generalization in Language Models
Oct 22, 2026Shin Yamaguchi, Masaya Taniguchi, Tatsuki Kuribayashi +2Cross-Lingual Transfer Language Model Generalization
NAIST · RIKEN · MBZUAI +2
From Parallels to Divergence: Language Models' Learning of Action Nominal Constructions
Oct 15, 2026Iskar Deng, Shane Steinert-ThrelkeldSyntactic Generalization in Language Models Language Modeling
University of Washington
A Scaled-Up Empirical Study of Syntactic Language Models
Oct 12, 2026Pengcheng Wang, Zhexuan Zhang, Chao Lou +1Syntactic Generalization in Language Models Language Modeling
School of Information Science and Technology, ShanghaiTech University · Shanghai Engineering Research Center of Intelligent Vision and Imaging · Independent researcher
Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languages
Oct 1, 2026Nadine El-Naggar, Tatsuki Kuribayashi, Ted BriscoeSyntactic Generalization in Language Models Language Modeling
Mohamed bin Zayed University of Artificial Intelligence · Tohoku University
Generalization and Memorization along the Learning Trajectory of Neural Language Models: A Geometric Account of Categorization
Sep 26, 2026Wang Bojun, Holly Jenkins, Elizabeth WonnacottMemorization in Language Models Language Modeling
Generalization through Lexical Abstraction in Transformer Models: The Case of Functional Words
Sep 17, 2026Giuseppe Samo, Vivi Nastase, Paola MerloTransformer Interpretability Word Embeddings
Idiap Research Institute, Martigny, Switzerland · University of Geneva, Swizerland
FreqBLiMP: Frequency-Controlled Minimal Pairs Reveal Robustness and Fragility of LLMs Under Lexical Rarity
Sep 7, 2026Tyrone White, Yuki AraseLLM Evaluation Syntactic Generalization in Language Models
School of Computing, Institute of Science Tokyo Tokyo, Japan
Disentangling Statistical Preemption from Entrenchment in Language Models' Avoidance of Overgeneralization
Sep 1, 2026Yixuan Wang, Freda Shi, Kanishka MisraSyntactic Generalization in Language Models Language Modeling
University of Waterloo Vector Institute · Department of Linguistics University of Texas at Austin
Does Episodic Memory Help Close the Lexical Frequency Gap in Sensitivity to Syntactic Contrasts? A Test Using Retrieval-Augmented Language Models
Aug 24, 2026Jing Liu, Najoung KimMemory-Augmented Language Models Syntactic Generalization in Language Models
CoML Team, LSCP ENS, Université PSL, EHESS, CNRS · Department of Linguistics Boston University
Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers
Aug 6, 2026Haris Riaz, Hyungji Kim, Mihai SurdeanuLanguage Model Pretraining Positional Encoding
Department of Computer Science University of Arizona Tucson, AZ, USA
Language Models Generalize to Human-like Word Order Preferences
Aug 5, 2026Amanda Popadich, Shane Steinert-ThrelkeldSyntactic Generalization in Language Models Language Modeling
Department of Linguistics, University of Washington
Exposure is Optional: Learning Unlike Coordination in Language Models
Jul 22, 2026Jiamu Luo, Shane Steinert-ThrelkeldSyntactic Generalization in Language Models Language Modeling
Department of Linguistics, University of Washington, Seattle, WA 98195, USA
Linear representations of grammaticality in neural language models
Jul 16, 2026Jane Li, Najoung KimSyntactic Generalization in Language Models Language Modeling
Department of Cognitive Science Johns Hopkins University Baltimore, MD · Department of Linguistics Boston University Boston, MA
A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
May 29, 2026Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande +1Language Model Pretraining Neural Network Generalization
University of Massachusetts Lowell
Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions
May 29, 2026Wesley Scivetti, Ethan Wilcox, Nathan Schneider +2LLM Evaluation Syntactic Generalization in Language Models
Georgetown University · The University of Texas at Austin · Leipzig University & ScaDS.AI Dresden/Leipzig
Measuring Form and Function in Language Models
May 27, 2026Héctor Javier Vázquez Martínez, Charles YangLLM Evaluation Syntactic Generalization in Language Models
University of Pennsylvania, Department of Linguistics and Computer and Information Science
Self-Training Doesn't Flatten Language -- It Restructures It: Surface Markers Amplify While Deep Syntax Dies
May 20, 2026Ming LiuSelf-Training for Language Models Syntactic Generalization in Language Models
Amazon
Filling in the Mechanisms: How do LMs Learn Filler-Gap Dependencies under Developmental Constraints?
Apr 15, 2026Atrey Desai, Sathvik NairSyntactic Generalization in Language Models Language Modeling
University of Maryland
Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italian
Mar 26, 2026Giuseppe Samo, Paola MerloSyntactic Generalization in Language Models Synthetic Data Evaluation
IDIAP Research Institute · University of Geneva
Current LLMs still cannot 'talk much' about grammar modules: Evidence from syntax
Mar 20, 2026Mohammed Q. Shormani, Yehia A. AlSohbaniSyntactic Generalization in Language Models Machine Translation
Ibb University, Yemen · Arab Open University, Saudia Arabia
The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models
Jan 9, 2026Nora Graichen, Iria de-Dios-Flores, Gemma BoledaTransformer Interpretability LLM Evaluation
Universitat Pompeu Fabra1 · ICREA2
Large Language Model probabilities cannot reliably distinguish between possible and impossible language
Sep 18, 2025Evelina Leivada, Raquel Montero, Paolo Morosi +5Syntactic Generalization in Language Models LLM Evaluation
Departament de Filologia Catalana, Centre de Lingüística Teòrica, Universitat Autònoma de Barcelona, Barcelona 08193, Spain · Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, 08010, Spain · Similarweb, Spain +1

Nov 5, 2026J/K move · Enter open · S save

Encoding and Propagation of Grammatical Number in Large Language Models

Jacob Solinsky, Serguei Pakhomov

University of Minnesota

LLM Interpretability Syntactic Generalization in Language Models

Open paper AskPDF arXiv

Sparse autoencoders (SAEs) have become an increasingly popular means of translating Large Language Model (LLM) hidden state vectors into a human-comprehensible list of features or symbols. As such, SAEs may provide insight into an LLM's knowledge representation and internal

mental'' grammar - topics of pragmatic interest for mechanistic interpretation of AI behavior and of theoretical interest for the study of in silico representations of human language. Language models with fewer than 100M parameters, trained on small corpora, nonetheless exhibit strong grammatical competence and discourse coherence, despite producing content of limited reliability. We use SAEs that we trained on one such model (Pythia-70M) to conduct an in-depth examination of how a single grammatical phenomenon, number agreement, is represented and processed in an LLM. We demonstrate the mechanism and the methodology by which SAEs can be used to find and propagate

plural noun'', plural matrix subject'', and generate a plural verb'' features which cause the LLM to produce plural verb forms such as are'', do'', or have'' instead of is'', does'', and has'' in response to prompts such as

the men in the park''. We show that intervening to remove these features from the hidden states of plural subject noun tokens causes the model to produce singular verbs instead, and vice versa. We also show that corresponding

singular noun'' and ``generate a singular verb'' features are not as effective for this task. This asymmetry suggests that Pythia-70m makes use of a privative representation of grammatical number in English. These findings replicated with larger LLMs Gemma-2-2b and Llama-3.1-8b yielded similar results. Our main contribution is a novel approach to the use of SAEs to understanding how LLMs encode human language that addresses a number of technical and methodological challenges.