cs.CLSep 24, 2026

Grammatical "grandmother neurons" are rare in LLMs

Authors: Linyang He, Nima Mesgarani

Organizations: Zuckerman Mind Brain Behavior Institute, Columbia University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Understanding how Large Language Models (LLMs) encode linguistic structures remains a fundamental challenge in interpretability research. While diagnostic classifiers (or "probes") are widely used for this task, they face significant methodological criticism: training auxiliary classifiers introduces capacity confounds and calibration issues, often making it difficult to distinguish the model's intrinsic representations from the probe's ability to learn the task. To address these limitations, we introduce a probe-free framework for localizing linguistic selectivity at the individual neuron level. Leveraging the controlled contrasts of linguistic minimal pairs, we propose a Neuron Separability Index (NSI), a metric that directly quantifies how reliably single neurons differentiate grammatical from ungrammatical constructions without parameter updates. Applying NSI across 68 linguistic paradigms and seven checkpoints reveals three main patterns: 1) raw separability reaches near-peak levels earlier for morphological and syntactic distinctions than for syntax-semantics interface and conceptual distinctions. 2) after permutation normalization, single-unit selectivity is sparse, weak, and narrowly tuned: only a small fraction of units are sensitive to an average paradigm, and strongly selective "grandmother neurons" are rare. 3) whole-vector linear separability, single-neuron selectivity, and behavioral competence are largely dissociated, and targeted ablations further separate activation selectivity from causal reliance.

Figures & tables

Figure 1: Minimal-pair neuron separability pipeline. Minimal pairs from BLiMP/COMPS are fed into a transformer LM (here, Qwen3-0.6B). For each minimal pair, we extract last-token activations, then focus on a single neuron and assemble paired activation vectors across items for the grammatical (positive) and ungrammatical (negative) sentences. We compute a raw separability score from the correlation between the two paired vectors. To control for lexical effects, we construct a null distribution by randomly swapping the grammatical/ungrammatical labels within half of the pairs and recomputing separability across many permutations; the observed raw score is then converted into a neuron separability index (NSI) value.
Figure 2: Colors indicate linguistic domains: Concept , Syntax–Semantics Interface , Syntax , and Morphology . Layer-wise average separability curves across linguistic phenomena. Each curve reports mean neuron separability for one phenomenon, averaged across its constituent paradigms.
Figure 3: (a) Proportion of sensitive neurons ( NSI>0\mathrm{NSI}{>}0 ) across layers, broken down by linguistic phenomenon. Lines show the mean proportion across paradigms within each phenomenon; shaded bands indicate variability across paradigms. Overall, neurons with positive sensitivity to the targeted contrasts are sparse throughout the network. Mean recruitment differs across the four domains: Concept ( ≈8.27%\approx 8.27\% ), Syntax–Semantics Interface ( ≈3.31%\approx 3.31\% ), Syntax ( ≈2.50%\approx 2.50\% ), and Morphology ( ≈3.88%\approx 3.88\% ). Sensitivity also tends to decrease in deeper layers, suggesting that later layers allocate relatively less capacity to local grammatical well-formedness and more to higher-level information (e.g., semantics, reasoning, and discourse). The right panel of Appendix Figure 11 provides the paradigm-level distributions. (b) Distribution of neuron breadth across paradigms. For each neuron (a unit at a specific layer), we count the paradigms for which it is responsive ( NSI>0\mathrm{NSI}{>}0 ). Left: histogram of the number of responsive paradigms per neuron. Right: empirical CDF of the same quantity. Vertical lines mark the mean and the 90th/95th percentiles, showing that most responsive neurons participate in only a small number of paradigms and that broad, multi-paradigm responsiveness is rare. Corresponding response distributions at the domain and phenomenon levels appear in Appendix Figures 14 and 14 .
Figure 4: (a) Mean NSI across layers computed over sensitive neurons only (NSI >0{>}0 ), grouped by linguistic phenomenon. Even within this subset, the average NSI remains small, indicating that sensitivity is generally weak rather than sharply selective. (b) For each BLiMP/COMPS paradigm, we compute the maximum NSI across neurons. Bar and point colors indicate each paradigm’s parent phenomenon; the outlined Others bar shows the mean over the remaining 65 paradigms. Only three paradigms contain a neuron with NSI >2{>}2 . Appendix Figure 12 gives paradigm- and neuron-level detail.
Figure 5: Layer-wise average tuning breadth at the paradigm, phenomenon, and domain levels. A neuron is counted as responsive to a phenomenon or domain when it has positive selectivity for at least one constituent paradigm. Shaded regions show standard error.
Figure 6: Functional modularity in tuning across linguistic phenomena. The heatmap displays mean phenomenon-level NSI for neurons responsive to at least three phenomena. Rows are grouped by domain, and columns are hierarchically clustered neurons. The block structure shows that even poly-selective neurons retain dominant within-domain preferences rather than behaving as uniform cross-domain grammar detectors.
Figure 7: Robustness and generalization of neuron-level selectivity. (a) Across four Qwen3 scales and three additional checkpoints, only 2.52.5 – 3.9%3.9\% of units have positive NSI for an average paradigm. (b) The average neuron responds to only 1.71.7 – 2.72.7 of 68 paradigms; diamonds mark the 95th percentile. (c) For Qwen3-0.6B, the fraction of units above threshold falls sharply, and only 3 of 68 paradigms contain any unit from NSI ≥1.645\geq 1.645 through NSI >3>3 . (d) Whole-vector probe accuracy is essentially uncorrelated with minimal-pair behavior across 67 BLiMP paradigms.
ParadigmkkTop ΔM\Delta M [95% CI]Random meanBottom ΔM\Delta Mprandp_{\mathrm{rand}}
Det.–noun agr.5−0.005-0.005 [ −0.014,0.003-0.014,0.003 ]−0.003-0.0030.1470.446
 100.0000.000 [ −0.011,0.011-0.011,0.011 ]−0.009-0.0090.0540.634
 200.1110.111 [ 0.084,0.1380.084,0.138 ]−0.034-0.034−0.022-0.0220.911
Det.–noun agr.+adj.5−0.006-0.006 [ −0.015,0.004-0.015,0.004 ]−0.005-0.0050.1610.545
 10−0.005-0.005 [ −0.020,0.012-0.020,0.012 ]−0.012-0.0120.1770.604
 200.0260.026 [ 0.004,0.0480.004,0.048 ]−0.009-0.0090.3510.733
Table 1: Primary targeted group-ablation results. ΔM\Delta M is the change in mean log⁡p(x+)−log⁡p(x−)\log p(x^{+})-\log p(x^{-}) ; brackets give paired-bootstrap 95% CIs, and prandp_{\mathrm{rand}} compares each top group with 100 same-size random groups.
Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Paradigm-first 90% saturation layers summarized by phenomenon. For each paradigm, we identify the earliest layer at which its median raw-separability score reaches 90% of its own maximum. Bars show the mean of these paradigm-level saturation layers within each phenomenon. The legend reports means computed directly over all constituent paradigms in each domain: 19.0 for Concept, 10.3 for Syntax–Semantics Interface, 9.5 for Syntax, and 8.4 for Morphology.
Figure 9: Correspondence between raw-separability and probing 90% saturation layers across the 68 paradigms. For each paradigm and measure, saturation is the earliest layer reaching 90% of that paradigm’s own maximum. Colors encode phenomena, marker shapes encode domains, the dashed diagonal indicates equal layers, and the solid line is the least-squares fit. The association is weakly positive (Pearson r=0.15r=0.15 ; Spearman ρ=0.23\rho=0.23 ), indicating limited paradigm-level agreement between the two localization measures. Probing embedding index 0 is excluded, and sampled hidden-state indices 2, 4, …, 28 are mapped to model layers 1, 3, …, 27.
ModelLayers ×\times widthNSI >0{>}0 (%)NSI >2{>}2 (%)Paradigms >2>2Mean BBP95/max BBB≥10B{\geq}10 (%)
Qwen3-0.6B28×102428\times 10243.5681.54×10−41.54{\times}10^{-4}32.436/130.18
Qwen3-1.7B28×204828\times 20483.2612.82×10−42.82{\times}10^{-4}62.225/130.09
Qwen3-4B36×256036\times 25602.5241.12×10−41.12{\times}10^{-4}21.725/130.06
Qwen3-8B36×409636\times 40962.5931.92×10−31.92{\times}10^{-3}41.765/150.24
Pythia-410M24×102424\times 10243.9425.39×10−45.39{\times}10^{-4}22.687/160.86
TinyLlama-1.1B22×204822\times 20482.9396.20×10−46.20{\times}10^{-4}22.005/170.35
Appendix
Table 2: Cross-model selectivity and tuning breadth over 68 paradigms. “Paradigms >2>2 ” is the number of paradigms containing any NSI >2{>}2 unit. Breadth BB counts the number of paradigms with NSI >0{>}0 for each neuron.
Figure 10: Cumulative distribution of paradigm-level tuning breadth across models. Even under the lenient NSI >0{>}0 definition, the high-breadth tail is small: the 95th percentile is 5–7 paradigms and fewer than 1% of neurons respond to 10 or more paradigms in every model.
Figure 11: Left : Distributions of non-zero NSI scores (i.e., over sensitive neurons with NSI>0\mathrm{NSI}{>}0 ) aggregated across all layers, shown separately for each paradigm. Across paradigms, the mean NSI is typically around ∼0.5\sim 0.5 , indicating that most sensitive neurons exhibit only weak separability. Right : Paradigm-wise proportion of sensitive neurons across all 68 BLiMP/COMPS paradigms for Qwen3-0.6B.
Figure 12: Paradigm-wise maximum NSI scores across all 68 BLiMP/COMPS paradigms for Qwen3-0.6B. Only three paradigms contain a unit with NSI >2{>}2 ; the bottom panels show that each case is realized by one unit across the network.
Figure 15
ThresholdUnits above (%)Paradigms with anyMean countMax count
03.7010681061.155229
10.09446827.06120
1.6451.54×10−41.54{\times}10^{-4}30.0441
21.54×10−41.54{\times}10^{-4}30.0441
2.3261.54×10−41.54{\times}10^{-4}30.0441
2.581.54×10−41.54{\times}10^{-4}30.0441
Appendix
Table 3: Threshold sweep over 68 paradigms and 28×102428\times 1024 units per paradigm.
ConditionMean rawNSI >0{>}0 (%)NSI >2{>}2 (%)Paradigms >2>2Median max
Matched good–bad0.14383.4696.67×10−46.67{\times}10^{-4}41.128
Random good–good0.5008100.000000.980
Random bad–bad0.5009100.000000.989
Cross-item good–bad0.500416.4640.0454503.325
Appendix
Table 4: Pairing controls over 68 paradigms.
SubsetConditionParadigmsMean rawMedian raw
One-prefixOriginal100.07530.0757
 Critical deleted100.00110.0007
 Random deleted100.07460.0772
Two-prefixOriginal60.10070.1069
 Shared word deleted60.18020.1500
 Random deleted60.09710.0933
Appendix
Table 5: Critical-token deletion controls. We emphasize raw separability because deletion changes the geometry and variance of the permutation null.
ComparisonnnPearson rrSpearman ρ\rho
Behavior vs. layer-14 probe67-0.0320.023
Behavior vs. peak probe67-0.0300.008
Behavior vs. mean NSI67-0.021-0.022
Behavior vs. maximum NSI670.1690.177
Layer-14 probe vs. mean NSI67-0.557-0.578
Layer-14 probe vs. maximum NSI670.092-0.178
Appendix
Table 6: Correlations across 67 BLiMP paradigms.
Figure 15: Additional comparisons among behavior, whole-vector probing, and neuron-level NSI. (a) Mean NSI is uncorrelated with behavioral accuracy. (b) Whole-vector probe accuracy and mean NSI are negatively correlated, further showing that distributed linear separability and average single-neuron selectivity capture different properties.
DomainParadigmsBehaviorProbeMean NSIMax NSI
Syntax–Semantics Interface230.7270.9070.0181.133
Syntax260.7360.9640.0141.424
Morphology180.8310.8610.0241.932
Appendix
Table 7: Domain-level averages over the 67 BLiMP paradigms. The legacy semantics , syntax_semantics , and syntax/semantics source labels are merged into the Syntax–Semantics Interface domain.
ParadigmDomainBehaviorProbeMean NSIMax NSI
principle_A_case_1Syntax–Semantics Interface0.9990.9360.0051.134
anaphor_number_agreementMorphology0.9860.9510.0111.118
principle_A_domain_1Syntax–Semantics Interface0.9851.0000.0101.224
wh_vs_that_with_gap_long_distanceSyntax0.2210.9730.0061.098
sentential_subject_islandSyntax0.2381.0000.0091.105
wh_vs_that_with_gapSyntax0.2980.9960.0051.064
Appendix
Table 8: Representative paradigms ranked by behavioral accuracy. Near-perfect linear probing can coexist with poor model behavior.
Figure 16: Targeted ablation at each frozen candidate’s actual layer. Red and gray lines show the top-score and bottom signed-score sets; the blue line and band show the mean and 2.5th–97.5th percentile interval across 100 same-layer random sets. Negative values indicate a reduced grammatical-over-ungrammatical log-probability margin.
ParadigmkkTop ΔM\Delta MTop CIRandom meanRandom intervalBottom ΔM\Delta MΔA\Delta Aprandp_{\mathrm{rand}}
Det.–noun agr.10.001[ −0.004,0.006-0.004,0.006 ]−0.001-0.001[ −0.038,0.017-0.038,0.017 ]0.1590.0010.554
 5−0.005-0.005[ −0.014,0.003-0.014,0.003 ]−0.003-0.003[ −0.152,0.119-0.152,0.119 ]0.147−0.001-0.0010.446
 100.000[ −0.011,0.011-0.011,0.011 ]−0.009-0.009[ −0.188,0.263-0.188,0.263 ]0.0540.0020.634
 200.111[ 0.084,0.1380.084,0.138 ]−0.034-0.034[ −0.236,0.218-0.236,0.218 ]−0.022-0.022−0.002-0.0020.911
Det.–noun agr.+adj.1−0.003-0.003[ −0.008,0.001-0.008,0.001 ]−0.005-0.005[ −0.077,0.021-0.077,0.021 ]0.1740.0010.396
 5−0.006-0.006[ −0.015,0.004-0.015,0.004 ]−0.005-0.005[ −0.106,0.110-0.106,0.110 ]0.1610.0000.545
Appendix
Table 9: Complete targeted-ablation results. Top CI is the paired-bootstrap 95% interval over items; Random interval is the 2.5th–97.5th percentile interval across 100 random sets. ΔA\Delta A is the change in total-log-probability minimal-pair accuracy.

Explore similar work

CardsList

  1. Do Language Models Encode Knowledge of Linguistic Constraint Violations?

    May 12, 2026Hardy, Sebastian PadóLarge Language Models FailLinguistics

    IMS, University of Stuttgart, Stuttgart, Germany · Universitas Mikroskil, Medan, Indonesia.

  2. Linear representations of grammaticality in neural language models

    Jul 16, 2026Jane Li, Najoung KimGrammaticalityLinguistics

    Department of Cognitive Science Johns Hopkins University Baltimore, MD · Department of Linguistics Boston University Boston, MA

  3. Implicit Representations of Grammaticality in Language Models

    May 6, 2026Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu +2GrammaticalityText Corpora

    Massachusetts Institute of Technology