Sparse autoencoders (SAEs) have become an increasingly popular means of translating Large Language Model (LLM) hidden state vectors into a human-comprehensible list of features or symbols. As such, SAEs may provide insight into an LLM's knowledge representation and internal mental'' grammar - topics of pragmatic interest for mechanistic interpretation of AI behavior and of theoretical interest for the study of in silico representations of human language. Language models with fewer than 100M parameters, trained on small corpora, nonetheless exhibit strong grammatical competence and discourse coherence, despite producing content of limited reliability. We use SAEs that we trained on one such model (Pythia-70M) to conduct an in-depth examination of how a single grammatical phenomenon, number agreement, is represented and processed in an LLM. We demonstrate the mechanism and the methodology by which SAEs can be used to find and propagate plural noun'', plural matrix subject'', and generate a plural verb'' features which cause the LLM to produce plural verb forms such as are'', do'', or have'' instead of is'', does'', and has'' in response to prompts such as the men in the park''. We show that intervening to remove these features from the hidden states of plural subject noun tokens causes the model to produce singular verbs instead, and vice versa. We also show that corresponding singular noun'' and ``generate a singular verb'' features are not as effective for this task. This asymmetry suggests that Pythia-70m makes use of a privative representation of grammatical number in English. These findings replicated with larger LLMs Gemma-2-2b and Llama-3.1-8b yielded similar results. Our main contribution is a novel approach to the use of SAEs to understanding how LLMs encode human language that addresses a number of technical and methodological challenges.
Figures & tables
Figure 1: The token “ man” bearing the Sg features
Figure 2: a“ yesterday”→“ man”(1,1) acquires a high value
Figure 3: Sg is copied to the output vector
Figure 4: Sg symbol read by the WU matrix to output singular verbs
Figure 5: 4:mlp:961 tracks embedded clause noun number. M = Matrix noun, E = Embedded noun
Figure 6: Illustration of the Zipfian nature of the verb token generation probabilities distribution
Figure 7: Illustration of the structure revealed by using log probabilities
Figure 8: Effect of Layer 4 MLP Matryoshka SAE feature number 961 on Pythia-70m verb conjugation behavior when applied to token “ organization” at a strength of 50
Figure 9: Asymmetry in plurality vs. singularity marking on noun tokens. Cliff’s delta scores (-1 to 1 range) are shown beneath each pair of ranges (blue=plural; green=singular) in the strength of the features on the x-axis.
Figure 10: The strength of the Fc = 0:mlp:134@noun subject →Fe = 5:attn:966@final token edge over S=rc . The numbers under the pluralized singular and the singularized plural violins are Cliff’s delta values.
Figure 11: The strength of the Fc = 0:mlp:134@noun subject →B edge over S=rc . The numbers above the pluralized singular and the singularized plural violins are Cliff’s delta values.
Figure 12: Variance explained by various SAEs on a calibration dataset of 1,000 hidden states. Our unique embedding and unembedding Matryoshka SAEs are for the sake of comparison denoted as “Attention layer -1” and “Attention layer 6” in the upper left subfigure. Our Gemma-2-2b and Llama-3.18b WE Matryoshka SAEs, not shown, had variance explained levels of 95.84% and 94.31%, respectively
S LLM
Dataset
Feature
strengths(Fc,B)
strengthp(Fc,Fe)
Gemma-2-2b
rc
we:533
0.56
0.26
”
”
we:18
0.34
0.03
”
”
we:2645
0.22
0.01
”
nounpp
we:533
0.47
0.25
”
within_rc
we:533
1.10
0.37
”
”
we:18
0.64
-0.04
Table 1: Encoding layer Matryoshka SAE features of all three LLMs. The feature we:5548 highlighted in red is the only singular positive feature we found.
Figure 13: Verb conjugation circuits identified in rc , nounpp , and within_rc . Sentences and verb continuations shown are merely examples, these circuits were derived by aggregating data from all sentence pairs and verb pairs in their respective datasets as described in the methods section. Single-high features are red, plural-high are blue. Black edges activate, red edges inhibit. The horizontal lines represent the layers of the LLM and the vertical blocks represent tokens in the prompt. Features belonging to the same sublayer and token which shared more than 60% of ingoing and outgoing edges were grouped into boxes.
Figure 14: The most strongly marked tokens for a number of prominent conjugation-involved plural noun features in the WE embedding matrix of Pythia-70m.
Figure 15: Average values of 4:attn:429 at different token positions in long_rc The bird(s) that I saw eating the worms in the park this morning. blue - subject is plural; orange - subject is singular
Figure 16: Removing 4:attn:429 from the final token of plural matrix singular embedded form sentences in within_rc (i.e., “The boys that the mother…”) increases singular verb conjugation preference
Figure 17: Tokens selected by the most prominent Singular and plural verb production features in the WU unembedding matrix
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Name
Variable
Dimension
Token vocabulary
V
∣V∣
Embedding matrix
WE
∣V∣×d
Unembedding matrix
WU
d×∣V∣
Query matrix at attention sublayer ℓ in head n
WQ(ℓ,n)
d×dh
Query bias at “
bQ(ℓ,n)
dh
Key matrix at “
WK(ℓ,n)
d×dh
Appendix
Table 2: Weights
Name
Variable
Dimension
Prompt Token indices
i or j
prompt length
Initial embeddings for token at index j or for token “ man”
\color[rgb]{1,0,1}{r_{j}^{0}} or \color[rgb]{1,0,1}{r_{\text{`` man''}}^{0}}
d
Residual stream for token j /“ man” after layer ℓ
\color[rgb]{1,0,1}{r_{j}^{\ell}} or \color[rgb]{1,0,1}{r_{\text{`` man''}}^{\ell}}
d
Normed attention layer input for layer ℓ
{\color[rgb]{1,0.5,0}r_{j}^{{\ell-1}:attnnorm}} or {\color[rgb]{1,0.5,0}r_{\text{`` man''}}^{{\ell-1}:attnnorm}}
d
Key for ” at attention head n of layer ℓ
\color[rgb]{0,0,1}{k_{j}^{(n,\ell)}} or \color[rgb]{0,0,1}{k_{\text{`` man''}}^{(n,\ell)}}
dh
Query for ”
\color[rgb]{0,0.6,0}{q_{j}^{(n,\ell)}} or \color[rgb]{0,0.6,0}{q_{\text{`` man''}}^{(n,\ell)}}
dh
Appendix
Table 3: States
Name
Index Variable
Size variable
Size in Pythia-70m
Vocab size
t
∣V∣
50304
Hidden size
d
512
Number of layers
ℓ
l
6
Number of attention heads
n
nh
8
Attention head size
dh=d/nh
64=512/8
MLP Intermediate size
dm
2048
Appendix
Table 4: Dimensions
Figure 18: Schematic of Pythia-70m
Figure 19: The 1st attention sublayer of Pythia-70m
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts. A core SAE training hyperparameter is L0: how many SAE features should fire per token on average. Existing work compares SAE algorithms using sparsity-reconstruction tradeoff plots, implying L0 is a free parameter with no inherently correct value aside from its effect on reconstruction. In this work we study the effect of L0 on SAEs, and show that if L0 is not set correctly, the SAE fails to disentangle the underlying features of the LLM. If L0 is too low, the SAE will mix correlated features to improve reconstruction. If L0 is too high, the SAE finds degenerate solutions that also mix features. Further, we present a proxy metric that can help guide the search for the correct L0 for an SAE on a given training distribution. We show that our method finds the correct L0 in toy models and coincides with peak sparse probing performance in LLM SAEs. We find that most commonly used SAEs have an L0 that is too low. Our work shows that practitioners must set L0 correctly to train SAEs with monosemantic features.
While multilingual language models successfully transfer factual and syntactic knowledge across languages, it remains unclear whether they process culture-specific pragmatic registers, such as slang, as isolated language-specific memorizations or as unified, abstract concepts. We study this by probing the internal representations of Gemma-2-9B-IT using Sparse Autoencoders (SAEs) across three typologically diverse source languages: English, Hebrew, and Russian. To definitively isolate pragmatic register processing from trivial lexical sensitivity, we introduce a novel dataset in which every target term is polysemous, appearing in both literal and informal contexts. We find that while much of the informal-register signal is distributed across language-specific features, a small but highly robust cross-linguistic core consistently emerges. This shared core forms a geometrically coherent ``informal register subspace'' that sharpens in the model's deeper layers. Crucially, these shared representations are not merely correlational: activation steering with these features causally shifts output formality across all source languages and transfers zero-shot to six unseen languages spanning diverse language families and scripts. Together, these results provide the first mechanistic evidence that multilingual LLMs internalize informal register not just as surface-level heuristics, but as a portable, language-agnostic pragmatic abstraction.
To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds of pythia-410m across training. We find that subject-verb agreement is learned early, and is initially handled by a diffuse collection of neurons. However, after agreement behavior has stabilized, the mechanisms responsible for it continue to reorganize. The attention heads which mediate agreement often shift during training: in the case of pythia-1b, behavior is controlled by two heads early in training but by a third (different) head later. Furthermore, the circuit shrinks over training, with pythia-1b's final circuit being strikingly sparse. We take these results to be informative about how linguistic knowledge emerges and is represented in a language model.