If the same neuron activates for both "lender" and "riverside," standard metrics attribute the overlap to superposition--the neuron must be compressing two unrelated concepts. This work explores how much of the overlap is due a lexical confound: neurons fire for a shared word form (such as "bank") rather than for two compressed concepts. A 2x2 factorial decomposition reveals that the lexical-only condition (same word, different meaning) consistently exceeds the semantic-only condition (different word, same meaning) across models spanning 110M-70B parameters. The confound carries into sparse autoencoders (18-36% of features blend senses), sits in <=1% of activation dimensions, and hurts downstream tasks: filtering it out improves word sense disambiguation and makes knowledge edits more selective (p = 0.002).
Figures & tables
Figure 1: Overview. Standard metrics (top) see that neurons n1 – n5 fire for both senses of bank and call all of it superposition. Our decomposition (bottom) shows that most shared neurons are sense-blind —they encode the word form, not compressed concepts. Only a small remainder is genuine superposition.
Figure 3: Top : Layer-averaged cosine similarity by condition across nine models (110M–70B). PS consistently exceeds SYN: word form drives more overlap than shared meaning. Bottom : Per-layer breakdown showing the pattern holds across layers. Rlex trends in Appendix B .
Figure 4: SAE collision analysis (GPT-2). Left : mean features per word by sense discriminability. Right : collision ratio across layers. 18–32% of features conflate senses of the same word (18–36% including Pythia-410M; Section 5.3 ).
Ablation
Δ ppl A
Δ ppl B
Specificity
Sense-A neurons
+0.55
−0.10
0.66
Sense-B neurons
+0.10
+2.62
2.52
Sense-blind
−0.07
+0.17
0.24
Random
−0.64
−0.32
0.32
Table 2: Sense-specific editing (GPT-2, 53 words, 142 neurons per group). Sense-selective ablation targets one sense ( 6.6× specificity gap over blind). Specificity =∣Δppltarget−Δpplother∣ .
Figure 5: Sense-selective ROME editing. Standard ROME modifies all activated neurons, disturbing both senses. Our approach classifies neurons first, shields sense-blind (lexical-form) neurons from modification, and edits only sense-selective ones—preserving the unedited sense.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Condition
Words
Pairs (total)
Pairs/word (mean)
Same-Lemma (SL)
407
∼ 40,700
100
Polysemantic (PS)
407
∼ 40,700
100
Synonym (SYN)
251
∼ 12,500
50
Cross-Lemma (CL)
407
∼ 40,700
100
Appendix
Table 3: Sentence pair counts by condition. SYN coverage is lower because synonyms must be attested in SemCor. All pairs are capped at 200 per word.
Layer
GPT-2
BERT
ELECTRA
GPT-2-Med
0
.735
.662
.686
.751
1
.629
.566
.528
.700
2
.560
.552
.482
.690
3
.494
.471
.428
.612
4
.446
.398
.367
.549
5
.441
.409
.307
.524
Appendix
Table 4: Rlex (cosine-based) by layer for 12-layer models and GPT-2-Medium (first 12 of 24 layers shown). GPT-2-Medium’s minimum of 0.27 occurs at layer 23. BERT achieves its minimum at layer 9 ( 0.23 ), ELECTRA at layer 6 ( 0.25 ). GPT-2’s Rlex remains above 0.29 throughout.
Layer
Pythia-12B
LLaMA-2-13B
0
.913
.911
5
.516
.406
10
.377
.262
15
.360
.456
20
.422
.476
25
.391
.348
Appendix
Table 5: Rlex (cosine-based) at representative layers for the larger models. Pythia-12B has 36 layers; LLaMA-2-13B has 40 layers. Both start with Rlex>0.9 and decline with depth, reaching minima comparable to the smaller models.
Model
Rlexemb
Rlex (L0)
Exceeds?
GPT-2
0.713
0.735
Yes
GPT-2-Med
0.720
0.751
Yes
BERT-base
0.778
0.662
No
ELECTRA-base
0.727
0.686
No
Pythia-1B
0.821
0.836
Yes
Pythia-6.9B
0.882
0.910
Yes
Appendix
Table 6: Token-embedding baseline comparison. All autoregressive models show MLP-level Rlex exceeding the embedding baseline at layer 0, confirming non-trivial amplification. Bidirectional models (BERT, ELECTRA) show MLP Rlex below the embedding baseline at layer 0, suggesting early-layer bidirectional processing partially resolves lexical overlap—but the confound nonetheless persists at all layers.
Layer
GPT-2
BERT
ELECTRA
GPT-2-Med
0
0.88
0.63
0.57
0.70
1
0.77
0.78
0.64
0.75
2
0.83
0.64
0.76
0.80
3
0.84
0.61
0.74
0.85
4
0.72
0.67
0.81
0.78
5
0.74
0.63
0.83
0.74
Appendix
Table 7: Fraction (%) of neurons with SSI>2 at each layer. The denominator is dmlp (3072 for GPT-2/BERT/ELECTRA, 4096 for GPT-2-Med). Pythia-1B’s layer 0 has the highest fraction (1.80%) due to its larger MLP producing more sense-differentiating neurons in absolute terms, but the fraction remains very small.
32k SAE
128k SAE
Layer
Active
Blind
Coll.%
Active
Blind
Coll.%
0
33.6
10.4
31.7
31.5
9.8
32.6
2
27.8
7.4
29.1
26.4
7.6
31.3
5
21.3
4.3
24.1
20.0
4.4
25.7
8
18.6
2.7
17.7
15.2
2.6
22.4
11
18.4
3.9
28.9
15.2
2.9
25.9
Appendix
Table 8: SAE collision analysis: 32k vs. 128k features. Active : mean features active per word ( > 30% firing rate in either sense). Blind : mean sense-blind features (Cohen’s d<0.5 , fires for both senses). Coll.% : collision ratio (blind/active). Both SAEs show similar collision ratios at every layer, confirming that increasing dictionary size does not resolve the confound.
Model
L0
L 4n
L 2n
L 43n
L n-1
min p
GPT-2
.43
.34
.45
.56
.37
.28
GPT-2-Med
.50
.28
.38
.42
.28
.28
BERT-base
.57
.31
.14
.18
.16
.14
ELECTRA
.48
.33
.50
.53
.26
.26
Pythia-1B
.62
.20
.33
.30
.16
.16
Pythia-6.9B
.80
.43
.26
.13
.25
.13
Appendix
Table 9: Mann-Whitney U test p -values comparing the per-word SL–PS cosine gap between synonym-available ( n=251 ) and synonym-absent ( n=156 ) words. No test reaches significance; the minimum p -value across all models and layers is 0.13 (Pythia-6.9B, layer 24).
Figure 6: Jaccard overlap of active-neuron sets across layers (GPT-2). The condition ordering \textscsl>\textscps>SYN>\textsccl is consistent with the cosine similarity results, confirming that the lexical confound affects which neurons fire, not just how strongly they fire.
Figure 7: Cross-sense (PS) cosine similarity per word per layer in GPT-2 (top 30 words by variance). Cooler colors indicate lower similarity (stronger sense-specific processing). Most words exhibit a U-shaped trajectory with a dip in mid-layers, though the depth and timing of the minimum varies substantially across words—sense separation is word-specific, not driven by a shared mechanism.
Figure 8: Rlex (solid) vs. Rlexno-syn (dashed) for all eight models. The alternative metric Rlexno-syn=(Mˉ\textscps−Mˉ\textsccl)/(Mˉ\textscsl−Mˉ\textsccl) does not require synonym data and can be computed for all 407 words. The qualitatively identical trajectories confirm that our findings do not depend on synonym coverage (62% of words, validated by subset invariance test; see Appendix B.5 ).
Figure 9: Mean magnitude divergence ( Δ=∣A∩B∣1∑i∈A∩B∣ai−bi∣ ) by condition and layer. Unlike cosine similarity (which shows a U-shaped trajectory), magnitude divergence increases monotonically with depth, reflecting growing activation scale differences as representations become more specialized. The condition ordering is reversed— \textsccl>SYN>\textscps>\textscsl —because more dissimilar conditions produce larger activation differences among their shared neurons.
Figure 10: Condition means at three representative layers in GPT-2 (layers 0, 6, 11). At layer 0, ps and sl are close while SYN and cl are distant. By layer 6, the gap between ps and sl widens. At layer 11, all conditions retain the same ordering with moderate separation.
Figure 11: Rlex (lexical contribution ratio) across layers for all nine models with 95% bootstrap CIs. The lexical contribution decreases with depth but never reaches zero in any model. Larger models (LLaMA-2-70B, cyan) show sustained Rlex≈0.35 – 0.45 across 80 layers.
PS–SYN gap
Rlex
Δ PS
Δ SYN
GPT-2 (d=3072, layer-averaged)
Baseline
0.119
0.65
—
—
k=10 (0.3%)
0.087
0.52
+ 0.008
− 0.032
k=20 (0.7%)
0.047
0.41
+ 0.019
− 0.054
k=50 (1.6%)
− 0.028
< 0
+ 0.031
− 0.116
Pythia-1B (d=8192, layer-averaged)
Appendix
Table 10: LIS dose-response ablation. Removing k=20 components ( ≤ 1% of dimensions) reduces Rlex by 37–51%. At k=50 , the PS–SYN gap reverses—the confound is eliminated.
Model
SL
PS
SYN
CL
PS > SYN
GPT-2
.90–.60
.89–.47
.31–.45
.26–.45
p<.001✓
BERT-base
.70–.58
.54–.30
.09–.32
.05–.35
p<.01✓
Pythia-1B
.99–.46
.99–.27
.09–.28
.09–.26
p<.001✓
Appendix
Table 11: Modern corpus validation on CoarseWSD-20 (Wikipedia sentences, 20 nouns). Ranges show cosine similarity across layers (layer 0 to final). The critical PS > SYN gap—indicating lexical form drives more overlap than shared meaning—is statistically significant at every layer in all three models (Wilcoxon signed-rank, Holm-Bonferroni corrected). Rlex bootstrap 95% CI remains above zero everywhere.
Glossaries, technical specifications, and system prompts routinely ask language models to use familiar words in unfamiliar ways. When this works, the local rule does not overwrite the old meaning; the pretrained prior keeps operating underneath, and its strength still shows through. We test this with a Stroop-style paradigm: a remapping rule (doctor means forest) pitted against the query word's lexical-prior distractor (hospital), with matched neutral controls. Across 11 open-weight models spanning four families and 1B-9B parameters, lexical-prior strength predicts interference even after item-level controls for answer prior, frequency, tokenization, and prompt wording. Activation patching on five models then locates where the override is repaired internally. Restoring three source positions that carry the redefinition (the definition subject, its new target, and the query word) almost fully recovers the effect (aggregate R∈[0.92,1.06]). The repair works by protecting the contextual target rather than by silencing the prior; the distractor's probability falls whenever these positions are perturbed, but the target survives only when the redefinition is restored intact. Behavior and mechanism converge on the same channel: the prior's strength both predicts which overrides fail and marks where the causal repair lands.
Lexical representations can encode both abstract information and information specific to particular contexts. We investigate how these forms of information are organized in language models by decomposing contextual word representations into item, class, and word-class interaction components. Across six pretrained transformers and two linguistic categories, we find that the interaction is consistently present, grows across layers, and contains information associated with both item and class. Unlike the nearly orthogonal item and class effects, the interaction overlaps with both and encodes class information beyond the shared class effect. An interaction also emerges in artificial-language learners even when the input contains only additive structure. Together, these findings show that lexical representations combine abstract information with structure specific to particular word-class combinations.
Representations extracted from large language models (LLMs) play an important role in many downstream applications. However, the structure of these representations is often influenced by lexical overlap rather than semantic content. Our understanding of the relationship between this lexical influence and semantic content, and its implications for downstream tasks, remains limited. In this work, we investigate representations to quantify the effect of lexical overlap relative to semantic content. We consider several adversarial semantic stress tests and further connect our findings to the information theory perspective. We find that lexical influence extends across the depth of models, consistently across architectures, training regimes, and objective functions, including the models trained for semantic similarity. Moreover, we observe a mid-depth region in which both lexical and semantic signals degrade simultaneously, indicating a transitional regime where representations are poor for both surface form and meaning. We further demonstrate the effect of lexical influence on downstream uses of LLMs using summarization and model editing as a case study.
Hammad Rizwan, Muhammad Umair Haider, Nishant Subramani +3
Dalhousie University Halifax, NS · University of Kentucky Lexington, KY · Carnegie Mellon University Pittsburgh, PA