Large-scale Neuron Community Discovery Reveals Neuron-Sparse Behaviors in Language Models
Organizations: Goodfire · Imperial College London
Abstract
What reusable units of computation can we discover within a language model's neurons? Prior work has shown that individual forward passes can often be sparsified post-hoc to small groups of neurons, but it has been unclear whether these groups form coherent computational entities reused across inputs. We introduce neuron community discovery, an unsupervised method for finding neuron groups that recur across prompts and are causally necessary and sufficient, within the layers they span, for common computations. The method combines gradient-based attribution, a graph connecting neurons that are repeatedly attributed together, and community detection. Applied to Llama-3.1-8B using ten million prompts, it discovers thousands of candidate communities. We evaluate them on held-out prompts through three causal tests: community necessity, sufficiency within a layer subset, and layer necessity. These tests identify groups whose removal disrupts predictions and whose retention preserves predictions despite ablating the remaining MLP neurons in the tested layers. Discovered communities support functions like grammatical agreement, token-probability calibration, fixed phrases, in-context copying, addition, and multiplication. With communities capped at 128 neurons across all layers, 10.3% of held-out prompts have at least one layer's contribution explained on a neuron-sparse basis. Coverage remains 8.7% when requiring at least four layers and 3.3% when requiring at least twelve. These results show that small groups of existing neurons can provide reusable computational components across prompts, and that such groups can be discovered at scale.
Figures & tables
| Function Neurons Layers Prompt Predictions ( base , ablated ) | |
|---|---|
| Grammar | Promote vowel-initial words after an 6 6 L2–L14 Diagnostic services play an ␣ essential (0.75) crucial (0.33) Select is or are after there 9 9 L0–L29 Ten cards are put into five envelopes so that there ␣ are (1.00) is (0.65) |
| Morphology | Detect plural -s 9 7 L0–L6 Finally, some NGOs ␣ are (0.13) leaders (0.53) Detect adverbial -ly 8 2 L0–L1 two independent normally ␣ distributed (1.00) variables (0.99) |
| Formatting | Complete LaTeX commands 11 10 L0–L25 sampling distribution of \subseteq$ L0–L8 class Solution(object ␣ ): (0.96) : (0.92) |
| In-context | Copy an entity from context 7 4 L23–L27 malware camouflages as Delta.exe … should check the ␣ Delta (1.00) location (0.85) Complete HTML closing tags 17 13 L0–L28 <div>JavaScript is disabled on your browser.</ ␣ div (1.00) </ (1.00) |
| Numerical operations | Multiplication 42 4 L18–L21 Prime factorization: 799 = 17 x ␣ 47 (1.00) 43 (0.51) Addition 8 2 L18–L19 are 43 active cases (40 at the border and ␣ three (0.88) four (0.65) Select ordinal suffix ( st/nd/rd ) 21 9 L1–L27 Last updated date: 22 ␣ nd (1.00) th (0.99) |
| Token-level | Suppress fraction 2 1 L29–L29 we could conceivably give ␣ fractions (0.21) fractions (0.74) Promote each other 6 5 L12–L29 farmers often come together to help ␣ each (0.81) their (0.67) |
| Class | Communities | Prompts |
|---|---|---|
| Common phrases | 42.3% | 31.3% |
| Token suppression | 19.5% | 19.6% |
| Formatting | 9.3% | 19.4% |
| Grammar | 2.9% | 15.3% |
| Sub-word completion | 8.0% | 6.0% |
| Topic | 9.0% | 3.0% |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| values | Train coverage (%) | Held-out coverage (%) |
|---|---|---|
| 5.70 | 5.74 | |
| 8.05 | 8.13 | |
| 8.98 | 9.03 | |
| 9.64 | 9.66 | |
| 10.09 | 10.13 | |
| {…} + 32 | 10.21 | 10.28 |
| 10M train | Held-out | ||||
|---|---|---|---|---|---|
| Slice | Target position | Sampled | Kept | Sampled | Kept |
| Grammar | Word-initial closed-class token (determiner, preposition, or pronoun) in French, German, Spanish, or Russian text. | 100,000 | 99,000 | 100,000 | 99,000 |
| Switch | First token after a language switch: Europarl [ Koehn, 2005 ] bilingual pairs in four layouts, plus Wikipedia “(Language: …)” glosses. | 29,000 | 29,000 | 29,000 | 25,000 |
| Code | Prompts sampled at uniformly chosen token locations in source files from the-stack-smol [ Kocetkov et al., 2023 ] , covering 16 languages. | 261,000 | 246,000 | 261,000 | 243,000 |
| QA | First answer token of the last question in a 5-shot quiz document (TriviaQA [ Joshi et al., 2017 ] and NQ-open [ Kwiatkowski et al., 2019 , Lee et al., 2019 ] , in five formats). | 50,000 | 50,000 | 50,000 | 49,000 |
| Memorized | Hits on famous public-domain text with where a 12-token context fails, plus controls from the same windows. | 48,000 | 46,000 | — | — |
| Label | Definition |
|---|---|
| Lexical / collocational | |
| Fixed phrase / collocation | Completes the same fixed expression across examples, such as “each other” or “take into account”. |
| Specific token promoted after a trigger family | Promotes the same output token after different, related triggers that motivate it. |
| Trigger lemma selects its complement | Selects different continuations after the same trigger word, according to context; for example, “demand” followed by “of”, “for”, or “to”. |
| Semantic content word / near-synonym choice | Selects a content word or near-synonym required by the sentence meaning, rather than merely related to its topic. |
| Default-token calibration | |