cs.LGNov 17, 2026

Large-scale Neuron Community Discovery Reveals Neuron-Sparse Behaviors in Language Models

Authors: Igor Shilov, Thomas Fel, Tal Haklay, Lucius Bushnaq, Owen Lewis

Organizations: Goodfire · Imperial College London

Abstract

What reusable units of computation can we discover within a language model's neurons? Prior work has shown that individual forward passes can often be sparsified post-hoc to small groups of neurons, but it has been unclear whether these groups form coherent computational entities reused across inputs. We introduce neuron community discovery, an unsupervised method for finding neuron groups that recur across prompts and are causally necessary and sufficient, within the layers they span, for common computations. The method combines gradient-based attribution, a graph connecting neurons that are repeatedly attributed together, and community detection. Applied to Llama-3.1-8B using ten million prompts, it discovers thousands of candidate communities. We evaluate them on held-out prompts through three causal tests: community necessity, sufficiency within a layer subset, and layer necessity. These tests identify groups whose removal disrupts predictions and whose retention preserves predictions despite ablating the remaining MLP neurons in the tested layers. Discovered communities support functions like grammatical agreement, token-probability calibration, fixed phrases, in-context copying, addition, and multiplication. With communities capped at 128 neurons across all layers, 10.3% of held-out prompts have at least one layer's contribution explained on a neuron-sparse basis. Coverage remains 8.7% when requiring at least four layers and 3.3% when requiring at least twelve. These results show that small groups of existing neurons can provide reusable computational components across prompts, and that such groups can be discovered at scale.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Oct 6, 2026

Where Do LLMs Retrieve Network Information? Identifying the Responsible Heads and Neurons

Large language models (LLMs) used for Web information processing rely on retrieving relationships between network entities, such as domain names and IP addresses, but the internal components that causally enable this retrieval remain unclear, hindering targeted safeguards against unsafe actions. Therefore, we investigate whether their retrieval capabilities can be localized to specific attention heads or individual neurons within these LLMs. Across five LLMs from three architecture families, we identify heads through causal ablation screening, test their selectivity against matched negative and context-free controls, rank neurons within these heads using causal ablation and correlational measures, and leverage features from both causally validated heads and individual neurons to build network information retrieval detectors, which we evaluate without retraining on an independent reverse-DNS dataset. In every model, a small set of causally validated heads (1--9 out of 128--1152 candidates) builds a detector with 99.5--100% held-out accuracy. At the neuron level, the results are model-specific, e.g., in one model, causal and correlational rankings identify the same neuron, matching full-head performance (AUC = 1.000), while in another, the best causally ranked neuron achieves only AUC = 0.665 despite the full head reaching 1.000. On independent CAIDA data, head detectors maintain AUC ≥0.971\geq 0.971, while single-neuron detectors fall below chance in three of five models. Overall, these findings highlight the importance of choosing an appropriate level of granularity when causally localizing how LLMs retrieve structured network information from context.
Aug 10, 2026

Decodable But Not Detachable: Training Data Granularity Determines Parametric Modularity in Large Language Models

Do large language models contain domain-specific parametric shells: concentrated, causally necessary neuron populations whose removal selectively degrades a target domain while sparing others? We apply a uniform causal methodology across two domain granularities, three model families (1.5B to 7B parameters), and eight domains. At the academic subject level, zero neurons exceed 60% domain selectivity across 939,008 combined FFN neurons and causal damage matrices are flat, despite domain identity being linearly decodable above 85% accuracy. At the language and modality level, 0.65--1.14% of neurons exceed 60% selectivity, damage matrices are near-perfectly diagonal (ratios up to 595:1), and shell neuron sets are essentially disjoint (IoU <0.003< 0.003). Masking code-selective neurons reduces mathematical reasoning accuracy by 16--24 percentage points across all models; masking Spanish or Chinese neurons leaves it at or below random. Shell strength increases monotonically with scale and shells are spatially interleaved in a pattern that precludes group-level selective quantization. Parametric shells form where and only where training data was modular at the token level.
Jul 8, 2026

Distributed Sparse Interventions in Language Models

Language models can perform a wide range of tasks at various levels of abstraction. They flexibly infer tasks from context, execute multiple tasks simultaneously, and select among competing goals. Understanding and controlling how models represent these tasks is central to aligning them with human objectives. Researchers can change model behaviour in many ways, but model steering, which manipulates intermediate task representations, links representation to behaviour most directly. Current steering methods typically intervene densely and assume linear effects, e.g., shifting every activation along a global direction in the residual stream. Because such interventions change thousands of neurons at once, they leave researchers to separate relevant from irrelevant neurons. We introduce Distributed Sparse Interventions (DSI), which steers models through a sparse set of individual neurons distributed across layers and attention heads. Finding such a set requires accounting for the selected neurons' interactions, even across layers; DSI achieves that by iteratively refining which neurons to intervene on and how strongly. Across 12 tasks and three instruction-tuned models, DSI activates task behaviour by identifying and intervening on fewer than 0.1% of neurons, outperforming dense steering vectors and often matching 10-shot in-context learning. Because DSI identifies explicit sets of neurons, it connects neuron-level computation to changes in latent representations and, ultimately, to desired outputs, allowing us to study how distributed computational components interact to produce steered behaviour. Our results demonstrate that steering can be viewed not only as a linear shift in representation space, but also as a targeted intervention on model computation, realised by adjusting a small set of interacting neurons whose coordinated activity spans multiple layers and contributes to behaviour.