Neural networks are more modular than single neurons suggest

Shreya Saha; Benjamin Bergen; Meenakshi Khosla

cs.AIOct 9, 2026

Neural networks are more modular than single neurons suggest

Authors: Shreya Saha, Benjamin Bergen, Meenakshi Khosla

Organizations: Electrical and Computer Engineering, University of California, San Diego · Cognitive Science Department, University of California, San Diego · Computer Science and Engineering, University of California, San Diego

Abstract

Neural networks routinely develop representations in which individual neurons contribute to multiple distinct functions, leading to the widespread view that such networks lack modular organization. We challenge this conclusion by introducing a formally nested hierarchy of modularity: unit-level, which requires distinct computations to occupy disjoint neurons; orthogonal, which asks whether a rotation reveals functionally independent subspaces; and linear, which asks whether a bounded linear map can separate task-relevant computations into independent subspaces. We develop a unified framework for learning decompositions at each level and evaluating them causally through selective ablation. Across vision models, language models and multi-task Recurrent Neural Networks (RNNs), we find that networks which lack unit-level modularity nonetheless often exhibit clear functional modularity at the subspace level. In vision and language models, content versus style and syntax versus semantics respectively dissociate under subspace ablation despite sharing overlapping populations of neurons. These learned subspaces also align differentially with brain data, with content and style subspaces predicting higher and lower visual cortex respectively, and semantic subspaces outperforming syntactic ones in predicting language cortex activity. In RNNs trained on multiple cognitive tasks and previously shown to lack modular structure, our decomposition recovers sharp double dissociations between tasks. The absence of unit-level modularity frequently coexists with clear subspace-level modularity across domains, enabling targeted intervention on specific computations even in networks where no individual neuron is specialized, with broad implications for mechanistic interpretability and for understanding functional organization in both biological and artificial neural systems.

Figures & tables

Figure 1: Illustration of functional modularity and our proposed decomposition framework. (A) In an entangled representation, individual dimensions participate in multiple functionalities, such that the representations of different functions overlap across dimensions. In an ideally modular representation, each functionality occupies a distinct, non-overlapping set of dimensions, allowing the functions to be independently manipulated. (B) Our framework applies an information-preserving transformation to an entangled representation to separate two target functionalities. Ideally, the transformed representation decomposes into three subspaces: two function-specific subspaces corresponding to the target functionalities being dissociated, and a residual subspace containing information associated with other, non-target functionalities.

Figure 2: Functional modularity in multitask RNNs (Yang et al., 2019). (a,f) FTV distributions for Anti vs DM1 in softplus (top) and tanh (bottom) networks. Softplus shows bimodal clustering; tanh shows a broad unimodal distribution centered near zero. (b–e) Double dissociation bars for softplus networks at each hierarchy level (unit, Sinkhorn, orthogonal, linear). All levels show selective ablation effects (DDI = 0.59–0.73). (g–j) Same for tanh networks. Unit-level DDI=0.00; orthogonal DDI = 0.37; linear DDI = 0.47. Subspace decomposition recovers modularity that is invisible at the unit level.

Transformation	Top-1 CM $\uparrow$	Top-5 CM $\uparrow$	Top-10 CM $\uparrow$	Top-1 SM $\uparrow$	Top-5 SM $\uparrow$	Top-10 SM $\uparrow$	$R_{s}$ $\uparrow$	$R_{c}$ $\uparrow$
Permutation	0.0063	0.0080	0.0125	0.0738	0.0625	0.0546	1.1032	1.0216
Orthogonal	0.1381	0.1686	0.2435	0.7542	0.7550	0.7613	7.2348	2.5603
Invertible	0.3107	0.3675	0.4884	0.8632	0.8785	0.8940	16.0815	5.2594

Table 1: Swin Base - Comparison of Top-K content and style summary metrics, along with content and style selectivity ratios, across permutation, orthogonal, and invertible transformations.

Figure 5: Top-10 nearest neighbor retrieval results under invertible transforms in the content subspace and style subspace

Transformation	Top-1 $S_{1}$ $\uparrow$	Top-5 $S_{1}$ $\uparrow$	Top-10 $S_{1}$ $\uparrow$	Top-1 $S_{2}$ $\uparrow$	Top-5 $S_{2}$ $\uparrow$	Top-10 $S_{2}$ $\uparrow$
Permutation	0.3333	0.3953	0.3971	0.1564	0.1681	0.1368
Orthogonal	0.8485	0.9286	0.9417	0.6667	0.5504	0.4763
Invertible	0.9375	0.9439	0.9259	0.6556	0.5675	0.5152

Table 2: Comparison of Top-K Syntax and Semantics summary metrics across permutation, orthogonal, and invertible transformations for Pythia 410M.

Figure 8: (A) Schematic illustrating how distinct subspace representations derived from vision and language models are used to predict neural activity in the human visual and language cortex. (B) Regions of the visual cortex that are better predicted by style versus content subspace representations from vision models. (C) Regions of the visual cortex that are better predicted by syntactic versus semantic subspace representations from language models. (D) Predictive performance of style and content subspace representations from vision models in explaining neural activity within the language cortex. (E) Predictive performance of syntactic and semantic subspace representations from language models in explaining neural activity within the language cortex.

Figure E1 : Examples of images belonging to same content (Top), and same colour (Bottom)

Figure E2 : Examples of sentences belonging to same syntax(top), and same semantics (bottom).

Figure E3 : For the semantic paraphrases and syntactic counterparts generated for each sentence in the Pereira dataset, we compare all sentence pairs within each semantic or syntactic set by measuring the number of tokens that remain identical at the same token positions across the two sentences. The resulting distributions reveal that semantic paraphrases exhibit a broader spread of token overlap values, reflecting greater variability in lexical structure, whereas syntactic counterparts are heavily concentrated near zero, indicating minimal token overlap at corresponding positions.

Panel of Table 12 — Table E1 : Prompt used for generating paraphrase using Gemini-2.5-Flash.

Panel of Table 13 — Table E2 : Prompt used for generating syntactic counterparts using Gemini-2.5-Flash.

	Motion context	Color context
Accuracy	$87.5\pm 1.2\%$	$87.1\pm 1.7\%$

Table E3 : Mante RNN accuracy on non-zero coherence trials (mean

\pm

s.d. across 10 seeds).

Figure E4 : a,b Accuracy on motion and color tasks after ablating the motion or color subspace (linear decomposition, $d=50$ ), shown for unit, orthogonal, and linear levels. Horizontal lines indicate intact baseline accuracy. Only linear decomposition produces selective impairment. (c) DDI as a function of subspace dimension for unit (gray), orthogonal (blue), and linear (red) decompositions. Unit and orthogonal DDI remain near zero at all dimensions; linear DDI rises to $\sim$ 0.28 by $d=15$ . (d) Accuracy after ablation as a function of dimension (linear decomposition). Solid lines show the targeted task (which should drop); dashed lines show the spared task (which should remain high). The dissociation emerges progressively with increasing subspace dimension.

Figure E5 : Swin Base: Top-

K

exact-category and broad-category retrieval for each silhouette image representation derived from edge-based inputs from Geirhos et al. [2018] .

Figure E6 : Top-

K

content and style retrieval across content and style subspaces for the conflicting-cues images from Geirhos et al. [2018] .

Yang task-pair category	Softplus: maximum DDI	Tanh: maximum DDI
Disjoint pair	0.704	0.472
Inclusive pair	0.052	0.000
Mixed pair	0.048	0.002

Table E4 : Maximum DDI values across Yang task-pair categories for softplus and tanh networks.

Mante decomposition	DDI (mean $\pm$ SD across 10 networks)
Learned map, true targets	$0.261\pm 0.084$
Spectrum-matched random maps	$0.0009\pm 0.0004$
Learned maps, shuffled targets	$0.010\pm 0.008$

Table E5 : DDI values for different Mante decomposition conditions. Values are reported as mean

\pm

SD across 10 networks.

Transformation	Content Matching (CM)			Style Matching (SM)			$R_{s}$	$R_{c}$
	Top 1	Top 5	Top 10	Top 1	Top 5	Top 10
Permutation	0.0007	0.0023	0.0002	0.0096	0.0154	0.0108	1.0141	1.0011
Orthogonal	0.0034	-0.0021	0.0014	-0.0024	-0.0080	0.0036	1.0275	1.0067
Invertible	0.0048	0.0065	0.0042	0.0148	-0.0034	0.0165	0.9727	1.0286

Table E6 : Retrieval metrics and information-preservation measures for different transformations.

Figure E7 : For each subspace (style/content/residual) in visual representations obtained from Swin Base, we plot content and style selectivity with different portions of units allocated to content and style subspaces

	Content Matching (CM)			Style/Category Matching
Dissociation	Top 1	Top 5	Top 10	Top 1	Top 5	Top 10
Content–Style	0.8157	0.8725	0.9351	0.5741	0.6635	0.7978
Content–Category	0.8157	0.8725	0.9351	-0.0800	-0.1966	-0.8368

Table E7 : Content dissociation results. Content matching (CM) measures preservation of content information, while the second set of metrics measures matching with the attribute being dissociated.

	Content Units			Style/Category Units
Dissociation	Top 1	Top 5	Top 10	Top 1	Top 5	Top 10
Content–Style	0.8424	0.9042	0.9269	0.7558	0.8450	0.8734
Content–Category	0.8330	0.9021	0.9235	0.8186	0.8864	0.9105

Table E8 : Content retrieval accuracy for content-style and content-category dissociation. Content units measure preservation of content information, while the second set of units corresponds to the attribute being dissociated.

Dissociation	Content			Category			Residual
Dissociation	T1	T5	T10	T1	T5	T10	T1	T5	T10
Content–Category	0.141	0.258	0.331	0.152	0.275	0.346	0.194	0.352	0.449

Table E9 : Top-

k

style retrieval accuracy for the Content–Category dissociation across content, category, and residual units.

Intervention	Motion Accuracy	Color Accuracy
Intact	$\mathbf{0.872\pm 0.016}$	$\mathbf{0.872\pm 0.022}$
Ablate motion subspace	$0.508\pm 0.072$	$0.836\pm 0.034$
Ablate color subspace	$0.846\pm 0.022$	$0.558\pm 0.083$
Ablate residual	$0.724\pm 0.050$	$0.720\pm 0.057$

Table E10 : Intervention-based ablation results for motion and color classification.

Figure E8 : For each subspace (style/content), we plot the distribution of average pairwise Pearson correlations computed within sets of images grouped by shared style or content, across invertible transforms for multiple vision models.

Figure E9 : For each subspace (style/content), we plot the distribution of average pairwise Pearson correlations computed within sets of images grouped by shared style or content, across orthogonal transforms for multiple vision models.

Figure E10 : For each subspace (style/content), we plot the distribution of average pairwise Pearson correlations computed within sets of images grouped by shared style or content, across permutation transforms for multiple vision models.

Figure E11 : Top-K content retrieval and Top-K style retrieval evaluated across content and style subspaces under invertible transforms for multiple vision models.

Figure E12 : Top-K content retrieval and Top-K style retrieval evaluated across content and style subspaces under orthogonal transforms for multiple vision models.

Figure E13 : Top-K content retrieval and Top-K style retrieval evaluated across content and style subspaces under permutation transforms for multiple vision models.

Figure E14 : We project the original visual representations from the punultimate layer of Swin Base, along with the dissociated content/style subspaces and residual subspaces, into 2D UMAPS to visualize how their manifolds are organized. Markers with the same color represent images with the same content, and markers with the same shape represent images with the same style. We observe that images sharing the same functional attribute form coherent, well-defined clusters within the corresponding functional subspace manifold, indicating that the decomposition induces meaningful geometric organization aligned with the target attributes.

Swin Base
Transformation	Top-1 CM $\uparrow$	Top-5 CM $\downarrow$	Top-10 CM $\uparrow$	Top-1 SM $\uparrow$	Top-5 SM $\uparrow$	Top-10 SM $\uparrow$	$R_{s}$ $\uparrow$	$R_{c}$ $\uparrow$
Permutation	0.0063	0.0080	0.0125	0.0738	0.0625	0.0546	1.1032	1.0216
Orthogonal	0.1381	0.1686	0.2435	0.7542	0.7550	0.7613	7.2348	2.5603
Invertible	0.3107	0.3675	0.4884	0.8632	0.8785	0.8940	16.0815	5.2594
Swin Small
Permutation	0.0138	0.0141	0.0180	0.0667	0.0633	0.0723	1.1016	1.0777

Table E11 : Comparison of Top-K content and style summary metrics, along with content and style selectivity ratios, across permutation, orthogonal, and invertible transformations across different vision models

Figure E15 : Top-

K

exact-category and broad-category retrieval for each silhouette image representation derived from edge-based inputs from Geirhos et al. [2018] across multiple vision models under invertible transforms

Figure E16 : (A) Top-K Semantic retrieval; (B) Top-K Syntax retrieval for Pythia 160M obtained using invertible transforms

Figure E17 : (A) Top-K Semantic retrieval; (B) Top-K Syntax retrieval for Pythia GPT2-XL obtained using invertible transforms

Transformation	Top-1 $S_{1}$ $\uparrow$	Top-5 $S_{1}$ $\uparrow$	Top-10 $S_{1}$ $\uparrow$	Top-1 $S_{2}$ $\uparrow$	Top-5 $S_{2}$ $\uparrow$	Top-10 $S_{2}$ $\uparrow$
Pythia 160M
Permutation	0.1707	0.1418	0.1271	0.0357	0.0563	0.0861
Orthogonal	0.9540	0.9058	0.8580	0.5574	0.5460	0.5259
Invertible	0.9806	0.9565	0.9536	0.7970	0.7955	0.7500
Pythia 410M
Permutation	0.3333	0.3953	0.3971	0.1564	0.1681	0.1368

Table E12 : Comparison of Top-K Syntax and Semantics summary metrics across permutation, orthogonal, and invertible transformations for Pythia 160M, Pythia 410M and GPT2-XL.

Figure E18 : (A) Top-K Semantic retrieval; (B) Top-K Syntax retrieval across different layers of GPT2-XL obtained using invertible transforms

Asset	Type	License / terms of use
ImageNet	Dataset	Official ImageNet access agreement / terms of use (not a standard open license).
THINGS	Dataset	Original THINGS images are for academic use only; the official site recommends using THINGSplus license-free images for publications.
MS-COCO	Dataset	Official COCO Terms of Use; images are subject to Flickr terms of use, and COCO annotations / website are distributed under CC BY 4.0.
Flickr8k-Expert	Dataset	Available for non-commercial academic research (images subject to original Flickr8k academic request terms).
NSD	Dataset	Access subject to the NSD Data Access Agreement.
Torchvision pretrained vision models	Model	BSD-3-Clause (official torchvision repository; includes standard pretrained models such as ResNet, VGG, AlexNet, and torchvision vision transformers where used).

Table E13 : Licenses and terms of use for existing assets used in this work.

Modular Neural Networks

Modular neural networks organize computation into specialized, reusable components that can be routed, composed, or updated independently.

See all 74 papers

Sparse Competition during Training For the Emergence of Specialized Modules
Aug 31, 2026Baptiste Rossigneux, Karim HarounSparse Neural Networks Neural Network Training Dynamics
Univ. Rennes · Inria, IRISA · University of Paris 8
Sparsity-driven emergence of functional modularity in multitask neural networks
Oct 23, 2026Bowen Shi, Qi Ouyang, Long QianModular Neural Networks Multi-Task Learning
Institute for Advanced Study in Physics, Zhejiang University, Hangzhou 310058, China · Center for Quantitative Biology, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China · ZJU–Hangzhou Global Scientific and Technological Innovation Center, Hangzhou 311200, China +1
Neural Functional Alignment Space: Brain-Referenced Representation of Artificial Neural Networks
Feb 28, 2026Ruiyu Yan, Hanqi Jiang, Yi Pan +4Neural Representation Geometry Cross-Modal Alignment
Tandon School of Engineering, New York University, Brooklyn, NY 11201, USA · School of Computing, University of Georgia, Athens, GA 30602, USA · Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA +1

Aug 31, 2026J/K move · Enter open · S save

Sparse Competition during Training For the Emergence of Specialized Modules

Baptiste Rossigneux, Karim Haroun

Univ. Rennes · Inria, IRISA · University of Paris 8

Sparse Neural Networks Neural Network Training Dynamics Modular Neural Networks

Open paper AskPDF arXiv

Modularity in deep neural networks has been proposed as a means of improving both interpretability and training by promoting disentangled representations and reducing redundancy. In this work, we study the emergence of modular structure through competition dynamics between groups of neurons during training. We introduce a method that (i) maintains near-baseline accuracy, (ii) induces usage-based modularity by sparsely routing inputs to neuron groups, and (iii) encourages specialization of these modules, such that their activations are correlated with input classes. We evaluate the proposed approach on ImageNet-100 and CIFAR-100 and show that with it, specialized modules emerge without module-level supervision. These modules capture a meaningful high-level structure in the data, with individual modules responding to semantic categories (e.g., dogs or vehicles). We also study the emergence of a hierarchical partition of sub-tasks depending on the number of modules. Our results suggest that competitive dynamics can serve as a simple mechanism for inducing functional modularity in standard architectures.

Abstract

Modular Neural Networks

Explore similar work

Sparse Competition during Training For the Emergence of Specialized Modules

Sparsity-driven emergence of functional modularity in multitask neural networks

Neural Functional Alignment Space: Brain-Referenced Representation of Artificial Neural Networks