SPICE: Simple Polysemantic Feature Interpretation via Clustering-based Explanation

Sehyun Lee; Dahee Kwon; Damin Lee; Jaesik Choi

cs.LGAug 14, 2026

SPICE: Simple Polysemantic Feature Interpretation via Clustering-based Explanation

Authors: Sehyun Lee, Dahee Kwon, Damin Lee, Jaesik Choi

Organizations: Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea · INEEJI, Seongnam, Republic of Korea

Abstract

One of the pivotal recent challenges in neural network interpretability is polysemanticity, where a single neuron is activated by multiple, often unrelated concepts, hindering clear functional understanding. Although prior work has explored this phenomenon, existing approaches remain architecture-specific and depend on manual heuristics such as a fixed number of concept clusters ( $K$ ), limiting their generality and scalability--especially for modern Transformer-based models. To address these limitations, we introduce SPICE (Simple Polysemantic Feature Interpretation via Clustering-based Explanation), a generalizable framework for analyzing polysemanticity in deep vision architectures. SPICE avoids architecture-dependent propagation rules, enabling the first systematic comparison of polysemanticity across both CNNs and Transformers, and automatically determines the number of concept clusters per neuron, eliminating reliance on a preset $K$ and supporting scalable analysis for large models. Using SPICE, we conduct a comprehensive investigation into how polysemanticity emerges, varies across depth and architecture, and forms through distinct computational pathways.

Figures & tables

Figure 1: An overview of the proposed method (SPICE). (a) A polysemantic neuron exhibits mixed selectivity across diverse image inputs. (b) SPICE traces the computational pathways of these activations by computing attribution footprints ( $\mathcal{A}_{u}$ ) from the preceding layer. (c) We iteratively cluster these footprints to disentangle the neuron’s representations. By strictly separating clusters that satisfy the condition $\text{Cohesion}(C)>\tau$ , SPICE successfully decomposes the entangled inputs into highly coherent, interpretable conceptual groups.

Model / Layer	PURE ∗	LE	CPE	Ours
ResNet-50 L2.1	$1.018{\tiny{}_{\pm 0.031}}$	$\textbf{1.112}{\tiny{}_{\pm 0.122}}$	$0.996{\tiny{}_{\pm 0.014}}$	$1.092{\tiny{}_{\pm 0.056}}$
ResNet-50 L3.3	$0.999{\tiny{}_{\pm 0.012}}$	$1.108{\tiny{}_{\pm 0.153}}$	$0.995{\tiny{}_{\pm 0.007}}$	$\textbf{1.160}{\tiny{}_{\pm 0.081}}$
ResNet-50 L4.2	$1.009{\tiny{}_{\pm 0.071}}$	$1.187{\tiny{}_{\pm 0.221}}$	$0.998{\tiny{}_{\pm 0.011}}$	$\textbf{1.291}{\tiny{}_{\pm 0.224}}$
ViT-B B2	$1.002{\tiny{}_{\pm 0.010}}$	$\textbf{1.199}{\tiny{}_{\pm 0.181}}$	$0.994{\tiny{}_{\pm 0.009}}$	$1.137{\tiny{}_{\pm 0.062}}$
ViT-B B6	$1.012{\tiny{}_{\pm 0.013}}$	$1.185{\tiny{}_{\pm 0.153}}$	$0.994{\tiny{}_{\pm 0.009}}$	$\textbf{1.240}{\tiny{}_{\pm 0.089}}$
ViT-B B11	$1.035{\tiny{}_{\pm 0.025}}$	$1.357{\tiny{}_{\pm 0.181}}$	$0.998{\tiny{}_{\pm 0.018}}$	$\textbf{1.577}{\tiny{}_{\pm 0.089}}$

Table 1: Separability comparison of baselines versus Ours across models and layers. As PURE was originally designed for CNNs, we adapted its CRP implementation for Transformer models for this comparison.

Figure 2: Simulation-based Semantic Evaluation in ViT-B

Figure 3: Qualitative comparisons with various baselines. Each row shows four exemplar images representing the visual characteristics of a discovered cluster, accompanied by its corresponding textual description.

Panel of Figure 4 — Figure 4: Dissecting a polysemantic neuron (#652) from ViT-B block 11. (a) UMAP embedding reveals distinct conceptual clusters within the neuron’s representation footprints. (b) A similarity heatmap quantifies the low inter-cluster and high intra-cluster similarity. (c) A CLIP similarity heatmap quantitatively shows that concepts are semantically disparate. (d) Exemplar images confirm the diverse and disparate visual concepts represented by each cluster; such as snakeskin (C1), penguins (C2), and various two-wheeled vehicles (C3-C5).

Figure 5: Top 20 co-occurrence concept pairs in ViT-B.

Figure 6: ( Left ) Inter-cluster CLIP similarity, where the x-axis denotes neurons in ViT-B block 11 sorted by similarity. ( Middle , Right ) Example of a polysemantic neuron with low inter-cluster similarity, and a high inter-cluster similarity, respectively.

Figure 7: Polysemanticity differs across network architectures. (Left) The number of concepts per neuron across layers. (Center) Intra-cluster similarity across layers. (Right) A structural evolution where the similarity distribution shifts from unimodal to bimodal.

Method	B2			B4			B6			B8			B10
Method	T	M	B	T	M	B	T	M	B	T	M	B	T	M	B
PURE	1.02	1.00	1.02	1.02	1.00	1.02	1.03	1.00	1.01	1.03	1.00	1.01	1.07	1.00	1.01
CPE	0.99	0.99	0.99	1.01	0.99	0.99	0.99	0.99	1.00	1.01	0.99	0.99	1.01	0.99	0.99
LE	1.20	1.11	1.12	1.15	1.15	1.13	1.19	1.18	1.13	1.21	1.14	1.13	1.33	1.13	1.04
Ours	1.14	1.09	1.12	1.19	1.14	1.16	1.24	1.18	1.21	1.20	1.31	1.29	1.48	1.27	1.41

Table 2: Separability of ViT-B/16 across activation ranges (Top/Mid/Bottom), showing the same depth-wise trend as Table 1 .

Figure 8: Robustness to attribution methods. SPICE consistently outperforms PURE across Integrated Gradients, GradientShap, Guided Backpropagation, and Input $\times$ Gradient.

	ResNet-50 (L3.3)		ViT-B (B6)
Variant	Score	Coverage	Score	Coverage
Fixed $K\!=\!2$ (no $\tau$ )	1.00	100/100	1.03	100/100
Static $\tau\geq 0.9$	1.56	6/100	2.18	7/100
Static $\tau\geq 0.8$	1.56	9/100	2.18	10/100
Static $\tau\geq 0.7$	1.39	27/100	1.88	21/100
Adaptive $\tau$ (SPICE)	1.15	100/100	1.25	100/100

Table 3: Adaptive vs. static thresholding. For each variant we report the separability score and the coverage (# of 100 randomly sampled neurons yielding at least one cohesive cluster). Fixed

K\!=\!2

accepts every cluster (no cohesion threshold

\Rightarrow

100% coverage by construction). Static

\tau

applies a global cosine-cohesion threshold; high

\tau

inflates separability but drops coverage to near zero because most neurons cannot meet the threshold. Adaptive (SPICE) sets

\tau

per neuron, achieving near-100% coverage with strong separability.

Panel of Figure 9 — Figure 9: Comparison of concept clusters under different attribution methods for ViT-B/16 and ResNet-50.

Attribution	B2	B6	B11
Input $\times$ Gradient (default)	1.137	1.240	1.577
LeGrad (layer-conditional)	1.111	1.196	1.493

Table 4: Compatibility of SPICE with different attribution backbones at ViT-B/16 (separability

\uparrow

Figure 10: Sensitivity analysis on the threshold of the last layer/block of ViT-B/16 and ResNet-50. Left (a) shows the number of concept clusters, and right (b) shows the attribution similarity of concept clusters at different thresholds.

Model	Layer	Normalization	Separability
ResNet-50	L2.1		1.058
ResNet-50	L2.1	✓	1.092
ResNet-50	L4.2		1.017
ResNet-50	L4.2	✓	1.293
ViT-B	B2		1.065
ViT-B	B2	✓	1.137

Table 5: Model Layer Attribution Normalization and Separability

Configuration	Sep.
$K\!=\!2$ , CRP ( $\approx$ PURE)	1.036
$K\!=\!2$ , IxG	1.027
Adaptive $K$ , IxG	1.308
+ Normalization ( $=$ SPICE)	1.577

Table 6: Ablation at ViT-B/16 block 11 (separability

\uparrow

). Left : stepwise PURE

\to

SPICE. Right : fixed-

K

baselines vs. adaptive cluster discovery.

Figure 11: Ablation study of the clustering components in our method, comparing the use of ISODATA, HDBSCAN, and K-Means for computing cluster means. The last column shows the default version of our algorithm, which employs iterative K-Means.

Figure 12: Changes in the distribution of attribution similarity across ViT-B’s blocks, shown from the first block (top-left) to the last block (bottom-right).

Figure 13: Changes in the distribution of attribution similarity across ResNet-50’s layers, shown from the first layer (top-left) to the last layer (bottom-right).

Depth	ViT-Base	ResNet50
Layer 1	$0.98\pm 0.06$	$0.95\pm 0.17$
Layer 2	$0.98\pm 0.07$	$0.97\pm 0.13$
Layer 3	$0.98\pm 0.09$	$0.96\pm 0.11$
Layer 4	$0.98\pm 0.06$	$0.97\pm 0.12$
Layer 5	$0.97\pm 0.09$	$0.97\pm 0.08$
Layer 6	$0.98\pm 0.06$	$0.97\pm 0.08$

Table 7: Comparison of distribution statistics between ViT-Base and ResNet-50. ResNet-50 shows consistently high Hartigan’s Dip Test

p

-values (

>0.9

), indicating stable unimodal distributions across all depths. In contrast, ViT-Base exhibits a precipitous drop in mean

p

-values accompanied by exceptionally high variance in its final layers ( bold ). This broad distribution indicates the emergence of a substantial subpopulation of highly multimodal (polysemantic) neurons, rather than a universal shift across the entire layer. Depth is measured in a block-wise or layer-wise manner, depending on the model architecture.

Sample Size ( $N$ )	Observed Time (s)	Complexity Check ( $\text{Time}/(N\log_{10}N)$ )
100	1.45	0.003149
200	4.63	0.004369
500	1.89	0.000608
1,000	3.01	0.000436
2,000	5.94	0.000391
5,000	17.71	0.000416

Table 8: Computational Efficiency of Iterative Divisive Clustering

Figure 14: Training accuracy plotted against training steps during transfer learning of ViT-L. The left panel displays Top-1 and Top-5 classification accuracy. The middle and right panels illustrate the attribution similarity and the average number of concept clusters, respectively.

Figure 15: Visualization of concept expansion in Neuron #550. The left panel shows the top-activating concepts at initialization, while the right panel shows the new concepts added after fine-tuning.

Model	ImageNet	YFCC100M	Places365
ViT-B/16 B11	1.577 $\pm$ 0.094	1.359 $\pm$ 0.133	1.415 $\pm$ 0.123
CLIP ViT-B/16 B11	1.410 $\pm$ 0.102	1.396 $\pm$ 0.103	1.423 $\pm$ 0.100

Table 9: Separability of SPICE at the last block of ViT-B/16 and CLIP ViT-B/16 across three probing datasets.

Figure 16: Examples of polysemantic neurons of ViT-B with different blocks. Each row in a neuron represents a distinct concept cluster.

Figure 17: Examples of polysemantic neurons of ResNet-50 with different blocks. Each row in a neuron represents a distinct concept cluster.

Figure 18: Examples of polysemantic neurons of ConvNeXt-B with different blocks. Each row in a neuron represents a distinct concept cluster.

Neural Network Interpretability

Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.

See all 330 papers

Interpretability Without Tradeoffs: Disentangling Polysemanticity At Equal Predictive Performance
May 29, 2026Doğukan Bağcı, Bernt Schiele, Simone Schaub-Meyer +2Disentangled Representation Learning Representation Disentanglement
Max Planck Institute for Informatics, SIC · Department of Computer Science, TU Darmstadt
Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability
Oct 14, 2026Anton Korznikov, Alexey Dontsov, Elena TutubalinaMechanistic Interpretability Neuron-Level Interpretability
Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision Transformers
Mar 16, 2026Siyu ZhangNeural Network Interpretability Transformer Interpretability
University of Texas at Austin, Austin TX 78712, USA

May 29, 2026J/K move · Enter open · S save

Interpretability Without Tradeoffs: Disentangling Polysemanticity At Equal Predictive Performance

Doğukan Bağcı, Bernt Schiele, Simone Schaub-Meyer, Jonas Fischer+1

Max Planck Institute for Informatics, SIC · Department of Computer Science, TU Darmstadt

Disentangled Representation Learning Representation Disentanglement Neural Network Interpretability Neuron-Level Interpretability

Open paper AskPDF arXiv

Deep neural networks (DNNs) are widely used, but interpreting what they actually learn remains difficult. A major obstacle is that individual neurons often encode multiple unrelated concepts, obscuring the decision process of the network. While prior work, such as sparse autoencoders, can separate these mixed signals into more meaningful, "monosemantic" features, this typically requires altering the model in ways that can degrade downstream performance. To overcome this, we introduce ELUDe (explicit, lossless, unsupervised disentanglement), a method for improving the interpretability of DNNs while preserving their functional equivalence. ELUDe breaks latent representations into clear, inspectable sub-units that behave like interpretable features, while guaranteeing that the model's outputs remain exactly the same. It requires no explicit training, no labels, and can be applied to pretrained models. ELUDe works by reorganizing how information flows between layers, re-routing concept-specific contributions while preserving the original computation by construction. Across several vision models, including DINOv2 and supervised ViT-B/16, ELUDe improves interpretability, keeps downstream accuracy unchanged, runs efficiently, and supports practical uses such as steering model representations. In short, ELUDe offers interpretability (almost) without a tradeoff: clearer, scalable, and actionable model insights with no loss in performance.

Abstract

Neural Network Interpretability

Explore similar work

Interpretability Without Tradeoffs: Disentangling Polysemanticity At Equal Predictive Performance

Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability

Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision Transformers