Select, Hypothesize and Verify: Towards Verified Neuron Concept Interpretation

ZeBin Ji; Yang Hu; Xiuli Bi; Bo Liu; Bin Xiao

cs.CVMar 26, 2026

Select, Hypothesize and Verify: Towards Verified Neuron Concept Interpretation

Authors: ZeBin Ji, Yang Hu, Xiuli Bi, Bo Liu, Bin Xiao

Organizations: Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China · Jinan Inspur Data Technology Co., Ltd., Jinan, China

Abstract

It is essential for understanding neural network decisions to interpret the functionality (also known as concepts) of neurons. Existing approaches describe neuron concepts by generating natural language descriptions, thereby advancing the understanding of the neural network's decision-making mechanism. However, these approaches assume that each neuron has well-defined functions and provides discriminative features for neural network decision-making. In fact, some neurons may be redundant or may offer misleading concepts. Thus, the descriptions for such neurons may cause misinterpretations of the factors driving the neural network's decisions. To address the issue, we introduce a verification of neuron functions, which checks whether the generated concept highly activates the corresponding neuron. Furthermore, we propose a Select-Hypothesize-Verify framework for interpreting neuron functionality. This framework consists of: 1) selecting activation samples that best capture a neuron's well-defined functional behavior through activation-distribution analysis; 2) forming hypotheses about concepts for the selected neurons; and 3) verifying whether the generated concepts accurately reflect the functionality of the neuron. Extensive experiments show that our method produces more accurate neuron concepts. Our generated concepts activate the corresponding neurons with a probability approximately 1.5 times that of the current state-of-the-art method.

Figures & tables

Figure 1: Comparison of previous methods and our proposed method with verification. The neurons are sampled from the second-to-last layer of ResNet-18. Existing methods assume that concepts inferred from activated neurons are all accurate, while our proposed method introduces verification to identify incorrect ones. Specifically, for neuron 162, both the concepts (Small Round Beards and Curly Dense Coat) are inferred from high-activation images. Verification shows that the activation rate for small round whiskers (0.26) suggests this concept may be incorrect, while curly dense hair achieves 0.98, confirming the correctness of the concept.

Figure 2: Overview of SIEVE. 1) Select: For each sample, we obtain the normalized activation of neuron $i$ from the network’s penultimate layer. Samples with activations above a predefined threshold are identified as high-activation samples. 2) These high-activation samples are clustered, and a vision-language model is used to generate the hypothesized concepts for each cluster. 3) Each concept is converted into a semantic image using text-to-image generation. The generated images are fed into the target model to measure the activation rate of neuron $i$ , thereby verifying the semantic features that the neuron reliably encodes.

Figure 3: Activation distributions and Concepts of high-discrimination neurons (Neuron 507) and low-discrimination neurons (Neuron 144). The high activation of high-discrimination neurons exhibits consistent patterns while the high activation of low-discrimination neurons fails to do so.

Figure 4: Functional descriptions of penultimate-layer neurons in ResNet-50 and ViT-B/16, generated by SIEVE, CLIP-Dissect [ 21 ] and WWW [ 1 ] . SIEVE provides more complete and accurate semantic explanations. It captures localized features and multiple concepts, rather than only broad attributes such as object category or color. For example, ViT-B/16 neuron 37 is described by SIEVE with specific cues like Short Dense Coat, whereas CLIP-Dissect and WWW may omit some relevant concepts or provide only coarse labels such as Dog. Overall, the neuron-level characterizations produced by SIEVE are more fine-grained and validated.

Method	$\bm{\mathcal{D}_{probe}}$	$\bm{\mathcal{D}_{concept}}$	Final Layer		Penultimate Layer
Method	$\bm{\mathcal{D}_{probe}}$	$\bm{\mathcal{D}_{concept}}$	CLIP cos	mpnet cos	mean AR (%)
Network Dissect	Broden	Broden	0.7073	0.3256	45.01
MILAN(b)	ImageNet val	–	0.7192	0.3089	46.10
FALCON	ImageNet val	LAION-400m	0.7031	0.2193	46.32
DnD	ImageNet val	GPT-3.5 Turbo	0.7595	0.4371	51.46
CLIP-Dissect	ImageNet val	Broden (1.2k)	0.7942	0.4543	55.16

Table 1: Quantitative comparison of neuron interpretation methods of ResNet-50 pre-trained on ImageNet-1K, using the ImageNet-1K validation set as

D_{probe}

. We evaluate the metrics proposed by CLIP-Dissect [ 21 ] in the final layer, and the mean Activation Rate in the penultimate layer. Bold numbers indicate the best results. We use the Common Words as the concept set following CLIP-Dissect.

Method	$\bm{\mathcal{D}_{probe}}$	$\bm{\mathcal{D}_{concept}}$	Final Layer		Penultimate Layer
Method	$\bm{\mathcal{D}_{probe}}$	$\bm{\mathcal{D}_{concept}}$	CLIP cos	mpnet cos	mean AR (%)
FALCON	ImageNet val	LAION-400m	0.7105	0.2281	45.95
DnD	ImageNet val	GPT-3.5 Turbo	0.7462	0.4354	52.47
CLIP-Dissect	ImageNet val	Broden (1.2k)	0.7651	0.4519	56.21
CLIP-Dissect	ImageNet val	Common Words (3k)	0.7538	0.4437	57.70
WWW	ImageNet val	Broden (1.2k)	0.7455	0.4261	49.19

Table 2: Quantitative comparison of neuron interpretation methods of ViT-B/16 pre-trained on ImageNet-1K.

Method	$\bm{\mathcal{D}_{probe}}$	$\bm{\mathcal{D}_{concept}}$	Final Layer		Penultimate Layer
Method	$\bm{\mathcal{D}_{probe}}$	$\bm{\mathcal{D}_{concept}}$	CLIP cos	mpnet cos	mean AR (%)
FALCON	ImageNet val	LAION-400m	0.6975	0.2203	47.15
DnD	Places365 test	GPT-3.5 Turbo	0.7458	0.4419	53.12
CLIP-Dissect	Places365 test	Broden (1.2k)	0.7761	0.4462	58.32
CLIP-Dissect	Places365 test	Common Words (3k)	0.7883	0.4528	56.90
WWW	Places365 test	Broden (1.2k)	0.7617	0.4587	52.23

Table 3: Quantitative comparison of neuron interpretation methods of ResNet-18 pre-trained on Places365, using the Places365 test set as

D_{probe}

. We evaluate the metrics proposed by CLIP-Dissect [ 21 ] in the final layer, and the mean Activation Rate in the penultimate layer. Bold numbers indicate the best scores under the same settings.

Select	Cluster	Verify	Final Layer		Penultimate Layer
Select	Cluster	Verify	CLIP cos	mpnet cos	mean AR (%)
			0.6738	0.2306	45.57
	✓	✓	0.7624	0.4301	77.90
✓		✓	0.7821	0.4423	81.52
✓	✓		0.7656	0.4189	72.87
✓	✓	✓	0.7914	0.4547	86.29

Table 4: Ablation study on Select, Cluster, and Verify components. Results show that all modules contribute positively to interpretation quality, with the full framework achieving the best performance across CLIP cos, mpnet cos and mean Activation Rate.

$\beta$	Final Layer		Penultimate Layer
$\beta$	CLIP cos	mpnet cos	mean AR (%)
4	0.7932	0.4421	86.05
6	0.7987	0.4517	85.61
8	0.7805	0.4432	85.72
10	0.7914	0.4547	86.29
12	0.7821	0.4478	85.98

Table 5: Performance under different activation thresholds

\beta

. The model achieves the best results at 10, which is therefore adopted throughout the experiments in this paper.

Method	Eurosat (domain shift)		Places365 (general)
Method	CLIP cos	mean AR (%)	CLIP cos	mean AR (%)
CLIP-Dis.	0.7152	43.16	0.7883	56.90
WWW	0.7047	37.43	0.7526	46.79
Ours	0.7393	75.45	0.7833	84.40

Table 6: Performance under domain shift. Eurosat represents a distribution significantly different from the generative model’s training data, while Places365 represents a general domain.

Figure 5: Visualization of the domain shift effect. Significant discrepancies exist between aerial and generated samples.

Figure 6: The heatmap (top) shows the part of the image that activates Neuron 39 the most, which is the cat’s eyes. The bar plot (bottom) quantifies the ablation effect, showing a targeted accuracy drop for semantically related feline categories.

Explore similar work

ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations
Apr 8, 2026Ricardo Knauer, Andre Beinrucker, Erik RodnerNeural Network Interpretability Mechanistic Interpretability
KI-Werkstatt, University of Applied Sciences Berlin, Berlin, Germany
LINE: LLM-based Iterative Neuron Explanations for Vision Models
Apr 9, 2026Vladimir Zaigrajew, Michał Piechota, Gaspar Sekula +2Neural Network Interpretability
Centre for Credible AI Warsaw University of Technology · Warsaw University of Technology Warsaw, Poland · Centre for Credible AI, Warsaw University of Technology University of Warsaw, Poland
A Case Study on Concept Induction for Neuron-Level Interpretability in CNN
Feb 27, 2026Moumita Sen Sarma, Samatha Ereshi Akkamahadevi, Pascal HitzlerNeural Network Interpretability Concept-Based Explanations
Department of Computer Science, Kansas State University, Manhattan, Kansas, USA

Apr 8, 2026J/K move · Enter open · S save

ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations

Ricardo Knauer, Andre Beinrucker, Erik Rodner

KI-Werkstatt, University of Applied Sciences Berlin, Berlin, Germany

Neural Network Interpretability Mechanistic Interpretability Concept-Based Explanations

Open paper AskPDF arXiv

Neural networks deliver impressive predictive performance across a variety of tasks, but they are often opaque in their decision-making processes. Despite a growing interest in mechanistic interpretability, tools for systematically exploring the representations learned by neural networks in general, and tabular foundation models in particular, remain limited. In this work, we introduce ConceptTracer, an interactive application for analyzing neural representations through the lens of human-interpretable concepts. ConceptTracer integrates two information-theoretic measures that quantify concept saliency and selectivity, enabling researchers and practitioners to identify neurons that respond strongly to individual concepts. We demonstrate the utility of ConceptTracer on representations learned by TabPFN and show that our approach facilitates the discovery of interpretable neurons. Together, these capabilities provide a practical framework for investigating how neural networks like TabPFN encode concept-level information. ConceptTracer is available at https://github.com/ml-lab-htw/concept-tracer.

Abstract

Explore similar work

ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations

LINE: LLM-based Iterative Neuron Explanations for Vision Models

A Case Study on Concept Induction for Neuron-Level Interpretability in CNN