A Case Study on Concept Induction for Neuron-Level Interpretability in CNN

Moumita Sen Sarma; Samatha Ereshi Akkamahadevi; Pascal Hitzler

cs.CVFeb 27, 2026

A Case Study on Concept Induction for Neuron-Level Interpretability in CNN

Authors: Moumita Sen Sarma, Samatha Ereshi Akkamahadevi, Pascal Hitzler

Organizations: Department of Computer Science, Kansas State University, Manhattan, Kansas, USA

Abstract

Deep Neural Networks (DNNs) have advanced applications in domains such as healthcare, autonomous systems, and scene understanding, yet the internal semantics of their hidden neurons remain poorly understood. Prior work introduced a Concept Induction-based framework for hidden neuron analysis and demonstrated its effectiveness on the ADE20K dataset. In this case study, we investigate whether the approach generalizes by applying it to the SUN2012 dataset, a large-scale scene recognition benchmark. Using the same workflow, we assign interpretable semantic labels to neurons and validate them through web-sourced images and statistical testing. Our findings confirm that the method transfers to SUN2012, showing its broader applicability.

Figures & tables

Neuron ID	ECII Concepts	Coverage Score	TLA %	Non-TLA %
0	snowy_mountain	0.986	95.00	52.57
7	snow, mountain	0.998	80.00	39.55
9	field	0.971	93.75	74.96
13	dish_rack	0.892	81.25	31.12
15	pillow, ceiling_fan	0.992	88.75	57.15
16	skyscraper, river_water	0.997	80.00	42.05

Table 1: Label confirmation via Google Images results with confirmed neurons (TLA %

\geq

80).

Neuron ID	ECII Concepts	TLA %	Non-TLA %	Target Median	Non-Target Median	Target Mean	Non-Target Mean	$z$ -score	$p$ -value
0	snowy_mountain	95	53.02	5.99	0.23	5.58	1.22	-6.18	$<0.00001$
7	snow, mountain	70	39.68	2.10	0.00	1.74	0.68	-3.04	$0.00061$
9	field	100	75.48	4.38	1.43	4.03	1.97	-3.64	$0.00025$
13	dish_rack	90	32.62	1.65	0.00	1.68	0.56	-4.66	$<0.00001$
15	pillow, ceiling_fan	90	54.21	3.03	0.21	2.36	0.85	-4.40	$<0.00001$
16	skyscraper, river_water	70	43.41	1.54	0.00	1.44	0.45	-3.15	$0.00052$

Table 2: Statistical evaluation results. Bold rows represents neurons with p-value

\geq

0.05, where we cannot reject the null hypothesis .

Explore similar work

Select, Hypothesize and Verify: Towards Verified Neuron Concept Interpretation
Mar 26, 2026ZeBin Ji, Yang Hu, Xiuli Bi +2Neural Network Interpretability Concept-Based Explanations
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China · Jinan Inspur Data Technology Co., Ltd., Jinan, China
ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations
Apr 8, 2026Ricardo Knauer, Andre Beinrucker, Erik RodnerNeural Network Interpretability Mechanistic Interpretability
KI-Werkstatt, University of Applied Sciences Berlin, Berlin, Germany
Interpretability Without Tradeoffs: Disentangling Polysemanticity At Equal Predictive Performance
May 29, 2026Doğukan Bağcı, Bernt Schiele, Simone Schaub-Meyer +2Disentangled Representation Learning Representation Disentanglement
Max Planck Institute for Informatics, SIC · Department of Computer Science, TU Darmstadt

Mar 26, 2026J/K move · Enter open · S save

Select, Hypothesize and Verify: Towards Verified Neuron Concept Interpretation

ZeBin Ji, Yang Hu, Xiuli Bi, Bo Liu+1

Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China · Jinan Inspur Data Technology Co., Ltd., Jinan, China

Neural Network Interpretability Concept-Based Explanations

Open paper AskPDF arXiv

It is essential for understanding neural network decisions to interpret the functionality (also known as concepts) of neurons. Existing approaches describe neuron concepts by generating natural language descriptions, thereby advancing the understanding of the neural network's decision-making mechanism. However, these approaches assume that each neuron has well-defined functions and provides discriminative features for neural network decision-making. In fact, some neurons may be redundant or may offer misleading concepts. Thus, the descriptions for such neurons may cause misinterpretations of the factors driving the neural network's decisions. To address the issue, we introduce a verification of neuron functions, which checks whether the generated concept highly activates the corresponding neuron. Furthermore, we propose a Select-Hypothesize-Verify framework for interpreting neuron functionality. This framework consists of: 1) selecting activation samples that best capture a neuron's well-defined functional behavior through activation-distribution analysis; 2) forming hypotheses about concepts for the selected neurons; and 3) verifying whether the generated concepts accurately reflect the functionality of the neuron. Extensive experiments show that our method produces more accurate neuron concepts. Our generated concepts activate the corresponding neurons with a probability approximately 1.5 times that of the current state-of-the-art method.

Abstract

Explore similar work

Select, Hypothesize and Verify: Towards Verified Neuron Concept Interpretation

ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations

Interpretability Without Tradeoffs: Disentangling Polysemanticity At Equal Predictive Performance