What is Missing? Explaining Neurons Activated by Absent Concepts
Authors: Robin Hesse, Simone Schaub-Meyer, Janina Hesse, Bernt Schiele, Stefan Roth
Organizations: Max Planck Institute for Informatics, SIC · Department of Computer Science, Technical University of Darmstadt · Leibniz Institute for Resilience Research · Institute for Quantitative and Computational Biosciences, Johannes Gutenberg University Mainz · University Medical Center Mainz
Explainable artificial intelligence (XAI) aims to provide human-interpretable insights into the behavior of deep neural networks (DNNs), typically by estimating a simplified causal structure of the model. In existing work, this causal structure often includes relationships where the presence of a concept is associated with a strong activation of a neuron. For example, attribution methods primarily identify input pixels that contribute most to a prediction, and feature visualization methods reveal inputs that cause high activation of a target neuron - the former implicitly assuming that the relevant information resides in the input, and the latter that neurons encode the presence of concepts. However, a largely overlooked type of causal relationship is that of encoded absences, where the absence of a concept increases neural activation. In this work, we show that such missing but relevant concepts are common and that mainstream XAI methods struggle to reveal them when applied in their standard form. To address this, we propose two simple extensions to attribution and feature visualization techniques that uncover encoded absences. Across experiments, we show how mainstream XAI methods can be used to reveal and explain encoded absences, how ImageNet models exploit them, and that debiasing can be improved when considering them.
Figures & tables
Figure 1: Encoded absence in image classification. (a) The model detects concepts present in the input image that are prototypical for the target class ( e.g. , the snout and feet). (b) The model can additionally encode the absence of snouts from other dog species to enhance evidence for the “Irish setter” class.
x~x~∧∧
¬x^¬x^
Figure 2: A mechanistic process to encode the absence of a concept. A neuron encoding the absence of concept x^ ( i.e. ¬x^ ) can be implemented by having a negative connection to a neuron encoding x^ and a positive potential through, e.g. , another activating concept x~ ( i.e. , the output encodes x~∧¬x^ ).
Figure 3: Hassenstein–Reichardt detector experiment. (a) Two example sequences showing a left-to-right and bi-directional movement. (b) A hand-crafted CNN to distinguish left-to-right motion from bi-directional motion. The first layer implements the spatio-temporal comparison of neighboring pixels, the second layer compares motion in opposing directions, followed by global average pooling (GAP). The first output node implements a Hassenstein-Reichardt detector (weights: 1/-1) and the second output averages both directions (weights: 0.5/0.5). (c) The outputs of the model for the two example sequences. (d) Visualizations of established XAI methods – target attribution and feature visualization via the highest activating patches, each consisting of two consecutive frames as CNN input (numbers indicate the activation strength). Both methods fail to highlight the absence encoded in the first output and thus lack a complete explanation of the CNN mechanisms. (e) Our proposed non-target attributions and feature visualization through minimization highlight that the first output encodes the absence of right-to-left motion.
Figure 4: Toy experiment. (a) Example RGB images from class 1 (green pixel) and class 2 (no green pixel) – zoom in for better visibility. (b) Architecture of the used toy model with the trained weights. (c) Average logit output for the two output nodes for images from class 1 and 2. Confidence intervals represent two times the standard deviation. (d) Integrated Gradients ( Sundararajan et al., 2017 ) target attributions for the above example images, and maximally activating patches for the two output nodes. (e) Non-target attributions for the two respective examples – note how the attributions switch from positive (blue) to negative (red) – and minimally activating patches for the two output nodes (numbers indicate the activation strength).
Table 1: Quantitative evaluation of encoded absences. We measure the activation of the 100 highest-activating images when inserting none, random, encoded logical NOT s ( Mu & Andreas, 2020 ) , most activating ( e.g. , as used in ( Ahn et al., 2024 ) ), or least activating 48×48 patches in a random corner.
Figure 6: Encoded presences (positive potential) and absences for three channels that have been found to be important for the corresponding class. We identify channels that are important for specific classes and visualize the positive potential from this class by showing the most activating patches. Further, we show the encoded absences by showing the least activating patches. In particular, for fine-grained classification, encoded absences of patterns from related species seem to be used.
Validation split (training bias)
Validation split (inverse bias)
Validation split (no bias)
Training bias
Model
Benign ∗
Malignant
Avg.
Attr.
Benign
Malignant ∗
Avg.
Attr.
Benign
Malignant
Avg.
None
X -ResNet-50
–
–
–
–
–
–
–
–
0.84
0.77
0.81
Benign*
No debiasing
1.00
0.99
0.99
0.40
0.04
0.00
0.02
0.47
0.04
0.99
0.51
Presence debiasing
0.96
0.88
0.92
0.08
0.66
0.17
0.41
0.13
0.66
0.88
0.77
Presence+absence debiasing (ours)
0.91
0.88
0.89
0.07
0.74
0.43
0.59
0.08
0.74
0.88
0.81
Table 2: Validation results for the ISIC dataset with varying biases. We report the accuracy (average over 5 runs) for the validation split of the ISIC dataset with different biases. In the “training bias” setup, the train bias is replicated with all the benign samples containing colorful patches, while in the “inverse bias” setup, the malignant samples contain colorful patches, as indicated by ∗ . A model with no debiasing learns the dataset bias and fails to classify samples when the bias is not present. A model with presence debiasing (existing attribution priors) can reduce this bias; however, it still fails to classify malignant samples when inserting colorful patches, indicating that it is biased based on the absence of colorful patches. Our proposed presence+absence debiasing results in the highest average accuracy for both setups without the training bias, and is similarly performant as a model trained without bias, suggesting that the model is largely debiased. “Attr.” shows the relative attribution within the colorful patches, confirming qualitative results from Figure 7 .
Figure 7: Images and attributions for our biased ISIC dataset. (a) We replicate the ISIC bias (real bias) that co-occurs with the benign samples with a synthetic bias. (b) Attributions of different (de)biased models for the benign sample with a synthetic bias. The target attribution is computed for the benign output logit and the non-target attribution for the malignant output. Only including absences in the debiasing prevents the model from relying on patch absence to predict malignancy.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Simplified illustration of the Hassenstein-Reichardt detector in Drosophila . The activation of two subunits – encoding right-to-left ( R ) and left-to-right ( L ) movements – is subtracted. The output neuron encodes the presence of left-to-right movements while encoding the absence of right-to-left movements ( L∧¬R ).
Figure 10: Counterfactual visual explanation. Counterfactual visual explanation ( Goyal et al., 2019 ) for the two images shown in Figure 4 and the corresponding trained toy model. The explanation correctly highlights the green pixel (zoom in) as the concept whose swap induces a class change, thereby identifying it as the most class-discriminative concept. However, this explanation does not clarify whether the model relies on the presence of the green pixel, its absence, or both, which is possible with our proposed modifications.
Model
Patch size
Nr. images
None
+Random
+Logical NOT ( Mu & Andreas, 2020 )
+Most act.
+Least act. (ours)
VGG19 ( Simonyan & Zisserman, 2015 )
32
100
2.98 ± 1.08
2.84 ± 1.09
2.70 ± 1.04 → 2.68 ± 1.04
3.39 ± 1.10
2.14 ± 1.13
VGG19
48
100
2.98 ± 1.08
2.68 ± 1.10
2.53 ± 1.06 → 2.51 ± 1.06
3.88 ± 1.16
0.94 ± 1.16
VGG19
64
100
2.98 ± 1.08
2.41 ± 1.12
2.25 ± 1.08 → 2.21 ± 1.08
4.38 ± 1.13
-0.38 ± 1.18
VGG19
48
50
3.72 ± 1.08
3.39 ± 1.10
3.21 ± 1.06 → 3.19 ± 1.06
4.59 ± 1.12
1.66 ± 1.16
VGG19
48
200
2.25 ± 1.07
1.98 ± 1.10
1.85 ± 1.04 → 1.84 ± 1.05
3.18 ± 1.10
0.25 ± 1.14
ResNet-50 ( He et al., 2016 )
32
100
0.18 ± 0.06
0.17 ± 0.06
0.16 ± 0.06 → 0.16 ± 0.06
0.21 ± 0.06
0.12 ± 0.07
Appendix
Table 3: Quantitative evaluation of encoded absences. We report the CNN results from Figure 5 alongside their standard deviations (indicated by “ ± ”) and under different hyperparameters (patch size and number of images). Please refer to Figure 5 for a detailed description.
Model
Layer
None
+Random
+Least act. (ours)
ResNet-50
layer4[2].conv3
0.18
0.16
0.03
ResNet-50
layer3[2].conv3
0.26
0.24
0.18
ResNet-50
layer2[2].conv3
0.47
0.45
0.38
ResNet-50
layer1[2].conv3
0.08
0.07
0.02
Appendix
Table 4: Quantitative evaluation of encoded absences in different layers. We report the results from Figure 5 in different layers of a ResNet-50. Please refer to Figure 5 for a detailed description.
Figure 11: The eight least activating patches for channels 2026 and 1470. For the most and least activating patches in Section 5.3 , we obtain eight candidate patches and manually select a monosemantic subset of three patches for more interpretable visualizations. Inspecting all eight patches for channels 2026 and 1470 in ResNet-50 reveals that these channels encode the absence of multiple concepts, consistent with prior work on polysemantic neurons ( Elhage et al., 2022 ) . The corresponding labels indicate the class of each patch.
Validation split (training bias)
Validation split (inverse bias)
Validation split (no bias)
Training bias
Model
Benign ∗
Malignant
Avg.
Attr.
Benign
Malignant ∗
Avg.
Attr.
Benign
Malignant
Avg.
None
X -ResNet-50
–
–
–
–
–
–
–
–
0.84
0.77
0.81
–
–
–
–
–
–
–
–
± 0.03
± 0.07
± 0.02
Benign*
No debiasing
1.00
0.99
0.99
0.40
0.04
0.00
0.02
0.47
0.04
0.99
0.51
± 0.00
± 0.02
± 0.01
± 0.02
± 0.02
± 0.00
± 0.01
± 0.02
± 0.02
± 0.02
± 0.00
Presence debiasing
0.96
0.88
0.92
0.08
0.66
0.17
0.41
0.13
0.66
0.88
0.77
Appendix
Table 5: Validation results for the ISIC dataset with varying biases. We report the results from Table 2 together with their standard deviations (indicated by “ ± ”). Please refer to Table 2 for a detailed description of the table.
Model
Train bias (Avg.)
Inv. bias (Avg.)
No bias (Avg.)
No debiasing
0.93
0.41
0.82
Presence debiasing
0.92
0.46
0.83
Pres.+abs. debiasing
0.90
0.64
0.83
Appendix
Table 6: Left: Weaker bias. We report the main results for the experimental setup from Section 5.4 when the bias is only present in 50 % of the training samples. Right: ViT. We report the main results for the experimental setup from Section 5.4 for a ViT-B/16.
Figure 12: Controlled concept intervention. For each experiment relying on non-target attributions, we select a sample from the class that is hypothesized to use an encoded absence (left-side sample of each group). The target attributions for these unmodified samples show no strong inhibitory signals, as indicated by the absence of pronounced red highlights. After inserting the corresponding absence concept into the right-side samples (a right-to-left motion pattern in the Hassenstein–Reichardt detector example; a green pixel in the toy example, bottom-left quadrant; and a colorful patch in the malignant sample from the debiasing experiment) and recomputing the target attribution (see Section B.5 for details), we observe markedly stronger inhibitory responses, visible as strong red attributions. This controlled intervention provides clear evidence that the inhibitory patterns identified in our experiments indeed reflect encoded absences.
Original sample without intervention
w/ controlled intervention
w/ patch intervention
Dataset
Avg. activation
Avg. neg. attr.
Avg. activation
Avg. neg. attr.
Avg. activation
Avg. neg. attr.
Toy
0.22
0
-0.11
10.44
-0.11
10.43
ISIC
-0.06
926.55
-0.73
1469.09
-0.98
1501.5
Appendix
Table 7: Quantitative evaluation of our intervention approximations. Average activation and negative attribution values across datasets and intervention settings.
Deep neural networks often exploit spurious associations, a failure known as shortcut learning. Before deployment, models should therefore be audited for reliance on concepts such as acquisition artifacts and patient demographics. Current methods, including linear probes and concept activation vectors, measure concept reliance by asking whether each concept is decodable from a layer. We show that this framing is misleading: decodability reflects not only concept reliance but also correlations among concepts and the outcome in the auditing dataset. We reframe shortcut auditing as a decomposition of the variance of intermediary representations with Independent Canonical cONcepts (ICON). ICON quantifies the share of a layer's variance explained by each concept conditional on all other concepts and the outcome, and also reports the variance left unexplained. Because ICON explanations are comparable across layers, their trajectories distinguish concepts that drive predictions from those that are merely encoded in the representations. In simulated data with ground truth, ICON recovers concept importance more accurately than seven baselines. In skin-cancer and neuroimaging applications, it separates learned shortcuts from merely correlated concepts, which we confirm with independent validation checks.
Roshan Prakash Rane, Marco Simnacher, Manuel Pfeuffer +7
Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany. · Department of Psychiatry and Neurosciences, Charit´e - Universitätsmedizin Berlin, Berlin, Germany. · Department of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany. +5
The demand for Explainable AI (XAI) has triggered an explosion of methods, producing a landscape so fragmented that we now rely on surveys of surveys. Yet, fundamental challenges persist: conflicting metrics, failed sanity checks, and unresolved debates over robustness and fairness. The only consensus on how to achieve explainability is a lack of one. This has led many to point to the absence of a ground truth for defining ``the'' correct explanation as the main culprit. This position paper posits that the persistent discord in XAI arises not from an absent ground truth but from a ground truth that exists, albeit as an elusive and challenging target: the causal model that governs the relevant system. By reframing XAI queries about data, models, or decisions as causal inquiries, we prove the necessity and sufficiency of causal models for XAI. We contend that without this causal grounding, XAI remains unmoored. Ultimately, we encourage the community to converge around advanced concept and causal discovery to escape this entrenched uncertainty.
Amir-Hossein Karimi
Department of Electrical & Computer Engineering, University of Waterloo & Vector Institute, Canada
Concept-based explanations offer a promising approach for explaining the predictions of deep neural networks in terms of high-level, human-understandable concepts. However, existing methods either do not establish a causal connection between the concepts and model predictions or are limited in expressivity and only able to infer causal explanations involving single concepts. At the same time, the parallel line of work on formal abductive and contrastive explanations computes the minimal set of input features causally relevant for model outcomes but only considers low-level features such as pixels. Merging these two threads, in this work, we propose the notion of concept-based abductive and contrastive explanations that capture the minimal sets of high-level concepts causally relevant for model outcomes. We then present a family of algorithms that enumerate all minimal explanations while using concept erasure procedures to establish causal relationships. By appropriately aggregating such explanations, we are not only able to understand model predictions on individual images but also on collections of images where the model exhibits a user-specified, common behavior. We evaluate our approach on multiple models, datasets, and behaviors, and demonstrate its effectiveness in computing helpful, user-friendly explanations.
Ronaldo Canizales, Divya Gopinath, Corina Păsăreanu +1
Colorado State University, Fort Collins CO, USA · KBR Inc., NASA Ames, Moffett Field CA, USA · Carnegie Mellon University, Pittsburgh PA, USA