cs.CVMar 10, 2026
SaveWhat is Missing? Explaining Neurons Activated by Absent Concepts
Organizations: Max Planck Institute for Informatics, SIC · Department of Computer Science, Technical University of Darmstadt · Leibniz Institute for Resilience Research · Institute for Quantitative and Computational Biosciences, Johannes Gutenberg University Mainz ·
Abstract
Explainable artificial intelligence (XAI) aims to provide human-interpretable insights into the behavior of deep neural networks (DNNs), typically by estimating a simplified causal structure of the model. In existing work, this causal structure often includes relationships where the presence of a concept is associated with a strong activation of a neuron. For example, attribution methods primarily identify input pixels that contribute most to a prediction, and feature visualization methods reveal inputs that cause high activation of a target neuron - the former implicitly assuming that the relevant information resides in the input, and the latter that neurons encode the presence of concepts. However, a largely overlooked type of causal relationship is that of encoded absences, where the absence of a concept increases neural activation. In this work, we show that such missing but relevant concepts are common and that mainstream XAI methods struggle to reveal them when applied in their standard form. To address this, we propose two simple extensions to attribution and feature visualization techniques that uncover encoded absences. Across experiments, we show how mainstream XAI methods can be used to reveal and explain encoded absences, how ImageNet models exploit them, and that debiasing can be improved when considering them.
Figures & tables
Figure 1: Encoded absence in image classification. (a) The model detects concepts present in the input image that are prototypical for the target class ( e.g. , the snout and feet). (b) The model can additionally encode the absence of snouts from other dog species to enhance evidence for the “Irish setter” class.
Figure 2: A mechanistic process to encode the absence of a concept. A neuron encoding the absence of concept ( i.e. ) can be implemented by having a negative connection to a neuron encoding and a positive potential through, e.g. , another activating concept ( i.e. , the output encodes ).
Figure 3: Hassenstein–Reichardt detector experiment. (a) Two example sequences showing a left-to-right and bi-directional movement. (b) A hand-crafted CNN to distinguish left-to-right motion from bi-directional motion. The first layer implements the spatio-temporal comparison of neighboring pixels, the second layer compares motion in opposing directions, followed by global average pooling (GAP). The first output node implements a Hassenstein-Reichardt detector (weights: 1/-1) and the second output averages both directions (weights: 0.5/0.5). (c) The outputs of the model for the two example sequences. (d) Visualizations of established XAI methods – target attribution and feature visualization via the highest activating patches, each consisting of two consecutive frames as CNN input (numbers indicate the activation strength). Both methods fail to highlight the absence encoded in the first output and thus lack a complete explanation of the CNN mechanisms. (e) Our proposed non-target attributions and feature visualization through minimization highlight that the first output encodes the absence of right-to-left motion.
Figure 4: Toy experiment. (a) Example RGB images from class 1 (green pixel) and class 2 (no green pixel) – zoom in for better visibility. (b) Architecture of the used toy model with the trained weights. (c) Average logit output for the two output nodes for images from class 1 and 2. Confidence intervals represent two times the standard deviation. (d) Integrated Gradients ( Sundararajan et al., 2017 ) target attributions for the above example images, and maximally activating patches for the two output nodes. (e) Non-target attributions for the two respective examples – note how the attributions switch from positive (blue) to negative (red) – and minimally activating patches for the two output nodes (numbers indicate the activation strength).
Table 1: Quantitative evaluation of encoded absences. We measure the activation of the highest-activating images when inserting none, random, encoded logical s ( Mu & Andreas, 2020 ) , most activating ( e.g. , as used in ( Ahn et al., 2024 ) ), or least activating patches in a random corner.
Figure 6: Encoded presences (positive potential) and absences for three channels that have been found to be important for the corresponding class. We identify channels that are important for specific classes and visualize the positive potential from this class by showing the most activating patches. Further, we show the encoded absences by showing the least activating patches. In particular, for fine-grained classification, encoded absences of patterns from related species seem to be used.
| Validation split (training bias) | Validation split (inverse bias) | Validation split (no bias) | ||||||||||
| Training bias | Model | Benign ∗ | Malignant | Avg. | Attr. | Benign | Malignant ∗ | Avg. | Attr. | Benign | Malignant | Avg. |
| None | -ResNet-50 | – | – | – | – | – | – | – | – | 0.81 | ||
| Benign* | No debiasing | 0.99 | ||||||||||
| Presence debiasing | ||||||||||||
| Presence+absence debiasing (ours) | 0.07 | 0.59 | 0.08 | 0.81 | ||||||||
Table 2: Validation results for the ISIC dataset with varying biases. We report the accuracy (average over 5 runs) for the validation split of the ISIC dataset with different biases. In the “training bias” setup, the train bias is replicated with all the benign samples containing colorful patches, while in the “inverse bias” setup, the malignant samples contain colorful patches, as indicated by ∗ . A model with no debiasing learns the dataset bias and fails to classify samples when the bias is not present. A model with presence debiasing (existing attribution priors) can reduce this bias; however, it still fails to classify malignant samples when inserting colorful patches, indicating that it is biased based on the absence of colorful patches. Our proposed presence+absence debiasing results in the highest average accuracy for both setups without the training bias, and is similarly performant as a model trained without bias, suggesting that the model is largely debiased. “Attr.” shows the relative attribution within the colorful patches, confirming qualitative results from Figure 7 .
Figure 7: Images and attributions for our biased ISIC dataset. (a) We replicate the ISIC bias (real bias) that co-occurs with the benign samples with a synthetic bias. (b) Attributions of different (de)biased models for the benign sample with a synthetic bias. The target attribution is computed for the benign output logit and the non-target attribution for the malignant output. Only including absences in the debiasing prevents the model from relying on patch absence to predict malignancy.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Simplified illustration of the Hassenstein-Reichardt detector in Drosophila . The activation of two subunits – encoding right-to-left ( ) and left-to-right ( ) movements – is subtracted. The output neuron encodes the presence of left-to-right movements while encoding the absence of right-to-left movements ( ).
Figure 10: Counterfactual visual explanation. Counterfactual visual explanation ( Goyal et al., 2019 ) for the two images shown in Figure 4 and the corresponding trained toy model. The explanation correctly highlights the green pixel (zoom in) as the concept whose swap induces a class change, thereby identifying it as the most class-discriminative concept. However, this explanation does not clarify whether the model relies on the presence of the green pixel, its absence, or both, which is possible with our proposed modifications.
| Model | Patch size | Nr. images | None | +Random | +Logical NOT ( Mu & Andreas, 2020 ) | +Most act. | +Least act. (ours) |
|---|---|---|---|---|---|---|---|
| VGG19 ( Simonyan & Zisserman, 2015 ) | 32 | 100 | 2.98 1.08 | 2.84 1.09 | 2.70 1.04 2.68 1.04 | 3.39 1.10 | 2.14 1.13 |
| VGG19 | 48 | 100 | 2.98 1.08 | 2.68 1.10 | 2.53 1.06 2.51 1.06 | 3.88 1.16 | 0.94 1.16 |
| VGG19 | 64 | 100 | 2.98 1.08 | 2.41 1.12 | 2.25 1.08 2.21 1.08 | 4.38 1.13 | -0.38 1.18 |
| VGG19 | 48 | 50 | 3.72 1.08 | 3.39 1.10 | 3.21 1.06 3.19 1.06 | 4.59 1.12 | 1.66 1.16 |
| VGG19 | 48 | 200 | 2.25 1.07 | 1.98 1.10 | 1.85 1.04 1.84 1.05 | 3.18 1.10 | 0.25 1.14 |
| ResNet-50 ( He et al., 2016 ) | 32 | 100 | 0.18 0.06 | 0.17 0.06 | 0.16 0.06 0.16 0.06 | 0.21 0.06 | 0.12 0.07 |
Table 3: Quantitative evaluation of encoded absences. We report the CNN results from Figure 5 alongside their standard deviations (indicated by “ ”) and under different hyperparameters (patch size and number of images). Please refer to Figure 5 for a detailed description.
| Model | Layer | None | +Random | +Least act. (ours) |
|---|---|---|---|---|
| ResNet-50 | layer4[2].conv3 | 0.18 | 0.16 | 0.03 |
| ResNet-50 | layer3[2].conv3 | 0.26 | 0.24 | 0.18 |
| ResNet-50 | layer2[2].conv3 | 0.47 | 0.45 | 0.38 |
| ResNet-50 | layer1[2].conv3 | 0.08 | 0.07 | 0.02 |
Table 4: Quantitative evaluation of encoded absences in different layers. We report the results from Figure 5 in different layers of a ResNet-50. Please refer to Figure 5 for a detailed description.
Figure 11: The eight least activating patches for channels 2026 and 1470. For the most and least activating patches in Section 5.3 , we obtain eight candidate patches and manually select a monosemantic subset of three patches for more interpretable visualizations. Inspecting all eight patches for channels 2026 and 1470 in ResNet-50 reveals that these channels encode the absence of multiple concepts, consistent with prior work on polysemantic neurons ( Elhage et al., 2022 ) . The corresponding labels indicate the class of each patch.
| Validation split (training bias) | Validation split (inverse bias) | Validation split (no bias) | ||||||||||
| Training bias | Model | Benign ∗ | Malignant | Avg. | Attr. | Benign | Malignant ∗ | Avg. | Attr. | Benign | Malignant | Avg. |
| None | -ResNet-50 | – | – | – | – | – | – | – | – | 0.84 | 0.77 | 0.81 |
| – | – | – | – | – | – | – | – | 0.03 | 0.07 | 0.02 | ||
| Benign* | No debiasing | 1.00 | 0.99 | 0.99 | 0.40 | 0.04 | 0.00 | 0.02 | 0.47 | 0.04 | 0.99 | 0.51 |
| 0.00 | 0.02 | 0.01 | 0.02 | 0.02 | 0.00 | 0.01 | 0.02 | 0.02 | 0.02 | 0.00 | ||
| Presence debiasing | 0.96 | 0.88 | 0.92 | 0.08 | 0.66 | 0.17 | 0.41 | 0.13 | 0.66 | 0.88 | 0.77 | |
Table 5: Validation results for the ISIC dataset with varying biases. We report the results from Table 2 together with their standard deviations (indicated by “ ”). Please refer to Table 2 for a detailed description of the table.
| Model | Train bias (Avg.) | Inv. bias (Avg.) | No bias (Avg.) |
|---|---|---|---|
| No debiasing | 0.93 | 0.41 | 0.82 |
| Presence debiasing | 0.92 | 0.46 | 0.83 |
| Pres.+abs. debiasing | 0.90 | 0.64 | 0.83 |
Table 6: Left: Weaker bias. We report the main results for the experimental setup from Section 5.4 when the bias is only present in % of the training samples. Right: ViT. We report the main results for the experimental setup from Section 5.4 for a ViT-B/16.
Figure 12: Controlled concept intervention. For each experiment relying on non-target attributions, we select a sample from the class that is hypothesized to use an encoded absence (left-side sample of each group). The target attributions for these unmodified samples show no strong inhibitory signals, as indicated by the absence of pronounced red highlights. After inserting the corresponding absence concept into the right-side samples (a right-to-left motion pattern in the Hassenstein–Reichardt detector example; a green pixel in the toy example, bottom-left quadrant; and a colorful patch in the malignant sample from the debiasing experiment) and recomputing the target attribution (see Section B.5 for details), we observe markedly stronger inhibitory responses, visible as strong red attributions. This controlled intervention provides clear evidence that the inhibitory patterns identified in our experiments indeed reflect encoded absences.
| Original sample without intervention | w/ controlled intervention | w/ patch intervention | ||||
| Dataset | Avg. activation | Avg. neg. attr. | Avg. activation | Avg. neg. attr. | Avg. activation | Avg. neg. attr. |
| Toy | 0.22 | 0 | -0.11 | 10.44 | -0.11 | 10.43 |
| ISIC | -0.06 | 926.55 | -0.73 | 1469.09 | -0.98 | 1501.5 |
Table 7: Quantitative evaluation of our intervention approximations. Average activation and negative attribution values across datasets and intervention settings.