Causal Interpretation of Neural Network Computations with Contribution Decomposition
Authors: Joshua Brendan Melander, Zaki Alaoui, Shenghua Liu, Surya Ganguli, Stephen A. Baccus
Organizations: Department of Neurobiology, Stanford University · Department of Physics, Stanford University · Department of Applied Physics, Stanford University
Understanding how neural networks transform inputs into outputs is crucial for interpreting and manipulating their behavior. Most existing approaches analyze internal representations by identifying hidden-layer activation patterns correlated with human-interpretable concepts. Here we take a direct approach to examine how hidden neurons act to drive network outputs. We introduce CODEC (Contribution Decomposition), a method that uses sparse autoencoders to decompose network behavior into sparse motifs of hidden-neuron contributions, revealing causal processes that cannot be determined by analyzing activations alone. Applying CODEC to benchmark image-classification networks, we find that contributions grow in sparsity and dimensionality across layers and, unexpectedly, that they progressively decorrelate positive and negative effects on network outputs. We further show that decomposing contributions into sparse modes enables greater control and interpretation of intermediate layers, supporting both causal manipulations of network output and human-interpretable visualizations of distinct image components that combine to drive that output. Finally, by analyzing state-of-the-art models of neural activity in the vertebrate retina, we demonstrate that CODEC uncovers combinatorial actions of model interneurons and identifies the sources of dynamic receptive fields. Overall, CODEC provides a rich and interpretable framework for understanding how nonlinear computations evolve across hierarchical layers, establishing contribution modes as an informative unit of analysis for mechanistic insights into artificial neural networks.
Figures & tables
Figure 1: Understanding the contribution of an intermediate component to downstream computation. (A) Biological and (B) artificial neural circuits construct computations by combining sets of upstream components in an input-dependent manner. The action of a network component Z is a composition of its receptive field, or sensitivity to input X , and its projective field, or effect on output Y . Measuring both is required to explain how the intermediate component contributes to the overall behavior of the system.
Figure 2: Hidden-neuron contributions in a deep convolutional network. (A) Pipeline of computing contributions for an image processed through ResNet-50. Gradient-based attribution methods are extended to compute the contributions of each hidden unit to scalar targets of network output such as entropy of logits, sum of top- k logits, and individual class logits. (B) Spatial map of activations and contributions of a single channel in Layer 5. (C) Mean positive, negative and net contribution for each channel. (D–E) Same as (B–C) for Layer 8.
Figure 3: Channel contributions through the network become more sparse, single-signed and high dimensional. (A) Example matrix of spatially-summed contributions and activations from one layer for all channels and images from four classes. (B) Hoyer sparsity index for contributions and activations across network layers. (C) Scatter plot of mean negative vs. mean positive contributions of each channel to network output. (D) Correlation coefficient between positive and negative contributions of individual channels across network depth. (E) Fraction of explained variance (FOEV) across all class-averaged channel weightings for layers 2 and 14. (F) Number of components required to reach 95 percent FOEV.
Figure 4: Sparse autoencoder decomposition of network contributions. (A) Schematic of contribution decomposition. Channel contributions are spatially summed and an autoencoder is trained to reconstruct the matrix of contributions by images by creating modes with sparse loadings (the weighting for each mode for particular image). Loadings are passed through a sigmoid then a threshold, and regularized to encourage sparsity. (B) Loadings from the mode that maximally correlated with the class “panda” for contributions (top, black) and activations (bottom, blue). Inset shows the loadings for 50 images of panda and 100 images of other classes.
Figure 5: Emergence of meaningful contribution modes in intermediate layers. (A) Histograms of each mode’s maximum correlation with binary class indicators for contributions (grey) and activations (blue) at hidden layer 13. (B) Same as (A) for the correlation of individual channel contributions or activations with class indicators. (C) Number of modes with a correlation to a class of greater than 0.2. (D) Mean of the maximal class-correlation as a function of layer.
Figure 6: Contribution-guided network control. (A) Example mode most correlated with the “black widow” class, showing channel weightings. (B) Sorted channel weightings from (A) with a few top channels highlighted for selection. (C) Results for “black widow” classification: Top shows preservation analysis (accuracy when keeping only selecte channels), bottom shows ablation analysis (accuracy when removing selecte channels). (D) Normalized accuracy change for the target class when ablating (left) or preserving (right) channels from the most correlated contribution mode (black), activation mode (blue), or random class (red, representing non-target/off-target performance). (E) Performance score quantifying perturbation effectiveness as the area between target and off-target curves from (D) across all blocks, normalized to the off-target area.
Figure 7: Visualizing stimuli acting through contribution modes. (A) Contribution mapping reveals how parallel pathways extract different information to drive network output. The contribution map is an intermediate stage for computing contributions prior to input summation defined by observing that the inner product of the image x and sensitivity A can be expanded into a product of 1 , the vector of ones, and an elementwise product that is the contribution map. Symbols are shown as their transpose to match the left to right schematic. (B) Visualization of 4 example stimuli from different classes through two modes highly correlated with violin at layer 7 of Resnet-50. The information conveyed by modes are interpretable features that can be identified across class and stimuli (shiny wood, hands, etc.). To preserve original colors, maps are shown averaged over color channels and as a weighted mask on the original image, though they contain color information not shown.
Figure 8: Contribution modes generate dynamic receptive fields in the retina. (A) CNN trained to predict retinal ganglion cell responses to natural stimuli, from Ding et al. (2023) . CODEC is performed for hidden units to single target outputs including surprisal (self-information) of the population or from single cell responses. (B) Model firing rate predictions for an example cell (top) aligned with the matrix of contributions, and with loadings for two example modes after SAE decomposition (bottom). Also shown are instantaneous receptive fields (IRFs) of the cell computed at two time points. (C) Clustering of cells using their contribution modes. Matrix shows the correlation of firing of each cell with the loading of each mode. Three clusters (number selected using silhouette value) computed using k -means remain consistent across layers. (D) IRFs during sparse mode combinations. Left column: IRFs when one specific mode was active alone for four different cells at different times. Right: IRFs for example cells at times when 5 or fewer modes were active and cells were firing more than 1 Hz. Each row is from a different cell.
Figure S1: Comparison of attribution methods and targets reveals consistent channel contribution patterns. (A) Spatial maps of activations, gradients, and contributions computed using different contribution algorithms (ActGrad, IntGrad) and targets (sum of all logits, most-active logit, most-active logit with softmax, entropy) for Channel 39. All methods show similar spatial sparsity. (B) Excitatory and inhibitory contribution maps for Channel 39 in Layer 5 for a single image demonstrating the separation of excitation and inhibition in an early layer. (C) Contribution values across channels for one image showing the distribution of net excitatory and inhibitory effects for Layer 5. (D) Same as (B) but for Channel 369 in Layer 11, illustrating consistent excitatory/inhibitory patterns across different channels. (E) Same as (C) but for Layer 11, showing how the excitatory/inhibitory decorrelation evolves across network depth.
Figure S2: Sparse autoencoder decomposition performance and sparsity analysis across network layers. (A) Reconstruction accuracy ( R2 ) of sparse autoencoder decompositions for contributions (black) and activations (blue) across network layers. Both show high reconstruction fidelity, with contributions maintaining slightly higher R2 values in intermediate layers. (B) Number of modes discovered by the sparse autoencoder decomposition as a function of network depth for contributions (black) and activations (blue). The number of modes increases through the network, with contributions yielding more modes than activations in deeper layers. (C) Comparison of the most correlated panda mode with the average contribution pattern during presentation of panda images. The decomposition identified a mode (black) that shows similar contribution patterns to the class-average contributions when panda images are presented (r=0.6). (D) Median Hoyer sparsity index of learned modes across network layers for contributions (black) and activations (blue). Contribution modes maintain higher sparsity than activation modes throughout the network.
Figure S3: SAE mode stability across hyperparameters and random seeds. SAEs were trained on Resnet-50 contributions for example layers 3, 7, 13, and 15, varying hyperparameters L1 sparsity penalty strength, threshold, dictionary size (N * number of channels), random seed, MLP encoder size, and a non-negativity constraint were varied across the indicated values. Each panel shows the correlation coefficient between baseline SAE modes (used in main text) and modes learned with varied hyperparameters as a function of network layer. Modes are aggregated for each class by summing all modes whose correlation with that ImageNet class exceeds 0.2; if no modes exceed this threshold, the single top-correlated mode is used. Shaded regions show mean ± standard error of correlation across 1000 ImageNet classes.
Figure S4: SAE performance metrics are robust across hyperparameter choices. Each panel shows a metric (rows) across network layers (x-axis) for different hyperparameter values (colored lines). For each hyperparameter column, values are averaged across all other hyperparameter configurations. Shaded regions indicate mean ± standard error across configurations. Top row: R2 score, reconstruction quality across all hyperparameters. Middle row: The number of modes having a correlation with classes > 0.2 for different hyperparameters. Bottom row: The number of classes having a correlation with modes > 0.2 for different hyperparameters.
Figure S5: Class-specific mode representations are consistent across hyperparameter choices. For the Black Widow class we summed all modes with correlation >0.2 to that class. Shown is the sum for SAEs trained with different hyperparameters.
Figure S6: Hyperparameter robustness of SAE ablation specificity across architectures. Ablation specificity measured as normalized AUC difference across ResNet-50 blocks (3,7,13,15) for SAEs trained with varying hyperparameters: dictionary size multiplier (N=1,3,5), activation threshold (0.5,0.7,0.9), mode L1 regularization (0, 1e-2, 1e-4), random seed (1001, 2002, 483), and MLP hidden layer size (128-4096). Threshold = 0.9 corresponds to our baseline set of hyperparameters used in the main text. Normalized AUC difference integrates performance ratios (perturbed/original accuracy) across ablation percentages (25%, 50%), with negative values indicating greater specificity (larger target class accuracy drop relative to off-target classes). Results demonstrate robustness across hyperparameter choices, with all configurations showing increased specificity in deeper layers. Error bars indicate SEM across ten trials with randomly sampled class pairs.
Figure S7: Application of CODEC on ViT-B. The columns correspond to the three layer types—token features, MLP, and attention. (A–C) Examples of contributions and activations computed for a single image at layer 9. (D–F) Hoyer sparsity as a function of depth for activations (blue) and contributions (black). Box plots show the distribution of sparsity values over the entire ImageNet validation set, omitting outliers for clarity. (G–I) Analogous to Fig. 5 D in the main text, the mean over units (pseudo-channels or modes) of the maximal class-correlation for each unit as a function of depth for four cases: contributions of single pseudo-channels, activations of single pseudo-channels, contribution modes, and activation modes. (J–L) Analogous to Fig. 6 E, performance score quantifying perturbation effectiveness as the area between target and off-target curves from Fig. 6 D normalized to the off-target area, as a function of depth. Six trials are performed for each layer, shown as scatter points, and the SEM is shown as the shade.
Figure S8: Attention head sparsity for ViT-B. (A) Hoyer sparsity as a function of layer after summing contributions (black) or activations (blue) within each attention head. Box plots show the distribution of sparsity values over the entire ImageNet validation set, omitting outliers for clarity. (B) Hoyer sparsity as a function of layer after summing contributions or activations within attention heads formed by a random grouping of hidden dimensions. (C) Distribution of the difference in Hoyer sparsity for the learned and randomly grouped attention heads as a function of layer after summing contributions or activations within each attention head.
Figure S9: Contribution modes capture set of dataset classes. To determine whether our SAE models learned modes that captured the full set of classes in the Imagenet dataset, we computed the full mode-by-class correlation matrix at each layer of the model for both activations and contributions. Sorting this matrix along either the rows or columns gave insights into how many modes individual classes were correlated with (top), and how many classes individual modes were correlated with (bottom). We found that in later layers, classes on average were well covered by learned modes; furthermore, this measure of class-coverage was higher for contribution modes than for activation modes at intermediate layers of the model. Error bars indicate standard deviation.
Figure S10: Manipulation of higher semantic classes using contribution modes. (A) Ablation experiment: A binary mask was created for all images with dog superclass labels, then correlated with mode loadings to identify dog-correlated modes. All modes with correlation > 0.2 were summed together. Removing channels from this aggregated dog mode in intermediate layers (12-15) eliminates the network’s ability to classify dog images while leaving other classes largely unaffected. (B) Preservation experiment: Using the same summed dog mode identified through correlation analysis, retaining only these mode channels allows the network to correctly identify dog images while reducing accuracy to near-zero for all other classes.
Figure S11: Layer-wise sparsity comparison of contrastive and top-1 contributions. Box plots showing the distribution of Hoyer sparsity of contributions across layers 0–15 of ResNet-50 on ImageNet. Black boxes represent contrastive contributions with respect to the difference between top-1 and top-2 predictions, while blue boxes show top-1 contributions with respect to top-1 predictions (left) or activations (right).
Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.
We present a method for diagnosing interpretation in neural networks by identifying an input subspace where a proposed interpretation is highly faithful. Our method is particularly useful for causal-abstraction-style interpretability, where a high-level causal hypothesis is evaluated by interchange interventions. Rather than treating interchange intervention accuracy as a single global summary, we refine this framework by partitioning the input space into well-interpreted and under-interpreted regions according to pairwise interchange-intervention behavior. This turns causal abstraction from a purely global evaluation into a more diagnostic tool: it not only measures whether an interpretation works, but also reveals where it works, where it fails, and what distinguishes the two cases. This diagnostic view also provides practical heuristics for improving interpretations. By analyzing the structure of the well-interpreted and under-interpreted regions, we can identify missing distinctions in a high-level hypothesis, discover previously unmodeled intermediate variables, and combine complementary partial interpretations into a stronger one. We instantiate this idea as a simple four-step recipe and show that it yields informative error analyses across multiple causal abstraction settings. In a toy logic task, recursively applying the recipe recovers a high-level hypothesis from scratch. More broadly, our results suggest that partitioning the input space is a useful step toward more precise, constructive, and scalable mechanistic interpretability.
Deep neural networks often exploit spurious associations, a failure known as shortcut learning. Before deployment, models should therefore be audited for reliance on concepts such as acquisition artifacts and patient demographics. Current methods, including linear probes and concept activation vectors, measure concept reliance by asking whether each concept is decodable from a layer. We show that this framing is misleading: decodability reflects not only concept reliance but also correlations among concepts and the outcome in the auditing dataset. We reframe shortcut auditing as a decomposition of the variance of intermediary representations with Independent Canonical cONcepts (ICON). ICON quantifies the share of a layer's variance explained by each concept conditional on all other concepts and the outcome, and also reports the variance left unexplained. Because ICON explanations are comparable across layers, their trajectories distinguish concepts that drive predictions from those that are merely encoded in the representations. In simulated data with ground truth, ICON recovers concept importance more accurately than seven baselines. In skin-cancer and neuroimaging applications, it separates learned shortcuts from merely correlated concepts, which we confirm with independent validation checks.
Roshan Prakash Rane, Marco Simnacher, Manuel Pfeuffer +7
Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany. · Department of Psychiatry and Neurosciences, Charit´e - Universitätsmedizin Berlin, Berlin, Germany. · Department of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany. +5
It is essential for understanding neural network decisions to interpret the functionality (also known as concepts) of neurons. Existing approaches describe neuron concepts by generating natural language descriptions, thereby advancing the understanding of the neural network's decision-making mechanism. However, these approaches assume that each neuron has well-defined functions and provides discriminative features for neural network decision-making. In fact, some neurons may be redundant or may offer misleading concepts. Thus, the descriptions for such neurons may cause misinterpretations of the factors driving the neural network's decisions. To address the issue, we introduce a verification of neuron functions, which checks whether the generated concept highly activates the corresponding neuron. Furthermore, we propose a Select-Hypothesize-Verify framework for interpreting neuron functionality. This framework consists of: 1) selecting activation samples that best capture a neuron's well-defined functional behavior through activation-distribution analysis; 2) forming hypotheses about concepts for the selected neurons; and 3) verifying whether the generated concepts accurately reflect the functionality of the neuron. Extensive experiments show that our method produces more accurate neuron concepts. Our generated concepts activate the corresponding neurons with a probability approximately 1.5 times that of the current state-of-the-art method.
ZeBin Ji, Yang Hu, Xiuli Bi +2
Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China · Jinan Inspur Data Technology Co., Ltd., Jinan, China