Shannon observables: A scalable approach to information decomposition
Authors: Aaron J. Gutknecht, Fernando E. Rosas, David A. Ehrlich, Abdullah Makkeh, Pedro A. M. Mediano, Michael Wibral
Organizations: Göttingen Campus Institute for Dynamics of Biological Networks, Georg-August University Göttingen · Sussex Centre for Consciousness Science and Sussex AI, Department of Informatics, University of Sussex · Center for Psychedelic Research and Centre for Complexity Science, Department of Brain Science, Imperial College London · Center for Eudaimonia and Human Flourishing, University of Oxford · Principles of Intelligent Behaviour in Biological and Social Systems (PIBBSS) · Max Planck Institute for Dynamics and Self-Organization, Göttingen · Department of Computing, Imperial College London · Division of Psychology and Language Sciences, University College London
Distributed systems, such as biological and artificial neural networks, process information via complex interactions engaging multiple subsystems, resulting in high-order structure of the information within the system with distinct properties across scales. Investigating how these systems process information remains challenging due to difficulties in defining appropriate multivariate metrics and ensuring their scalability to large systems. To address these challenges, we introduce a novel framework based on what we call "Shannon observables": quantities that capture essential properties of high-order information processing, yet can be computed directly from Shannon entropies and scale efficiently to large systems. Our theoretical results demonstrate how Shannon observables can be used to resolve long-standing ambiguities regarding the interpretation of widely used multivariate information-theoretic measures. On the practical side, we demonstrate the framework's versatility across several domains. An analytical study of error-correcting codes shows how Shannon observables trace out distinct encoding regimes as channel noise varies; analyses of deep classifiers and convolutional autoencoders reveal characteristic information-processing signatures across layers and over training; and applied to magnetoencephalography recordings from the human brain, the framework yields a cortical atlas exposing pronounced regional structure in directed information transfer. Overall, our framework resolves fundamental limitations in analysing high-order information phenomena and offers broad opportunities for theoretical developments and empirical analyses.
Figures & tables
Figure 1: Two complementary perspectives on how information is distributed across source variables. (a) For two sources, PID decomposes the joint information I(X1,X2;Y) (outer ellipse) into information redundantly available from each of the sources (blue), information unique to each source (grey), and synergistic information accessible only from the joint observation (red). (b) Each information atom in PID has a degree of redundancy r , counting how many individual sources can provide it. Atoms with r=0 are source-level synergistic (not accessible from any single source); atoms with r≥2 are source-level redundant. The average degree of redundancy rˉ is the weighted average of r across all atoms. The grey triangle at r=1 acts like the balance point of an asymmetric seesaw. (c) Dually, each atom has a degree of vulnerability v , counting how many sources it critically depends on. Atoms with v=0 are robust (no single source is critical); atoms with v≥2 have multi-source vulnerability. The same seesaw structure applies in mirror. (d) The two measures span complementary axes of information structure, illustrated for three canonical systems with n=3 binary sources each: COPY (maximal redundancy), XOR (maximal synergy), and UNIQUE (purely unique information).
Method
Directed
PID-independent
Scaling
Directed methods (source → target)
Full PID [ 24 ]
✓
×
Super-exponential ( D(n)−2 terms)
Φ ID [ 26 ]
✓
×
Super-exponential ( (D(n)−2)2 terms)
Backbone decomposition [ 35 ]
✓
×
Super-exponential (redundancy-based); linear (synergy-based) †
Representational complexity [ 29 ]
✓
×
Super-exponential (redundancy-based); linear (synergy-based) †
α -synergy backbone [ 48 ]
✓∗
(✓)∗
Exponential
Table 1: Comparison of approaches for quantifying redundancy and synergy. ‘Directed’ indicates whether the method distinguishes source and target variables. ‘PID-independent’ indicates whether the result is invariant to the choice of PID measure. ‘Scaling’ refers to the asymptotic growth, with the number of source variables n , of the number of information-theoretic functionals that must be evaluated, whether Shannon entropies, mutual informations, or PID atoms. The cost of evaluating each individual functional is a separate concern shared by all methods. D(n) stands for the Dedekind numbers.
Figure 2: Information decomposition of error correcting coding. a) Schematic of a Hamming (7,4) code transmitted over a noisy channel. A message of four independent random bits is encoded using a Hamming (7,4) code into a seven-bit codeword C=(C1,…,C7) , transmitted through a binary symmetric channel (BSC) with bit-flip probability ϵ , and received as X=(X1,…,X7) . We analyse how information about the first message bit Y=C3 is distributed across the received bits. b) Evolution of the average degree of redundancy rˉ (blue) and average degree of vulnerability vˉ (red) as functions of the channel error probability ϵ . The measures exhibit diverse behaviours across different noise regimes. c) Mutual information terms underlying the Shannon observables: joint mutual information I(X;Y) (black), marginal mutual informations I(Xi;Y) for i=3 (solid blue) and I(X3;Y) (dashed blue), and conditional mutual informations I(Xi;Y∣X−i) for i=3 (solid red) and I(X3;Y∣X−3) (dashed red).
Figure 3: Average degree of redundancy and vulnerability in a deep classification network. These analyses showcase how Shannon observables can track layer-wise representational structure across training in a PID-independent and computationally tractable way. The degree of redundancy about the label increases from the first equal-sized hidden layer to the deeper ones, while the degree of vulnerability decreases across the hidden layers and over training. In panels (b) and (e)–(h), lines represent the median, shaded regions the maximum and minimum of 10 runs with random weight initialisations; dashed lines show networks trained with one-neuron dropout applied to the correspondingly coloured five-neuron layer. (a) The architecture of the MNIST classification network comprises five fully-connected hidden layers with activation values quantised to eight levels, three of which have the same size of 5 neurons. (b) Evolution of classifier accuracy over training epochs; the test accuracy of the per-layer dropout networks (dashed, coloured) nearly coincides with that of the baseline networks. (c) Mean drop in test-set accuracy caused by lesioning single neurons at the end of training, for the standard networks (solid borders) and for the networks trained with one-neuron dropout on that layer (lighter fill, dashed borders); bars show the mean over neurons and runs, dots the individual run means. (d) Per-neuron vulnerability vi at the end of training (epoch 104 ) against the drop in test-set accuracy caused by lesioning that neuron in the standard networks. Each dot corresponds to one neuron of the three equal-sized hidden layers in one run; coloured lines show per-layer fits and the black line the pooled least-squares fit as visual guides (Spearman ρ=0.72 , p<10−24 ). (e), (f) Degree of redundancy and degree of vulnerability computed on the training set for the three equal-sized hidden layers of the standard networks. (g), (h) The same quantities for the networks trained with one-neuron dropout on the respective layer (dashed).
Figure 4: Average degree of redundancy and vulnerability in a convolutional autoencoder. The analyses illustrate the applicability of Shannon observables with up to 2048 source variables, which would be intractable with standard information decomposition approaches. The degree of redundancy about the input is larger in decoder layers than in the size-matched encoder layers. Across varying bottleneck sizes, redundancy increases with bottleneck width while vulnerability decreases. In all plots, lines represent the median, shaded regions the maximum and minimum of 10 runs with random weight initialisations. (a) The architecture comprises a three-layer convolutional encoder, a fully-connected bottleneck layer with a varying number of neurons nb , and a three-layer convolutional decoder. Numbers below the layers reflect the size of the activation matrices as well as the number of channels. (b) Performance against training epochs for a bottleneck size of nb=128 . Five original images (upper row) and their reconstruction (lower row) are shown for the converged networks. (c) Degree of redundancy computed on the training set; all activations from all convolutional filters have been treated as individual source variables. The corresponding degree of vulnerability (not shown) is equal to zero up to numerical error for all layers. (d), (e) Degree of redundancy and degree of vulnerability of the bottleneck layer for varying small bottleneck sizes nb .
Figure 5: Population-level cortical maps of the Shannon observables from resting-state MEG in healthy adults. Shannon observables can be extended to study directed information flow, remaining tractable for whole-cortex analyses with 61 source regions per target. The analysis reveals pronounced regional structure in the redundancy and vulnerability of incoming information transfer (as approximated by linear-Gaussian state-space Granger causality) across the cortex. (a) Sample-mean degree of redundancy rˉ(i) for each of the 62 DKT atlas regions (four views per row: left lateral, left medial, right medial, right lateral). (b) Corresponding sample-mean degree of vulnerability vˉ(i) . (c) Scatter of the group-mean rˉ versus vˉ across the 62 cortical regions (Pearson ρ=−0.73 , p<10−10 ); each dot is one region and the line shows the least-squares fit. (d) Inter-subject consistency. For each subject, the Spearman correlation between that subject’s 62-region profile and the leave-one-out group mean is shown separately for rˉ (red) and vˉ (blue). Violins show the cross-subject distribution ( n=54 ); horizontal bars indicate the median; individual subjects are overlaid as dots. The dashed line marks zero.
Figure 6: Mereological representation of the mutual information and PID atoms of n=3 sources, with colouring according to a) degree of redundancy, and b) degree of vulnerability.
nb
Final train loss
Final test loss
128
9.43(9)×10−3
9.5(1)×10−3
20
1.70(4)×10−2
1.70(4)×10−2
16
1.86(3)×10−2
1.86(3)×10−2
12
2.12(4)×10−2
2.12(4)×10−2
Table 2: Final train and test set mean square error losses for the face autoencoder network for different bottleneck sizes nb .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Lesion damage versus standard neuron-importance measures. Single-neuron lesion damage (drop in test-set accuracy, cf. Figure 3 d) of the fully trained MNIST classifiers, plotted against the per-neuron vulnerability vi and five standard importance measures, for every neuron of the three five-neuron hidden layers and all 10 runs. All importance measures are computed on the training set, as is vi . Coloured lines show per-layer least-squares fits as visual guides, with per-layer Spearman rank correlations in the legends; black lines show the pooled fit for measures that are comparable across layers. The raw incoming and outgoing L2 weight norms are not — the dimensions of the weight vectors depend on the widths of the adjacent layers — so for these only within-layer fits are shown.
measure
L3
L4
L5
pooled
vulnerability vi
+0.47(6×10−4)
+0.61(<10−5)
+0.83(<10−12)
+0.72(<10−24)
incoming L2 norm
+0.36(0.011)
−0.25(0.080)
+0.27(0.063)
+0.19(0.017)†
outgoing L2 norm
+0.12(0.424)
+0.57(<10−4)
+0.28(0.049)
−0.40(<10−6)†
activation std
+0.40(0.004)
−0.25(0.084)
+0.20(0.165)
+0.13(0.105)
mean ∣ activation ∣
+0.24(0.092)
−0.20(0.175)
+0.29(0.038)
+0.13(0.103)
Taylor saliency
+0.21(0.147)
+0.20(0.174)
+0.30(0.032)
−0.30(2×10−4)†
Appendix
Table 3: Spearman rank correlations ρ between the importance measures of Figure 7 and the single-neuron lesion damage, within each of the three equal-sized hidden layers and pooled across them ( n=50 neuron-lesions per layer, 150 pooled; two-sided p -values in parentheses). Only the per-neuron vulnerability vi correlates significantly within every layer. The pooled values of the raw weight norms and of the Taylor saliency (marked † ) are dominated by between-layer differences in scale and are reported for completeness only.
Figure 8: Robustness of the MNIST results to the number of quantisation levels. The analysis of Figure 3 e, f repeated for networks trained and measured with six ( (a) , (d) ), eight ( (b) , (e) ; the setting of the main text), and ten ( (c) , (f) ) quantisation levels, with 10 runs per setting. Lines show the median, shaded regions the maximum and minimum over runs; colours denote the three equal-sized hidden layers as in Figure 3 . The qualitative features of the main-text analysis are preserved at every level count.
Mandlebrot's coastline paradox reframes measurement as a joint property of an object and its ruler. Here we ask how observational resolution shapes the apparent structure of neural computation. We introduce a scale-indexed decomposition that separates the predictive information captured by direct dependencies from the residual outside that model family. Exact constructions show that projection can convert higher-order dependence into pairwise structure. Projection can increase the direct fraction of surviving information even as total predictive information decreases. A schematic branch-to-soma cascade grounds these effects in local coincidence detection, saturating integration and somatic mixing. Resolving organized nonlinear subunits improves approximation and raises effective representational rank. In a nested hierarchy, these gains occur without increasing information about the target. This distinction separates information availability from access by a restricted readout. A point-neuron or direct population model therefore establishes an effective description at a specified scale. It does not uniquely identify the finer computation that generated the observed relation. Which resolution is adequate depends on the observable and perturbation of interest.
Nima Dehghani
McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA · The NSF AI Institute for Artificial Intelligence and Fundamental Interactions (IAIFI), Cambridge, Massachusetts, USA
We investigate how internal representations emerge across hierarchical processing systems by introducing a neuroscience-inspired framework for analyzing deep spiking neural networks (SNN) through the lens of functional connectivity. Drawing on concepts from systems neuroscience and information theory, we form the first-order functionally-connected (1FC) group of a neuron based on its statistically significant pairwise correlations with neurons from the previous layer of a trained SNN architecture. We then track its response properties during inference under various conditions. Our analysis shows that several principles of functional connectivity previously observed in biological cortex are preserved in spiking ResNet architectures. These 1FC ensembles display interesting properties: their aggregate cofiring reliably predicts downstream neuronal responses through a robust, ReLU-like input-output relationship, whose gain scales systematically with ensemble size. Reliable encoding of the presented class emerges only during high 1FC cofiring events, which themselves occur infrequently, indicating that informative representations are concentrated in rare but highly coordinated activity patterns. Under uniform random noise or adversarial perturbations, these response profiles are disrupted, particularly in early and intermediate layers. This enables a targeted high-resolution interrogation at specific nodes and pathways. We showed that the functional connectivity structure is shaped by learning and this structure breaks under weight permutation. These establish 1FC ensembles as a functionally meaningful substrate for input encoding and information transfer, with potential implications in designing targeted fine-grained diagnostics on the information flow.
Aditi Aravind, Konstantinos Ladakis, Mario Alexios Savaglio +2
University of Crete, Greece · 2Foundation of Research & Technology - Hellas (FORTH), Greece · 4Harvard Medical School, USA +2
A unified framework is proposed that characterizes supervised deep classification as the decoding of error-correcting codes in the Shannon sense. Cross-entropy optimization is shown to align softmax classifiers with maximum-likelihood decoding, where logits map to Hamming distances to class-specific codewords. This correspondence explains the emergence of standard divergence objectives (e.g., Kullback-Leibler, Jeffreys) and provides a geometric basis for the stability of decision boundaries. Furthermore, the geometries of Neural Collapse and LeJEPA are identified as optimal constellations for decoding under isotropic noise. Unlike classical Error-Correcting Output Codes, deep networks are shown to learn effective codebooks in the feature space implicitly. This insight is leveraged to derive novel criteria for lossless quantization, improved knowledge distillation, and architectural regularization based on code redundancy. Finally, this perspective is extended to self-supervised encoders, offering a cohesive explanation for the geometry of latent representations in modern neural models.
Sergey Salishev, Anton Makarov, Oleg Granichin +2
St. Petersburg State University, Saint Petersburg, 199034, Russia · Department of Software Engineering, Braude College, 21982, Karmiel, Israel