Shannon observables: A scalable approach to information decomposition
Organizations: Göttingen Campus Institute for Dynamics of Biological Networks, Georg-August University Göttingen · Sussex Centre for Consciousness Science and Sussex AI, Department of Informatics, University of Sussex · Center for Psychedelic Research and Centre for Complexity Science, Department of Brain Science, Imperial College London · Center for Eudaimonia and Human Flourishing, University of Oxford ·
Abstract
Distributed systems, such as biological and artificial neural networks, process information via complex interactions engaging multiple subsystems, resulting in high-order structure of the information within the system with distinct properties across scales. Investigating how these systems process information remains challenging due to difficulties in defining appropriate multivariate metrics and ensuring their scalability to large systems. To address these challenges, we introduce a novel framework based on what we call "Shannon observables": quantities that capture essential properties of high-order information processing, yet can be computed directly from Shannon entropies and scale efficiently to large systems. Our theoretical results demonstrate how Shannon observables can be used to resolve long-standing ambiguities regarding the interpretation of widely used multivariate information-theoretic measures. On the practical side, we demonstrate the framework's versatility across several domains. An analytical study of error-correcting codes shows how Shannon observables trace out distinct encoding regimes as channel noise varies; analyses of deep classifiers and convolutional autoencoders reveal characteristic information-processing signatures across layers and over training; and applied to magnetoencephalography recordings from the human brain, the framework yields a cortical atlas exposing pronounced regional structure in directed information transfer. Overall, our framework resolves fundamental limitations in analysing high-order information phenomena and offers broad opportunities for theoretical developments and empirical analyses.
Figures & tables
| Method | Directed | PID-independent | Scaling |
| Directed methods (source target) | |||
| Full PID [ 24 ] | ✓ | Super-exponential ( terms) | |
| ID [ 26 ] | ✓ | Super-exponential ( terms) | |
| Backbone decomposition [ 35 ] | ✓ | Super-exponential (redundancy-based); linear (synergy-based) † | |
| Representational complexity [ 29 ] | ✓ | Super-exponential (redundancy-based); linear (synergy-based) † | |
| -synergy backbone [ 48 ] | Exponential | ||
| Final train loss | Final test loss | |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| measure | pooled | |||
| vulnerability | ||||
| incoming L2 norm | ||||
| outgoing L2 norm | ||||
| activation std | ||||
| mean activation | ||||
| Taylor saliency |
Mutual Information
Mutual information measures how much one variable reveals about another, supporting analysis of dependence, prediction, and learned representations.
See all 28 papers