Neural Network Interpretability

Follow this topic Ask about this topic

Neural network interpretability asks what a model has learned, which internal computations support its predictions, and whether those computations are understandable and reliable. Post-hoc methods include saliency and attribution, concept-based explanations, counterfactuals, and formal logical explanations; mechanistic work uses probes, sparse autoencoders, circuit analysis, and interventions on neurons or features. Another approach builds interpretability into models through sparse or additive architectures, prototypes, symbolic rules, and concept bottlenecks, with evaluations spanning image, language, time-series, and scientific tasks. A central challenge is showing that an explanation is both human-meaningful and faithful to the model: readable features or sparse circuits alone do not establish causal importance, and methods can vary in robustness and reliability across settings.

Latest papers

333 papers37 papers in the last four weeks, up 23% on the four weeks before. 0.2% of all new papers.

Concept-Based Abductive and Contrastive Explanations for Behaviors of Vision Models
May 7, 2026Ronaldo Canizales, Divya Gopinath, Corina Păsăreanu +1Abductive Reasoning Concept-Based Explanations
Colorado State University, Fort Collins CO, USA · KBR Inc., NASA Ames, Moffett Field CA, USA · Carnegie Mellon University, Pittsburgh PA, USA
Hyperbolic Concept Bottleneck Models
May 7, 2026Daniel Uyterlinde, Swasti Shreya Mishra, Pascal MettesConcept Bottleneck Models Hierarchical Representation Learning
Informatics Institute, University of Amsterdam, The Netherlands
The Metagame of Interpretability and Meta-Attributions
May 7, 2026Hubert Baniecki, Przemyslaw Biecek, Fabian FumagalliTransformer Interpretability Feature Interaction Modeling
University of Warsaw · Centre for Credible AI, Warsaw University of Technology · Bielefeld University +1
AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networks
May 7, 2026Yi Wei, Xuan Qi, Furao Shen +3Neural Network Interpretability Neural Network Approximation Theory
State Key Laboratory of Novel Software Technology School of Intelligence Science and Technology Nanjing University, Jiangsu, China · AI for Good Istituto Italiano di Tecnologia, Genoa, Italy · DITEN University of Genoa, Genoa, Italy +2
Temporal Functional Circuits: From Spline Plots to Faithful Explanations in KAN Forecasting
May 7, 2026Naveen MysoreExplainable Time Series Forecasting Explainable Artificial Intelligence
University of California, Santa Barbara
Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization
May 7, 2026Andy Zeyi Liu, Elliot Paquette, John SousRepresentation Learning Decoder-Only Language Models
Department of Applied Physics, Yale University, New Haven, CT 06511, United States · Department of Mathematics and Statistics, McGill University · Department of Applied Physics, Yale University, New Haven, Connecticut 06511, USA +1
Interpreting V1 Population Activity via Image-Neural Latent Representation Alignment
May 5, 2026Xin Wang, Zhuangzhi Gao, Hongyi Qin +3Contrastive Learning Neural Processes
School of Psychological and Cognitive Sciences, Peking University, Beijing, China · Department of Primary Care and Mental Health, University of Liverpool, Liverpool, United Kingdom · Liverpool Centre for Cardiovascular Science, University of Liverpool, Liverpool, United Kingdom +2
KANs need curvature: penalties for compositional smoothness
May 4, 2026James BagrowNeural Network Activation Functions Neural Network Interpretability
Mathematics & Statistics, University of Vermont, Burlington, VT, United States · Vermont Complex Systems Center, University of Vermont, Burlington, VT, United States
A framework for analyzing concept representations in neural models
May 2, 2026Burin Naowarat, Hao Tang, Sharon GoldwaterRepresentation Disentanglement Neural Network Interpretability
The Centre for Speech Technology Research School of Informatics, University of Edinburgh, United Kingdom
HyCOP: Hybrid Composition Operators for Interpretable Learning of PDEs
May 1, 2026Jinpai Zhao, Nishant Panda, Yen Ting Lin +3Neural Surrogate Modeling Neural Network Interpretability
Oden Institute University of Texas at Austin Austin, TX, USA · Information Sciences, CAI-3 Los Alamos National Laboratory Los Alamos, NM 87545, USA · Norwegian University of Life Sciences Ås, Norway / Simula Research Lab. Oslo, Norway
From Local to Global to Mechanistic: An iERF-Centered Unified Framework for Interpreting Vision Models
May 1, 2026Yearim Kim, Sangyu Han, Nojun KwakSaliency Map Evaluation Mechanistic Interpretability
Seoul National University, Seoul 08826, South Korea
Do Sparse Autoencoders Capture Concept Manifolds?
Apr 30, 2026Usha Bhalla, Thomas Fel, Can Rager +9Neural Representation Geometry Sparse Autoencoders
aHarvard University · bNortheastern University · cTechnion IIT +1
Adjoint Inversion Reveals Holographic Superposition and Destructive Interference in CNN Classifiers
Apr 30, 2026Kaixiang ShuFeature Superposition Neural Network Interpretability
Independent Researcher Chongqing, China
Towards interpretable AI with quantum annealing feature selection
Apr 28, 2026Francesco Aldo Venturelli, Emanuele Costa, Sikha O K +3Quantum Annealing Neural Network Interpretability
1BCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain · 2Barcelona Supercomputing Center (BSC) · 3Departament de F´ısica Qu`antica i Astrof´ısica, Facultat de F´ısica, Universitat de Barcelona, E-08028 Barcelona, Spain +2
SaliencyDecor: Enhancing Neural Network Interpretability through Feature Decorrelation
Apr 28, 2026Ali Karkehabadi, Jamshid Hassanpour, Houman Homayoun +1Gradient-Based Attribution Neural Network Interpretability
University of California, Davis, USA · Georgia Institute of Technology, USA
On the explainability of max-plus neural networks
Apr 27, 2026Ikhlas Enaieh, Olivier Fercoq, García ÁngelFeature Attribution Neural Network Interpretability
S2A, LTCI · LTCI, T´el´ecom Paris · Institut Polytechnique de Paris +5
Explainable AI in Speaker Recognition -- Making Latent Representations Understandable
Apr 25, 2026Yanze Xu, Wenwu Wang, Mark D. PlumbleyClustering Hierarchical Clustering
Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, GU2 7XH, UK · Department of Informatics, King’s College London, London, WC2R 2LS, UK
Explanation of Dynamic Physical Field Predictions using WassersteinGrad: Application to Autoregressive Weather Forecasting
Apr 24, 2026Younes Essafouri, Laure Raynaud, Luciano Drozda +1Gradient-Based Attribution Neural Network Interpretability
Univ. Toulouse, INSA Toulouse, CNRS UMR 5219, IMT, Toulouse, France · Météo-France, CNRS, Univ. Toulouse, CNRM, Toulouse, France · Cerfacs, CNRS/Cerfacs/IRD, CECI, Toulouse, France
Contrastive Semantic Projection: Faithful Neuron Labeling with Contrastive Examples
Apr 24, 2026Oussama Bouanani, Jim Berend, Wojciech Samek +2Neural Network Interpretability
Fraunhofer Heinrich Hertz Institute, Berlin, Germany · Technische Universität Berlin, Berlin, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Berlin, Germany +1
H-Sets: Hessian-Guided Discovery of Set-Level Feature Interactions in Image Classifiers
Apr 23, 2026Ayushi Mehrotra, Dipkamal Bhusal, Michael Clifford +1Gradient-Based Attribution Feature Attribution
Rochester Institute of Technology · Toyota InfoTech Labs
Foundation models for discovering robust biomarkers of neurological disorders from dynamic functional connectivity
Apr 23, 2026Deepank Girish, Yi Hao Chan, Sukrit Gupta +2Medical Imaging Foundation Models Functional Connectivity
College of Computing and Data Science, Nanyang Technological University, Singapore
An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modeling
Apr 22, 2026Anif N. Shikder, Ramit Dey, Sayantan Auddy +6Dynamical Systems Neural Network Interpretability
Department of Mathematics, Western University, London ON, Canada · Fields Lab for Network Computation, Fields Institute, Toronto ON, Canada · King’s University College at Western University, London ON, Canada +1
Rethinking Intrinsic Dimension Estimation in Neural Representations
Apr 22, 2026Rickmer Schulte, David RügamerIntrinsic Dimensionality Neural Representation Geometry
LMU Munich, MCML
Bridging Traditional Explainability Methods and Multimodal Multilingual Models: An XAI-Based Analysis
Apr 21, 2026Paweł Pozorski, Jakub Muszyński, Maria GanzhaAudio-Text Alignment Multimodal Large Language Models
student record book number 327304 · student record book number 327246 · thesis supervisor
Decision-Aware Attention Propagation for Vision Transformer Explainability
Apr 20, 2026Sehyeong Jo, Gangjae Jang, Haesol ParkTransformer Interpretability Vision Transformer
Aarhus University · University of Colorado Boulder · KIST
The Topological Dual of a Dataset: A Logic-to-Topology Encoding for AlphaGeometry-Style Data
Apr 20, 2026Anthony BordgNeural Network Interpretability Neuro-Symbolic Reasoning
Huawei Lagrange Center, Paris, France
A Sugeno Integral View of Binarized Neural Network Inference
Apr 20, 2026Ismaïl Baaj, Henri PradeBinary Neural Networks Neural Network Interpretability
LEMMA, Paris-Panthéon-Assas University · IRIT - Toulouse University
From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Models
Apr 20, 2026Qidong Wang, Junjie Hu, Ming JiangMulti-Task Learning Causal Attribution
Tongji University · University of Wisconsin-Madison
Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
Apr 20, 2026Charles Ye, Bo Yuan, Lee SharkeyMixture-of-Experts Interpretability LLM Interpretability
Independent · Georgia Institute of Technology · Goodfire
Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
Apr 18, 2026Xinru Yan, Boxi Cao, Yaojie Lu +4Unified Multimodal Models Neural Network Interpretability
University of Chinese Academy of Sciences, Beijing, China · Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China

May 7, 2026J/K move · Enter open · S save

Concept-Based Abductive and Contrastive Explanations for Behaviors of Vision Models

Ronaldo Canizales, Divya Gopinath, Corina Păsăreanu, Ravi Mangal

Colorado State University, Fort Collins CO, USA · KBR Inc., NASA Ames, Moffett Field CA, USA · Carnegie Mellon University, Pittsburgh PA, USA

Abductive Reasoning Concept-Based Explanations Neural Network Interpretability Explainable Image Classification Causal Interpretability

Open paper AskPDF arXiv

Concept-based explanations offer a promising approach for explaining the predictions of deep neural networks in terms of high-level, human-understandable concepts. However, existing methods either do not establish a causal connection between the concepts and model predictions or are limited in expressivity and only able to infer causal explanations involving single concepts. At the same time, the parallel line of work on formal abductive and contrastive explanations computes the minimal set of input features causally relevant for model outcomes but only considers low-level features such as pixels. Merging these two threads, in this work, we propose the notion of concept-based abductive and contrastive explanations that capture the minimal sets of high-level concepts causally relevant for model outcomes. We then present a family of algorithms that enumerate all minimal explanations while using concept erasure procedures to establish causal relationships. By appropriately aggregating such explanations, we are not only able to understand model predictions on individual images but also on collections of images where the model exhibits a user-specified, common behavior. We evaluate our approach on multiple models, datasets, and behaviors, and demonstrate its effectiveness in computing helpful, user-friendly explanations.