Neural Network Interpretability

Follow this topic Ask about this topic

Neural network interpretability asks what a model has learned, which internal computations support its predictions, and whether those computations are understandable and reliable. Post-hoc methods include saliency and attribution, concept-based explanations, counterfactuals, and formal logical explanations; mechanistic work uses probes, sparse autoencoders, circuit analysis, and interventions on neurons or features. Another approach builds interpretability into models through sparse or additive architectures, prototypes, symbolic rules, and concept bottlenecks, with evaluations spanning image, language, time-series, and scientific tasks. A central challenge is showing that an explanation is both human-meaningful and faithful to the model: readable features or sparse circuits alone do not establish causal importance, and methods can vary in robustness and reliability across settings.

Latest papers

333 papers37 papers in the last four weeks, up 23% on the four weeks before. 0.2% of all new papers.

Sparse Crosscoders for diffing MoEs and Dense models
Mar 6, 2026Marmik Chaudhari, Nishkal Hundia, Idhant GulatiMixture-of-Experts Interpretability Mixture-of-Experts Models
University of California, Berkeley · University of Maryland
Model Medicine: A Clinical Framework for Understanding, Diagnosing, and Treating AI Models
Mar 5, 2026Jihoon JeongExplainable Artificial Intelligence Neural Network Interpretability
Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST) · ModuLabs
SCAN: Visual Explanations with Self-Confidence and Analysis Networks
Mar 4, 2026Gwanghee Lee, Sungyoon Jeong, Kyoungson JhangNeural Network Interpretability Explainable Image Classification
the Department of Computer Science and Engineering, Chungnam National University, Daejeon 34134, Republic of Korea · the Department of Artificial Intelligence, Chungnam National University, Daejeon 34134, Republic of Korea
Towards Explainable Deep Learning for Ship Trajectory Prediction in Inland Waterways
Mar 4, 2026Tom Legel, Dirk Söffker, Roland Schätzle +1Vessel Trajectory Prediction Multi-Agent Trajectory Prediction
Department Hydraulic Engineering in Inland Areas, Federal Waterways Engineering and Research Institute, Germany · Chair of Dynamics and Control, University of Duisburg-Essen, Germany · Department of Technology (Computer Science), Baden-Wuerttemberg Cooperative State University, Germany +1
Sparse autoencoders reveal organized biological knowledge but minimal regulatory logic in single-cell foundation models: a comparative atlas of Geneformer and scGPT
Mar 3, 2026Ihor KendiukhovGenomic Foundation Models Single-Cell Transcriptomics
Department of Computer Science, University of Tübingen, Tübingen, Germany
Enhancing Physics-Informed Neural Networks with Domain-aware Fourier Features: Towards Improved Performance and Interpretable Results
Mar 3, 2026Alberto Miño Calero, Luis Salamanca, Konstantinos E. TatsisFourier Feature Embeddings Neural Network Interpretability
Department of Engineering Cybernetics NTNU, Trondheim, Norway · Swiss Data Science Center (SDSC) ETH Zürich, Zürich, Switzerland
Embedding interpretable $\ell_1$ -regression into neural networks for uncovering temporal structure in cell imaging
Mar 3, 2026Fabian Kabus, Maren Hackenberg, Julia Hindel +6Autoregressive Modeling Neural Network Interpretability
Institute of Medical Biometry and Statistics (IMBI), Faculty of Medicine and Medical Center, University of Freiburg, Germany · Department of Computer Science, University of Freiburg, Germany · Institute of Physiology I, Faculty of Medicine, University of Freiburg, Germany +2
Explanation-Guided Adversarial Training for Robust and Interpretable Models
Mar 2, 2026Chao Chen, Yanhui Chen, Shanshan Lin +4Adversarial Training Adversarial Robustness
Harbin Institute of Technology (Shenzhen) · Fuzhou University · NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences
Bound Propagation meets Constraint Simplification: Improving Logic-based XAI for Neural Networks
Mar 2, 2026Ronaldo Gomes, Jairo Ribeiro, Luiz Queiroz +1Explainable Artificial Intelligence Neural Network Interpretability
Department of Computer Science, Instituto Federal do Ceará (IFCE), Maracanaú, Brazil
Interpretable Cross-Network Attention for Resting-State fMRI Representation Learning
Feb 28, 2026Karanpartap Singh, Adam Turnbull, Mohammad Abbasi +3Brain Network Analysis Functional Connectivity
Stanford University, Stanford, CA 94305
A Case Study on Concept Induction for Neuron-Level Interpretability in CNN
Feb 27, 2026Moumita Sen Sarma, Samatha Ereshi Akkamahadevi, Pascal HitzlerNeuron-Level Interpretability Neural Network Interpretability
Department of Computer Science, Kansas State University, Manhattan, Kansas, USA
Slice and Explain: Logic-Based Explanations for Neural Networks through Domain Slicing
Feb 25, 2026Luiz Fernando Paulino Queiroz, Carlos Henrique Leitão Cavalcante, Thiago Alves RochaNeural Network Interpretability
Instituto Federal do Ceará (IFCE), Maracanaú, Ceará, Brazil
Physics-Informed Machine Learning for Vessel Shaft Power and Fuel Consumption Prediction: Interpretable KAN-based Approach
Feb 25, 2026Hamza Haruna Mohammed, Dusica Marijan, Arnbjørn MaressaMaritime Transportation Physics-Informed ML
Simula Research Laboratory, Oslo, Norway · Navtor AS, Egersund, Norway
SymTorch: Symbolic Distillation of Neural Networks
Feb 24, 2026Elizabeth S. Z. Tan, Adil Soubki, Miles CranmerNeural Network Interpretability Scientific Discovery
Department of Applied Mathematics and Theoretical Physics University of Cambridge
Beyond Accuracy: An Explainability-Driven Analysis of Harmful Content Detection
Feb 24, 2026Trishita Dhara, Siddhesh ShethContent Moderation Neural Network Interpretability
Upper Hand, Indianapolis, IN 46204, USA · Ace Rent a Car, Indianapolis, IN 46204, USA
The Confusion is Real: GRAPHIC -- A Network Science Approach to Confusion Matrices in Deep Learning
Feb 23, 2026Johanna S. Fröhlich, Bastian Heinlein, Jan U. Claar +3Neural Network Interpretability Explainable Artificial Intelligence
Friedrich-Alexander-Universität Erlangen-Nürnberg Erlangen, Germany · Technical University of Darmstadt Darmstadt, Germany
Systematic Evaluation of Single-Cell Foundation Model Interpretability Reveals Attention Captures Co-Expression Rather Than Unique Regulatory Signal
Feb 19, 2026Ihor KendiukhovMechanistic Interpretability Neural Network Interpretability
Department of Computer Science, University of Tübingen, Tübingen, Germany
Provably Explaining Neural Additive Models
Feb 19, 2026Shahaf Bassan, Yizhak Yisrael Elboher, Tobias Ladner +4Interpretable ML Neural Network Interpretability
Hebrew University of Jerusalem, Israel · Technical University of Munich, Germany · Masaryk University, Czech Republic
Inelastic Constitutive Kolmogorov-Arnold Networks: A generalized framework for automated discovery of interpretable inelastic material models
Feb 19, 2026Chenyi Ji, Kian P. Abdolazizi, Hagen Holthusen +2Materials Science Neural Network Interpretability
Computational Mechanics in Medicine, Applied Medical Engineering, RWTH Aachen University Pauwelsstraße 20, 52074 Aachen, Germany · Institute for Continuum and Material Mechanics, Hamburg University of Technology Eißendorfer Straße 42, 21073 Hamburg, Germany · Institute of Applied Mechanics, University of Erlangen-Nuremberg Egerlandstrasse 5, 91058 Erlangen, Germany +1
What Do Neurons Listen To? A Neuron-level Dissection of a General-purpose Audio Model
Feb 17, 2026Takao Kawamura, Daisuke Niizumi, Nobutaka OnoAudio Representation Learning Neural Network Interpretability
Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo, Japan
Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks
Jan 29, 2026Kaustubh Sharma, Srijan Tiwari, Ojasva Nema +1Prior-Data Fitted Networks Neural Network Interpretability
Indian Institute of Technology Roorkee, India
Architecture Before the Formula: Training-Dependent Computational Semantics in Transformers
Jan 10, 2026Luis F. Rosario FreytesTransformer Training Dynamics Transformer Attention
When, How Long and How Much? Interpretable Neural Networks for Time Series Regression by Learning to Mask and Aggregate
Dec 3, 2025Florent Forest, Amaury Wei, Olga FinkConcept-Based Explanations Neural Network Interpretability
1IMOS Laboratory, EPFL, Lausanne, CH-1015, Switzerland.
Activation-Deactivation: A General Framework for Robust Post-hoc Explainable AI
Oct 1, 2025Akchunya Chanchal, David A. Kelly, Hana ChocklerExplainable Artificial Intelligence Explanation Stability
Department of Informatics King’s College London Strand, WC2R 2LS, London, UK
The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network
Aug 29, 2025Elias Sandmann, Sebastian Lapuschkin, Wojciech SamekNeural Algorithmic Reasoning Neural Network Interpretability
Fraunhofer HHI · TU Dublin · TU Berlin
Neural Logic Networks for Interpretable Classification
Aug 11, 2025Vincent Perreault, Katsumi Inoue, Richard Labib +1Boolean Function Learning Neural Network Interpretability
Department of Mathematics and Industrial Engineering Polytechnique Montréal · National Institute of Informatics, Tokyo
Position: Use Sparse Autoencoders to Discover Unknowns
Jun 30, 2025Kenny Peng, Rajiv Movva, Jon Kleinberg +2Explainable Artificial Intelligence Sparse Autoencoders
1Cornell University · 2UC Berkeley
Learning Interpretable Differentiable Logic Networks for Tabular Regression
May 29, 2025Chang Yue, Niraj K. JhaEfficient Neural Network Inference Differentiable Logic Gate Networks
Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USA
Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks
May 26, 2025Safa Hamreras, Sukhbinder Singh, Román OrúsTensor Networks Neural Network Interpretability
Donostia International Physics Center, Paseo Manuel de Lardizabal 4, E-20018 San Sebastián, Spain · Multiverse Computing, Spadina Ave., Toronto, ON M5T 2C2, Canada · Multiverse Computing, Paseo de Miramón 170, E-20014 San Sebastián, Spain
Explainable embeddings with Distance Explainer
May 21, 2025Christiaan Meijer, E. G. Patrick BosFeature Attribution Perturbation-Based Feature Attribution
Netherlands eScience Center, Science Park 402, 1098 XH Amsterdam, Netherlands · Center for Information Technology, University of Groningen, P.O. Box 11044, 9700 CA Groningen, Netherlands

Mar 6, 2026J/K move · Enter open · S save

Sparse Crosscoders for diffing MoEs and Dense models

Marmik Chaudhari, Nishkal Hundia, Idhant Gulati

University of California, Berkeley · University of Maryland

Mixture-of-Experts Interpretability Mixture-of-Experts Models Neural Network Interpretability Sparse Autoencoders Representation Geometry in Language Models

Open paper AskPDF arXiv

Mixture of Experts (MoE) achieve parameter-efficient scaling through sparse expert routing, yet their internal representations remain poorly understood compared to dense models. We present a systematic comparison of MoE and dense model internals using crosscoders, a variant of sparse autoencoders, that jointly models multiple activation spaces. We train 5-layer dense and MoEs (equal active parameters) on 1B tokens across code, scientific text, and english stories. Using BatchTopK crosscoders with explicitly designated shared features, we achieve

\sim 87\%

fractional variance explained and uncover concrete differences in feature organization. The MoE learns significantly fewer unique features compared to the dense model. MoE-specific features also exhibit higher activation density than shared features, whereas dense-specific features show lower density. Our analysis reveals that MoEs develop more specialized, focused representations while dense models distribute information across broader, more general-purpose features.