Neural Network Interpretability

Ask about this topic

Overview

Neural network interpretability asks what a model has learned, which internal computations support its predictions, and whether those computations are understandable and reliable. Post-hoc methods include saliency and attribution, concept-based explanations, counterfactuals, and formal logical explanations; mechanistic work uses probes, sparse autoencoders, circuit analysis, and interventions on neurons or features. Another approach builds interpretability into models through sparse or additive architectures, prototypes, symbolic rules, and concept bottlenecks, with evaluations spanning image, language, time-series, and scientific tasks. A central challenge is showing that an explanation is both human-meaningful and faithful to the model: readable features or sparse circuits alone do not establish causal importance, and methods can vary in robustness and reliability across settings.

Latest papers

CardsList

333 papers37 papers in the last four weeks, up 23% on the four weeks before. 0.2% of all new papers.

  1. Improving the Intrinsic Interpretability of Differentiable Decision Trees through Euclidean Feature-Concentration Regularisation

    Oct 29, 2026Qiqi YangInterpretable MLDecision Tree Learning

  2. Interpretable neural equation learning for high-dimensional materials data

    Oct 27, 2026Philip Jacobson, Prasad Iyer, Remi Dingreville +1Symbolic RegressionNeural Network Interpretability

    Sandia National Laboratories, Livermore, CA, USA. · Center for Integrated Nanotechnologies, Sandia National Laboratories, Albuquerque , NM, USA.

  3. Structure, Not Sequence: Tracking and Validating Folding Mechanisms in ESMFold's Trunk with Sparse Autoencoders

    Oct 26, 2026Bohan Liu, Xiaohan Fan, Wenqian Ye +1Protein Structure PredictionNeural Network Interpretability

    University of Virginia Charlottesville VA 22903, USA

  4. Distilling neural networks into equations to predict planetary instability

    Oct 26, 2026Simon Alford, Edward Gu, Omar Costilla-Reyes +3Dynamical SystemsScientific ML

    Department of Computer Science, Cornell University, Ithaca, NY 14850, USA · Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA 02139, USA · Department of Physics, Harvey Mudd College, Claremont, CA 91711, USA +3

  5. Interpreting Physics in Video Encoders using Multi-Object Interaction Videos

    Oct 22, 2026Ayushi Mehrotra, Sai Vidyaranya NuthalapatiPhysical Reasoning in VideoNeural Network Interpretability

    Caltech

  6. Post-Hoc Feature Selection Layer for Neural Networks Interpretability

    Oct 22, 2026João Kenji Suwa, João Pedro Silveira e Silva, Bruno Iochins GrisciNeural Network InterpretabilityFeature Attribution

    Institute of Informatics, UFRGS, Porto Alegre, Brazil

  7. Sufficient Frame Subsets for Video Understanding

    Oct 18, 2026Lahiru WijayasinghaNeural Network InterpretabilityVideo Understanding

    Independent Researcher

  8. MoCKEFA: Model-Conditioned Keyframe Extraction and Fair Attribution for Video Model Explanation

    Oct 18, 2026Lahiru WijayasinghaNeural Network Interpretability

    Independent Researcher

  9. Interpretability of AI weather models via intermediate decoding

    Oct 16, 2026Matthias Beylich, Kirsten I. Tempest, Tobias Selz +1Interpretable MLNeural Network Interpretability

    Meteorological Institute, LMU Munich

  10. A Feature Selection Layer Approach for Multi-Class Classification on Neural Networks

    Oct 16, 2026João Pedro Silveira e Silva, Márcio Dorn, Bruno Iochins GrisciFeature SelectionFeature Attribution

    Institute of Informatics, Federal University of Rio Grande do Sul, Porto Alegre, RS, Brazil · Center for Biotechnology, Federal University of Rio Grande do Sul, Porto Alegre, RS, Brazil · National Institute of Science and Technology - Forensic Science, Porto Alegre, RS, Brazil

  11. Complete or Sparse: A Tale of Two Identifiabilities

    Oct 14, 2026Junze Zhou, David KlindtRepresentation IdentifiabilityRepresentation Learning

    Cold Spring Harbor Laboratory

  12. KANLab: TensorFlow Playground for Kolmogorov-Arnold Networks

    Oct 14, 2026Lukas Niehaus, Robin Rawiel, Ulf KrumnackKolmogorov-Arnold NetworksNeural Network Interpretability

    University Osnabrueck · University of Applied Sciences Osnabrueck

  13. Decomposing A Neural Learner

    Oct 9, 2026Zelong BiNeural Network InterpretabilityUncertainty Quantification

    Department of Mathematics & Statistics University of New South Wales High St Kensington NSW 2052 Australia

  14. Layer-wise Input Recoverability in Deep Neural Networks

    Oct 9, 2026Maxwell Spiridonov, Alireza Rahmansetayesh, Pierre BaldiConvolutional Neural NetworksNeural Network Interpretability

    Department of Computer Science, University of California, Irvine

  15. Neural networks are more modular than single neurons suggest

    Oct 9, 2026Shreya Saha, Benjamin Bergen, Meenakshi KhoslaModular Neural NetworksNeural Network Interpretability

    Electrical and Computer Engineering, University of California, San Diego · Cognitive Science Department, University of California, San Diego · Computer Science and Engineering, University of California, San Diego

  16. Kernelized Parameter Sensitivity Atlases for Amortized Conditional Interpretability

    Oct 8, 2026Davide Wiest, Ahanaf Hasan Ariq, Abdul Monaf ChowdhuryNeural Network InterpretabilityGradient-Based Attribution

    Technische Universität Darmstadt · Daffodil International School · University of Dhaka

  17. Bridging Human and Model Attention: Explainability Analysis of CNN, Mamba, and ViT Architectures with Gaze-Based Validation

    Oct 8, 2026Marco Stefani, Edoardo Parpaiola, Matteo Bino +5Neural Network InterpretabilitySaliency Map Evaluation

    Department of Information Engineering (DEI), University of Padova, Padova, Italy

  18. The Polytopal Neural Network

    Oct 8, 2026A. Emilie J. Wedenborg, Anders V. Nørskov, Teresa Dorszewski +2Neural Network InterpretabilityGeometric Representation Learning

    Technical University of Denmark · UiT The Arctic University of Norway

  19. RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing

    Oct 8, 2026Ilya Lasy, Nora Yinuo Cai, Kola AyonrindeMixture-of-Experts InterpretabilityMixture of Experts

    Faculty of Informatics, TU Wien · Independent · UK AI Security Institute

  20. For Those Who Believe in Faithfulness: Optimizing the Area Under Insertion and Deletion Curves for Ranking Relative Feature Importance

    Oct 7, 2026Bjørn Leth Møller, Bulat Ibragimov, Christian IgelFeature AttributionNeural Network Interpretability

    Department of Computer Science, University of Copenhagen, Denmark

  21. Feature Encoding in VAE-based Audio Decoders: Effects of Input, Depth and Distribution

    Oct 6, 2026Louis McCallum, Mick GriersonVariational AutoencodersAudio Representation Learning

    Creative Computing Institute, University of the Arts London, London, UK

  22. Interpretable Hypergraph Learning via Neural Additive Models

    Oct 5, 2026Shihan Feng, Xin Zheng, Shiyi Yang +3Neural Network InterpretabilityHypergraph Neural Networks

    School of Data and Information Sciences, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA · Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA · Department of Electrical and Computer Engineering, Illinois Institute of Technology, Chicago, IL 60616, USA +2

  23. Weight Oracles: Reading Neural Network Weights with Language Models

    Oct 5, 2026Krishna Kabra, Constantin Venhoff, Christian Schroeder de WittLLM AuditingBackdoor Detection

    University of Oxford

  24. A Graph-Based Inspection and Intervention Tool for Assessing Mechanistic Learning in PINNs

    Oct 4, 2026Adwait Patkhedkar, Alifaraz Lakhani, Prathmesh Mohite +1Physics-Informed MLMechanistic Interpretability

    Sardar Patel Institute of Technology, Mumbai, India

  25. Specific Algorithmic Interpretability of Neural Networks: A Case Study on Textures

    Oct 3, 2026Yanglin Zhang, Anneke von Seeger, Gilad Lerman +1Wavelet Scattering TransformsNeural Network Interpretability

    Technion – Israel Institute of Technology, Faculty of Mathematics · University of Minnesota, School of Mathematics

  26. The Independence Prior of SAEs Fragments Visual Concepts

    Oct 2, 2026Tommaso Mencattini, Giorgos Nikolaou, Donato Crisostomi +4Sparse AutoencodersNeural Network Interpretability

    ISTA · EPFL · Goodfire +2

  27. On architectural choices for interpretability and thermodynamic consistency in Physically Recurrent Neural Networks in the low-data regime

    Oct 2, 2026M. A. Maia, K. A. Meyer, A. M. C. M. van Gils +2Recurrent Neural NetworksNeural Network Interpretability

    Delft University of Technology, Department of Civil Engineering and Geosciences, P.O.Box 5048, 2600GA, Delft, The Netherlands · Chalmers University of Technology, Department of Mechanical Engineering, Gothenburg, Sweden

  28. Shapley-based Structural Analysis of Neural Calibration for Stochastic Volatility Models

    Oct 2, 2026Shaïn Afzali, Serena Della Corte, Antonis PapapantoleonModel CalibrationFeature Selection

  29. Temporal Geometry of Deep Networks: Hyperbolic Representations of Training Dynamics for Intrinsic Explainability

    Oct 2, 2026Ambarish MoharilNeural Network InterpretabilityNeural Network Training Dynamics

    Data Governance Department Jheronimus Academy of Data Science Eindhoven University of Technology

  30. CRISP: A Framework for Clause-Reconstructed Interpretable NeuroSymbolic Propositions

    Oct 1, 2026Alex Chan, Shafi Muhtasim Chowdhury, Ekin Can Erkuş +3Neural Network InterpretabilityNeuro-Symbolic Reasoning

    School of Engineering, Newcastle University, Newcastle upon Tyne, UK · Northumbria University, Newcastle upon Tyne, UK · Centre for Artificial Intelligence Research, University of Agder, Grimstad, Norway