Feature Attribution

Ask about this topic

Overview

Feature attribution studies how to assign responsibility for a model’s prediction to parts of its input, such as image regions, text tokens, time-series measurements, or graph features. Common approaches include gradients and relevance propagation, perturbation and masking, Shapley values, and counterfactual or causal interventions; some methods instead build explanations into the model. Researchers evaluate explanations for faithfulness, stability, sparsity, localization, and usefulness across tasks such as image classification, language modeling, medical prediction, and time-series forecasting. A central challenge is that attributions can change with the model architecture, baseline, perturbation choices, or correlations among features, so no single method or metric is reliable in every setting.

Latest papers

CardsList

234 papers20 papers in the last four weeks, down 17% on the four weeks before. 0.1% of all new papers.

Apr 15, 2026

Post-Hoc Uncertainty-Aware Explanations for Deployed Power Quality Disturbance Classifiers via Laplace Approximation

Deep learning classifiers achieve high accuracy in power quality disturbance (PQD) recognition, but existing explanation methods return a single deterministic attribution map and provide no measure of its reliability. This paper develops a post-hoc Bayesian explanation (B-explanation) method for trained PQD classifiers. A computationally efficient Laplace approximation converts the trained network into an approximate parameter posterior without retraining, and occlusion sensitivity is propagated through posterior samples to produce a distribution over disturbance-localization maps. Percentile summaries of this distribution yield explanations with distribution-free coverage bands: consensus summaries at low percentiles sharpen localization significantly for distinctive events such as sags, swells, and oscillatory transients, the band width indicates the reliability of each attribution, and the remaining disturbance types show class-dependent behavior. Explanation dispersion also increases under injected measurement noise and synthetic-to-field transfer, complementing predictive uncertainty. Experiments on a synthetic benchmark of 15 disturbance classes and on field-recorded sags compare the method with Monte Carlo dropout and deep ensembles under a common evaluation protocol, evaluate it against deterministic occlusion, LIME, and SHAP with localization and faithfulness metrics, and characterize the computational cost of explanation generation for grid monitoring applications.

Yinsong Chen, Samson S. Yu, Kashem M. Muttaqi

School of Engineering, Deakin University, Melbourne, 3216, VIC, Australia · ARC Training Centre in Energy Technologies for Future Grids, School of Engineering, University of Wollongong, Wollongong, 2522, NSW, Australia

Explainable Artificial IntelligenceFeature AttributionUncertainty QuantificationExplainability Evaluation
AskPDF
Apr 14, 2026

Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs

Attribution methods seek to explain language model predictions by quantifying the contribution of input tokens to generated outputs. However, most existing techniques are designed for encoder-based architectures and rely on linear approximations that fail to capture the causal and semantic complexities of autoregressive generation in decoder-only models. To address these limitations, we propose Hessian-Enhanced Token Attribution (HETA), a novel attribution framework tailored for decoder-only language models. HETA combines three complementary components: a semantic transition vector that captures token-to-token influence across layers, Hessian-based sensitivity scores that model second-order effects, and KL divergence to measure information loss when tokens are masked. This unified design produces context-aware, causally faithful, and semantically grounded attributions. Additionally, we introduce a curated benchmark dataset for systematically evaluating attribution quality in generative settings. Empirical evaluations across multiple models and datasets demonstrate that HETA consistently outperforms existing methods in attribution faithfulness and alignment with human annotations, establishing a new standard for interpretability in autoregressive language models.

Vishal Pramanik, Maisha Maliha, Nathaniel D. Bastian +1

Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL 32611, USA · School of Computer Science, University of Oklahoma, Norman, OK 73019, USA · Department of Electrical Engineering and Computer Science, United States Military Academy, West Point, NY 10996, USA

Feature AttributionLLM InterpretabilityFaithfulness of Language Model ExplanationsDecoder-Only Language Models
AskPDF
Apr 13, 2026

Trustworthy Feature Importance Avoids Unrestricted Permutations

Feature importance methods using unrestricted permutations are flawed due to extrapolation errors; such errors appear in all non-trivial variable importance approaches. We propose three new approaches: conditional model reliance and Knockoffs with Gaussian transformation, and restricted ALE plot designs. Theoretical and numerical results show our strategies reduce/eliminate extrapolation.

Emanuele Borgonovo, Francesco Cappelli, Xuefei Lu +2

Department of Decision Sciences and Bocconi Institute for Data Science and Analytics, Bocconi University, Milan, Italy · Department of Decision Science, Bocconi University, Milan, Italy · SKEMA Business School, Université Côte d’Azur, France +2

Feature AttributionPerturbation-Based Feature Attribution
AskPDF
Apr 13, 2026

Regional Explanations: Bridging Local and Global Variable Importance

We analyze two widely used local attribution methods, Local Shapley Values and LIME, which aim to quantify the contribution of a feature value xix_i to a specific prediction f(x1,…,xp)f(x_1, \dots, x_p). Despite their widespread use, we identify fundamental limitations in their ability to reliably detect locally important features, even under ideal conditions with exact computations and independent features. We argue that a sound local attribution method should not assign importance to features that neither influence the model output (e.g., features with zero coefficients in a linear model) nor exhibit statistical dependence with functionality-relevant features. We demonstrate that both Local SV and LIME violate this fundamental principle. To address this, we propose R-LOCO (Regional Leave Out COvariates), which bridges the gap between local and global explanations and provides more accurate attributions. R-LOCO segments the input space into regions with similar feature importance characteristics. It then applies global attribution methods within these regions, deriving an instance's feature contributions from its regional membership. This approach delivers more faithful local attributions while avoiding local explanation instability and preserving instance-specific detail often lost in global methods.

Salim I. Amoukou, Nicolas J-B. Brunel

J.P. Morgan AI Research · LaMME, ENSIIE, University Paris Saclay & Capgemini Invent

Feature AttributionExplainable Artificial IntelligenceShapley Value Attribution
AskPDF
Apr 13, 2026

ShapShift: Explaining Model Prediction Shifts with Subgroup Conditional Shapley Values

Changes in input distribution can induce shifts in the average predictions of machine learning models. Such prediction shifts may impact downstream business outcomes (e.g. a bank's loan approval rate), so understanding their causes can be crucial. We propose \ours{}: a Shapley value method for attributing prediction shifts to changes in the conditional probabilities of interpretable subgroups of data, where these subgroups are defined by the structure of decision trees. We initially apply this method to single decision trees, providing exact explanations based on conditional probability changes at split nodes. Next, we extend it to tree ensembles by selecting the most explanatory tree and accounting for residual effects. Finally, we propose a model-agnostic variant using surrogate trees grown with a novel objective function, allowing application to models like neural networks. While exact computation can be intensive, approximation techniques enable practical application. We show that \ours{} provides simple, faithful, and near-complete explanations of prediction shifts across model classes, aiding model monitoring in dynamic environments.

Tom Bewley, Salim I. Amoukou, Emanuele Albini +2

J.P. Morgan AI Research

Shapley Value AttributionFeature AttributionCovariate Shift
AskPDF
Apr 13, 2026

Diffusion-CAM: Faithful Visual Explanations for dMLLMs

While diffusion Multimodal Large Language Models (dMLLMs) have recently achieved remarkable strides in multimodal generation, the development of interpretability mechanisms has lagged behind their architectural evolution. Unlike traditional autoregressive models that produce sequential activations, diffusion-based architectures generate tokens via parallel denoising, resulting in smooth, distributed activation patterns across the entire sequence. Consequently, existing Class Activation Mapping (CAM) methods, which are tailored for local, sequential dependencies, are ill-suited for interpreting these non-autoregressive behaviors. To bridge this gap, we propose Diffusion-CAM, the first interpretability method specifically tailored for dMLLMs. We derive raw activation maps by differentiably probing intermediate representations in the transformer backbone, accordingly capturing both latent features and their class-specific gradients. To address the inherent stochasticity of these raw signals, we incorporate four key modules to resolve spatial ambiguity and mitigate intra-image confounders and redundant token correlations. Extensive experiments demonstrate that Diffusion-CAM significantly outperforms SoTA methods in both localization accuracy and visual fidelity, establishing a new standard for understanding the parallel generation process of diffusion multimodal systems.

Haomin Zuo, Yidi Li, Luoxiao Yang +1

Sun Yat-sen University · Northwestern University · Technion - Israel Institute of Technology +1

Multimodal Model InterpretabilityFeature AttributionMultimodal Diffusion Models
AskPDF
Apr 10, 2026

Investigating Trustworthiness of Nonparametric Deep Survival Models for Alzheimer's Disease Progression Analysis

Alzheimer's Dementia (AD) is a progressive neurodegenerative disease marked by irreversible decline, making reliable modeling of its progression essential for effective patient care. Progression-aware methods such as survival analysis are therefore crucial tools for the early detection and monitoring of AD. Recent advancements in deep learning have demonstrated remarkable performance in survival tasks, but alarmingly fewer studies have been conducted in the domain of AD. Further, the studies that do exist do not consider learned bias within the model itself, which could result in unfair and unreliable predictions toward certain marginalized groups. As such, we conduct a rigorous study of fairness in AD progression analysis along with a thorough feature importance study to determine the characteristics which are most important for reliable AD predictions. Furthermore, we propose two novel fairness metrics, called Time-Dependent Concordance Impurity and Kaplan-Meier Fairness, to quantify bias with respect to sensitive attributes such as sex, race, and education in nonparametric survival models. Our study demonstrates that while deep learning powered survival models are robust tools which can aid clinicians in AD care decisions, they often exhibit considerable bias, representing important avenues for future research.

Jacob Thrasher, Kaitlyn Heintzelman, Peter Martone +4

West Virginia University Morgantown, USA · University of Aberdeen Aberdeen, Scotland

Alzheimer's Disease Progression PredictionAlgorithmic FairnessSurvival AnalysisFeature Attribution
AskPDF
Apr 8, 2026

ADAG: Automatically Describing Attribution Graphs

In language model interpretability research, circuit tracing aims to identify which internal features causally contributed to a particular output and how they affected each other, with the goal of explaining the computations underlying some behaviour. However, all prior circuit tracing work has relied on ad-hoc human interpretation of the role that each feature in the circuit plays, via manual inspection of data artifacts such as the dataset examples the component activates on. We introduce ADAG, an end-to-end pipeline for describing these attribution graphs which is fully automated. To achieve this, we introduce attribution profiles which quantify the functional role of a feature via its input and output gradient effects. We then introduce a novel clustering algorithm for grouping features, and an LLM explainer--simulator setup which generates and scores natural-language explanations of the functional role of these feature groups. We run our system on known human-analysed circuit-tracing tasks and recover interpretable circuits, and further show ADAG can find steerable clusters which are responsible for a harmful advice jailbreak in Llama 3.1 8B Instruct.

Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt +1

Stanford University · Transluce

Mechanistic InterpretabilityCircuits in Language ModelsFeature Attribution
AskPDF
Apr 8, 2026

Non-identifiability of Explanations from Model Behavior in Deep Networks of Image Authenticity Judgments

Deep neural networks can predict human judgments, but this does not imply that they rely on human-like information or reveal the cues underlying those judgments. Prior work has addressed this issue using attribution heatmaps, but their explanatory value in itself depends on robustness. Here we tested the robustness of such explanations by evaluating whether models that predict human authenticity ratings also produce consistent explanations within and across architectures. We fit lightweight regression heads to multiple frozen pretrained vision models and generated attribution maps using Grad-CAM, LIME, and multiscale pixel masking. Several architectures predicted ratings well, reaching about 80% of the noise ceiling. VGG models achieved this by tracking image quality rather than authenticity-specific variance, limiting the relevance of their attributions. Among the remaining models, attribution maps were generally stable across random seeds within an architecture, especially for EfficientNetB3 and Barlow Twins, and consistency was higher for images judged as more authentic. Crucially, agreement in attribution across architectures was weak even when predictive performance was similar. To address this, we combined models in ensembles, which improved prediction of human authenticity judgments and enabled image-level attribution via pixel masking. We conclude that while deep networks can predict human authenticity judgments well, they do not produce identifiable explanations for those judgments. More broadly, our findings suggest that post hoc explanations from successful models of behavior should be treated as weak evidence for cognitive mechanism.

Icaro Re Depaolini, Uri Hasson

Center for Mind/Brain Sciences, The University of Trento, Trento, Italy

Explainability EvaluationFeature AttributionPerturbation-Based Feature Attribution
AskPDF
Apr 8, 2026

The Attribution Impossibility: No Feature Ranking Is Faithful, Stable, and Complete Under Collinearity

No feature ranking can be simultaneously faithful, stable, and complete when features are collinear. For collinear pairs, ranking reduces to a coin flip. We prove this impossibility, quantify it for four model classes, resolve it via ensemble averaging (DASH), and machine-verify it with 305 Lean 4 theorems. We characterize the complete attribution design space: exactly two families of methods exist -- faithful-complete methods (unstable, with rankings that flip up to 50% of the time) and ensemble methods like DASH (stable, reporting ties for symmetric features) -- and no method lies outside this dichotomy. The impossibility is quantitative: the attribution ratio diverges as 1/(1-rho^2) for gradient boosting, is infinite for Lasso, and converges for random forests. DASH (Diversified Aggregation of SHAP) is provably Pareto-optimal among unbiased aggregations, achieving the Cramer-Rao variance bound with a tight ensemble size formula. In a survey of 77 public datasets, 68% exhibit attribution instability. Switching to conditional SHAP does not escape the impossibility when features have equal causal effects. The framework includes practical diagnostics -- a Z-test workflow and single-model screening tool -- and has direct consequences for fairness auditing: SHAP-based proxy discrimination audits are provably unreliable under collinearity. The design space theorem, diagnostics, and impossibility are mechanically verified in Lean 4 (305 theorems from 16 axioms, 0 sorry) -- to our knowledge, the first formally verified impossibility in explainable AI.

Drake Caraker, Bryan Arnold, David Rhoads

Independent Researchers

Feature AttributionSHAP Feature Attribution
AskPDF
Apr 7, 2026

Learn to Rank: Visual Attribution by Learning Importance Ranking

Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical domains. An established approach to interpretability is generating visual attribution maps that highlight regions of the input most relevant to the model's prediction. However, existing methods face a three-way trade-off. Propagation-based approaches are efficient, but they can be biased and architecture-specific. Meanwhile, perturbation-based methods are causally grounded, yet they are expensive and for vision transformers often yield coarse, patch-level explanations. Learning-based explainers are fast but usually optimize surrogate objectives or distill from heuristic teachers. We propose a learning scheme that instead optimizes deletion and insertion metrics directly. Since these metrics depend on non-differentiable sorting and ranking, we frame them as permutation learning and replace the hard sorting with a differentiable relaxation using Gumbel-Sinkhorn. This enables end-to-end training through attribution-guided perturbations of the target model. During inference, our method produces dense, pixel-level attributions in a single forward pass with optional, few-step gradient refinement. Our experiments demonstrate consistent quantitative improvements and sharper, boundary-aligned explanations, particularly for transformer-based vision models. Code and pretrained models are available at https://github.com/dschinagl/AHA.

David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch +2

Institute of Visual Computing (IVC), Graz University of Technology, Graz, Austria · Nomadic AI, USA · Amazon, USA

Feature AttributionExplainable Image ClassificationTransformer Interpretability
AskPDF
Apr 7, 2026

Attribution-Driven Explainable Intrusion Detection with Encoder-Based Large Language Models

Software-Defined Networking (SDN) improves network flexibility but also increases the need for reliable and interpretable intrusion detection. Large Language Models (LLMs) have recently been explored for cybersecurity tasks due to their strong representation learning capabilities; however, their lack of transparency limits their practical adoption in security-critical environments. Understanding how LLMs make decisions is therefore essential. This paper presents an attribution-driven analysis of encoder-based LLMs for network intrusion detection using flow-level traffic features. Attribution analysis demonstrates that model decisions are driven by meaningful traffic behavior patterns, improving transparency and trust in transformer-based SDN intrusion detection. These patterns align with established intrusion detection principles, indicating that LLMs learn attack behavior from traffic dynamics. This work demonstrates the value of attribution methods for validating and trusting LLM-based security analysis.

Umesh Biswas, Shafqat Hasan, Syed Mohammed Farhan +2

Computer Science & Engineering, Mississippi State University

Explainable Intrusion DetectionLLMs for CybersecurityNetwork Intrusion DetectionLLM InterpretabilityFeature Attribution
AskPDF
Mar 31, 2026

Structural Compactness as a Complementary Criterion for Explanation Quality

In the evaluation of attribution quality, the quantitative assessment of explanation legibility is particularly difficult, as it is influenced by varying shapes and internal organization of attributions not captured by simple statistics. To address this issue, we introduce Minimum Spanning Tree Compactness (MST-C), a graph-based structural metric that captures higher-order geometric properties of attributions, such as spread and cohesion. These components are combined into a single score that evaluates compactness, favoring attributions with salient points spread across a small area and spatially organized into few but cohesive clusters. We show that MST-C reliably distinguishes between explanation methods, exposes fundamental structural differences between models, and provides a robust, self-contained diagnostic for explanation compactness that complements existing notions of attribution complexity.

Mohammad Mahdi Mesgari, Jackie Ma, Wojciech Samek +2

Fraunhofer Heinrich Hertz Institute, Berlin, Germany · Technische Universität Berlin, Berlin, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Berlin, Germany +1

Explainability EvaluationExplanation EvaluationFeature AttributionSaliency Map Evaluation
AskPDF
Mar 27, 2026

TTE-CAM: Self-Explainable Class Activation Maps for Pretrained Black-Box CNNs

Convolutional neural networks (CNNs) achieve state-of-the-art performance in medical image analysis yet remain opaque, limiting adoption in high-stakes clinical settings. Existing approaches face a fundamental trade-off: post-hoc methods provide unfaithful approximate explanations, while inherently interpretable architectures are faithful but often sacrifice predictive performance. We introduce TTE-CAM, a test-time framework that bridges this gap by converting pretrained black-box CNNs into self-explainable models via a convolution-based replacement of their classification head, initialized from the original weights. The resulting model preserves black-box predictive performance while delivering built-in faithful explanations competitive with post-hoc methods, both qualitatively and quantitatively. The code is available at https://github.com/kdjoumessi/Test-Time-Explainability

Kerol Djoumessi, Philipp Berens

Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Tübingen AI Center, University of Tübingen, Germany

Explainable Image ClassificationNeural Network InterpretabilityExplainable Medical Image AnalysisFeature AttributionConvolutional Neural Networks
AskPDF
Mar 26, 2026

End-to-end Feature Alignment: A Simple CNN with Intrinsic Class Attribution

We present Feature-Align CNN (FA-CNN), a prototype CNN architecture with intrinsic class attribution through end-to-end feature alignment. Our intuition is that the use of unordered operations such as Linear and Conv2D layers cause unnecessary shuffling and mixing of semantic concepts, thereby making raw feature maps difficult to understand. We introduce two new order preserving layers, the dampened skip connection, and the global average pooling classifier head. These layers force the model to maintain an end-to-end feature alignment from the raw input pixels all the way to final class logits. This end-to-end alignment enhances the interpretability of the model by allowing the raw feature maps to intrinsically exhibit class attribution. We prove theoretically that FA-CNN penultimate feature maps are identical to Grad-CAM saliency maps. Moreover, we prove that these feature maps slowly morph layer-by-layer over network depth, showing the evolution of features through network depth toward penultimate class activations. FA-CNN performs well on benchmark image classification datasets. Moreover, we compare the averaged FA-CNN raw feature maps against Grad-CAM and permutation methods in a percent pixels removed interpretability task. We conclude this work with a discussion and future, including limitations and extensions toward hybrid models.

Parniyan Farvardin, David Chapman

University of Miami

Neural Network InterpretabilityExplainable Image ClassificationConvolutional Neural NetworksFeature AttributionGradient-Based AttributionSaliency Map Evaluation
AskPDF
Mar 25, 2026

Combi-CAM: A Novel Multi-Layer Approach for Explainable Image Geolocalization

Planet-scale photo geolocalization involves the intricate task of estimating the geographic location depicted in an image purely based on its visual features. While deep learning models, particularly convolutional neural networks (CNNs), have significantly advanced this field, understanding the reasoning behind their predictions remains challenging. In this paper, we present Combi-CAM, a novel method that enhances the explainability of CNN-based geolocalization models by combining gradient-weighted class activation maps obtained from several layers of the network architecture, rather than using only information from the deepest layer as is typically done. This approach provides a more detailed understanding of how different image features contribute to the model's decisions, offering deeper insights than the traditional approaches.

David Faget, José Luis Lisani, Miguel Colom

CB · Centre Borelli, ENS Paris-Saclay, Université de Paris, CNRS, INSERM, SSA, France · CB, CMLA +1

Feature AttributionGradient-Based Attribution
AskPDF
Mar 23, 2026

Toward Faithful Segmentation Attribution via Benchmarking and Dual-Evidence Fusion

Attribution maps for semantic segmentation are almost always judged by visual plausibility. Yet looking convincing does not guarantee that the highlighted pixels actually drive the model's prediction, nor that attribution credit stays within the target region. These questions require a dedicated evaluation protocol. We introduce a reproducible benchmark that tests intervention-based faithfulness, off-target leakage, perturbation robustness, and runtime on Pascal VOC and SBD across three pretrained backbones. To further demonstrate the benchmark, we propose Dual-Evidence Attribution (DEA), a lightweight correction that fuses gradient evidence with region-level intervention signals through agreement-weighted fusion. DEA increases emphasis where both sources agree and retains causal support when gradient responses are unstable. Across all completed runs, DEA consistently improves deletion-based faithfulness over gradient-only baselines and preserves strong robustness, at the cost of additional compute from intervention passes. The benchmark exposes a faithfulness-stability tradeoff among attribution families that is entirely hidden under visual evaluation, providing a foundation for principled method selection in segmentation explainability. Code is available at https://github.com/anmspro/DEA.

Abu Noman Md Sakib, OFM Riaz Rahman Aranya, Kevin Desai +1

The University of Texas at San Antonio

Feature AttributionExplainability EvaluationGradient-Based AttributionPerturbation-Based Feature Attribution
AskPDF
Mar 22, 2026

CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphs

Mechanistic interpretability seeks to understand how Large Language Models (LLMs) represent and process information. Recent approaches based on dictionary learning and transcoders enable representing model computation in terms of sparse, interpretable features and their interactions, giving rise to feature attribution graphs. However, these graphs are often large and redundant, limiting their interpretability in practice. Cross-Layer Transcoders (CLTs) address this issue by sharing features across layers while preserving layer-specific decoding, yielding more compact representations, but remain difficult to train and analyze at scale. We introduce an open-source library for end-to-end training and interpretability of CLTs. Our framework integrates scalable distributed training with model sharding and compressed activation caching, a unified automated interpretability pipeline for feature analysis and explanation, attribution graph computation using Circuit-Tracer, and a flexible visualization interface. This provides a practical and unified solution for scaling CLT-based mechanistic interpretability. Our code is available at: https://github.com/LLM-Interp/CLT-Forge.

Florent Draye, Vedant Palit, Abir Harrasse +8

Max Planck Institute for Intelligent Systems, Tübingen, Germany · Jinesis AI Lab, University of Toronto & Vector Institute · CMU +2

LLM InterpretabilityMechanistic InterpretabilityFeature Attribution
AskPDF
Mar 20, 2026

Shift-Invariant Feature Attribution in the Application of Wireless Electrocardiograms

Assigning relevance scores to the input features of a machine learning model enables to measure the contributions of the features in achieving a correct outcome. It is regarded as one of the approaches towards developing explainable models. For biomedical assignments, this is very useful for medical experts to comprehend machine-based decisions. In the analysis of electro cardiogram (ECG) signals, in particular, understanding which of the electrocardiogram samples or features contributed most for a given decision amounts to understanding the underlying cardiac phases or conditions the machine tries to explain. For the computation of relevance scores, determining the proper baseline is important. Moreover, the scores should have a distribution which is at once intuitive to interpret and easy to associate with the underline cardiac reality. The purpose of this work is to achieve these goals. Specifically, we propose a shift-invariant baseline which has a physical significance in the analysis as well as interpretation of electrocardiogram measurements. Moreover, we aggregate significance scores in such a way that they can be mapped to cardiac phases. We demonstrate our approach by inferring physical exertion from cardiac exertion using a residual network. We show that the ECG samples which achieved the highest relevance scores (and, therefore, which contributed most to the accurate recognition of the physical exertion) are those associated with the P and T waves. Index Terms Attribution, baseline, cardiovascular diseases, electrocardiogram, activity recognition, machine learning

Yalemzerf Getnet, Abiy Tasissa, Waltenegus Dargie

Department of Electrical and Computer Engineering at Addis Ababa University, Ethiopia · Department of Mathematics, Tufts University, USA · Faculty of Computer Science, Technische Universität Dresden, 01062 Dresden, Germany

Feature AttributionElectrocardiography
AskPDF
Mar 20, 2026

Ensembles-based Feature Guided Analysis

Recent Deep Neural Networks (DNN) applications ask for techniques that can explain their behavior. Existing solutions, such as Feature Guided Analysis (FGA), extract rules on their internal behaviors, e.g., by providing explanations related to neurons activation. Results from the literature show that these rules have considerable precision (i.e., they correctly predict certain classes of features), but the recall (i.e., the number of situations these rule apply) is more limited. To mitigate this problem, this paper presents Ensembles-based Feature Guided Analysis (EFGA). EFGA combines rules extracted by FGA into ensembles. Ensembles aggregate different rules to increase their applicability depending on an aggregation criterion, a policy that dictates how to combine rules into ensembles. Although our solution is extensible, and different aggregation criteria can be developed by users, in this work, we considered three different aggregation criteria. We evaluated how the choice of the criterion influences the effectiveness of EFGA on two benchmarks (i.e., the MNIST and LSC datasets), and found that different aggregation criteria offer alternative trade-offs between precision and recall. We then compare EFGA with FGA. For this experiment, we selected an aggregation criterion that provides a reasonable trade-off between precision and recall. Our results show that EFGA has higher train recall (+28.51% on MNIST, +33.15% on LSC), and test recall (+25.76% on MNIST, +30.81% on LSC) than FGA, with a negligible reduction on the test precision (-0.89% on MNIST, -0.69% on LSC).

Federico Formica, Stefano Gregis, Andrea Rota +3

McMaster University, Canada · University of Bergamo, Italy

Neural Network InterpretabilityFeature AttributionEnsemble Learning
AskPDF
Mar 17, 2026

Attribution Upsampling should Redistribute, Not Interpolate

Attribution methods in explainable AI rely on upsampling techniques that were designed for natural images, not saliency maps. Standard bilinear and bicubic interpolation systematically corrupts attribution signals through aliasing, ringing, and boundary bleeding, producing spurious high-importance regions that misrepresent model reasoning. We identify that the core issue is treating attribution upsampling as an interpolation problem that operates in isolation from the model's reasoning, rather than a mass redistribution problem where model-derived semantic boundaries must govern how importance flows. We present Universal Semantic-Aware Upsampling (USU), a principled method that reformulates upsampling through ratio-form mass redistribution operators, provably preserving attribution mass and relative importance ordering. Extending the axiomatic tradition of feature attribution to upsampling, we formalize four desiderata for faithful upsampling and prove that interpolation structurally violates three of them. These same three force any redistribution operator into a ratio form; the fourth selects the unique potential within this family, yielding USU. Controlled experiments on models with known attribution priors verify USU's formal guarantees; evaluation across ImageNet, CIFAR-10, and CUB-200 confirms consistent faithfulness improvements and qualitatively superior, semantically coherent explanations.

Vincenzo Buono, Peyman Sheikholharam Mashhadi, Mahmoud Rahat +2

Halmstad University, Halmstad, Sweden

Feature AttributionSaliency Map Evaluation
AskPDF
Mar 16, 2026

Hypothesis Class Determines Explanation: Why Accurate Models Disagree on Feature Attribution

The assumption that prediction-equivalent models produce equivalent explanations underlies many practices in explainable AI, including model selection, auditing, and regulatory evaluation. In this work, we show that this assumption does not hold. Through a large-scale empirical study across 24 datasets and multiple model classes, we find that models with identical predictive behavior can produce substantially different feature attributions. This disagreement is highly structured: models within the same hypothesis class exhibit strong agreement, while cross-class pairs (e.g., tree-based vs. linear) trained on identical data splits show substantially reduced agreement, consistently near or below the lottery threshold. We identify hypothesis class as the structural driver of this phenomenon, which we term the Explanation Lottery. We theoretically show that the resulting Agreement Gap persists under interaction structure in the data-generating process. This structural finding motivates a post-hoc diagnostic, the Explanation Reliability Score R(x), which predicts when explanations are stable across architectures without additional training. Our results demonstrate that model selection is not explanation-neutral: the hypothesis class chosen for deployment can determine which features are attributed responsibility for a decision.

Thackshanaramana B

SRM Institute of Science and Technology, India

Feature AttributionExplainability EvaluationModel Selection
AskPDF
Mar 16, 2026

GradCFA: A Hybrid Gradient-Based Counterfactual and Feature Attribution Explanation Algorithm for Local Interpretation of Neural Networks

Explainable Artificial Intelligence (XAI) is increasingly essential as AI systems are deployed in critical fields such as healthcare and finance, offering transparency into AI-driven decisions. Two major XAI paradigms, counterfactual explanations (CFX) and feature attribution (FA), serve distinct roles in model interpretability. This study introduces GradCFA, a hybrid framework combining CFX and FA to improve interpretability by explicitly optimizing feasibility, plausibility, and diversity - key qualities often unbalanced in existing methods. Unlike most CFX research focused on binary classification, GradCFA extends to multi-class scenarios, supporting a wider range of applications. We evaluate GradCFA's validity, proximity, sparsity, plausibility, and diversity against state-of-the-art methods, including Wachter, DiCE, CARE for CFX, and SHAP for FA. Results show GradCFA effectively generates feasible, plausible, and diverse counterfactuals while offering valuable FA insights. By identifying influential features and validating their impact, GradCFA advances AI interpretability. The code for implementation of this work can be found at: https://github.com/jacob-ws/GradCFs .

Jacob Sanderson, Hua Mao, Wai Lok Woo

Department of Computer and Information Sciences, Northumbria University, Newcastle, UK, NE1 8ST

Counterfactual ExplanationsNeural Network InterpretabilityFeature AttributionGradient-Based Attribution
AskPDF
Mar 16, 2026

xplainfi: Feature Importance and Statistical Inference for Machine Learning in R

We introduce xplainfi, an R package built on top of the mlr3 ecosystem for global, loss-based feature importance methods for machine learning models. Various feature importance methods exist in R, but significant gaps remain, particularly regarding conditional importance methods and associated statistical inference procedures. The package implements permutation feature importance, conditional feature importance, relative feature importance, leave-one-covariate-out, and generalizations thereof, and both marginal and conditional Shapley additive global importance methods. It provides a modular conditional sampling architecture based on Gaussian distributions, adversarial random forests, conditional inference trees, and knockoff-based samplers, which enable conditional importance analysis for continuous and mixed data. Statistical inference is available through multiple approaches, including variance-corrected confidence intervals and the conditional predictive impact framework. We demonstrate that xplainfi produces importance scores consistent with existing implementations across multiple simulation settings and learner types, while offering competitive runtime performance. The package is available on CRAN and provides researchers and practitioners with a comprehensive toolkit for feature importance analysis and model interpretation in R.

Lukas Burk, Fiona Katharina Ewald, Giuseppe Casalicchio +2

Feature AttributionPerturbation-Based Feature AttributionSHAP Feature Attribution
AskPDF
Mar 14, 2026

Aumann-SHAP: The Geometry of Counterfactual Interaction Explanations in Machine Learning

We introduce Aumann-SHAP, an interaction-aware framework that decomposes counterfactual transitions by restricting the model to a local hypercube connecting baseline and counterfactual features. Each hypercube is discretized into a grid to construct an induced micro-player cooperative game in which elementary grid-step moves become players. Shapley and LES values on this TU-micro-game yield geometry-aware within-pot attributions that converge to the diagonal Aumann--Shapley / Integrated Gradients limit under grid refinement, and recover equal-split Shapley as the degenerate m=1m=1 special case. An exact grid-state closed form gives polynomial-time computation for fixed interaction order. On a synthetic benchmark with known ground truth, equal-split Shapley carries an irreducible bias while Aumann-SHAP converges to the correct decomposition. On German Credit, interaction geometry changes feature priority rankings in 12.3%12.3\% of instances. On UCI Heart Disease, equal-split misattributes a cholesterol suppressor as a positive contributor, which is a sign error Aumann-SHAP corrects. On MNIST, game-theoretic attribution reaches target confidence with 3.5×3.5\times fewer edits than magnitude-based ordering, with micro-game Shapley achieving the best efficiency across all budgets.

Adam Belahcen, Stéphane Mussard

AIRESS & UM6P

Feature AttributionShapley Value Attribution
AskPDF
Mar 13, 2026

Causal Attribution via Activation Patching

Attribution methods for Vision Transformers (ViTs) aim to identify image regions that influence model predictions, but producing faithful and well-localized attributions remains challenging. Existing attribution methods face several limitations, with gradient-based, relevance-propagation, and attention-based methods relying on local approximations, while perturbation or optimization-based methods intervene on inputs, tokens, or surrogates rather than internal patch representations. The key challenge is that class-relevant evidence is formed through interactions between patch tokens across layers; methods that operate only on input changes, attention weights, or backward relevance signals may therefore provide indirect proxies for patch importance rather than directly testing the predictive effect of contextualized patch representations. We propose Causal Attribution via Activation Patching (CAAP), which estimates the contribution of individual image patches to the ViT's prediction by directly intervening on internal activations rather than using learned masks or synthetic perturbation patterns. For each patch, CAAP inserts the corresponding source-image activations into a neutral target context over an intermediate range of layers and uses the resulting target-class score as the attribution signal. The resulting attribution map reflects the causal contribution of patch-associated internal representations on the model's prediction. The causal intervention serves as a principled measure of patch influence by capturing semantic evidence after initial representation formation, while avoiding late-layer global mixing that can reduce spatial specificity. Across multiple ViT backbones and standard metrics, CAAP consistently outperforms existing methods in various settings and produces more faithful and localized attributions.

Amirmohammad Izadi, Mohammadali Banayeeanzade, Alireza Mirrokni +4

Sharif University of Technology

Feature AttributionTransformer InterpretabilityExplainable Image ClassificationActivation PatchingVision Transformer
AskPDF
Mar 12, 2026

Trust Oriented Explainable AI for Fake News Detection

This article examines the application of Explainable Artificial Intelligence (XAI) in NLP based fake news detection and compares selected interpretability methods. The work outlines key aspects of disinformation, neural network architectures, and XAI techniques, with a focus on SHAP, LIME, and Integrated Gradients. In the experimental study, classification models were implemented and interpreted using these methods. The results show that XAI enhances model transparency and interpretability while maintaining high detection accuracy. Each method provides distinct explanatory value: SHAP offers detailed local attributions, LIME provides simple and intuitive explanations, and Integrated Gradients performs efficiently with convolutional models. The study also highlights limitations such as computational cost and sensitivity to parameterization. Overall, the findings demonstrate that integrating XAI with NLP is an effective approach to improving the reliability and trustworthiness of fake news detection systems.

Krzysztof Siwek, Daniel Stankowski, Maciej Stodolski

Warsaw University of Technology, Warsaw, Poland · Student of Warsaw University of Technology, Warsaw, Poland

Fake News DetectionExplainable Artificial IntelligenceFeature AttributionExplainability Evaluation
AskPDF
Mar 10, 2026

What is Missing? Explaining Neurons Activated by Absent Concepts

Explainable artificial intelligence (XAI) aims to provide human-interpretable insights into the behavior of deep neural networks (DNNs), typically by estimating a simplified causal structure of the model. In existing work, this causal structure often includes relationships where the presence of a concept is associated with a strong activation of a neuron. For example, attribution methods primarily identify input pixels that contribute most to a prediction, and feature visualization methods reveal inputs that cause high activation of a target neuron - the former implicitly assuming that the relevant information resides in the input, and the latter that neurons encode the presence of concepts. However, a largely overlooked type of causal relationship is that of encoded absences, where the absence of a concept increases neural activation. In this work, we show that such missing but relevant concepts are common and that mainstream XAI methods struggle to reveal them when applied in their standard form. To address this, we propose two simple extensions to attribution and feature visualization techniques that uncover encoded absences. Across experiments, we show how mainstream XAI methods can be used to reveal and explain encoded absences, how ImageNet models exploit them, and that debiasing can be improved when considering them.

Robin Hesse, Simone Schaub-Meyer, Janina Hesse +2

Max Planck Institute for Informatics, SIC · Department of Computer Science, Technical University of Darmstadt · Leibniz Institute for Resilience Research +2

Feature VisualizationNeural Network InterpretabilityNeuron-Level InterpretabilityExplainable Image ClassificationNeural EncodingFeature Attribution
AskPDF
Mar 6, 2026

Causal Interpretation of Neural Network Computations with Contribution Decomposition

Understanding how neural networks transform inputs into outputs is crucial for interpreting and manipulating their behavior. Most existing approaches analyze internal representations by identifying hidden-layer activation patterns correlated with human-interpretable concepts. Here we take a direct approach to examine how hidden neurons act to drive network outputs. We introduce CODEC (Contribution Decomposition), a method that uses sparse autoencoders to decompose network behavior into sparse motifs of hidden-neuron contributions, revealing causal processes that cannot be determined by analyzing activations alone. Applying CODEC to benchmark image-classification networks, we find that contributions grow in sparsity and dimensionality across layers and, unexpectedly, that they progressively decorrelate positive and negative effects on network outputs. We further show that decomposing contributions into sparse modes enables greater control and interpretation of intermediate layers, supporting both causal manipulations of network output and human-interpretable visualizations of distinct image components that combine to drive that output. Finally, by analyzing state-of-the-art models of neural activity in the vertebrate retina, we demonstrate that CODEC uncovers combinatorial actions of model interneurons and identifies the sources of dynamic receptive fields. Overall, CODEC provides a rich and interpretable framework for understanding how nonlinear computations evolve across hierarchical layers, establishing contribution modes as an informative unit of analysis for mechanistic insights into artificial neural networks.

Joshua Brendan Melander, Zaki Alaoui, Shenghua Liu +2

Department of Neurobiology, Stanford University · Department of Physics, Stanford University · Department of Applied Physics, Stanford University

Neural Network InterpretabilityMechanistic InterpretabilityNeuron-Level InterpretabilityCausal InterpretabilityFeature Attribution
AskPDF
Mar 5, 2026

Fusion-CAM: Integrating Gradient and Region-Based Class Activation Maps for Robust Visual Explanations

Interpreting the decision-making process of deep convolutional neural networks remains a central challenge in achieving trustworthy and transparent artificial intelligence. Explainable AI (XAI) techniques, particularly Class Activation Map (CAM) methods, are widely adopted to visualize the input regions influencing model predictions. Gradient-based approaches (e.g. Grad-CAM) provide highly discriminative, fine-grained details by computing gradients of class activations but often yield noisy and incomplete maps that emphasize only the most salient regions rather than the complete objects. Region-based approaches (e.g. Score-CAM) aggregate information over larger areas, capturing broader object coverage at the cost of over-smoothing and reduced sensitivity to subtle features. We introduce Fusion-CAM, a novel framework that bridges this explanatory gap by unifying both paradigms through a dedicated fusion mechanism to produce robust and highly discriminative visual explanations. Our method first denoises gradient-based maps, yielding cleaner and more focused activations. It then combines the refined gradient map with region-based maps using contribution weights to enhance class coverage. Finally, we propose an adaptive similarity-based pixel-level fusion that evaluates the agreement between both paradigms and dynamically adjusts the fusion strength. This adaptive mechanism reinforces consistent activations while softly blending conflicting regions, resulting in richer, context-aware, and input-adaptive visual explanations. Extensive experiments on standard benchmarks show that Fusion-CAM consistently outperforms existing CAM variants in both qualitative visualization and quantitative evaluation, providing a robust and flexible tool for interpreting deep neural networks.

Hajar Dekdegue, Moncef Garouani, Josiane Mothe +1

IRIT, UMR5505 CNRS Universit´e de Toulouse · IRIT, UMR5505 CNRS Universit´e Toulouse Capitole · INRAE, Centre Occitanie-Toulouse Unit´e Exp´erimentale APC

Explainable Image ClassificationFeature AttributionGradient-Based AttributionSaliency Map EvaluationExplainability Evaluation
AskPDF