A unified framework for evaluating the robustness of machine-learning interpretability for prospect risking
Authors: Prithwijit Chowdhury, Ahmad Mustafa, Mohit Prabhushankar, Ghassan AlRegib
Organizations: Center for Energy and Geo Processing (CeGP), Omni Lab for Intelligent Visual Engineering and Science (OLIVES), School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA.
In geophysics, hydrocarbon prospect risking involves assessing the risks associated with hydrocarbon exploration by integrating data from various sources. Machine learning-based classifiers trained on tabular data have been recently used to make faster decisions on these prospects. The lack of transparency in the decision-making processes of such models has led to the emergence of explainable AI (XAI). LIME and SHAP are two such examples of these XAI methods which try to generate explanations of a particular decision by ranking the input features in terms of importance. However, explanations of the same scenario generated by these two different explanation strategies have shown to disagree or be different, particularly for complex data. This is because the definitions of "importance" and "relevance" differ for different explanation strategies. Thus, grounding these ranked features using theoretically backed causal ideas of necessity and sufficiency can prove to be a more reliable and robust way to improve the trustworthiness of the concerned explanation strategies.We propose a unified framework to generate counterfactuals as well as quantify necessity and sufficiency and use these to perform a robustness evaluation of the explanations provided by LIME and SHAP on high dimensional structured prospect risking data. This robustness test gives us deeper insights into the models capabilities to handle erronous data and which XAI module works best in pair with which model for our dataset for hydorcarbon indication.
Figures & tables
With all features
With seven high level features
Model
Training Accuracy
Validation Accuracy
Training Accuracy
Validation Accuracy
Logistic Regression
0.7949
0.7327
0.7906
0.7129
Gaussian NB
0.7778
0.7329
0.7991
0.7426
Random Forests
1.00
0.7525
1.00
0.7228
Voting Classifier (1:5:1)
0.7821
0.7330
0.8034
0.7426
Table 1: The training and the test accuracies of the binary classifiers used for the experiments
Figure 1: (a) shows the traditional feature attribute based XAI framework to evaluate local feature importance for a particular decision made on a datapoint by a trained model. (b) is our unified framework to generate forward counterfactuals and global necessity and sufficiency score to do a robustness evaluation (described in Fig x) of given feature attribution (FA) methods (here: LIME and SHAP)
Figure 2: This Figure explains how necessity and sufficiency calculation for a feature, in a classifier fitted distribution space, works. (a) Shows the distribution is the Context space U . This distribution is then fitted to a binary classifier model in (b) while separates the space based on the class prediction of y=y∗&y=y∗ . (c) and (d) captures the diagrammatic representation of how intervention is done on the feature xj to a′ and a respectively, to calculate the conditional probability α and β based on the change in prediction results.
Figure 3: Toy example to validate that the necessity and sufficiency scores generated by the selected method are synonymous with logically calculated impact score of a cause leading to an effect.
Figure 4: Robustness Analysis of LIME scores: Calculating mean Necessity Score for Rank 2 . (a) Record the Feature names which occupy the same rank in all cases. (b) Match them with their corresponding Global Necessity Scores. (c) The average of the Global Scores of the features in a particular rank (here 2 ) gives us the robustness analysis of LIME explanation of the decisions made by the model on the dataset.
Figure 5: (a) A plot of the well outcomes by DHI Index vs. Initial Pg. (b) The risk factors assessment flow chart: This figure presents the main steps of the Pg calculation for a given prospect.
Figure 6: Global Necessity Score for model (a) Logistic Regression, (b) Gaussian NB, (c) Random Forest and (d) Voting Classifier (1:5:1), for two different settings: “blue” when model is trained on all features and “orange” when model is trained on only the top 7 features.
Figure 7: Global Sufficiency Score for model (a) Logistic Regression, (b) Gaussian NB, (c) Random Forest and (d) Voting Classifier (1:5:1), for two different settings: “green” when model is trained on all features and “red” when model is trained on only the top 7 features.
Figure 8: Necessity Analysis of LIME explanations of predictions made by a (a) Logistic Regression model, a (b) Gaussian Naive Bayes model, a (c) Random Forest Classifier and a (d) Voting Classifier (1:5:1) for two different settings: “blue” when model is trained on all features and “orange” when model is trained on only the top 7 features. The necessity scores of the #2 ranked feature in both (b) and (d) are higher than rank #1 thus disagreeing with our hypothesis and failing to be an robust explanation. The LIME explanations conducted on the fitted Random Forest model (c) is observed to be the most robust of the four sets of experiments that have been conducted.
Figure 9: Necessity Analysis of SHAP explanations of predictions made by a (a) Logistic Regression model, a (b) Gaussian Naive Bayes model, a (c) Random Forest Classifier and a (d) Voting Classifier (1:5:1) for two different settings: “blue” when model is trained on all features and “orange” when model is trained on only the top 7 features. The necessity scores of the #2 ranked feature in both (b) and (d) are higher than rank #1 thus disagreeing with our hypothesis and failing to be an robust explanation. The SHAP explanations conducted on the fitted Logistic Regression (a) and Random Forest models (c) are observed to be the robust ones among the four sets of experiments that have been conducted.
Figure 10: Sufficiency Analysis of LIME explanations of predictions made by a (a) Logistic Regression model, a (b) Gaussian Naive Bayes model, a (c) Random Forest Classifier and a (d) Voting Classifier (1:5:1) for two different settings: “green” when model is trained on all features and “red” when model is trained on only the top 7 features.
Figure 11: Sufficiency Analysis of SHAP explanations of predictions made by a (a) Logistic Regression model, a (b) Gaussian Naive Bayes model, a (c) Random Forest Classifier and a (d) Voting Classifier (1:5:1) for two different settings: “green” when model is trained on all features and “red” when model is trained on only the top 7 features. The evaluation of the SHAP explanations of all the fitted models are observed to be quite robust to our sufficiency test proving that the feature rankings generated by SHAP correspond to how sufficient a feature is in generating the outcome.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Top k-th Occurrences of high level features in LIME explanations (Model = Logistic Regression
For All Features
For Top 7 Features
Feature Name
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Feature 1
0
0
0
1
1
96
101
101
Feature 2
0
0
0
0
0
5
61
91
Feature 3
0
0
0
0
0
0
27
77
Feature 4
49
49
50
57
100
101
101
101
Appendix
Table 2: Top k occurrence for the Top 7 features of the DHI dataset as ranked by LIME during explanation of 101 test cases for model = Logistic Regression.
Top k-th Occurrences of high level features in LIME explanations (Model= Gaussian NB)
For All Features
For Top 7 Features
Feature Name
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Feature 1
4
13
29
53
20
43
81
101
Feature 2
10
15
27
52
10
19
47
76
Feature 3
6
44
49
63
2
49
72
98
Feature 4
48
61
77
96
69
91
100
101
Appendix
Table 3: Top k occurrence for the top seven features of the DHI dataset as ranked by LIME during explanation of 101 test cases for model = Gaussian NB
Top k-th Occurrences of high level features in LIME explanations (Model= Random Forest)
For All Features
For Top 7 Features
Feature Name
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Feature 1
1
1
7
23
0
2
16
78
Feature 2
24
34
80
93
0
42
82
101
Feature 3
48
70
71
71
1
52
75
80
Feature 4
25
76
94
101
100
101
101
101
Appendix
Table 4: Top k occurrence for the Top 7 features of the DHI dataset as ranked by LIME during explanation of 101 test cases for model = Random Forest
Top k-th Occurrences of high level features in LIME explanations (Model= Voting Classifier(1:5:1))
For All Features
For Top 7 Features
Feature Name
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Feature 1
15
28
55
83
2
22
68
101
Feature 2
12
17
38
54
11
22
56
93
Feature 3
8
52
55
76
5
65
73
99
Feature 4
60
83
92
100
83
92
100
101
Appendix
Table 5: Top k occurrence for the top seven features of the DHI dataset as ranked by LIME during explanation of 101 test cases for model = Voting Classifier (1:5:1)
Top k-th Occurrences of high level features in SHAP explanations (Model= Logistic Regression)
For All Features
For Top 7 Features
Feature Name
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Feature 1
0
0
0
0
3
38
56
96
Feature 2
0
0
1
6
8
45
70
85
Feature 3
0
0
0
0
0
18
48
78
Feature 4
56
67
72
84
90
92
95
97
Appendix
Table 6: Top k occurrence for the Top 7 features of the DHI dataset as ranked by SHAP during explanation of 101 test cases for model = Logistic Regression
Top k-th Occurrences of high level features in SHAP explanations (Model= Gaussian NB)
For All Features
For Top 7 Features
Feature Name
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Feature 1
3
6
8
27
1
9
36
95
Feature 2
8
17
50
71
17
31
78
98
Feature 3
3
53
75
84
7
69
87
99
Feature 4
71
86
88
91
74
89
95
100
Appendix
Table 7: Top k occurrence for the top seven features of the DHI dataset as ranked by SHAP during explanation of 101 test cases for model = Gaussian NB
Top k-th Occurrences of high level features in SHAP explanations (Model= Random Forest)
For All Features
For Top 7 Features
Feature Name
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Feature 1
0
0
0
0
0
1
7
25
Feature 2
19
34
68
80
11
25
74
92
Feature 3
20
63
79
85
14
78
86
98
Feature 4
55
74
83
89
71
85
91
98
Appendix
Table 8: Top k occurrence for the Top 7 features of the DHI dataset as ranked by SHAP during explanation of 101 test cases for model = Random Forest
Top k-th Occurrences of high level features in SHAP explanations (Model= Voting Classifier(1:5:1))
For All Features
For Top 7 Features
Feature Name
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Top 1 occurrence
Top 2 occurrence
Top 3 occurrence
Top 5 occurrence
Feature 1
0
2
9
15
1
2
6
25
Feature 2
10
21
42
65
19
28
79
96
Feature 3
2
62
77
84
1
73
88
96
Feature 4
76
83
87
92
79
93
98
100
Appendix
Table 9: Top k occurrence for the top seven features of the DHI dataset as ranked by SHAP during explanation of 101 test cases for model = Voting Classifier (1:5:1)
SHAP and LIME are now standard tools for interpreting black-box predictions, yet their outputs can vary substantially when the input is perturbed by small amounts of noise--a problem we observed firsthand in our previous work on food security in Madagascar (Ralinirina et al., 2025). This variability raises the question of whether such explanations can be trusted at all. We address it by constructing an auditing protocol that measures two properties of any post-hoc explainer: robustness (how stable the explanation is under input perturbation) and fidelity (whether the features deemed important actually drive the model's prediction). These two quantities are combined into a single Trust Score. We run the protocol on a multi-sectoral dataset from Madagascar (83 features, 253 records, 4 malnutrition classes) using three classifiers and two explainers, plus their regularized counterparts. The results are sobering: models with AUC above 0.99 can produce numerically degenerate or flatly uninformative explanations, and fidelity scores lose discriminative power when the model is overfitted. These findings suggest that auditing XAI outputs is not optional but necessary, particularly when they inform decisions in sensitive domains.
Rosa Elysabeth Ralinirina, Jean Christian Ralaivao, Niaiko Michaël Ralaivao +2
Doctoral School Modeling–Computer Science, University of Fianarantsoa, Madagascar
Despite the wide use of explainability techniques to attempt to understand the behavior of Artificial Intelligence (AI), the generated explanations may not always be reliable. An explanation can appear plausible to humans but fail to capture the internal reasoning of a model, particularly when dealing with complex tabular data. This paper studies the trustworthiness of local explainability techniques when applied to complex tabular classification tasks, considering evaluated metrics for three main properties: faithfulness to the model's predictions, robustness to input data variations, and complexity of the explanation itself. A benchmark was performed for Local Interpretable Model-Agnostic Explanations (LIME), Kernel SHapley Additive exPlanations (SHAP), and Feature Ablation techniques, across 32 datasets and different types of machine learning models. Model performance ranges were analyzed to identify two groups: consensus-correct, which are samples that all models predicted correctly, and consensus-wrong, samples that all models predicted incorrectly. The obtained results demonstrate that that the explanations are not always correlated with a model's predictive performance. Instead, dataset complexity and feature distributions seem to be the main factors affecting explanation quality and reliability.
Tomás Pereira, João Vitorino, Eva Maia +1
GECAD, ISEP, Polytechnic of Porto, Porto, Portugal
For around a decade, non-symbolic methods have been the option of choice when explaining complex machine learning (ML) models. Unfortunately, such methods lack rigor and can mislead human decision-makers. In high-stakes uses of ML, the lack of rigor is especially problematic. One prime example of provable lack of rigor is the adoption of Shapley values in explainable artificial intelligence (XAI), with the tool SHAP being a ubiquitous example. This paper overviews the ongoing efforts towards using rigorous symbolic methods of XAI as an alternative to non-rigorous non-symbolic approaches, concretely for assigning relative feature importance.