A unified framework for evaluating the robustness of machine-learning interpretability for prospect risking

Prithwijit Chowdhury; Ahmad Mustafa; Mohit Prabhushankar; Ghassan AlRegib

cs.LGFeb 16, 2026

A unified framework for evaluating the robustness of machine-learning interpretability for prospect risking

Authors: Prithwijit Chowdhury, Ahmad Mustafa, Mohit Prabhushankar, Ghassan AlRegib

Organizations: Center for Energy and Geo Processing (CeGP), Omni Lab for Intelligent Visual Engineering and Science (OLIVES), School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA.

Abstract

In geophysics, hydrocarbon prospect risking involves assessing the risks associated with hydrocarbon exploration by integrating data from various sources. Machine learning-based classifiers trained on tabular data have been recently used to make faster decisions on these prospects. The lack of transparency in the decision-making processes of such models has led to the emergence of explainable AI (XAI). LIME and SHAP are two such examples of these XAI methods which try to generate explanations of a particular decision by ranking the input features in terms of importance. However, explanations of the same scenario generated by these two different explanation strategies have shown to disagree or be different, particularly for complex data. This is because the definitions of "importance" and "relevance" differ for different explanation strategies. Thus, grounding these ranked features using theoretically backed causal ideas of necessity and sufficiency can prove to be a more reliable and robust way to improve the trustworthiness of the concerned explanation strategies.We propose a unified framework to generate counterfactuals as well as quantify necessity and sufficiency and use these to perform a robustness evaluation of the explanations provided by LIME and SHAP on high dimensional structured prospect risking data. This robustness test gives us deeper insights into the models capabilities to handle erronous data and which XAI module works best in pair with which model for our dataset for hydorcarbon indication.

Figures & tables

	With all features		With seven high level features
Model	Training Accuracy	Validation Accuracy	Training Accuracy	Validation Accuracy
Logistic Regression	0.7949	0.7327	0.7906	0.7129
Gaussian NB	0.7778	0.7329	0.7991	0.7426
Random Forests	1.00	0.7525	1.00	0.7228
Voting Classifier (1:5:1)	0.7821	0.7330	0.8034	0.7426

Table 1: The training and the test accuracies of the binary classifiers used for the experiments

Figure 1: (a) shows the traditional feature attribute based XAI framework to evaluate local feature importance for a particular decision made on a datapoint by a trained model. (b) is our unified framework to generate forward counterfactuals and global necessity and sufficiency score to do a robustness evaluation (described in Fig x) of given feature attribution (FA) methods (here: LIME and SHAP)

Figure 2: This Figure explains how necessity and sufficiency calculation for a feature, in a classifier fitted distribution space, works. (a) Shows the distribution is the Context space $U$ . This distribution is then fitted to a binary classifier model in (b) while separates the space based on the class prediction of $y=y^{*}\,\&\,y\neq y^{*}$ . (c) and (d) captures the diagrammatic representation of how intervention is done on the feature $x_{j}$ to $a^{\prime}$ and $a$ respectively, to calculate the conditional probability $\alpha$ and $\beta$ based on the change in prediction results.

Figure 3: Toy example to validate that the necessity and sufficiency scores generated by the selected method are synonymous with logically calculated impact score of a cause leading to an effect.

Figure 4: Robustness Analysis of LIME scores: Calculating mean Necessity Score for Rank $2$ . (a) Record the Feature names which occupy the same rank in all cases. (b) Match them with their corresponding Global Necessity Scores. (c) The average of the Global Scores of the features in a particular rank (here $2$ ) gives us the robustness analysis of LIME explanation of the decisions made by the model on the dataset.

Figure 5: (a) A plot of the well outcomes by DHI Index vs. Initial Pg. (b) The risk factors assessment flow chart: This figure presents the main steps of the Pg calculation for a given prospect.

Figure 6: Global Necessity Score for model (a) Logistic Regression, (b) Gaussian NB, (c) Random Forest and (d) Voting Classifier (1:5:1), for two different settings: “blue” when model is trained on all features and “orange” when model is trained on only the top 7 features.

Figure 7: Global Sufficiency Score for model (a) Logistic Regression, (b) Gaussian NB, (c) Random Forest and (d) Voting Classifier (1:5:1), for two different settings: “green” when model is trained on all features and “red” when model is trained on only the top 7 features.

Figure 8: Necessity Analysis of LIME explanations of predictions made by a (a) Logistic Regression model, a (b) Gaussian Naive Bayes model, a (c) Random Forest Classifier and a (d) Voting Classifier (1:5:1) for two different settings: “blue” when model is trained on all features and “orange” when model is trained on only the top 7 features. The necessity scores of the $\#2$ ranked feature in both (b) and (d) are higher than rank $\#1$ thus disagreeing with our hypothesis and failing to be an robust explanation. The LIME explanations conducted on the fitted Random Forest model (c) is observed to be the most robust of the four sets of experiments that have been conducted.

Figure 9: Necessity Analysis of SHAP explanations of predictions made by a (a) Logistic Regression model, a (b) Gaussian Naive Bayes model, a (c) Random Forest Classifier and a (d) Voting Classifier (1:5:1) for two different settings: “blue” when model is trained on all features and “orange” when model is trained on only the top 7 features. The necessity scores of the $\#2$ ranked feature in both (b) and (d) are higher than rank $\#1$ thus disagreeing with our hypothesis and failing to be an robust explanation. The SHAP explanations conducted on the fitted Logistic Regression (a) and Random Forest models (c) are observed to be the robust ones among the four sets of experiments that have been conducted.

Figure 10: Sufficiency Analysis of LIME explanations of predictions made by a (a) Logistic Regression model, a (b) Gaussian Naive Bayes model, a (c) Random Forest Classifier and a (d) Voting Classifier (1:5:1) for two different settings: “green” when model is trained on all features and “red” when model is trained on only the top 7 features.

Figure 11: Sufficiency Analysis of SHAP explanations of predictions made by a (a) Logistic Regression model, a (b) Gaussian Naive Bayes model, a (c) Random Forest Classifier and a (d) Voting Classifier (1:5:1) for two different settings: “green” when model is trained on all features and “red” when model is trained on only the top 7 features. The evaluation of the SHAP explanations of all the fitted models are observed to be quite robust to our sufficiency test proving that the feature rankings generated by SHAP correspond to how sufficient a feature is in generating the outcome.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Top k-th Occurrences of high level features in LIME explanations (Model = Logistic Regression
	For All Features				For Top 7 Features
Feature Name	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence
Feature 1	0	0	0	1	1	96	101	101
Feature 2	0	0	0	0	0	5	61	91
Feature 3	0	0	0	0	0	0	27	77
Feature 4	49	49	50	57	100	101	101	101

Appendix

Table 2: Top

k

occurrence for the Top 7 features of the DHI dataset as ranked by LIME during explanation of 101 test cases for model = Logistic Regression.

Top k-th Occurrences of high level features in LIME explanations (Model= Gaussian NB)
	For All Features				For Top 7 Features
Feature Name	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence
Feature 1	4	13	29	53	20	43	81	101
Feature 2	10	15	27	52	10	19	47	76
Feature 3	6	44	49	63	2	49	72	98
Feature 4	48	61	77	96	69	91	100	101

Appendix

Table 3: Top

k

occurrence for the top seven features of the DHI dataset as ranked by LIME during explanation of 101 test cases for model = Gaussian NB

Top k-th Occurrences of high level features in LIME explanations (Model= Random Forest)
	For All Features				For Top 7 Features
Feature Name	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence
Feature 1	1	1	7	23	0	2	16	78
Feature 2	24	34	80	93	0	42	82	101
Feature 3	48	70	71	71	1	52	75	80
Feature 4	25	76	94	101	100	101	101	101

Appendix

Table 4: Top

k

occurrence for the Top 7 features of the DHI dataset as ranked by LIME during explanation of 101 test cases for model = Random Forest

Top k-th Occurrences of high level features in LIME explanations (Model= Voting Classifier(1:5:1))
	For All Features				For Top 7 Features
Feature Name	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence
Feature 1	15	28	55	83	2	22	68	101
Feature 2	12	17	38	54	11	22	56	93
Feature 3	8	52	55	76	5	65	73	99
Feature 4	60	83	92	100	83	92	100	101

Appendix

Table 5: Top

k

occurrence for the top seven features of the DHI dataset as ranked by LIME during explanation of 101 test cases for model = Voting Classifier (1:5:1)

Top k-th Occurrences of high level features in SHAP explanations (Model= Logistic Regression)
	For All Features				For Top 7 Features
Feature Name	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence
Feature 1	0	0	0	0	3	38	56	96
Feature 2	0	0	1	6	8	45	70	85
Feature 3	0	0	0	0	0	18	48	78
Feature 4	56	67	72	84	90	92	95	97

Appendix

Table 6: Top

k

occurrence for the Top 7 features of the DHI dataset as ranked by SHAP during explanation of 101 test cases for model = Logistic Regression

Top k-th Occurrences of high level features in SHAP explanations (Model= Gaussian NB)
	For All Features				For Top 7 Features
Feature Name	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence
Feature 1	3	6	8	27	1	9	36	95
Feature 2	8	17	50	71	17	31	78	98
Feature 3	3	53	75	84	7	69	87	99
Feature 4	71	86	88	91	74	89	95	100

Appendix

Table 7: Top

k

occurrence for the top seven features of the DHI dataset as ranked by SHAP during explanation of 101 test cases for model = Gaussian NB

Top k-th Occurrences of high level features in SHAP explanations (Model= Random Forest)
	For All Features				For Top 7 Features
Feature Name	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence
Feature 1	0	0	0	0	0	1	7	25
Feature 2	19	34	68	80	11	25	74	92
Feature 3	20	63	79	85	14	78	86	98
Feature 4	55	74	83	89	71	85	91	98

Appendix

Table 8: Top

k

occurrence for the Top 7 features of the DHI dataset as ranked by SHAP during explanation of 101 test cases for model = Random Forest

Top k-th Occurrences of high level features in SHAP explanations (Model= Voting Classifier(1:5:1))
	For All Features				For Top 7 Features
Feature Name	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence	Top 1 occurrence	Top 2 occurrence	Top 3 occurrence	Top 5 occurrence
Feature 1	0	2	9	15	1	2	6	25
Feature 2	10	21	42	65	19	28	79	96
Feature 3	2	62	77	84	1	73	88	96
Feature 4	76	83	87	92	79	93	98	100

Appendix

Table 9: Top

k

occurrence for the top seven features of the DHI dataset as ranked by SHAP during explanation of 101 test cases for model = Voting Classifier (1:5:1)

Explainability Evaluation

Explainability evaluation tests whether AI explanations are faithful, stable, understandable, useful, and reliable across models and settings.

See all 319 papers

A Formal Methodological Framework for Auditing Robustness and Fidelity in Explainable AI: From Application to Trust Certification
Aug 24, 2026Rosa Elysabeth Ralinirina, Jean Christian Ralaivao, Niaiko Michaël Ralaivao +2Explainability Evaluation Black-Box Model Auditing
Doctoral School Modeling–Computer Science, University of Fianarantsoa, Madagascar
Evaluating Local Explainability Metrics for Machine Learning Models on Tabular Data
May 26, 2026Tomás Pereira, João Vitorino, Eva Maia +1Feature Attribution Explainable Artificial Intelligence
GECAD, ISEP, Polytechnic of Porto, Porto, Portugal
Towards Rigorous Explainability by Feature Attribution
Apr 17, 2026Olivier Létoffé, Xuanxiang Huang, Joao Marques-SilvaFeature Attribution Explainable Artificial Intelligence
IRIT, University of Toulouse France · Nanyang Technological University, Singapore · ICREA & Univ. Lleida, Spain

Aug 24, 2026J/K move · Enter open · S save

A Formal Methodological Framework for Auditing Robustness and Fidelity in Explainable AI: From Application to Trust Certification

Rosa Elysabeth Ralinirina, Jean Christian Ralaivao, Niaiko Michaël Ralaivao, Alain Josué Ratovondrahona+1

Doctoral School Modeling–Computer Science, University of Fianarantsoa, Madagascar

Explainability Evaluation Black-Box Model Auditing Feature Attribution SHAP Feature Attribution

Open paper AskPDF arXiv

SHAP and LIME are now standard tools for interpreting black-box predictions, yet their outputs can vary substantially when the input is perturbed by small amounts of noise--a problem we observed firsthand in our previous work on food security in Madagascar (Ralinirina et al., 2025). This variability raises the question of whether such explanations can be trusted at all. We address it by constructing an auditing protocol that measures two properties of any post-hoc explainer: robustness (how stable the explanation is under input perturbation) and fidelity (whether the features deemed important actually drive the model's prediction). These two quantities are combined into a single Trust Score. We run the protocol on a multi-sectoral dataset from Madagascar (83 features, 253 records, 4 malnutrition classes) using three classifiers and two explainers, plus their regularized counterparts. The results are sobering: models with AUC above 0.99 can produce numerically degenerate or flatly uninformative explanations, and fidelity scores lose discriminative power when the model is overfitted. These findings suggest that auditing XAI outputs is not optional but necessary, particularly when they inform decisions in sensitive domains.

Abstract

Explainability Evaluation

Explore similar work

A Formal Methodological Framework for Auditing Robustness and Fidelity in Explainable AI: From Application to Trust Certification

Evaluating Local Explainability Metrics for Machine Learning Models on Tabular Data

Towards Rigorous Explainability by Feature Attribution