A Synthetic Ground-Truth Framework for the Evaluation of Explainable AI Methods
Authors: Miquel Miró-Nicolau, Francesco Spinnato, Riccardo Guidotti
Organizations: Universitat de les Illes Balears, Cra. de Valldemossa, km 7.5., Palma, 07122, Spain · Uiniversity of Pisa, Largo B. Pontecrovo, 3, 56127, Pisa, Italy · ISTI-CNR, Via G. Moruzzi, 1, Pisa, 56127, Italy
Evaluating explainable Artificial Intelligence (XAI) methods is a challenging task due to the lack of reliable evaluation procedures and, in particular, the absence of ground truth explanations. In the literature, existing evaluation approaches typically assess explanations by measuring their fidelity with respect to the predictions of a black-box model. However, such evaluation strategies only quantify the degree to which an explanation reproduces the model's output, without ensuring that the explanation correctly reflects the underlying decision process. As a consequence, different explanations may achieve similar fidelity scores while providing inconsistent or misleading interpretations of the model behavior. In this paper, we propose a framework for the evaluation of XAI methods based on synthetic ground truth. The proposed approach relies on controlled interventions to generate synthetic datasets in which the importance of input components can be determined by design. This enables the construction of ground truth explanations that are directly aligned with the behavior of the model under analysis. The framework is instantiated across three data domains, namely binary images, tabular data, and time series, allowing a comprehensive assessment of explanation methods in heterogeneous settings. Experimental results obtained by evaluating nine widely used XAI methods show significant limitations in current techniques and highlight the importance of synthetic, intervention-based benchmarks for a reliable assessment of explanation quality.
Figures & tables
Figure 1: Illustration of the intervention procedure across data modalities: color images, binary images, and time series. The latter two are constructed from the original color image using the proposed framework.
Figure 2: Representative samples from the FunnyBirds dataset across distinct classes. Each category is defined by a unique combination of geometric and color parts: wing color and foot shape (left image), wing and tail color (center image), and the foot shape (right image).
Figure 3: Illustration of class ambiguity induced by binarization. Two images belonging to different classes (first and second) become identical when converted to binary representations (third and fourth), as color information is removed.
Figure 4: Step-by-step procedure for extracting a time series from a color image. From left to right: the original image, the binarized image, the measurement of radial distances from the object center to its contour, and the resulting time series plotted from these measurements
Data Modality
Model
Precision
Recall
F1-Score
Binary images
ResNet50
0.91
0.88
0.89
Time series
Multirocket w. Ridge
0.64
0.63
0.63
Tabular data
MLP
0.98
0.98
0.98
Table 1: Performance measure results for each AI model used in each experiment.
Method
Hyperparameter
Value
RISE
N Masks
600
S Masks
8
Probability remaining
0.1
LIME
Occlusion Value
0
N Samples
1500
Kernel Width
0.25
Table 2: XAI method hyperparameter values for binary image experiments.
Method
Hyperparameter
Value
RISE
N Masks
1000
Grid resolution
16
Cell activation probability
0.5
Mask generation
Linear upsampling + random shift
LIME
N Samples
5000
Sampling
Gaussian
Table 3: XAI method hyperparameters values for time series experiments.
Method
Hyperparameter
Value
LIME
Occlusion Value
0
N Samples
1500
Kernel Width
0.25
Feature Selection
Highest Weights
Kernel
Similarity kernel
Distance Metric
Cosine
Table 4: XAI method hyperparameters values for tabular data experiments.
\minf↑
\auc↑
\kl↓
RISE [ 42 ]
0.343±0.382
0.573±0.273
1.915±1.505
LIME [ 43 ]
0.293±0.164
0.932±0.083
1.217±0.664
Gradient [ 48 ]
0.378±0.427
0.500±0.026
1.903±1.517
DeepLIFT [ 47 ]
0.380±0.429
0.555±0.123
1.895±1.510
KernelSHAP [ 31 ]
0.342±1.210
0.848±0.165
1.551±1.210
Integrated Gradients [ 51 ]
0.380±0.430
0.510±0.098
1.905±1.519
Table 5: Metrics results to compare XAI methods output and proposed Ground Truth in the Binary images task. All reported value are the mean and standard deviation for the test set. Best values for each metric in bold.
Figure 5: Comparison of different XAI saliency map techniques for the binary image dataset The more intense the color the more important the pixel.
Figure 6: Time series attribution for an instance of class 2 . From top to bottom, the ground truth (GT), and the predicted attributions from LIME, RISE, TSHAP (ROI and WIN), and KernelSHAP. The more intense the color the more important the time series observation.
\minf↑
\auc↑
\kl↓
RISE
0.522±0.136
0.709±0.161
0.790±0.405
LIME
0.442±0.199
0.646±0.134
0.907±0.510
Kernel SHAP
0.007±0.003
0.500±0.003
5.315±0.509
T-SHAP ROI
0.386±0.170
0.490±0.089
1.784±0.767
T-SHAP Window
0.424±0.191
0.490±0.132
1.055±0.562
Table 6: Metrics results to compare XAI methods output and proposed Ground Truth in the Time series task. All reported value are the mean and standard deviation for the validation set. Best values for each metric in bold.
\minf↑
\kl↓
LORE [ 15 ]
0.784±0.177
1.204±1.855
LIME [ 43 ]
0.560±0.269
3.806±3.535
Gradient [ 48 ]
0.980±0.099
0.092±0.618
DeepLIFT [ 47 ]
0.636±0.221
2.011±2.559
Kernel SHAP [ 31 ]
0.601±0.229
2.315±2.701
Integrated Gradients [ 51 ]
0.642±0.221
1.983±2.559
Table 7: Metrics results to compare XAI methods output and proposed Ground Truth in the Tabular data task. All reported value are the mean and standard deviation for the validation set. Best values for each metric in bold.
Explainable AI (XAI) techniques are increasingly important for the validation and responsible use of modern deep learning models, but are difficult to evaluate due to the lack of good ground-truth to compare against. We propose a framework that serves as a quantifiable metric for the quality of XAI methods, based on continuous input perturbation. Our metric formally considers the sufficiency and necessity of the attributed information to the model's decision-making, and we illustrate a range of cases where it aligns better with human intuitions of explanation quality than do existing metrics. To exploit the properties of this metric, we also propose a novel XAI method, considering the case where we fine-tune a model using a differentiable approximation of the metric as a supervision signal. The result is an adapter module that can be trained on top of any black-box model to output causal explanations of the model's decision process, without degrading model performance. We show that the explanations generated by this method outperform those of competing XAI techniques according to a number of quantifiable metrics.
Amritpal Singh, Andrey Barsky, Mohamed Ali Souibgui +2
Computer Vision Center, Barcelona, Spain · Autonomous University of Barcelona, Spain
The evaluation of explainable AI (XAI) approaches often relies on three families of methods: subjective measures (e.g., questionnaires on trust or satisfaction), objective measures (e.g., task performance metrics), and mathematical metrics (e.g., for faithfulness). Yet, it remains unclear how these families align or diverge in practice. In a{preregistered} between-subjects study (N=166), we use three established saliency map techniques (LIME, Grad-CAM, Guided Backpropagation) as a testbed to examine this issue. We find that each family of methods leads to different conclusions: participants reported no differences in trust or satisfaction, Grad-CAM improved user performance, while mathematical metrics favored Guided Backpropagation. At the same time, mathematical metrics were only partially related to user performance, and these relationships were sometimes counterintuitive. Our findings highlight the methodological importance of comparing subjective, objective, and mathematical approaches when evaluating XAI, illustrating both tensions and aspects that are aligned. We discuss implications for XAI evaluation frameworks.
Felix Kares, Timo Speith, Hanwei Zhang +1
Saarland University, Germany · University of Bayreuth, Germany · University of Freiburg, Germany
In this paper, we present a comprehensive framework for assessing the explainability of various XAI methods, such as LIME and SHAP, across multiple datasets and machine learning models, with the ultimate goal of creating a unified multidimensional explainability score. Our methodology focuses on three key aspects of explainability: fidelity, simplicity, and stability. We leverage benchmarking experiments to systematically evaluate these aspects and use the insights gained to construct an offline knowledge base. This knowledge base captures the explainability scores for each registered model and serves as a valuable resource for context-dependent evaluation of explainability. By analyzing the complementary characteristics and metadata of AI models, datasets, and XAI methods, the knowledge base will enable the estimation of explainability scores for previously unseen datasets and models. Properties like fidelity, simplicity, and stability may vary significantly based on the dataset, underlying model, and domain expertise of the end user. We demonstrate our framework by applying it to three open-source datasets, discussing the implications of the obtained results in relation to the characteristics of the datasets. Our work contributes to the growing field of XAI by providing a robust and versatile tool for evaluating and comparing the explainability of various XAI methods, ultimately supporting the development of more transparent and trustworthy AI systems.
Department of Digital Systems University of Piraeus Piraeus, Greece · Department of Business Administration University of Piraeus Piraeus, Greece · Innov-Acts Ltd Innov-Acts Nicosia, Cyprus