Stop Using Plausibility as the Criterion for Explainable AI
Organizations: Simon Fraser University · The University of British Columbia
Abstract
Explainable artificial intelligence (XAI) is motivated by the problem of making AI predictions understandable, transparent, and responsible, as AI becomes increasingly impactful in society and high-stakes domains. The evaluation and optimization criteria of XAI are gatekeepers for XAI algorithms to achieve their expected goals and should withstand rigorous inspection. To improve the scientific rigor of XAI, we conduct the first comprehensive and critical examination of a common XAI criterion: plausibility. Plausibility assesses how convincing the AI explanation is to humans, and is usually quantified by metrics of feature localization or feature correlation. Our examination shows that plausibility is an invalid measure of explainability, and human explanations are not ground truth for XAI, because both ignore the necessary assumptions of explanation. One fundamental assumption is that AI explanations are supposed to behave adversarially, analogous to auditors' or doctors' roles of identifying potential problems. Evaluating or optimizing XAI to generate more plausible explanations is analogous to incentivizing (i.e., bribing) auditors/doctors to generate more positive reports, which is different from encouraging AI models to learn more plausible features (analogous to encouraging companies/patients to improve their conduct/health). Our examination further reveals the consequences of using plausibility as the XAI criterion, including increasing misleading explanations that manipulate users, deteriorating users' trust in the AI system, undermining human autonomy, being unable to achieve complementary human-AI task performance, and abandoning other possible approaches of enhancing understandability. Due to the invalidity of measurements and the unethical issues, this paper argues that the AI community should stop using plausibility as the criterion to evaluate and optimize XAI algorithms.
Figures & tables
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| XAI | Analogous to | |
| The AI model learns plausible features | Companies improve business conduct | Patients improve health |
| The XAI algorithm presents plausible features | Auditors generate positive auditing reports | Doctors generate positive health reports |
| Illegitimate: Using plausibility as the XAI criterion (i.e., encouraging the XAI algorithm to present as many plausible features as possible) | Illegitimate: Encouraging auditors to generate as many positive auditing reports as possible | Illegitimate: Encouraging doctors to generate as many positive health reports as possible |
| Legitimate: Optimizing the AI model to learn more plausible features such that their prediction-making process is more understandable as presented by the XAI algorithm | Legitimate: Encouraging companies to improve their business conduct such that they will receive more positive auditing reports from auditors accordingly | Legitimate: Encouraging patients to improve their health such that they will receive more positive health reports from doctors accordingly |
| Cases where plausibility is inappropriately used | Cases where plausibility is appropriately used |
| Case 1 : Plausibility is used to evaluate various post-hoc XAI algorithms regarding their capability of presenting plausible features, e.g. [ 73 ] . | Case 2 : Plausibility is used as an intermediate measure to evaluate XAI utility to users on whether users’ judgment on the explanation reasonableness can help them discern correct and incorrect predictions, e.g., the informative plausibility in [ 42 ] . Case 3 : When evaluating faithfulness of XAI algorithms using synthetic data, plausibility metric may be used to measure the similarity between AI explanation and the known ground truth explanation as specified in the synthetic data, e.g., [ 48 ] . |
| Rationale : It occludes the non-ideal signals in the model prediction process that explainability is supposed to reveal, and can increase misleading explanations, see Section 3.1 and Section 3.2.1 . | Rationale : Plausibility is used as an intermediate measure to evaluate XAI utility or faithfulness. Plausibility is not used as an end goal or a criterion, see Section 4 . |
| Case 4 : Plausibility is used to evaluate the interpretability capability of an inherently interpretable model for presenting plausible features, e.g.: the interpretability metric in [ 10 ] . | Case 5 : Plausibility (or localization accuracy) is used to optimize the AI model’s learning of human-interpretable features, e.g., the fine annotations in the Method section in [ 10 ] . |
| Rationale : It occludes the non-ideal signals in the model prediction process that explainability is supposed to reveal, and can increase misleading explanations, see Section 3.1 and Section 3.2.1 . | Rationale : The optimization of AI models to learn plausible features should be clearly stated, and should not be regarded as the optimization of XAI algorithms to present plausible features, see Section 3.2.2 . |
| Term | Definition |
| Accountability | According to Doshi-Velez et al. [ 24 ] , accountability is “the ability to determine whether a decision was made in accordance with procedural and substantive standards and to hold someone responsible if those standards are not met.” |
| Explainability | In this work, we use explainability and interpretability interchangeably to denote the functionality of an AI system that can provide rationales of AI predictions to users in understandable ways. Explainability/interpretability differs from AI model visualization in that explainability emphasizes the intention and behavior of “explaining” and complies with all the social assumptions in human explanatory communication [ 30 , 77 ] . |
| Faithfulness | Faithfulness is the degree to which the explanation accurately represents the prediction process of an AI model [ 38 ] . In the literature, it is also called fidelity or truthfulness [ 28 , 42 ] . |
| Plausibility | According to Jacovi and Goldberg [ 38 ] , plausibility is how convincing the explanation is to humans, and they differentiated plausibility from faithfulness, where the former relies on human judgment or human-provided explanations. |
| Transparency | According to Markus et al. [ 54 ] , a model is transparent “if the inner workings of the model are visible and one can understand how inputs are mathematically mapped to outputs.” While some XAI literature uses transparency as a synonym for understandability [ 11 , 6 ] , we use transparency to emphasize exposing the inner workings of the AI model, and use understandability to emphasize the human factors in comprehending the decision rationales of an AI model. |
| Trustworthiness | According to the Oxford English Dictionary, trustworthiness is “the ability to be relied on as honest or truthful.” |
| Symbol | Definition | Where introduced |
| Important feature set | Section 2 | |
| Explanation | Eq. 1 | |
| Explanation form | Section 2 | |
| Probability | Theorem 1 | |
| A data instance in the dataset | Theorem 2 | |
| A real-valued r.v. of the plausibility measure of an explanation . Its subscript denotes is for the explanation of an instance | Eq. 1 |
| Reason number and its description in Section 2 | Evidence in the literature | How we arrived at the reason and how widespread it is | Summary of our identified flaws in using the reason to support the opposing view |
|---|---|---|---|
| 1. Plausibility is exactly how we humans gauge the goodness of an explanation from a human explainer. | [ 91 ] discusses criteria on how people judge explanation quality, and plausibility is one of them: “A good explanation should also be coherent; it should be compatible with our existing beliefs, and consistent with the evidence and with itself.” | It is one of the main intuitions and a widespread assumption to support the use of plausibility as the XAI criterion. | We agree with this reason, but argue it should not be used to support the plausibility criterion, because XAI is expected to truthfully reveal both ideal and non-ideal signals in AI decision process. |
| 2. AI explanations are designed the same way to mimic human explanations as the ground truth. | By using human-annotated important features as the ground truth to evaluate and compare XAI algorithms, this evaluation practice, such as a specific example in [ 73 ] , assumes that human explanation is the ground truth for AI explanations. | It is a widespread assumption, as evidenced in multiple XAI benchmarks that use human-annotated important features as the ground truth for XAI algorithms [ 73 , 60 , 94 , 21 , 49 ] . | We argue against this reason due to the epistemic analysis in Appendix D that human explanation is not ground truth for XAI. |
| 3. More plausible explanations indicate better predictions of the AI model. | In “Section U2.3 Verifying AI decision” in Supplementary Material S1 of [ 42 ] , doctors mentioned that they identify AI mistakes or adjust their confidence in an AI prediction based on their plausibility assessments. | This is a widespread assumption because this is how humans judge a conclusion based on the reasonableness of its rationale (Assumption 2). | We agree with this reason, but point out in Section 3.2.1 that this assumption does not automatically hold when using plausibility as the XAI criterion. |
| 4. More plausible explanations indicate the AI model learns more effective features as humans. | The abstract of [ 90 ] mentioned that “to improve visual reasoning in models by supervising feature importance (estimated by model explanation techniques) with human annotations such as highlights of important image regions”, “improved accuracy is through improved explanation plausibility”. | This may be a common assumption that motivates work on “right for the right reason” [ 90 , 71 , 10 , 89 ] , but it may not be as widespread as the previous reasons. | We agree with this reason, but point out in Section 3.2.2 that this reason should not be used to support the use of the plausibility criterion, because doing so reverses the causation. |
| 5. Plausible explanations can make AI systems more transparent. | [ 38 ] refers to transparency as faithfulness, and states “many authors do not clearly make the distinction, and sometimes conflate the two [faithfulness and plausibility].” | The link between plausibility and transparency may not be as prevalent as the previous reasons. However, because transparency is a main outcome commonly associated with explainability, we included it in our analysis to understand if it can benefit from the plausibility criterion. | We refer to previous work [ 38 ] to state that the two should be treated independently and cannot be confused to measure each other. |
| 6. Plausible explanations can improve the trustworthiness of an AI system. | [ 74 ] used plausibility to assess trust level in Section 5.2. Evaluating Trust. | Ditto. | We show plausibility can increase local trust, but destroy global trust or trustworthiness in Section 3.3.1 . |
| Author, year | Title of the review | Number of included papers | Year range of included papers | Detail |
| Dembinsky et al. 2026 | Unifying VXAI: A Systematic Review and Framework for the Evaluation of Explainable AI | 362 | before 2025 | Plausibility is included in one of the seven XAI evaluation desiderata and is also included in “Metric V.40: GT Dataset Evaluation”. |
| Kaya et al. 2026 | Explainable Artificial Intelligence (XAI): Concepts, Applications, Challenges, and Future Perspectives | NA | NA | Plausibility (called localization in their paper) is included as one of the XAI evaluation criteria. |
| Ghauth & Kustiawan, 2026 | Explainable Artificial Intelligence (XAI) for Cancer Classification in Medical Imaging: A Systematic Review | 46 | 2020-2026 | Section 4.2 reports that 21/46 (45%) papers used the plausibility criterion. |
| Zhang et al. 2026 | A comprehensive review of explainable artificial intelligence in healthcare methods, evaluation, and clinical integration | 170 | 2016-2025 | The plausibility criterion is included in their proposed “Practice-focused evaluation system” as one of the “three core categories of metrics”. |
| Lago et al. 2026 | Evaluating Explainability: A Framework for Systematic Assessment of Explainable AI Features in Medical Imaging | NA | NA | Plausibility is one of the four evaluation criteria in their proposed evaluation framework of XAI. |
| Salih et al. 2024 | A review of evaluation approaches for explainable AI with applications in cardiology | NA | NA | Two (Plausibility and Clinical relevance) out of the seven XAI evaluation criteria listed in “Section 3.1 Human and application-grounded evaluations” are plausibility measures. |
| Group | Number of decisions | Plausibility | |||||
| M SD | Min | 25% | Median | 75% | Max | ||
| Agree | 649 | 6.45 2.82 | 0 | 5 | 7 | 9 | 10 |
| Disagree | 95 | 3.82 2.56 | 0 | 2 | 4 | 5.5 | 10 |
| Data ID | Human prediction | AI prediction | Accept AI | Team prediction | GT | ||
| 01 | 5 | 5 | 2.47 | 0.92 | True | 5 | 5 |
| 02 | 2 | 2 | 1.14 | 0.76 | True | 2 | 2 |
| 03 | 5 | 5 | 0.79 | 0.69 | True | 5 | 5 |
| 04 | 3 | 3 | 3.75 | 0.98 | True | 3 | 3 |
| 05 | 3 | 4 | 1.46 | 0.81 | True | 4 | 4 |
| 06 | 1 | 2 | 1.13 | 0.76 | True | 2 | 2 |
| Event: and | Values of the r.v. | Probability of the event: | Who is the decision-maker? | Likelihood of the decision-maker correctly predicting ( ) | |
| i. | Human accepts AI | AI | 1 | ||
| AI predicts correctly for | |||||
| ii. | Human accepts AI | AI | 0 | ||
| AI predicts incorrectly for | |||||
| iii. | Human rejects AI | Human | |||
| AI predicts correctly for | |||||
| Event: and | Values of the r.v. | Probability of the event: | Who is the decision-maker? | Likelihood of the decision-maker correctly predicting ( ) | |
| i. | Human accepts AI | AI | 1 | ||
| AI predicts correctly for | |||||
| ii. | Human accepts AI | AI | 0 | ||
| AI predicts incorrectly for | |||||
| iii. | Human rejects AI | Human | |||
| AI predicts correctly for | |||||
| Event: and | Values of the r.v. | Probability of the event: | Who is the decision-maker? | Likelihood of the decision-maker correctly predicting ( ) | |
| v. | Human accepts AI | AI | 1 | ||
| AI predicts correctly for | |||||
| vi. | Human accepts AI | AI | 0 | ||
| AI predicts incorrectly for | |||||
| vii. | Human rejects AI | Human | |||
| AI predicts correctly for | |||||
Explainable Artificial Intelligence
Explainable artificial intelligence develops ways to make AI decisions understandable, testable, and useful to people.
See all 346 papers