cs.AI · 2606.14838 Copy arXiv ID · Jun 12, 2026 Save A Definition of Good Explanations and the Challenges Explaining LLM Outputs Authors: Louis Mahon , Elliot Ford , Callum Hackett
Organizations: UnlikelyAI, London, UK
Abstract How to define a good explanation is a long-standing philosophical debate which has found recent renewed interest in the context of AI outputs. Explainability is crucial for AI adoption in many contexts, but in order to produce good explanations of AI systems, we must first have an understanding of what good explanations are. In this paper we propose a definition inspired by the notion of counterfactual explanations, however we argue that one must also take into account the interlocutor's prior beliefs in each fact that could be offered in an explanation. We explore the ramifications of this definition for AI explainability and, in particular, why LLM outputs are difficult to produce good explanations for.
Explore similar work Jun 26, 2026 · Nhi Nguyen, Shauli Ravfogel, Rajesh Ranganath Sufficient Conditions Belief
Jul 17, 2026 · Elize Herrewijnen, Benedetta Muscato, Gizem Gezici +1 Explainable Artificial Intelligence Rationales
Mar 17, 2025 · Noah Y. Siegel, Nicolas Heess, Maria Perez-Ortiz +1 Explanation Faithfulness Faithfulness
Jun 26, 2026 · cs.LG J/K move · Enter open · S save
Nhi Nguyen, Shauli Ravfogel, Rajesh Ranganath
New York University
Large language models (LLMs) are increasingly deployed in high-stakes domains, where free-text explanations such as chain-of-thought and post-hoc rationales are used to justify model outputs. Yet it remains unclear whether these explanations are sufficient, i.e., if they contain enough information to explain the model's output-generating process. We generalize classical sufficiency from feature attributions to arbitrary explanations and prove that explanation sufficiency can change depending on the input distribution, which must be explicitly defined for LLM explanations. We propose using the LLM itself to generate alternative inputs conditioned on an explanation, capturing its beliefs about possible inputs. We formalize self-consistent sufficiency as a goal for free-text explanations and introduce an information-theoretic metric, SCSuff, that enables evaluation of free-text explanations without relying on predefined biases or shortcuts. Our experiments show that SCSuff agrees with targeted perturbation tests where applicable and demonstrate that explanation sufficiency can vary with the input distribution. We find LLM explanations are generally insufficient and weakly correlated with model size, accuracy, or output entropy. Analysis of final-token hidden states shows that top and bottom SCSuff scores can be predicted from internal representations, suggesting that SCSuff can guide detection and improvement of sufficient LLM explanations. The code for this paper is available at https://github.com/rajesh-lab/self-consistent-sufficiency .