Position: Explainable AI is Causality in Disguise
Organizations: Department of Electrical & Computer Engineering, University of Waterloo & Vector Institute, Canada
Abstract
The demand for Explainable AI (XAI) has triggered an explosion of methods, producing a landscape so fragmented that we now rely on surveys of surveys. Yet, fundamental challenges persist: conflicting metrics, failed sanity checks, and unresolved debates over robustness and fairness. The only consensus on how to achieve explainability is a lack of one. This has led many to point to the absence of a ground truth for defining ``the'' correct explanation as the main culprit. This position paper posits that the persistent discord in XAI arises not from an absent ground truth but from a ground truth that exists, albeit as an elusive and challenging target: the causal model that governs the relevant system. By reframing XAI queries about data, models, or decisions as causal inquiries, we prove the necessity and sufficiency of causal models for XAI. We contend that without this causal grounding, XAI remains unmoored. Ultimately, we encourage the community to converge around advanced concept and causal discovery to escape this entrenched uncertainty.
Figures & tables
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Question | Approach | References | ||
| Causal Abstraction & Representation Learning | RQ1 | How can we systematically identify and extract human-interpretable concepts from deep neural networks (DNNs) to construct meaningful causal abstractions? | Develop unsupervised or semi-supervised algorithms leveraging neuro-symbolic methods to identify concept-level abstractions from latent spaces. | ( Geiger et al., 2023 ; Marconato et al., 2023b ; Zhang, 2024 ) |
| RQ2 | Under what conditions can we guarantee that causal abstractions discovered from low-level neural representations remain faithful to the original neural network’s causal structure? | Study theoretical guarantees for causal abstraction under approximate causal modeling frameworks. Quantify trade-offs between abstraction fidelity and interpretability. | ( Beckers and Halpern, 2019 ; Karimi et al., 2023 ; Rubenstein et al., 2017 ) | |
| Approximate and Partial Causal Models | RQ3 | How can we formulate and efficiently compute approximate causal explanations that balance computational complexity with explanation accuracy? | Formalize a notion of -approximate causal explanations and develop scalable algorithms to derive such explanations from large-scale models. | ( Glymour et al., 2019 ; Montagna et al., 2024 ) |
| RQ4 | Can we establish rigorous bounds on how approximations in causal discovery methods affect downstream XAI tasks such as model debugging or user trust? | Empirically and theoretically analyze sensitivity to approximation errors; develop robustness metrics for causal XAI methods. | ( Janzing et al., 2020 ) | |
| RQ5 | What computational trade-offs arise when using approximate causal models in realistic XAI scenarios? | Study whether relaxing accuracy requirements yields tangible computational benefits or if computational complexity remains a core barrier. | ( Adolfi et al., 2024 ; Barceló et al., 2020 ; Bassan et al., 2024 ) | |
| Human-Aligned Causal Queries & Interactive XAI | RQ6 | How should interactive XAI systems leverage causal interventions to enhance users’ mental models about AI systems? | Design and validate interactive interfaces (e.g., visual tools or agents) enabling causal interventions, and assess user understanding. | ( Miller, 2019 ) |
Explainable Artificial Intelligence
Explainable artificial intelligence develops ways to make AI decisions understandable, testable, and useful to people.
See all 320 papers