B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data
Organizations: University of Edinburgh · Jagiellonian University, Faculty of Mathematics and Computer Science · Jagiellonian University, Faculty of Chemistry · Ardigen SA
Abstract
Understanding the reasoning behind deep learning model predictions is crucial in cheminformatics and drug discovery, where molecular design determines their properties. However, current evaluation frameworks for Explainable AI (XAI) in this domain often rely on artificial datasets or simplified tasks, employing data-derived metrics that fail to capture the complexity of real-world scenarios and lack a direct link to explanation faithfulness. To address this, we introduce B-XAIC, a novel benchmark constructed from real-world molecular data and diverse tasks with known ground-truth rationales for assigned labels. Through a comprehensive evaluation using B-XAIC, we reveal limitations of existing XAI methods for Graph Neural Networks (GNNs) in the molecular domain. This benchmark provides a valuable resource for gaining deeper insights into the faithfulness of XAI, facilitating the development of more reliable and interpretable models.
Figures & tables
| Feature | MUTAG | Benzene | Flu.-Carb. | Alk.-Carb. | BA-2Mot. | ShapeGGen | B-XAIC |
|---|---|---|---|---|---|---|---|
| Size | 0.2K | 12K | 8.7K | 1.1K | 1K | config. | 50K |
| Tasks | 1 | 1 | 1 | 1 | 1 | 1 | 7 |
| Real mol. | Yes | Yes | Yes | Yes | No | No | Yes |
| NE eval | No | No | No | No | No | No | Yes |
| Multi-pat. | No | No | No | No | No | No | Yes |
| Edge+Node | No | Yes | Yes | Yes | Varies | Varies | Yes |
| task | % of graphs with pos. label | % of graphs with NE | % of nodes in SE | % of edges in SE |
|---|---|---|---|---|
| B | 2.23 | 97.77 | 4.1 ± 2.1 | — |
| P | 13.22 | 86.78 | 4.3 ± 2.4 | — |
| X | 55.87 | 44.13 | 6.0 ± 4.0 | — |
| indole | 36.81 | 63.19 | 31.3 ± 12.1 | 31.4 ± 12.1 |
| PAINS | 32.89 | 67.11 | 34.1 ± 14.3 | 31.4 ± 14.7 |
| rings-count | 30.12 | 1.34 | 64.0 ± 16.0 | 61.8 ± 14.8 |
| nodes | edges | ||||||
| Class | Explainer | NE | SE | avg | NE | SE | avg |
| GB | Saliency | 0.51 0.12 | 0.85 0.12 | 0.68 | 0.31 0.15 | 0.67 0.06 | 0.49 |
| Deconvolution | 0.79 0.10 | 0.81 0.09 | 0.80 | 0.38 0.20 | 0.69 0.04 | 0.53 | |
| InputXGradient | 0.54 0.14 | 0.83 0.14 | 0.68 | 0.37 0.14 | 0.65 0.05 | 0.51 | |
| GuidedBackprop | 0.41 0.12 | 0.87 0.09 | 0.64 | 0.30 0.08 | 0.70 0.10 | 0.50 | |
| IntegratedGradients | 0.39 0.26 | 0.85 0.13 | 0.62 | 0.32 0.18 | 0.70 0.06 | 0.51 | |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | 5K Samples | 10K Samples | 25K Samples | 50K Samples | 100K Samples |
|---|---|---|---|---|---|
| B | 99.92 | 99.92 | 99.96 | 99.96 | 99.94 |
| P | 99.97 | 99.99 | 99.98 | 99.98 | 100.00 |
| X | 99.77 | 99.85 | 99.94 | 99.94 | 99.96 |
| indole | 94.11 | 95.54 | 97.65 | 98.32 | 98.24 |
| PAINS | 82.79 | 85.81 | 91.91 | 92.90 | 93.20 |
| rings-count | 86.86 | 93.14 | 99.64 | 99.62 | 99.94 |
| Explainer | IQR | IQR | IQR | IQR |
|---|---|---|---|---|
| PGMExplainer | ||||
| Deconvolution | ||||
| FlowX | ||||
| GNNExplainer | ||||
| GraphMaskExplainer | ||||
| InputXGradient |
| Explainer | IQR | IQR | IQR | IQR |
|---|---|---|---|---|
| GNNExplainer | ||||
| FlowX | ||||
| Deconvolution | ||||
| InputXGradient | ||||
| GraphMaskExplainer | ||||
| IntegratedGradients |
| Node | Edge | |||
|---|---|---|---|---|
| Multiplier | ||||
| IQR | 0.952 | 0.867 | 0.855 | 0.778 |
| IQR | 0.952 | 0.867 | 0.867 | 0.778 |
| IQR | 0.867 | 0.689 | 0.818 | 0.644 |
| Task | Node | Edge |
|---|---|---|
| B | 0.048 | — |
| P | -0.128 | — |
| X | 0.199 | — |
| indole | 0.800 | 0.020 |
| PAINS | 0.504 | -0.269 |
| rings-count | 0.932 | 0.925 |
| Intrinsic interaction order | |||||
| Task | # positive | min | median | mean | max |
| B | 1,116 | 1 | 1 | 1.0 | 5 |
| P | 6,612 | 1 | 1 | 1.5 | 29 |
| X | 27,934 | 1 | 1 | 1.8 | 20 |
| indole | 18,403 | 9 | 9 | 10.3 | 108 |
| PAINS | 16,446 | 2 | 9 | 9.7 | 90 |
| Task | GIN F1 | # mols | intrinsic order | atom-recovery AUROC (mean / min) | model-error | trunc. |
|---|---|---|---|---|---|---|
| B | 1.00 | 87 | 1 | 1.00 / 1.00 | 2.07 | 1.61 |
| P | 1.00 | 100 | 1 | 1.00 / 1.00 | 2.42 | 1.96 |
| X | 1.00 | 100 | 1 | 1.00 / 0.91 | 1.52 | 0.98 |
| indole | 0.98 | 78 | 9 | 0.98 / 0.82 | 1.81 | 1.64 |
| PAINS | 0.93 | 100 | 2–12 | 0.77 / 0.10 | 3.41 | 3.12 |
| rings-max | 0.92 | 22 | 7 | 0.76 / 0.54 | 1.84 | 1.47 |
GNN Explainability
Graph neural network explainability studies how to identify and communicate the features, nodes, edges, and subgraphs behind graph-model predictions.
See all 88 papers