Dropping Tokens Without Dropping Them: Causal Attribution for Diabetic Retinopathy
Organizations: Indian Institute of Science
Abstract
In this work, we investigate whether attention-based attribution maps in a vision transformer provide causal explanations for diabetic retinopathy (DR) classification. We introduce a principled decomposition of attention into two attribution maps: the contribution (), which satisfies the completeness axiom in explainable artificial intelligence (xAI), and the pull (), which quantifies causal explanations. We define an identity relating and that characterizes when fails to capture causality. We test whether each map is causally faithful on the MAPLES-DR dataset by dropping/inserting tokens, and report Necessity (), Sufficiency (), and Faithfulness Correlation (). The pull achieves the highest causal alignment (, , ), outperforming Attention Rollout, Grad-CAM, and AttnLRP. Occlusion retains better sufficiency () but is about 800 times slower. Furthermore, on MAPLES-DR, achieves of the average precision (AP) of HRDecoder, a state-of-the-art supervised lesion segmentation model, despite the absence of pixel-level supervision. Together, these findings show that the pull and the contribution form computationally efficient attribution maps for reliable causal interpretation in safety-critical medical diagnostics. The code and demo are available at https://github.com/pramath-haritz/dr-xai.
Figures & tables
| Causal tests (MAPLES-DR) | Lesion | Cost per image | ||||||||
| Map | (%) | (%) | MAPLES | DDR | TFLOPs | Time (s) | ||||
| Attention Decomposition (Ours) | ||||||||||
| Attention weights | 0.61 † | [0.50, 0.76] | 3.9 | [3.0, 5.2] | 0.47 † | [0.44, 0.50] | 0.275 | 0.274 | 4.7 | 0.14 |
| Contribution | 0.96 | [0.61, 1.74] | 3.5 | [2.6, 4.8] | 0.26 | [0.01, 0.42] | 0.244 | 0.263 | 4.7 | 0.23 |
| Pull | 0.71 † | [0.57, 0.89] | 3.0 | [2.2, 4.6] | 0.53 † | [0.49, 0.57] | 0.277 | 0.252 | 4.7 | 0.27 |
| Map | Loss | (%) | (%) | ||
|---|---|---|---|---|---|
| Attention | CORAL [ 6 ] | 0.61 | 3.9 | 0.47 † | 0.275 |
| weights | CE | 1.10 | 3.9 | 0.29 | 0.211 |
| Pull | CORAL [ 6 ] | 0.71 | 3.0 | 0.53 | 0.277 |
| CE | 0.76 | 2.6 | 0.45 | 0.269 |