Beyond Attention Heatmaps: How to Get Better Explanations for Multiple Instance Learning Models in Histopathology
Authors: Mina Jamshidi Idaji, Julius Hense, Tom Neuhäuser, Augustin Krause, Yanqing Luo, Oliver Eberle, Thomas Schnake, Laure Ciernik, Farnoush Rezaei Jafari, Reza Vahidimajd, Jonas Dippel, Christoph Walz, Frederick Klauschen, Andreas Mock, Klaus-Robert Müller
Organizations: Berlin Institute for the Foundations of Learning and Data, Berlin, Germany · Machine Learning Group, Technische Universität Berlin, Berlin, Germany · Department of Chemistry, Chemical Physics Theory Group, University of Toronto, Canada · Vector Institute for Artificial Intelligence, Toronto, Canada · Acceleration Consortium, University of Toronto, Canada · Department of Computer Science and Engineering, The Chinese University of Hong Kong · Aignostics GmbH, Berlin, Germany · Institute of Pathology, Ludwig Maximilian University, Munich, Germany · German Cancer Research Center, Heidelberg, and German Cancer Consortium, Munich, Germany · Institute of Pathology, Charité Universitätsmedizin, Berlin, Germany · Department of Artificial Intelligence, Korea University, Seoul, Korea · Max-Planck Institute for Informatics, Saarbrücken, Germany
Multiple instance learning (MIL) has enabled substantial progress in computational histopathology, where a large amount of patches from gigapixel whole slide images are aggregated into slide-level predictions. Heatmaps are widely used to validate MIL models and to discover tissue biomarkers. Yet, the validity of these heatmaps has barely been investigated. In this work, we introduce a general framework for evaluating the quality of MIL heatmaps without requiring additional labels. We conduct a large-scale benchmark experiment to assess six explanation methods across histopathology task types (classification, regression, survival), MIL model architectures (Attention-, Transformer-, Mamba-based), and patch encoder backbones (UNI2, Virchow2). Our results show that explanation quality mostly depends on MIL model architecture and task type, with perturbation ("Single"), layer-wise relevance propagation (LRP), and integrated gradients (IG) consistently outperforming attention-based and gradient-based saliency heatmaps, which often fail to reflect model decision mechanisms. We further demonstrate the advanced capabilities of the best-performing explanation methods: (i) We provide a proof-of-concept that MIL heatmaps of a bulk gene expression prediction model can be correlated with spatial transcriptomics for biological validation, and (ii) showcase the discovery of distinct model strategies for predicting human papillomavirus (HPV) infection from head and neck cancer slides. Our work highlights the importance of validating MIL heatmaps and establishes that improved explainability can enable more reliable model validation and yield biological insights, making a case for a broader adoption of explainable AI in digital pathology. Our code is provided in a public GitHub repository: https://github.com/bifold-pathomics/xMIL/tree/xmil-journal
Figures & tables
Figure 1: Overview of our explanation pipeline for multiple instance learning (MIL) . a Block diagram of a general MIL approach to make predictions directly from whole slide images, and to obtain heatmaps to interpret the model predictions. The explanation methods produce patch-wise scores by explaining (only) the task head and the aggregation model, but not the patch encoder. In this work, we assess six explanation methods in the context of two patch encoders, three aggregation model architectures, and three types of prediction heads. b Composition of the task heads. In the survival head, the discrete-time model predicts hazard probabilities {hk} for survival time intervals k=1,⋯,K , and subsequently aggregates them into a patient risk score. See Section B for details. σ = sigmoid function; FNN = feedforward neural network.
Figure 2: Overview of the evaluation framework for comparing heatmaps and explanation methods in multiple instance learning (MIL) .
Figure 3: The patch flipping procedure . Patches are sorted based on their heatmap scores either in ascending or descending order. They are then grouped into 100 mini-bags. The mini-bags are removed progressively from the main bag of patches, and the remaining patches are fed into the model. Two perturbation curves (model prediction vs. dropped fraction) are formed for the two orderings. A larger area under the curve (AUC) of the ascending (Asc.) curve is desired, whereas a smaller AUC of the descending (Desc.) curve is better. Therefore, the area between the two curves (called Symmetric Relevance Gain–SRG) quantifies the faithfulness of the heatmap. A larger SRG indicates a more faithful heatmap.
Figure 4: Datasets and model performances. The bar plots in each panel show the mean test performance over the cross-validation folds with the error bars depicting the standard deviations (see Tables 2 and 3 for numerical model performance results). For the classification tasks in panel (a), the ratio of the cases in each class is depicted. In panels (b) and (c), the endpoint target’s histograms are illustrated.
Figure 5: Comparison of explanation methods for two example settings . a / b Faithfulness comparison from three complementary perspectives: ( Left ) The distribution of Symmetric Relevance Gain (SRG) faithfulness scores per explanation method computed for each slide in the test set. ( Center ) Wilcoxon’s signed-rank test effect sizes comparing SRG scores between explanation methods for each slide in the test set. Colors and marker sizes indicate effect size magnitude as shown on the colorbar. Triangle, square, and circle markers are used to denote the negligible ( <0.2 ), weak to moderate ( 0.2−0.5 ), and moderate to strong ( ≥0.5 ) effect sizes. These effect size scores assess the magnitude of the faithfulness differences between explanation methods. ( Right ) Mean Rank Score (MRS) per explanation method (vertical axis). Lower averaged MRS = better position in the faithfulness ranking . c / d Heatmaps for example slides from the two experimental settings. For LRP, values greater/lower than zero indicate evidence for/against the model prediction (visualized by shades of red/blue). Single and Attention (Attn) have no natural boundary between positive and negative evidence, but only provide an ordering (visualized by shades of red). In the HRD regression task, however, the 0 value for Single may be interpreted as such a boundary; therefore, we visualize this heatmap using a red/blue color scheme as well.
Figure 6: Comparisons of explanation methods aggregated across experimental settings . a Pairwise Wilcoxon signed-rank effect sizes comparing SRG scores between explanation methods, averaged over various experimental settings. “Overall” considers the mean effect size across all experimental settings, “Classification” the mean over all classification experiments, etc. The filling colors and marker sizes indicate effect size magnitude as shown on the colorbar. Triangle, square, and circle markers are used to denote the negligible ( <0.2 ), weak to moderate ( 0.2−0.5 ), and moderate to strong ( ≥0.5 ) effect sizes. Notably, two consistent groups emerge: LRP, IG, Single vs. Attn, G × I, Grad2, Rand, with the first group generally outperforming the second. All aggregated comparisons were additionally replicated using a robust median/MAD effect size, and non-significant pairwise comparisons after FDR correction are reported separately in Section E . b Mean Rank Score (MRS) per explanation method (vertical axis), averaged across selected MIL pipeline settings (horizontal axis). Lower averaged MRS = better position in the faithfulness ranking .
Figure 7: Proof-of-concept for spatial mapping of bulk gene expression using explanation heatmaps of a regression MIL model . a Experimental pipeline. Whole-slide H&E images are subdivided into cutouts, from which patch-level feature vectors are extracted. For each cutout, pseudo-bulk gene expression targets are derived by aggregating spatial transcriptomics (ST) measurements spatially aligned to the same region. Model explanation methods are then applied to obtain patch-level relevance scores, which are then realigned to build a WSI-level heatmap and compared against the original spatial gene expression. b Example heatmaps for gene FASN on a slide, shown alongside the corresponding ST ground truth. Heatmaps indicate the patch-level model-attributed relevance for predicting bulk expression. c Pearson correlation coefficients between patch scores derived from explanation heatmaps and the corresponding ST ground truth. Results are shown for four explanation methods: Layer-wise Relevance Propagation (LRP), Integrated Gradients (IG), Single, and Attention. Individual slides are indicated by distinct symbols on the violin plots.
Figure 8: Histological analysis of HPV heatmaps for discovery . a We extract the most relevant heatmap regions (=top 10% LRP patch scores) and characterize them by computing interpretable and quantifiable histological features. b The cell composition of the most relevant regions determined via CellViT++ ( 95 ) and tissue compartment segmentation exposes clusters among HPV-positive slides. See also Figure 14 for a 2D scatter matrix of the five most relevant features. The samples in panels c-f are marked. c-f Example heatmaps and relevant histological features for HPV-positive prediction, consisting of a thumbnail of the most relevant regions (top row) and a selected zoomed-in slide region (middle row) with CellViT-annotated neoplastic (red) and inflammatory cells (blue) (bottom row).
Table 1: Hyperparameter grids for all models trained in this work
Classification
Regression
Survival
CAMELYON
NSCLC
HPV
TP53
AURKA
HRD
BRCA
GBMLGG
KIRC
COADREAD
TransMIL
1.00(0.00)
0.97(0.02)
0.90(0.04)
0.71(0.05)
0.66(0.02)
0.62(0.02)
0.65(0.08)
0.74(0.04)
0.70(0.03)
0.63(0.06)
AttnMIL
1.00(0.00)
0.98(0.01)
0.92(0.04)
0.68(0.07)
0.69(0.03)
0.60(0.02)
0.62(0.05)
0.76(0.04)
0.71(0.03)
0.61(0.07)
MambaMIL
0.97(0.03)
0.90(0.04)
0.89(0.03)
0.63(0.08)
0.65(0.03)
0.60(0.03)
0.59(0.05)
0.76(0.12)
0.68(0.06)
0.64(0.06)
Appendix
Table 2: Performance results for the models in faithfulness experiments, with backbone Virchow2. The numbers are mean (std) of the performance over the cross-validation folds.
Classification
Regression
Survival
CAMELYON
NSCLC
HPV
TP53
AURKA
HRD
BRCA
GBMLGG
KIRC
COADREAD
TransMIL
1.00(0.00)
0.98(0.01)
0.92(0.04)
0.68(0.08)
0.66(0.03)
0.62(0.05)
0.65(0.07)
0.75(0.04)
0.71(0.06)
0.62(0.08)
AttnMIL
1.00(0.00)
0.97(0.02)
0.92(0.04)
0.65(0.07)
0.69(0.03)
0.63(0.03)
0.58(0.05)
0.75(0.03)
0.71(0.04)
0.58(0.06)
MambaMIL
0.98(0.02)
0.86(0.03)
0.88(0.04)
0.62(0.07)
0.66(0.04)
0.61(0.04)
0.62(0.06)
0.75(0.12)
0.71(0.07)
0.62(0.09)
Appendix
Table 3: Performance results for the models in faithfulness experiments, with backbone UNI2. The numbers are mean (std) of the performance over the cross-validation folds.
Figure 9: Experimental configurations with at least one non-significant pairwise faithfulness differences after multiple comparison correction. Each panel corresponds to one experimental configuration (dataset × task × backbone × MIL architecture) in which at least one pairwise Wilcoxon signed-rank test between explanation methods was non-significant after FDR correction ( α=0.05 ). Cells indicate method pairs, and the red color marks the pairs for which no statistically significant difference in SRG faithfulness was detected. Most non-significant comparisons occur within the two groups {Single, LRP, IG} and {Attention, G×I, Grad2, Random}. Only two configurations in panels 4 and 7 (both CAMELYON16 tumor detection experiments) exhibit non-significant differences across groups, consistent with the simplicity of the task and the aggregation-level results reported in Section 4.4 .
Figure 10: Pairwise median/MAD-based comparison of SRG scores between explanation methods, averaged over various experimental factors. This figure corresponds to Figure 6 . The filling colors indicate the magnitude of median/MAD as shown on the colorbar. Triangle, square, and circle markers are used to denote the negligible ( <0.2 ), weak to moderate ( 0.2−0.5 ), and moderate to strong ( ≥0.5 ) effect sizes.
Figure 11: Ground truth and heatmaps for the ERBB2 prediction model in Section 5 . The ground-truth spatial transcriptomics and the heatmaps for the four explanation methods used in the ST validation study presented in Section 5 .
Figure 12: Ground truth and heatmaps for the FASN prediction model in Section 5 . The ground-truth spatial transcriptomics and the heatmaps for the four explanation methods used in the ST validation study presented in Section 5 .
Figure 13: Most relevant features for HPV prediction in TCGA HNSC . Mean and standard deviation of the cell type shares per tissue compartment in the top-10% heatmap regions across all HPV positive samples.
Figure 14: Comparison of histological feature profiles for HPV prediction in TCGA HNSC . Scatter matrix displaying the distribution of cell type shares per tissue compartment in the HL10 regions across all HPV positive samples for five selected histological features. The prediction scores indicate the model prediction for these samples (closer to 1 = higher HPV-positive likelihood). The cluster markers represent the clusters revealed with Agglomerative Clustering. See also Figure 8 -b.
Explanations of multiple instance learning (MIL) models are widely used for validation and discovery in digital histopathology. Existing methods primarily rely on heatmaps that highlight influential regions but do not explain how evidence from different tissue regions is combined to produce a prediction. This limits interpretability, especially when decisions depend on interactions between tissue features. We introduce Symbolic explainable MIL (Symb-xMIL), a post-hoc explanation framework that quantifies how a MIL model's behavior aligns with human-readable decision rules, expressed as logical relationships (e.g., AND, OR, NOT) between input features. These alignment scores reveal semantic patterns underlying the model's predictions. We evaluate Symb-xMIL on synthetic and real-world histopathology datasets. On synthetic MIL data, Symb-xMIL reliably recovers ground-truth logical rules. In a clinical tumor detection task, the best-aligned rules uncover heterogeneous decision patterns and expose hidden model errors. On an HPV-prediction task on TCGA-HNSCC, a cohort of head and neck cancer, our framework refines patient survival stratification beyond HPV status with potential clinical relevance. Overall, Symb-xMIL extends MIL explainability beyond visual attribution toward structured, rule-based reasoning, enabling more transparent and semantically grounded interpretation of model predictions.
Yanqing Luo, Julius Hense, Niklas Prenißl +4
Berlin Institute for the Foundations of Learning and Data, Berlin, Germany · Machine Learning Group, Technische Universität Berlin, Berlin, Germany · Institute of Pathology, Charité Universitätsmedizin, Berlin, Germany +10
Attention-based multiple instance learning (ABMIL) is the predominant approach for slide-level prediction in computational pathology, yet its attention maps provide only local explanations: they indicate where a model focuses but not which histological features drive its predictions or how the model behaves across a patient cohort. We present Semantic Attention Global Explanations (SAGE), a post-hoc framework that extracts global, language-grounded explanations from a frozen ABMIL model. Using a pathology vision-language model, SAGE scores image patches against a dictionary of 25 histological concepts, aggregates these scores according to the model's learned attention, and quantifies how each concept relates to prediction risk across a cohort. Applied to survival prediction using seven TCGA cancer cohorts and three foundation models, SAGE recovered established prognostic features, such as the adverse association of necrosis, while revealing cancer-specific biology, including a favorable angiogenic signature in renal cell carcinoma consistent with known molecular subtypes. Ablation studies demonstrated that these associations depend on the model's learned attention rather than concept prevalence alone, and that the concept dictionary captures much of the prognostic information encoded by the foundation model features. Through semantically-grounded explanations, SAGE provides a scalable, model-agnostic framework for understanding what ABMIL survival models learn, enabling pathologists to interpret model behavior at the cohort level and offering the potential for biomarker identification.
Abdallah Lamane, Abdul Rahman Diab, Ren-Chin Wu +1
Massachusetts Institute of Technology, Cambridge, MA, USA · Department of Data Science, Dana-Farber Cancer Institute, Boston, MA · Department of Pathology, Dana-Farber Cancer Institute, Boston, MA +2
Multiple Instance Learning (MIL) is the predominant framework for classifying gigapixel whole-slide images in computational pathology. MIL follows a sequence of 1) extracting patch features, 2) applying a linear layer to obtain task-specific patch features, and 3) aggregating the patches into a slide feature for classification. While substantial efforts have been devoted to optimizing patch feature extraction and aggregation, none have yet addressed the second point, the critical layer which transforms general-purpose features into task-specific features. We hypothesize that this layer constitutes an overlooked performance bottleneck and that stronger representations can be achieved with a low-rank transformation tailored to each patch's phenotype, yielding synergistic effects with any of the existing MIL approaches. To this end, we introduce MAMMOTH, a parameter-efficient, multi-head mixture of experts module designed to improve the performance of any MIL model with minimal alterations to the total number of parameters. Across eight MIL methods and 19 different classification tasks, we find that such task-specific transformation has a larger effect on performance than the choice of aggregation method. For instance, when equipped with MAMMOTH, even simple methods such as max or mean pooling attain higher average performance than any method with the standard linear layer. Overall, MAMMOTH improves performance in 130 of the 152 examined configurations, with an average +3.8% change in performance. Code is available at https://github.com/mahmoodlab/mammoth.
Daniel Shao, Joel Runevic, Richard J. Chen +4
Massachusetts Institute of Technology, Cambridge MA · Harvard University, Cambridge MA · Emory University, Atlanta GA +1