Mitigating Spatial Blindness in Multiple Instance Learning for Whole-Slide Image Analysis
Authors: Xiangyu Li, Jincheng Li, Xing Xie, Bin Guan, Shaohua Kevin Zhou, Rui Yan, Ran Su
Organizations: College of Intelligence and Computing, Tianjin University, Tianjin, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE) Lab, USTC Suzhou Institute for Advanced Research, Suzhou, China · School of Biomedical Engineering, University of Science and Technology of China, Hefei, China · Engineering Training Center, Nantong University, Nantong, China · School of Information Science and Engineering, University of Jinan, Jinan, China
Whole-slide multiple instance learning can predict diagnostic and prognostic labels from tissue composition while making limited use of spatial organization. Providing coordinates or a graph does not establish that a trained predictor uses tissue neighborhoods. We examine this gap as spatial blindness and hypothesize that early composition learning can weaken supervision of a spatial pathway. We introduce ResTopoMIL, which freezes a learned composition predictor before training a graph residual with a coordinate-reassignment separation loss. Permutation and information analyses distinguish architectural symmetry, available spatial information and the proposed optimization mechanism. On a controlled spatial task, staged training achieved an AUC of 0.987, compared with 0.684 for the same jointly trained architecture. Performance remained high on a composition-only task. Across four WSI classification and five survival benchmarks, ResTopoMIL improved mean primary performance on eight tasks; four improvements remained significant after Holm-adjusted patient-cluster tests. Coordinate-free residual controls recovered some of the gains, while neighborhood destruction reduced performance close to the composition anchor. Neighborhood-preserving transformations and anatomy-stratified perturbations further characterized this dependence. These results support fixing a composition reference as a practical intervention for learning spatial corrections, while retaining additional appearance capacity as a contributing explanation.
Figures & tables
Fig. 1 : Motivation and coordinate-reassignment diagnostic. (a) Coordinate-free MIL receives the same appearance multiset after coordinate reassignment and therefore gives the same prediction; joint relabeling of paired features and coordinates is a different operation. The schematic AUC values are illustrative. (b) ResTopoMIL separates a composition reference from a graph correction constrained to distinguish intact from disrupted neighborhoods. (c) The TCGA-BRCA diagnostic holds patch embeddings fixed while reassigning coordinates. Near-invariance is expected when a model’s effective inputs do not change; it does not itself establish failed spatial learning. Table 6 tests spatial baselines with rebuilt coordinate-derived inputs.
Fig. 2 : ResTopoMIL framework. A frozen patch encoder supplies shared appearance features. The statistical stream averages soft prototype assignments into a composition histogram and is trained in Stage 1, then frozen in Stage 2. The topological stream constructs KNN graphs from intact and reassigned coordinates and processes both views with a shared, trainable two-layer GCN. A margin loss separates their pooled representations. The statistical and intact-graph prediction logits are added before probability conversion, as specified in Eq. ( 15 ). The reassigned view and separation loss are used only during training; inference uses the intact graph.
Fig. 3 : Gradient dynamics on BRCA and PANDA. Upper panels compare joint and staged optimization; the dashed vertical line marks the transition after ten warm-up epochs. The statistical stream is then frozen and graph training begins. Lower panels compare joint graph training with and without Ltexture . The plotted gradient norms are not separated into supervised and auxiliary components and provide descriptive optimization evidence.
Fig. 4 : Progressive coordinate shuffling with fixed patch embeddings. Panels report AUC on TCGA-BRCA/PANDA and C-index on TCGA-KIRP/TCGA-LUAD. Curves are representative-seed trajectories, with no uncertainty bands; they are not five-seed aggregate estimates.
Method
Dice
Specificity
FROC
AB-MIL
0.412
0.985
0.3952
CLAM-SB
0.459
0.987
0.4257
DS-MIL
0.259
0.863
0.4506
TransMIL
0.103
0.999
0.4866
DTFD-MIL
0.525
0.999
0.4712
MHIM-MIL
0.548
0.992
0.4815
Table 8: CAMELYON-16 localization (upper panel) and post-hoc anatomy-stratified graph perturbation (lower panel). FROC uses the reported seven-point patch-grid summary. Masks stratify perturbations after training; higher values are better. Best and second-best values are bold and underlined within each panel; ties share formatting.
Fig. 5 : Qualitative H&E and native heatmap comparisons across five MIL methods. Warmer colors indicate higher within-slide normalized scores. Non-tissue patches are masked, with no smoothing, morphological refinement, or tumor-shape prior. The examples illustrate contiguity and background leakage, rather than segmentation accuracy. Case labels are internal to this figure.
PANDA
BRCA
Schedule
Stat.
Full
Stat.
Full
10 epochs
0.9027
0.9426
0.9486
0.9838
20 epochs
0.9029
0.9425
0.9488
0.9839
Early stopping
0.9028
0.9427
0.9487
0.9838
Table 9: Warm-up schedule analysis. Cells give reported AUC without run-level dispersion. Stat./Full denote the anchor and complete model. Bold and underlined AUCs mark the best and second-best distinct values per column; ties share formatting.
Fig. 6 : BRACS feature-space projections. The upper panel uses PCA and the lower uses t-SNE; each shows statistical-stream features above topological-stream features across five reference splits. Colors denote classes, hollow circles denote statistically hard samples, and stars denote cases corrected by the topological stream. Class separation is descriptive evidence, rather than a test of coordinate dependence.
Whole-slide MIL models are often called context-aware once graphs, Transform ers, or state-space modules are placed above patch embeddings. We show that this label can be deceptive. On pathology tasks where tissue architecture is part of the diagnostic signal, several strong MIL baselines retain nearly unchanged slide level AUC after patch coordinates are permuted. Their predictions are accurate, but largely compositional. We refer to this failure mode as spatial blindness. Our explanation is optimization-based: dense appearance statistics are learned early under slide-level supervision, leaving weak gradients for sparse spatial relations. ResTopoMIL addresses the issue by first fitting a permutation-invariant prototype histogram and then freezing it while a lightweight graph branch learns the residual under a coordinate-shuffling constraint. The architecture is simple by design; the intervention is in how the spatial branch is trained. Across 9 public WSI bench marks, ResTopoMIL improves classification and survival prediction with 1.15M parameters, restores sensitivity to coordinate perturbation, and gives stronger lo calization evidence on CAMELYON-16.
Xiangyu Li, Ran Su
College of Intelligence and Computing Tianjin University Tianjin, China
Multiple instance learning (MIL) is the standard approach for whole-slide image (WSI) classification and survival prediction, where attention-based models ag gregate patch features into slide-level predictions. These models treat attention weights as evidence for their predictions, but attention is optimized for classi fication, not for identifying which patches actually support the diagnosis. This conflation leads to three failures: selected patches are insufficient (keeping them alone drops Macro-F1 by 0.078), unnecessary (removing them barely changes the prediction), and unrecoverable (continuous attention scores disagree with discrete patch subsets used at inference). The central premise is that evidence quality should be optimized directly through explicit criteria- Sufficiency, Necessity, and Recov erability (S/N/R)- rather than inherited as a byproduct of classification. GCE-MIL is a backbone-agnostic wrapper implemented through three injection modes and three evidence components: a grounding mechanism that aligns selection with domain-specific concepts, noisy-OR coverage that acts as a differentiable proxy for interventional evidence search, and threshold-plus-repair recovery that converts continuous selectors into discrete subsets through marginal-guided repair. Across 9 backbones and 9 datasets (81 configurations), GCE-MIL improves average Macro-F1 by 0.024 and C-index by 0.014, reduces the continuous-discrete gap by 4-7, and increases complement degradation by 2-4. With optional tile prefiltering after discrete recovery, inference runs up to 5 faster while retaining 0.989 full-bag utility.
Xiangyu Li, Ran Su
College of Intelligence and Computing Tianjin University
Computational pathology involves the digitization of stained tissues into whole-slide images (WSIs) that contain billions of pixels arranged as contiguous patches. Statistical analysis of WSIs largely focuses on classification via multiple instance learning (MIL), in which slide-level labels are inferred from unlabeled patches. Most MIL methods treat patches as exchangeable, overlooking the rich spatial and topological structure that underlies tissue images. This work builds on recent graph-based methods that aim to incorporate spatial awareness into MIL. Our approach is new in two regards: (1) we deploy vision foundation-model embeddings to incorporate local spatial structure within each patch, and (2) achieve across-patch spatial awareness using graph neural networks together with a novel {\em jigsaw regularization}. We find that a combination of these two features markedly improves classification over state-of-the-art attention-based MIL approaches on benchmark datasets in breast, head-and-neck, and colon cancer.
So Won Jeong, Veronika Ročková
Booth School of Business, The University of Chicago, IL, USA.