Dictionary-based Pathology Mining with Hard-instance-assisted Classifier Debiasing for Genetic Biomarker Prediction from WSIs
Authors: Ling Zhang, Boxiang Yun, Ting Jin, Qingli Li, Xinxing Li, Yan Wang
Organizations: Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai, China · Department of General Surgery, Tongji Hospital, Tongji University School of Medicine, Shanghai 200065, China
Prediction of genetic biomarkers, e.g., microsatellite instability in colorectal cancer is crucial for clinical decision making. But, two primary challenges hamper accurate prediction: (1) It is difficult to construct a pathology-aware representation involving the complex interconnections among pathological components. (2) WSIs contain a large proportion of areas unrelated to genetic biomarkers, which make the model easily overfit simple but irrelative instances. We hereby propose a Dictionary-based hierarchical pathology mining with hard-instance-assisted classifier Debiasing framework to address these challenges, dubbed as D2Bio. Our first module, dictionary-based hierarchical pathology mining, is able to mine diverse and very fine-grained pathological contextual interaction without the limit to the distances between patches. The second module, hard-instance-assisted classfier debiasing, learns a debiased classifier via focusing on hard but task-related features, without any additional annotations. Experimental results on five cohorts show the superiority of our method, with over 4% improvement in AUROC compared with the second best on the TCGA-CRC-MSI cohort. Our analysis further shows the clinical interpretability of D2Bio in genetic biomarker diagnosis and potential clinical utility in survival analysis. Code will be available at https://github.com/DeepMed-Lab-ECNU/D2Bio.
Figures & tables
Fig. 1: Motivation of our dictionary-based strategy. (a) Attention-based MIL methods: simply model the relationships between instances. (b) Graph-based methods: model instance relationships without construction of real pathological components. (c) Proposed dictionary-based method: utilizes a learnable dictionary to group instances into pathological components and hierarchically mine pathological contextual interaction.
Fig. 2: Motivation of our hard-instance-assisted classifier debiasing strategy. Sufferring from redundant task-irrelevant instances, during inference, biased classifier misjudges the bag-level labels by overfitting simple instances. Learning hard instances which include task-specific instances can assist the classifier to reduce the bias.
Fig. 3: Illustration of D 2 Bio. The overall framework consists of two parts: 1) dictionary-based hierarchical pathology mining, 2) hard-instance assisted-classifier debiasing. Given a WSI X , our D 2 Bio first initialize a learnable dictionary to extract pathological information of X via cross attention operation. Then our D 2 Bio groups instances into fine-grained pathological groups according to the similarity matrix. To hierarchically mine the interaction, Multi-head Self-attention (MSA) is first employed in each group. Features of X are updated by ungrouping these groups, which is further used to update the dictionary. After repeating the above steps L times, inter-group ViT is employed. Finally, our D 2 Bio assigns hard instance pseudo labels via unsupervised clustering to supervise the classfication head to reduce the bias in WSIs. The purple arrow indicates the classification branch and the orange arrow indicates the classifier debiasing branch.
CtransPath [ 50 ]
Method
TCGA-CRC-MSI
TCGA-CRC-BRAF
CPTAC-MSI
CPTAC-BRAF
TCGA-STAD-MSI
AUC
Acc
PRC
Bal
AUC
Acc
PRC
Bal
AUC
Acc
PRC
Bal
AUC
Acc
PRC
Bal
AUC
Acc
PRC
Bal
ABMIL [ 11 ]
88.00 ∗
80.80 ∗
64.80
86.20
68.40 ∗
80.80
24.80
54.90 ∗
87.50
76.20 ∗
49.10
50.00 ∗
85.20
85.70 ∗
56.90
50.00 ∗
77.80
65.70 ∗
46.80 ∗
67.80 ∗
(75.5-96.6)
(73.3-87.5)
(46.0-83.1)
(77.6-92.4)
(46.1-87.6)
(74.2-87.5)
(6.4-55.7)
(41.5-71.0)
(68.8-100.0)
(76.2-76.2)
(29.8-100.0)
(50.0-50.0)
(72.6-99.4)
(85.7-85.7)
(21.8-95.8)
(50.0-50.0)
(67.8-87.2)
(56.6-74.7)
(35.2-68.2)
(57.7-77.8)
DSMIL [ 12 ]
88.20 ∗
87.50 ∗
62.70
81.40
63.80 ∗
76.70 ∗
24.70
52.70 ∗
91.20
76.20
57.30
56.90 ∗
87.00
85.70
38.80
50.00
77.10
63.60 ∗
47.70 ∗
63.80 ∗
(76.1-97.5)
(81.7-93.3)
(40.2-87.9)
(68.6-92.4)
(40.5-83.8)
(70.0-83.3)
(5.6-56.8)
(39.3-69.2)
(76.2-100.0)
(61.9-85.7)
(34.4-100.0)
(40.6-76.9)
(78.1-92.2)
(85.7-85.7)
(22.1-75.2)
(50.0-50.0)
(66.7-86.6)
(54.5-72.7)
(35.2-69.3)
(53.1-73.8)
TABLE I: Comparisons with state-of-the-art MIL methods on five cohorts using two feature extractors, i.e. , CtransPath [ 50 ] and UNI v2 [ 51 ] . We report Mean (95% CI) in percentage (%). “*” denotes statistical significance (p < 0.05). Best results are shown in bold .
Fig. 4: Heatmap visualization of D 2 Bio on the MSI prediction task and identified pathological patterns.
Pathology Mining
Debiasing
TCGA-CRC-MSI
TCGA-STAD-MSI
×
×
88.63
64.05
×
✓
91.49
79.73
×
88.70
64.32
✓
×
93.46
80.00
✓
✓
96.70
81.73
TABLE II: Ablation study of two key components on TCGA-CRC-MSI cohort. w/o Mining: All instances are fed into two ViT layers without dictionary-guided instance grouping. The symbol “*” indicates only inter-group interaction mining is conducted without intra-group interaction mining. w/o Debiasing: Remove the hard-instance-assisted classifier debiasing module.
Cohort
Rand.
Text
K-means
k-NN (dist)
k-NN (cos)
Dic.
TCGA-CRC-MSI
89.46
91.75
92.83
92.00
90.86
96.70
TCGA-STAD-MSI
78.97
78.70
75.62
65.73
66.49
81.73
TABLE III: Ablation study of grouping strategies on the TCGA-CRC-MSI and TCGA-STAD-MSI cohorts. Rand.: Randomly dividing instances. K-means: Clustering instances using K-means. Text: using GPT-4 TO GENERATE AS MANY DESCRIPTIONS AS POSSIBLE, i.e. , 15 descriptions and grouping instances into 15 groups following [ 28 ] . k-NN (dist): k-NN grouping based on Euclidean distance. k-NN (cos): k-NN grouping based on cosine similarity. Dic.: Dictionary-guided grouping. The best result for each cohort is highlighted .
Fig. 5: Pathological group distribution and corresponding pathological patterns on a WSI of MSI cancer.
Fig. 6: Performance changes by varying the (a) dictionary size ( K ), (b) number of groups ( G ) and (c) number of dictionary updates ( L ) on the TCGA-CRC-MSI and TCGA-STAD-MSI cohort.
Fig. 7: t-SNE visualization of bag representations in different methods on the testing set of TCGA-CRC-MSI cohort. Our dictionary-based method constructs more discriminative representations of WSIs.
Fig. 8: (a) AUROC and (b) accuracy improvement on the TCGA-CRC-MSI cohort after employing dictionary-based hierarchical pathology mining to four MIL baselines, i.e. , ABMIL [ 11 ] , DSMIL [ 12 ] , CLAM [ 13 ] and TransMIL [ 14 ] . Performance improvement is marked red .
Fig. 9: Performance changes by varying the (a) hard threshold ( α ), (b) threshold for the number of hard instances ( β ) and (c) weight in loss function ( λ ) on the TCGA-CRC-MSI and the TCGA-STAD-MSI cohort.
Fig. 10: Distributions of ∣2p^i−1∣ on the (a) TCGA-CRC-MSI and (b) TCGA-STAD-MSI cohort.
Fig. 11: Heatmap visualizations of D 2 Bio w/o or w/ the hard-instance-assisted classifier debiasing module.
Fig. 12: (a) AUROC and (b) accuracy improvement on the TCGA-CRC-MSI cohort by employing hard-instance-assisted debiasing module to three MIL baselines: ABMIL [ 11 ] , CLAM [ 13 ] and TransMIL [ 14 ] . Performance improvement is marked red .
Extractor
TCGA-CRC-MSI
TCGA-STAD-MSI
DSMIL [ 12 ]
WiKG [ 25 ]
D 2 Bio (ours)
DSMIL [ 12 ]
WiKG [ 25 ]
D 2 Bio (ours)
AUROC
Acc.
AUROC
Acc.
AUROC
Acc.
AUROC
Acc.
AUROC
Acc.
AUROC
Acc.
CtransPath [ 50 ]
88.19
87.50
90.22
90.00
96.70
92.50
77.14
63.64
79.84
70.71
81.73
77.78
Virchow2 [ 59 ]
91.49
91.67
93.10
90.83
97.52
93.33
84.22
78.79
82.97
77.78
86.65
83.84
UNI [ 51 ]
90.54
89.17
90.92
90.83
94.35
94.17
65.46
72.73
82.49
73.74
86.32
74.75
UNI v2 [ 51 ]
89.21
90.83
95.49
94.10
98.48
96.67
84.05
74.75
84.22
72.73
86.16
82.83
TABLE IV: Performance of DSMIL [ 12 ] , WiKG [ 25 ] and our D 2 Bio using different pre-trained feature extractors on TCGA-CRC-MSI cohort and TCGA-SATD-MSI chohort. CtransPath [ 50 ] : Trained on over 32,000 WSIs of TCGA and PAIP datasets. Virchow2 [ 59 ] : Trained on 3.1 million WSIs from globally diverse institutions. UNI [ 51 ] : Trained on approximately 100,000 WSIs of the Mass-100k dataset. UNI v2 [ 51 ] : The extended version of UNI [ 51 ] , trained on approximately 350,000 WSIs. Best results are in bold and second-best results are underlined .
Fig. 13: KM plot of OS and PFS according to three methods, i.e. , our D 2 Bio, CLAM [ 13 ] and WiKG [ 25 ] , predicted patient-level “High MSI & Low SH-MSI” vs other subtypes. “SH-MSI” refers to spatial heterogeneity of MSI.
Method
Time per Epoch
CLAM-SB [ 13 ]
∼ 9 s
TransMIL [ 14 ]
∼ 10 s
ABMIL [ 11 ]
∼ 27 s
D 2 Bio (Ours)
∼ 60 s
TABLE V: Average training time per epoch on TCGA-CRC-MSI.
Institute of Pathology, Technical University of Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Munich Data Science Institute (MDSI), Munich, Germany +6