Mixed Magnification Aggregation for Generalizable Region-Level Representations in Computational Pathology
Authors: Eric Zimmermann, Julian Viret, Michal Zelechowski, James Brian Hall, Neil Tenenholtz, Adam Casson, George Shaikovski, Eugene Vorontsov, +2 more
Organizations: Microsoft Research, Cambridge, MA, United States · Paige, NYC, NY, United States
In recent years, a standard computational pathology workflow has emerged where whole slide images are cropped into tiles, these tiles are processed using a foundation model, and task-specific models are built using the resulting representations. At least 15 different foundation models have been proposed, and the vast majority are trained exclusively with tiles using the 20× magnification. However, it is well known that certain histologic features can only be discerned with larger context windows and requires a pathologist to zoom in and out when analyzing a whole slide image. Furthermore, creating 224×224 pixel crops at 20× leads to a large number of tiles per slide, which can be gigapixel in size. To more accurately capture multi-resolution features and investigate the possibility of reducing the number of representations per slide, we propose a region-level mixing encoder. Our approach jointly fuses image tile representations of a mixed magnification foundation model using a masked embedding modeling pretraining step. We explore a design space for pretraining the proposed mixed-magnification region aggregators and evaluate our models on transfer to biomarker prediction tasks representing various cancer types. Results demonstrate cancer dependent improvements in predictive performance, highlighting the importance of spatial context and understanding.
Figures & tables
Figure 1: a An example of a WSI. b A 1344 × 1344 micron region of the WSI. c, d, e Successively zoomed-in 224 × 224 pixel regions of panel b corresponding to 5 × , 10 × , 20 × , respectively. These regions depict the various features at various magnification ranging from tissue organization to individual cells.
Figure 2: An overview of the pretraining framework for the mixing encoder. The basic setting takes masked embeddings from 5 × , 10 × , and 20 × magnifications corresponding to a 3 × 3 region at 5 × magnification as input. a An encoder-decoder architecture trained using a masked reconstruction loss on patch embeddings. b An encoder-projector architecture using crops from a larger context as input augmentations trained with a contrastive loss on CLS embeddings denoted with *. During experimentation, two aspects of the masked reconstruction base setup are investigated: the masking rate and the addition of the contrastive branch.
Breast
Bladder
Colon
Endo
Gastric
Lung
Skin
Average
CDH1
FGFR
MSI
PTEN
HER2
EGFR
BRAF
AB-MIL
5 ×
94.7
82.8
96.6
81.9
79.5
79.2
86.5
85.9
10 ×
95.7
84.4
97.3
81.6
82.5
80.0
63.4
83.6
20 ×
93.1
83.0
97.8
83.6
82.3
79.2
85.5
86.3
all ×
90.9
84.7
98.0
82.1
83.1
78.6
88.9
86.6
Random
5 ×
94.3
86.9
95.8
85.0
78.2
76.8
85.3
86.0
Table 1: Results measured by AUROC using the patch tokens, i.e., uncompressed representations, of the various models. The best and second best performance for each biomarker are bolded and underlined, respectively. On average, pretrained models outperform all baselines.
Breast
Bladder
Colon
Endo
Gastric
Lung
Skin
Average
CDH1
FGFR
MSI
PTEN
HER2
EGFR
BRAF
Random
5 ×
89.1
86.5
95.5
86.5
78.7
76.3
85.5
85.4
10 ×
93.1
83.4
96.5
84.2
78.5
78.3
91.5
86.5
20 ×
95.6
85.5
97.5
84.2
80.5
77.5
94.7
87.9
all ×
96.1
82.9
95.3
83.8
80.6
77.9
95.2
87.4
MEM
0.25
94.7
87.9
98.2
85.4
85.8
83.5
95.2
90.1
Table 2: Results measured by AUROC using the CLS tokens, i.e. compressed representations, of the various models. The best and second best performance for each biomarker are bolded and underlined, respectively. On average, MEM models perform the best. Notably, MEM performance using the CLS token has only about 1.0 decrease in AUROC compared to the uncompressed patch token performance.
Figure 3: Summarized results of various aspects of the pretraining framework as measured by difference in AUROC (higher is better). “Pretrained” refers to MEM and CMEM results averaged over biomarker tasks and removal ratio r and source size c , respectively. Random refers to the setting with no pretraining, averaged over possible magnifications. Overall, we observe improvements by using pretraining whether compared to AB-MIL or a randomly initialized model. MEM generally outperforms CMEM and the contextualized region (Patch) embeddings outperformed the compressed (CLS) embedding.
Biomarker
Subset
Cases
Slides
Positive
Breast-CDH1
train
648
673
0.15
tune
214
219
0.14
test
214
228
0.15
Colon-MSI
train
2008
2268
0.10
tune
335
381
0.12
test
331
373
0.13
Table 3: A summary of the data used for the biomarker prediction tasks. For each biomarker the number of cases, slides (whole slide images) and positivity rate for the train, tune, and test splits is reported.
Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France · Université Paris-Saclay, CentraleSupélec, MICS Laboratory, France