SpaCRD: Multimodal Deep Fusion of Histology and Spatial Transcriptomics for Cancer Region Detection
Authors: Shuailin Xue, Jun Wan, Lihua Zhang, Wenwen Min
Organizations: School of Information Science and Engineering, Yunnan University, Kunming 650500, China · Yunnan Key Laboratory of Intelligent Systems and Computing, Yunnan University, Kunming 650500, China · School of Information Engineering, Zhongnan University of Economics and Law, Wuhan 430073, China · School of Artificial Intelligence, School of Computer Science, Wuhan University, Wuhan 430072, China
Accurate detection of cancer tissue regions (CTR) enables deeper analysis of the tumor microenvironment and offers crucial insights into treatment response. Traditional CTR detection methods, which typically rely on the rich cellular morphology in histology images, are susceptible to a high rate of false positives due to morphological similarities across different tissue regions. The groundbreaking advances in spatial transcriptomics (ST) provide detailed cellular phenotypes and spatial localization information, offering new opportunities for more accurate cancer region detection. However, current methods are unable to effectively integrate histology images with ST data, especially in the context of cross-sample and cross-platform/batch settings for accomplishing the CTR detection. To address this challenge, we propose SpaCRD, a transfer learning-based method that deeply integrates histology images and ST data to enable reliable CTR detection across diverse samples, platforms, and batches. Once trained on source data, SpaCRD can be readily generalized to accurately detect cancerous regions across samples from different platforms and batches. The core of SpaCRD is a category-regularized variational reconstruction-guided bidirectional cross-attention fusion network, which enables the model to adaptively capture latent co-expression patterns between histological features and gene expression from multiple perspectives. Extensive benchmark analysis on 23 matched histology-ST datasets spanning various disease types, platforms, and batches demonstrates that SpaCRD consistently outperforms existing eight state-of-the-art methods in CTR detection.
Figures & tables
Figure 1: SpaCRD achieves accurate CTR detection across diverse samples and platforms&batches by deeply integrating histology images and ST data through transfer learning.
Figure 2: The framework of SpaCRD. Stage I: UNI is used to extract histology features, while modality-alignment representation learning aligns histology and ST modalities into a shared embedding space. Stage II: VRBCA integrates aligned histology and ST features into a compact and class-consistent embedding that captures biologically relevant cross-modal interactions. Stage III: The learned representation is used to estimate cancer likelihood scores for each spot.
Metric
Method
Cross-samples
CRC_A1
CRC_A2
CRC_B1
CRC_B2
CRC_C1
CRC_C2
CRC_D1
CRC_D2
CRC_E1
CRC_E2
CRC_G1
CRC_G2
AUC
SimpleNet
0.491 ± 0.163
0.518 ± 0.098
0.488 ± 0.036
0.481 ± 0.055
0.440 ± 0.047
0.488 ± 0.070
0.518 ± 0.050
0.435 ± 0.036
0.429 ± 0.129
0.580 ± 0.103
0.506 ± 0.082
0.463 ± 0.047
Spatial-ID
0.594 ± 0.068
0.563 ± 0.091
0.518 ± 0.016
0.496 ± 0.043
0.513 ± 0.057
0.503 ± 0.082
0.510 ± 0.029
0.526 ± 0.040
0.457 ± 0.109
0.453 ± 0.093
0.458 ± 0.042
0.525 ± 0.033
STAGE
0.544 ± 0.088
0.516 ± 0.116
0.517 ± 0.037
0.520 ± 0.078
0.488 ± 0.084
0.502 ± 0.059
0.498 ± 0.086
0.490 ± 0.055
0.354 ± 0.129
0.466 ± 0.086
0.488 ± 0.106
0.537 ± 0.117
STANDS
0.540 ± 0.045
0.530 ± 0.043
0.466 ± 0.030
0.506 ± 0.021
0.587 ± 0.053
0.549 ± 0.057
0.564 ± 0.053
0.510 ± 0.022
0.609 ± 0.075
0.643 ± 0.098
0.504 ± 0.027
0.600 ± 0.070
TESLA
0.496 ± 0.016
0.527 ± 0.004
0.436 ± 0.006
0.548 ± 0.007
0.716 ± 0.011
0.593 ± 0.002
0.664 ± 0.006
0.582 ± 0.036
0.517 ± 0.007
0.635 ± 0.024
0.662 ± 0.004
0.721 ± 0.006
Table 1: Quantitative evaluation of SpaCRD for CTR detection on twelve colorectal cancer datasets compared to baselines. Each reported values are means ± standard deviations over five independent runs. Best results in bold, second-best underlined.
Metric
Method
Cross-samples
Cross-platforms&batches
STHBC_A
STHBC_B
STHBC_C
STHBC_D
STHBC_E
STHBC_F
STHBC_G
STHBC_H
ViHBC
XeHBC
IDC
AUC
SimpleNet
0.536 ± 0.126
0.468 ± 0.157
0.476 ± 0.132
0.528 ± 0.143
0.429 ± 0.122
0.536 ± 0.063
0.571 ± 0.101
0.454 ± 0.132
0.468 ± 0.076
0.531 ± 0.106
0.612 ± 0.098
Spatial-ID
0.544 ± 0.045
0.534 ± 0.135
0.471 ± 0.083
0.451 ± 0.085
0.477 ± 0.102
0.496 ± 0.107
0.558 ± 0.094
0.438 ± 0.117
0.436 ± 0.067
0.593 ± 0.081
0.502 ± 0.080
STAGE
0.536 ± 0.126
0.468 ± 0.157
0.476 ± 0.132
0.528 ± 0.143
0.429 ± 0.122
0.536 ± 0.063
0.571 ± 0.101
0.474 ± 0.190
0.444 ± 0.067
0.575 ± 0.054
0.605 ± 0.068
STANDS
0.535 ± 0.053
0.449 ± 0.058
0.510 ± 0.008
0.507 ± 0.038
0.524 ± 0.027
0.575 ± 0.036
0.564 ± 0.046
0.451 ± 0.072
0.629 ± 0.102
0.532 ± 0.037
0.466 ± 0.092
TESLA
0.724 ± 0.008
0.586 ± 0.018
0.551 ± 0.012
0.605 ± 0.011
0.627 ± 0.002
0.705 ± 0.022
0.697 ± 0.014
0.729 ± 0.009
0.672 ± 0.025
0.748 ± 0.026
0.576 ± 0.021
Table 2: Quantitative evaluation of SpaCRD for CTR detection on eleven breast cancer datasets compared to baselines. Each reported values are means ± standard deviations over five independent runs. Best results in bold, second-best underlined.
Figure 3: ATR detection results of SpaCRD and other baselines on the STHBC_G dataset. Green outlines indicate pathologist-annotated CTR. Gray and red dots represent normal and cancerous spots, respectively.
Figure 4: Comparison of the KS distances between predicted cancer likelihood distributions in healthy and tumor regions.
Figure 5: Visualization of cancer likelihood scores predicted by SpaCRD and other baselines on the ViHBC dataset.
Figure 6: Cancer likelihood scores predicted by SpaCRD and the best-performing baseline method, SpaCell-Plus on the XeHBC dataset.
Figure 7: In the IDC dataset, SpaCRD-predicted high-score spots labeled non-cancerous exhibit elevated breast cancer marker expression, suggesting potential early lesions.
Extractor
Cross-samples
Cross-platforms&batches
STHBC
CRC
10XHBC
IDC
XeHBC
AUC
AP
F1
AUC
AP
F1
AUC
AP
F1
AUC
AP
F1
AUC
AP
F1
w/ Swin-Tiny
0.880
0.923
0.838
0.820
0.795
0.793
0.689
0.751
0.728
0.501
0.601
0.601
0.718
0.465
0.501
w/ ResNet50
0.878
0.908
0.836
0.830
0.800
0.794
0.637
0.697
0.701
0.596
0.649
0.657
0.656
0.698
0.691
w/ HIPT
0.885
0.912
0.827
0.822
0.796
0.793
0.784
0.829
0.786
0.399
0.525
0.546
0.771
0.736
0.742
w/ UNI (ours)
0.929
0.946
0.880
0.869
0.854
0.810
0.900
0.930
0.867
0.891
0.914
0.854
0.931
0.859
0.795
Table 3: Ablation study of histology feature extractors conducted across all datasets used in this study.
Model
HBC(eleven datasets)
CRC(twelve datasets)
AUC
AP
F1
AUC
AP
F1
Image-based
0.789
0.752
0.733
0.606
0.557
0.553
ST-based
0.832
0.815
0.747
0.782
0.793
0.755
w/o BCA
0.849
0.833
0.788
0.797
0.774
0.759
w/o RVAE
0.887
0.898
0.817
0.831
0.816
0.795
w/o VRBCA
0.815
0.796
0.734
0.771
0.746
0.717
Table 4: Ablation study of modalities and fusion modules conducted across all datasets used in this study.
Metric
Parameter α
Parameter β
0.0
0.5
1.0
0.1
0.5
1.0
AUC
0.908
0.923
0.916
0.863
0.923
0.920
AP
0.912
0.934
0.929
0.895
0.934
0.931
F1
0.843
0.869
0.863
0.828
0.869
0.867
Metric
Parameter γ
neighboring spots
0.0
0.1
1.0
4
6
10
Table 5: Sensitivity analysis of key hyperparameter across eleven breast cancer datasets. Gray : default settings.
Datasets
Tissue
Spots
Cancer Ratio
Platform
STHBC_A
Breast cancer (Invasive cancer & Cancer in situ)
346
89.9%
ST
STHBC_B
Breast cancer (Invasive cancer)
295
22.0%
STHBC_C
Breast cancer (Invasive cancer)
176
73.9%
STHBC_D
Breast cancer (Invasive cancer)
306
48.1%
STHBC_E
Breast cancer (Invasive cancer)
587
57.9%
STHBC_F
Breast cancer (Invasive cancer)
691
85.7%
Table S1: Summary of all experimental datasets used in this study.
Figure S1: ATR detection results of SpaCRD and other baselines on the eight STHBC datasets. Gray and red dots represent normal and cancerous spots, respectively.
Figure S2: Visualization of cancer likelihood scores predicted by SpaCRD and other baselines on the XeHBC dataset.
Figure S3: Visualization of cancer likelihood scores predicted by SpaCRD and other baselines on the IDC dataset.
Figure S4: Histograms of predicted cancer likelihood scores by SpaCRD and other baselines on the ViHBC dataset, grouped by three region types: healthy, tumor, and tumor boundary, based on ground truth.
Figure S5: Violin plots of predicted cancer likelihood scores by SpaCRD and other baselines on the ViHBC dataset, grouped by three region types: healthy, tumor, and tumor boundary, based on ground truth.
Method
Params
Training Time (s)
Inference Time (s)
Memory Usage (MB)
SpaCRD (Ours)
8.70M
33.86
1.15
535.68
MEATRD
48.61M
1378.70
13.35
8401.39
STANDS
68.19M
986.77
12.89
18426.91
SpaCell-Plus
1.72M
44.25
0.06
94.14
iStar
2.41M
222.03
11.98
325.15
TESLA
5.68M
5790.68
18.71
7619.64
Table S2: Comparison of computational cost on the ViHBC dataset among all methods.
Training data
ViHBC
XeHBC
IDC
AUC
AP
F1
AUC
AP
F1
AUC
AP
F1
C (176 spots)
0.861 ± 0.029
0.882 ± 0.035
0.817 ± 0.018
0.875 ± 0.022
0.749 ± 0.047
0.731 ± 0.038
0.779 ± 0.034
0.783 ± 0.36
0.757 ± 0.41
D (306 spots)
0.888 ± 0.016
0.913 ± 0.016
0.863 ± 0.010
0.911 ± 0.016
0.820 ± 0.037
0.759 ± 0.025
0.867 ± 0.021
0.886 ± 0.26
0.833 ± 0.36
Full (3,481 spots)
0.900 ± 0.029
0.930 ± 0.037
0.867 ± 0.014
0.931 ± 0.008
0.859 ± 0.021
0.795 ± 0.013
0.891 ± 0.006
0.914 ± 0.006
0.854 ± 0.008
Table S3: Small-sample training analysis results on the ViHBC, XeHBC, and IDC datasets.
Yunnan University, Kunming 650500, China · Shenzhen Research Institute of Big Data, Shenzhen 518172, China · Zhongnan University of Economics and Law, Wuhan 430073, China
Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA · Department of Statistics and Data Science, Yale University, New Haven, CT, USA · Department of Medicine (Oncology), Stanford University School of Medicine, Stanford, CA, USA +4
School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China · Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China · Department of Pathology, Sichuan Provincial People’s Hospital, University of Electronic Science and Technology of China, Chengdu 610072, China