Organizations: SWJTU-Leeds Joint School, Southwest Jiaotong University, Chengdu, China · Yu-Yue Pathology Scientific Research Center, Jinfeng Laboratory, Chongqing, China · Department of Epidemiology and Medical Statistics, West China School of Public Health and West China Fourth Hospital, Sichuan University, Chengdu, China · School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, China · Department of Epidemiology and Biostatistics, School of Public Health, Imperial College London, London, UK
Cancer pathological analysis requires modeling tumor heterogeneity across multiple modalities, primarily through transcriptomics and whole slide imaging (WSI), along with their spatial relations. On one hand, bulk transcriptomics and WSI images are largely available but lack spatial mapping; on the other hand, spatial transcriptomics (ST) data can offer high spatial resolution, yet facing challenges of high cost, low sequencing depth, and limited sample sizes. Therefore, the data foundation of either side is flawed and has its limit in accurately finding the mapping between the two modalities. To this end, we propose BiTro, a bidirectional transfer learning framework that can enhance bulk and spatial transcriptomics prediction from pathological images. Our contributions are twofold. First, we design a universal and transferable model architecture that works for both bulk+WSI and ST data. A major highlight is that we model WSI images on the cellular level to better capture cells' visual features, morphological phenotypes, and their spatial relations; to map cells' features to their transcriptomics measured in bulk or ST, we adopt multiple instance learning. Second, by using LoRA, our model can be efficiently transferred between bulk and ST data to exploit their complementary information. To test our framework, we conducted comprehensive experiments on five cancer datasets. Results demonstrate that 1) our base model can achieve better or competitive performance compared to existing models on bulk or spatial transcriptomics prediction, and 2) transfer learning can further improve the base model's performance.
Figures & tables
Figure 1. Overview of the BiTro framework. There are four sequential phases: (a) WSI preprocessing : segment a WSI into cells by HoverNet or CellViT; (b) Cell feature extraction : extract cells’ visual features via Dinov3, capture cell phenotypes by K-means clustering, and add spatial coordinates; (c) Spatial enhancement : enhance cell features with both local and global spatial relations; (d) MIL and Bidirectional transfer learning : map cellular features to their bulk or spot-level transcriptomics by multiple instance learning. The learning process is enhanced by both levels of data via bidirectional transfer learning.
Figure 2. Model architecture. Inputs are cellular visual features H and their local spatial graph G . The GAT module and the Transformer encoder are to enhance H with spatial relations both locally and globally. The enhanced features Hcell are used to predict gene expression y^ by multiple instance pooling.
Method
BRCA
COAD
PRAD
LIHC (HCC)
PCCoverall↑
JS ↓
PCCoverall↑
JS ↓
PCCoverall↑
JS ↓
PCCoverall↑
JS ↓
Original
0.502 ± 0.198
0.181 ± 0.046
0.869 ± 0.015
0.040 ± 0.015
0.659 ± 0.148
0.092 ± 0.034
0.662 ± 0.248
0.214 ± 0.076
UNI+MLP
0.652 ± 0.275
0.132 ± 0.059
0.920 ± 0.098
0.021 ± 0.015
0.806 ± 0.127
0.053 ± 0.021
0.735 ± 0.177
0.174 ± 0.062
HE2RNA
0.710 ± 0.032
0.086 ± 0.002
0.911 ± 0.005
0.024 ± 0.002
0.830 ± 0.157
0.051 ± 0.005
0.827 ± 0.234
0.119 ± 0.084
SEQUOIA
0.738 ± 0.199
0.094 ± 0.059
0.932 ± 0.043
0.020 ± 0.022
0.820 ± 0.068
0.056 ± 0.019
0.738 ± 0.109
0.149 ± 0.014
BiTro (w/o trans.)
0.711 ± 0.174
0.107 ± 0.081
0.933 ± 0.007
0.021 ± 0.009
0.814 ± 0.127
0.050 ± 0.022
0.811 ± 0.263
0.139 ± 0.093
Table 1. Performance comparison on bulk transcriptomics prediction. Best results are bolded, and the second best underlined.
Model
BRCA
COAD
PRAD
HCC
CSCC
PCCoverall↑
PCCgene↑
PCCoverall↑
PCCgene↑
PCCoverall↑
PCCgene↑
PCCoverall↑
PCCgene↑
PCCoverall↑
PCCgene↑
HisToGene
0.541 ± 0.079
0.114 ± 0.079
0.515 ± 0.161
0.102 ± 0.062
0.449 ± 0.120
0.110 ± 0.087
0.282 ± 0.020
0.072 ± 0.005
0.560 ± 0.049
0.157 ± 0.086
Hist2ST
0.574 ± 0.090
0.141 ± 0.084
0.621 ± 0.080
0.133 ± 0.132
0.438 ± 0.160
0.071 ± 0.057
0.274 ± 0.013
-0.001 ± 0.005
0.610 ± 0.109
0.131 ± 0.061
THItoGene
0.558 ± 0.099
0.094 ± 0.089
0.614 ± 0.076
0.075 ± 0.081
0.484 ± 0.104
0.134 ± 0.106
0.247 ± 0.051
0.005 ± 0.003
0.599 ± 0.039
0.241 ± 0.044
M2OST
0.558 ± 0.660
0.117 ± 0.081
0.514 ± 0.057
0.122 ± 0.117
0.515 ± 0.094
0.256 ± 0.070
0.124 ± 0.002
0.102 ± 0.030
0.450 ± 0.084
0.142 ± 0.057
BiTro (w/o trans.)
0.587 ± 0.101
0.154 ± 0.136
0.565 ± 0.066
0.103 ± 0.092
0.502 ± 0.129
0.186 ± 0.075
0.341 ± 0.012
0.128 ± 0.025
0.667 ± 0.041
0.429 ± 0.154
Table 2. Performance comparison on spatial transcriptomics prediction. Best results are bolded, and the second best underlined.
Model
BRCA
COAD
PRAD
HCC
PCCoverall↑
PCCgene↑
PCCoverall↑
PCCgene↑
PCCoverall↑
PCCgene↑
PCCoverall↑
PCCgene↑
GHIST
0.499 ± 0.024
0.348 ± 0.038
0.450 ± 0.092
0.209 ± 0.056
0.241 ± 0.010
0.141 ± 0.009
0.509 ± 0.070
0.204 ± 0.108
iStar
0.562 ± 0.037
0.223 ± 0.018
0.639 ± 0.024
0.153 ± 0.012
0.289 ± 0.020
0.094 ± 0.016
0.699 ± 0.033
0.166 ± 0.016
BiTro (w/o trans.)
0.512 ± 0.026
0.089 ± 0.034
0.513 ± 0.041
0.084 ± 0.042
0.328 ± 0.031
0.051 ± 0.021
0.587 ± 0.042
0.084 ± 0.017
BiTro (trans)
0.544 ± 0.041
0.135 ± 0.024
0.533 ± 0.047
0.103 ± 0.041
0.342 ± 0.019
0.082 ± 0.019
0.542 ± 0.036
0.113 ± 0.012
Table 3. Performance comparison on cellular transcriptomics prediction. Best results are bolded, and the second best underlined.
Figure 3. Gene expression visualization on the SPA148 sample of BRCA for gene in log1p space. A visualization showing the gene expression heatmaps for GNAS on the SPA148 dataset.
Figure 4. Visualization of BiTro’s inference on super-resolved spatial gene expression for gene EPCAM in the COAD sample TENX139 at both spot and cellular levels in the log1p space.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Set
BRCA
COAD
LIHC (HCC)
PRAD
Training
200
246
104
152
Test
50
59
27
38
Total
250
305
131
190
Genes
734
388
185
175
Note: CSCC has no Bulk transcriptomics data in this study.
Appendix
Table 4. Dataset statistics of bulk transcriptomics and WSI cases.
Metric
BRCA
COAD
CSCC
LIHC (HCC)
PRAD
Technology
ST
Visium
ST
Visium
Visium
Samples
36
15
12
2
23
Total Spots
13,620
44,823
8,671
4248
62,710
Min Spots/Sample
178
328
462
1987
1,418
Genes
785
388
469
200
200
Appendix
Table 5. Dataset statistics of spot-level ST datasets.
Metric
BRCA
COAD
LIHC (HCC)
PRAD
Technology
Xenium
Xenium
Xenium
Xenium
ID
NCBI785
TENX139
TENX120
TENX157
Total cells
167,780
388,175
162,628
193,000
Median transcripts/cell
166.0
78.0
114.0
144.0
Transcripts per 100\SIUnitSymbolMicrom2
106.0
182.8
85.0
371.7
Genes
278
465
474
300
Appendix
Table 6. Dataset statistics of super-resolution (cellular) ST datasets.
Configuration
BRCA
COAD
PRAD
LIHC
O.
G.
O.
G.
O.
G.
O.
G.
BiTro.
0.743 ± 0.186
0.091 ± 0.058
0.946 ± 0.007
0.017 ± 0.014
0.831 ± 0.112
0.043 ± 0.012
0.844 ± 0.194
0.117 ± 0.102
w/o Trans.
0.711 ± 0.174
0.107 ± 0.065
0.937 ± 0.009
0.019 ± 0.008
0.821 ± 0.143
0.047 ± 0.024
0.823 ± 0.247
0.131 ± 0.074
w/o Clu+Trans.
0.708 ± 0.183
0.104 ± 0.062
0.933 ± 0.007
0.021 ± 0.009
0.814 ± 0.127
0.050 ± 0.022
0.811 ± 0.263
0.139 ± 0.093
Appendix
Table 7. Ablation study on bulk transcriptomics prediction. We compare our base and transfer models under three configurations: standard (Trans+Cluster), without clustering (w/o Clu.), without Clustering and Trans (w/o Clu+Trans). O. and G. represent Overall and Gene Pearson correlations. The best result for each dataset is bolded.
Type
Config.
BRCA
COAD
PRAD
HCC
CSCC
O.
G.
O.
G.
O.
G.
O.
G.
O.
G.
Base
Norm+Clu.
0.578 ± 0.11
0.136 ± 0.13
0.566 ± 0.07
0.094 ± 0.10
0.572 ± 0.15
0.100 ± 0.12
0.340 ± 0.01
0.128 ± 0.03
0.574 ± 0.01
0.231 ± 0.08
w/o Clu.
0.587 ± 0.10
0.154 ± 0.14
0.565 ± 0.07
0.103 ± 0.09
0.502 ± 0.13
0.186 ± 0.08
0.341 ± 0.02
0.128 ± 0.03
0.667 ± 0.04
0.429 ± 0.15
w/o Norm.
0.618 ± 0.12
0.120 ± 0.09
0.485 ± 0.13
0.073 ± 0.07
0.526 ± 0.15
0.091 ± 0.10
0.074 ± 0.03
0.133 ± 0.04
0.202 ± 0.01
0.250 ± 0.02
w/o Norm+Clu.
0.589 ± 0.09
0.088 ± 0.09
0.456 ± 0.13
0.104 ± 0.14
0.514 ± 0.12
0.127 ± 0.07
0.081 ± 0.04
0.134 ± 0.04
0.598 ± 0.06
0.424 ± 0.16
Trans.
Norm+Clu.
0.588 ± 0.10
0.162 ± 0.14
0.573 ± 0.06
0.116 ± 0.10
0.516 ± 0.13
0.204 ± 0.13
0.342 ± 0.01
0.143 ± 0.01
—
—
Appendix
Table 8. Ablation study on spatial transcriptomics prediction. We compare our base and transfer models under three configurations: standard (Norm+Cluster), without clustering (w/o Clu.), without normalization (w/o Norm.), and without clustering and normalization (w/o Norm+Clu.). O. and G. represent Overall and Gene Pearson correlations. The best result for each dataset is bolded.
Automating the analysis of whole-slide images (WSIs), a key step in cancer diagnosis, has high clinical value, as it can reduce pathologist's workload while improving diagnosis accuracy. Recently, vision-language models have shown promising performance for patch-level classification without requiring any annotation, yet these zero-shot (ZS) predictions remain noisy on fine-grained tasks and must be further refined. A promising direction is to refine all predictions jointly, i.e., a transductive approach. However, most existing methods are not tailored to WSIs. We thus propose SlideTIM, an adaptation to WSIs of the recent transductive approach LC-TIM, which introduces a combined spatial--latent regularizer together with a prior on the patch class distribution. The former enforces spatially and semantically close patches to receive the same predictions, while the prior calibrates the predicted class proportions. Together, they address the complex spatial organization and the strong class imbalance of WSIs. Evaluated on four histology datasets, SlideTIM consistently outperforms all TIM variants, improving the macro-F1 by +8.1pp over the best competing baseline at 1 shot. Compared to the ZS, it raises the macro-F1 by +19.4pp at 1 shot. The code will be made available after submission.
Tiffanie Godelaine, Manon Dausort, Karim El Khoury +3
ICTEAM, Universit´e Catholique de Louvain, Belgium
Spatial transcriptomics (ST) enables gene expression mapping within anatomical context but remains costly and low-throughput. Hematoxylin and eosin (H&E) staining offers rich morphology yet lacks molecular resolution. We present \ours (Spatial Transcriptomics and histOlogy Representation Model), a foundation model trained on 1.2 million spatially resolved transcriptomic profiles with matched histology across 18 organs. Using a hierarchical architecture integrating morphological features, gene expression, and spatial context, STORM bridges imaging and omics through robust molecular--morphological representations. STORM enhances spatial domain discovery, producing biologically coherent tissue maps, and outperforms existing methods in predicting spatial gene expression from H&E images across 11 tumor types. The model is platform-agnostic, performing consistently across Visium, Xenium, Visium HD, and CosMx. Applied to 23 independent cohorts comprising 7,245 patients, STORM significantly improves immunotherapy response prediction and prognostication over established biomarkers, providing a scalable framework for spatially informed discovery and clinical precision medicine.
Jinxi Xiang, Siyu Hou, Yuchen Li +17
Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA · Department of Statistics and Data Science, Yale University, New Haven, CT, USA · Department of Medicine (Oncology), Stanford University School of Medicine, Stanford, CA, USA +4
Spatial Transcriptomics (ST) profiles thousands of gene expression values at discrete spots with precise coordinates on tissue sections, preserving spatial context essential for clinical and pathological studies. With rising sequencing throughput and advancing platforms, the expanding data volumes motivate large-scale ST pretraining. However, the fundamental unit for pretraining, i.e., what constitutes a single training sample, remains ill-posed. Existing choices fall into two camps: (1) treating each spot as an independent sample, which discards spatial dependencies and collapses ST into single-cell transcriptomics; and (2) treating an entire slide as a single sample, which produces prohibitively large inputs and drastically fewer training examples, undermining effective pretraining. To address this gap, we propose treating spatial transcriptomics as croppable images. Specifically, we define a multi-channel image representation with fixed spatial size by cropping patches from raw slides, thereby preserving spatial context while substantially increasing the number of training samples. Along the channel dimension, we define gene subset selection rules to control input dimensionality and improve pretraining stability. Extensive experiments show that the proposed image-like dataset construction for ST pretraining consistently improves downstream performance, outperforming conventional pretraining schemes. Ablation studies verify that both spatial patching and channel design are necessary, establishing a unified, practical paradigm for organizing ST data and enabling large-scale pretraining.
Yishun Zhu, Jiaxin Qi, Jian Wang +2
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of the Chinese Academy of Sciences, Zhejiang, China