Organizations: kaiko.ai, Zürich, Switzerland · Biomedical Data Science Center, Lausanne University Hospital, Lausanne, Switzerland · Swiss Institute of Bioinformatics (SIB), Lausanne, Switzerland · ETOP IBCSG Partners Foundation · Vall d’Hebron University Hospital, Barcelona, Spain · The Royal Marsden, Londen, United Kingdom · Department of Oncology, Lausanne University Hospital, Lausanne, Switzerland · Faculty of Biology and Medicine, University of Lausanne, Lausanne, Switzerland · School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
Spatial transcriptomics maps gene expression across tissue sections and, when integrated with whole slide images, jointly captures both morphological architecture and molecular activity to provide a rich, multimodal view of tissue biology. ML approaches combining spatial transcriptomics and WSIs typically learn a mapping from morphological features to molecular profiles via contrastive alignment or supervised fine-tuning of pathology foundation models. In both cases, transcriptomics serves as a training signal rather than a true input modality, leaving inference fundamentally image-driven. We propose an early-fusion vision transformer architecture in which transcript tokens are constructed at arbitrary spatial resolution, merged with image patch tokens, and jointly processed through a shared transformer stack for deep cross-modal interaction. Starting from a pretrained vision-only pathology foundation model, we train HistoTx by continuing self-supervised pretraining on paired image and spatial transcriptomics data and demonstrate that: 1) at inference with image only, performance remains on par with the baseline on most tasks while improving on tasks that inherently benefit from molecular knowledge, such as gene expression prediction; 2) when both modalities are available at inference, jointly providing image and transcript inputs outperforms either modality alone.
Figures & tables
Figure 1: Early-fusion multimodal model architecture and cross-modal loss. A) Input patches contain the H&E image and the corresponding spatial transcriptomics. Both modalities are tokenized into smaller token regions and embedded with modality-specific token encoders. The tokens of both modalities are concatenated and fed through a ViT block. The snow flake emoji indicates the frozen part of the model and the flame the part where model weights are optimized. B) DINO transforms created n local views and 2 global views. By removing transcripts from the local views, the DINO local view loss on the CLS tokens becomes a cross-modality loss.
Benchmark
Metric
Image only
HistoTx
SEAL+CONCH
Supervised
HBench
PC
0.31
0.38
0.35
-
Image Hexpr
PC
-
0.36
-
-
HexprIO
PC
0.26
0.30
0.28
-
BEAT
PC
-
0.66
-
0.82
BEATIO
PC
0.53
0.58
-
-
C16
BBAcc
0.79
0.87
0.89
-
Table 1: Gene expression tasks: Hbench: HEST-Bench, Hexpr: HEST expression prediction data from ( Jaume et al., 2024 ) ; HexprIO: Hest expression task with image only input. BEAT: ETOP BEAT-meso prediction of 100 held-out genes from image+remaining 380 genes. BEATIO: ETOP BEAT-meso prediction of 100 genes from image only. Classification tasks: C16: Camelyon16, BACH, BRACS, BH: BreakHis, GTMA: Gleason TMA, PR: PANDA Radboud; Segmentation tasks: MNS: MoNuSAC, TTS: Tan Tock Seng, CNP: Consep. See Appendix B for more task details. Supervised model as described in ( Martinelli et al., 2026 ) , using vit small patch16 224.augreg in21k as morphology encoder and a trainable/unfrozen MLP as expression encoder. MCAcc: Multiclass accuracy, BBAcc: Binary Balance accuracy, MDS: Monai Dice Score, PC: Mean Pearson correlation coefficient.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
16 tokens
256 tokens
Expression aggregation kernel
56×56
14×14
Expression aggregation stride
56
14
Expression tokens per patch
16 ( 4×4 )
256 ( 16×16 )
GPUs
2
4
Gradient accumulation steps
128
64
Appendix
Table 2: Configuration-specific hyperparameters for the four expression token variants.
Model
BACH
BRACS
BH
CRC
GTMA
MHIST
PC
C16
Panda
CNP
Image only
0.88
0.64
0.82
0.96
0.77
0.77
0.91
0.79
0.63
0.61
16 expr tokens
0.82
0.60
0.73
0.96
0.69
0.74
0.89
0.87
0.64
0.62
256 expr tokens
0.86
0.63
0.77
0.95
0.71
0.75
0.90
0.92
0.63
0.61
Conch + SEAL
0.82
0.61
0.68
0.95
0.73
0.73
0.91
0.89
0.62
0.62
Appendix
Table 3: Benchmark results of all models on all tasks Bach ( Polónia et al., 2019 ; Aresta et al., 2019 ; Araújo et al., 2017 ; Fondón et al., 2018 ) , BRACS ( Brancati et al., 2022 ) , BH: BreakHis ( Spanhol et al., 2016 ) , CRC: CRC-100K ( Kather et al., 2018 ) , GTMA: Gleason TMA ( Arvaniti et al., 2018 ) , MHIST ( Wei et al., 2021 ) , PC: PatchCamelyon ( Veeling et al., 2018 ) , C16: Camelyon16 ( Ehteshami Bejnordi et al., 2017 ) , Panda ( Bulten et al., 2022 ) , CNP: CoNSeP ( Graham et al., 2019 ) , MNS: MoNuSAC ( Verma et al., 2021 ) , Hbench: Hest-bench ( Jaume et al., 2024 ) , Hexpr: Hest expression prediction data from ( Jaume et al., 2024 ) , HexprIO: Hest expression task with image only input, PR: panda radboud ( Bulten et al., 2022 ) , PK: panda karolinska ( Bulten et al., 2022 ) . TTS: tan tock seng ( Oner et al., 2022a ; Oner et al., 2022b ) . Conch+SEAL: ( Hemker et al., 2026 ) . Avg: Average over all tasks, excluding Hexpr and BEAT, but including HexprIO and BEATIO
Model
MNS
HBench
Hexpr
HexprIO
PK
PR
TTS
BEAT
BEATIO
Avg
Image only
0.61
0.31
-
0.26
0.70
0.61
0.31
-
0.53
0.65
16 expr tokens
0.62
0.38
0.36
0.30
0.72
0.67
0.37
0.66
0.58
0.66
256 expr tokens
0.64
0.38
0.37
0.30
0.72
0.67
0.38
0.67
0.59
0.67
Conch + SEAL
0.63
0.35
-
0.28
0.71
0.66
0.37
-
-
-
Appendix
Table 4:
Model
CCRCC
COAD
HCC
IDC
LUNG
LYMPH_IDC
PAAD
PRAD
READ
SKCM
Image only
0.096
0.270
0.083
0.406
0.513
0.298
0.350
0.309
0.171
0.610
16 expr tokens
0.192
0.334
0.082
0.439
0.615
0.370
0.431
0.422
0.245
0.707
256 expr tokens
0.198
0.348
0.091
0.438
0.623
0.362
0.4237
0.399
0.248
0.689
Conch + SEAL
0.182
0.288
0.047
0.425
0.553
0.340
0.385
0.361
0.228
0.652
Appendix
Table 5: HEST-bench benchmark results for all models, shown in the average Pearson Correlation coefficient
Pathology foundation models learn morphological representations through self-supervised pretraining on large-scale whole-slide images, yet they do not explicitly capture the underlying molecular state of the tissue. Spatial transcriptomics technologies bridge this gap by measuring gene expression in situ, offering a natural cross-modal supervisory signal. We propose MINT (Molecularly Informed Training), a fine-tuning framework that incorporates spatial transcriptomics supervision into pretrained pathology Vision Transformers. MINT appends a learnable ST token to the ViT input to encode transcriptomic information separately from the morphological CLS token, preventing catastrophic forgetting through DINO self-distillation and explicit feature anchoring to the frozen pretrained encoder. Gene expression regression at both spot-level (Visium) and patch-level (Xenium) resolutions provides complementary supervision across spatial scales. Trained on 577 publicly available HEST samples, MINT achieves the best overall performance on both HEST-Bench for gene expression prediction (mean Pearson r = 0.440) and EVA for general pathology tasks (0.803), demonstrating that spatial transcriptomics supervision complements morphology-centric self-supervised pretraining.
Spatial transcriptomics (ST) enables gene expression mapping within anatomical context but remains costly and low-throughput. Hematoxylin and eosin (H&E) staining offers rich morphology yet lacks molecular resolution. We present \ours (Spatial Transcriptomics and histOlogy Representation Model), a foundation model trained on 1.2 million spatially resolved transcriptomic profiles with matched histology across 18 organs. Using a hierarchical architecture integrating morphological features, gene expression, and spatial context, STORM bridges imaging and omics through robust molecular--morphological representations. STORM enhances spatial domain discovery, producing biologically coherent tissue maps, and outperforms existing methods in predicting spatial gene expression from H&E images across 11 tumor types. The model is platform-agnostic, performing consistently across Visium, Xenium, Visium HD, and CosMx. Applied to 23 independent cohorts comprising 7,245 patients, STORM significantly improves immunotherapy response prediction and prognostication over established biomarkers, providing a scalable framework for spatially informed discovery and clinical precision medicine.
Jinxi Xiang, Siyu Hou, Yuchen Li +17
Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA · Department of Statistics and Data Science, Yale University, New Haven, CT, USA · Department of Medicine (Oncology), Stanford University School of Medicine, Stanford, CA, USA +4
Spatial transcriptomics provides a molecularly rich description of tissue organization, enabling unsupervised discovery of tissue niches -- spatially coherent regions of distinct cell-type composition and function that are relevant to both biological research and clinical interpretation. However, spatial transcriptomics remains costly and scarce, while H&E histology is abundant but carries a less granular signal. We propose to leverage paired spatial transcriptomics and H&E data to transfer transcriptomics-derived niche structure to a histology-only model via cross-modal distillation. Across multiple tissue types and disease contexts, the distilled model achieves substantially higher agreement with transcriptomics-derived niche structure than unsupervised morphology-based baselines trained on identical image features, and recovers biologically meaningful neighborhood composition as confirmed by cell-type analysis. The resulting framework leverages paired spatial transcriptomic and H&E data during training, and can then be applied to held-out tissue regions using histology alone, without any transcriptomic input at inference time.
Arbel Hizmi, Artemii Bakulin, Shai Bagon +1
Weizmann Institute of Science · Reichman University