HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning

Alice Driessen; Emil Malmsten; Steffen Wolf; Bernd Illing; Isinsu Katircioglu; Adriano Martinelli; Rolf Stahel; Solange Peters

cs.LGNov 2, 2026

HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning

Authors: Alice Driessen, Emil Malmsten, Steffen Wolf, Bernd Illing, Isinsu Katircioglu, Adriano Martinelli, Rolf Stahel, Solange Peters,

Organizations: kaiko.ai, Zürich, Switzerland · Biomedical Data Science Center, Lausanne University Hospital, Lausanne, Switzerland · Swiss Institute of Bioinformatics (SIB), Lausanne, Switzerland · ETOP IBCSG Partners Foundation ·

Abstract

Spatial transcriptomics maps gene expression across tissue sections and, when integrated with whole slide images, jointly captures both morphological architecture and molecular activity to provide a rich, multimodal view of tissue biology. ML approaches combining spatial transcriptomics and WSIs typically learn a mapping from morphological features to molecular profiles via contrastive alignment or supervised fine-tuning of pathology foundation models. In both cases, transcriptomics serves as a training signal rather than a true input modality, leaving inference fundamentally image-driven. We propose an early-fusion vision transformer architecture in which transcript tokens are constructed at arbitrary spatial resolution, merged with image patch tokens, and jointly processed through a shared transformer stack for deep cross-modal interaction. Starting from a pretrained vision-only pathology foundation model, we train HistoTx by continuing self-supervised pretraining on paired image and spatial transcriptomics data and demonstrate that: 1) at inference with image only, performance remains on par with the baseline on most tasks while improving on tasks that inherently benefit from molecular knowledge, such as gene expression prediction; 2) when both modalities are available at inference, jointly providing image and transcript inputs outperforms either modality alone.

Figures & tables

Figure 1: Early-fusion multimodal model architecture and cross-modal loss. A) Input patches contain the H&E image and the corresponding spatial transcriptomics. Both modalities are tokenized into smaller token regions and embedded with modality-specific token encoders. The tokens of both modalities are concatenated and fed through a ViT block. The snow flake emoji indicates the frozen part of the model and the flame the part where model weights are optimized. B) DINO transforms created $n$ local views and 2 global views. By removing transcripts from the local views, the DINO local view loss on the CLS tokens becomes a cross-modality loss.

Benchmark	Metric	Image only	HistoTx	SEAL+CONCH	Supervised
HBench	PC	0.31	0.38	0.35	-
Image Hexpr	PC	-	0.36	-	-
HexprIO	PC	0.26	0.30	0.28	-
BEAT	PC	-	0.66	-	0.82
BEATIO	PC	0.53	0.58	-	-
C16	BBAcc	0.79	0.87	0.89	-

Table 1: Gene expression tasks: Hbench: HEST-Bench, Hexpr: HEST expression prediction data from ( Jaume et al., 2024 ) ; HexprIO: Hest expression task with image only input. BEAT: ETOP BEAT-meso prediction of 100 held-out genes from image+remaining 380 genes. BEATIO: ETOP BEAT-meso prediction of 100 genes from image only. Classification tasks: C16: Camelyon16, BACH, BRACS, BH: BreakHis, GTMA: Gleason TMA, PR: PANDA Radboud; Segmentation tasks: MNS: MoNuSAC, TTS: Tan Tock Seng, CNP: Consep. See Appendix B for more task details. Supervised model as described in ( Martinelli et al., 2026 ) , using vit small patch16 224.augreg in21k as morphology encoder and a trainable/unfrozen MLP as expression encoder. MCAcc: Multiclass accuracy, BBAcc: Binary Balance accuracy, MDS: Monai Dice Score, PC: Mean Pearson correlation coefficient.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	16 tokens	256 tokens
Expression aggregation kernel	$56\times 56$	$14\times 14$
Expression aggregation stride	56	14
Expression tokens per patch	16 ( $4\times 4$ )	256 ( $16\times 16$ )
GPUs	2	4
Gradient accumulation steps	128	64

Appendix

Table 2: Configuration-specific hyperparameters for the four expression token variants.

Model	BACH	BRACS	BH	CRC	GTMA	MHIST	PC	C16	Panda	CNP
Image only	0.88	0.64	0.82	0.96	0.77	0.77	0.91	0.79	0.63	0.61
16 expr tokens	0.82	0.60	0.73	0.96	0.69	0.74	0.89	0.87	0.64	0.62
256 expr tokens	0.86	0.63	0.77	0.95	0.71	0.75	0.90	0.92	0.63	0.61
Conch + SEAL	0.82	0.61	0.68	0.95	0.73	0.73	0.91	0.89	0.62	0.62

Appendix

Table 3: Benchmark results of all models on all tasks Bach ( Polónia et al., 2019 ; Aresta et al., 2019 ; Araújo et al., 2017 ; Fondón et al., 2018 ) , BRACS ( Brancati et al., 2022 ) , BH: BreakHis ( Spanhol et al., 2016 ) , CRC: CRC-100K ( Kather et al., 2018 ) , GTMA: Gleason TMA ( Arvaniti et al., 2018 ) , MHIST ( Wei et al., 2021 ) , PC: PatchCamelyon ( Veeling et al., 2018 ) , C16: Camelyon16 ( Ehteshami Bejnordi et al., 2017 ) , Panda ( Bulten et al., 2022 ) , CNP: CoNSeP ( Graham et al., 2019 ) , MNS: MoNuSAC ( Verma et al., 2021 ) , Hbench: Hest-bench ( Jaume et al., 2024 ) , Hexpr: Hest expression prediction data from ( Jaume et al., 2024 ) , HexprIO: Hest expression task with image only input, PR: panda radboud ( Bulten et al., 2022 ) , PK: panda karolinska ( Bulten et al., 2022 ) . TTS: tan tock seng ( Oner et al., 2022a ; Oner et al., 2022b ) . Conch+SEAL: ( Hemker et al., 2026 ) . Avg: Average over all tasks, excluding Hexpr and BEAT, but including HexprIO and BEATIO

Model	MNS	HBench	Hexpr	HexprIO	PK	PR	TTS	BEAT	BEATIO	Avg
Image only	0.61	0.31	-	0.26	0.70	0.61	0.31	-	0.53	0.65
16 expr tokens	0.62	0.38	0.36	0.30	0.72	0.67	0.37	0.66	0.58	0.66
256 expr tokens	0.64	0.38	0.37	0.30	0.72	0.67	0.38	0.67	0.59	0.67
Conch + SEAL	0.63	0.35	-	0.28	0.71	0.66	0.37	-	-	-

Appendix

Table 4:

Model	CCRCC	COAD	HCC	IDC	LUNG	LYMPH_IDC	PAAD	PRAD	READ	SKCM
Image only	0.096	0.270	0.083	0.406	0.513	0.298	0.350	0.309	0.171	0.610
16 expr tokens	0.192	0.334	0.082	0.439	0.615	0.370	0.431	0.422	0.245	0.707
256 expr tokens	0.198	0.348	0.091	0.438	0.623	0.362	0.4237	0.399	0.248	0.689
Conch + SEAL	0.182	0.288	0.047	0.425	0.553	0.340	0.385	0.361	0.228	0.652

Appendix

Table 5: HEST-bench benchmark results for all models, shown in the average Pearson Correlation coefficient

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 800 papers

MINT: Molecularly Informed Training with Spatial Transcriptomics Supervision for Pathology Foundation Models
Mar 9, 2026Minsoo Lee, Jonghyun Kim, Juseung Yun +2Pathology Foundation Models Gene Expression Prediction from Histopathology Images
LG AI Research
A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction
Apr 4, 2026Jinxi Xiang, Siyu Hou, Yuchen Li +17Spatial Transcriptomics Gene Expression Prediction from Histopathology Images
Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA · Department of Statistics and Data Science, Yale University, New Haven, CT, USA · Department of Medicine (Oncology), Stanford University School of Medicine, Stanford, CA, USA +4
Cross-Modal Knowledge Distillation from Spatial Transcriptomics to Histology
Apr 10, 2026Arbel Hizmi, Artemii Bakulin, Shai Bagon +1Computational Pathology Cross-Modal Knowledge Distillation
Weizmann Institute of Science · Reichman University

Mar 9, 2026J/K move · Enter open · S save

MINT: Molecularly Informed Training with Spatial Transcriptomics Supervision for Pathology Foundation Models

Minsoo Lee, Jonghyun Kim, Juseung Yun, Sunwoo Yu+1

LG AI Research

Pathology Foundation Models Gene Expression Prediction from Histopathology Images Vision Foundation Model Adaptation Spatial Gene Expression Prediction

Open paper AskPDF arXiv

Pathology foundation models learn morphological representations through self-supervised pretraining on large-scale whole-slide images, yet they do not explicitly capture the underlying molecular state of the tissue. Spatial transcriptomics technologies bridge this gap by measuring gene expression in situ, offering a natural cross-modal supervisory signal. We propose MINT (Molecularly Informed Training), a fine-tuning framework that incorporates spatial transcriptomics supervision into pretrained pathology Vision Transformers. MINT appends a learnable ST token to the ViT input to encode transcriptomic information separately from the morphological CLS token, preventing catastrophic forgetting through DINO self-distillation and explicit feature anchoring to the frozen pretrained encoder. Gene expression regression at both spot-level (Visium) and patch-level (Xenium) resolutions provides complementary supervision across spatial scales. Trained on 577 publicly available HEST samples, MINT achieves the best overall performance on both HEST-Bench for gene expression prediction (mean Pearson r = 0.440) and EVA for general pathology tasks (0.803), demonstrating that spatial transcriptomics supervision complements morphology-centric self-supervised pretraining.

Abstract

Multimodal Fusion

Explore similar work

MINT: Molecularly Informed Training with Spatial Transcriptomics Supervision for Pathology Foundation Models

A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction

Cross-Modal Knowledge Distillation from Spatial Transcriptomics to Histology