cs.AIAug 14, 2026

Disentangled Shared Representations Improve Morpho-Transcriptomic Integration

Authors: Julian Ostermaier, Swann Ruyter, Reuben Dorent, Daniel Racoceanu

Organizations: ESPCI Paris, PSL University, Paris, France · Sorbonne Université, CNRS, Inserm, AP-HP, Inria, Paris Brain Institute – ICM, Paris, France

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Spatial transcriptomics (ST) enables the simultaneous profiling of gene expression and tissue morphology, creating an opportunity to learn multimodal representations capturing shared morpho-transcriptomic structure. However, standard multimodal models often compress modalities into a common latent space without explicitly separating shared and modality-specific sources of variation, which may limit downstream utility. We investigate whether explicit disentanglement of shared and private latent components improves multimodal representation learning for paired Hematoxylin & Eosin (H&E) and ST data. We compare VAE-based and contrastive approaches, each in standard and disentangled variants, across two cancer cohorts under matched experimental conditions. Representations are evaluated using cross-modal reconstruction, downstream probing and cross-modal probe transfer. The experiments suggest two main trends. First, contrastive objectives yield higher downstream probing performance than VAE-based models. Second, disentangled variants improve the selected reconstruction and probing metrics, although the gains depend on the model family, task, direction, and disentanglement strength. Overall, our results suggest that explicitly factorizing shared and modality-specific information can improve multimodal representation learning for spatial transcriptomics and provides a useful evaluation framework for future foundation models.

Figures & tables

Figure 1: Multimodal representation learning pipeline. (A) Pre-processing of the paired gene expression and H&E image patch data. (B) Representations are learned using different model families including contrastive and VAE methods as well as their disentangled versions. Representations are evaluated using (C) cross-modal reconstruction and (D) prediction of downstream labels.
 FamilyModeltop-50 PCC (HE →\to G)cosine similarity (G →\to HE)
   KK 2005001000KK 2005001000
ColonUni- modalPCA47.9 ±\pm 1.551.4 ±\pm 1.052.7 ±\pm 0.859.2 ±\pm 1.060.7 ±\pm 0.560.7 ±\pm 0.8
VAE43.7 ±\pm 0.646.7 ±\pm 0.847.2 ±\pm 1.057.0 ±\pm 0.558.5 ±\pm 0.458.9 ±\pm 0.5
VAEMMVAE42.3 ±\pm 0.448.6 ±\pm 0.249.7 ±\pm 0.560.0 ±\pm 0.359.9 ±\pm 0.361.4 ±\pm 0.2
MMVAE+47.5 ±\pm 0.650.8 ±\pm 0.852.5 ±\pm 0.861.5 ±\pm 0.460.8 ±\pm 0.362.5 ±\pm 0.5
MMVAE+ sg\mathrm{sg}47.6 ±\pm 0.551.8 ±\pm 0.453.2 ±\pm 0.662.3 ±\pm 0.361.5 ±\pm 0.562.7 ±\pm 0.4
Table 1: Cross-modal reconstruction results on Colon and GBM datasets for different gene panel sizes KK . Metrics are reported as percentages and presented as mean ±\pm standard deviation over five repeated runs with different random seeds. For disSSL, 10−310^{-3} , 10−210^{-2} , and 10−110^{-1} denote the value of β\beta . Bold and underline indicate the best and second-best values per column within each model family.
Figure 2: Downstream task performance for tissue niche classification ( HE→GHE\rightarrow G ) and tissue type classification ( G→HEG\rightarrow HE ). AUC prediction performance (five repeated runs with random seeds) with gene panel sizes KK for (A) tissue niche classification derived from scRNA-seq reference Belgian cohort and (B) pathologist tissue types classification. Stars indicate non-disentangled and squares disentangled versions. The gray arrow connects contrastive models from CLIP ( β=0\beta=0 ) to increasing disentanglement strength.
ModelContrastiveGenerativeShared latentPrivate latent
MMVAE [ 16 ]–✓✓–
MMVAE+ [ 13 ]–✓✓✓
MMVAE+ sg\mathrm{sg} [ 11 ]–✓✓✓
CLIP [ 14 ]✓–✓–
disSSL [ 19 ]✓–✓✓
Table 2: Summary of the evaluated representation learning models.
ComparisonMetricpp (BH)Sig.
Colon
——— 
GBM
MMVAE vs MMVAE+sgPCC (HE →\to G) 5000.035∗*
MMVAE vs MMVAE+Cosine (G →\to HE) 2000.015∗*
MMVAE vs MMVAE+Cosine (G →\to HE) 5000.019∗*
Table 3: Statistically significant Wilcoxon signed-rank test results after Benjamini–Hochberg FDR correction ( p<0.05p<0.05 ) across Colon and GBM dataset (HVG, log1p preprocessing). For each slide performance metrics were averaged across repeated runs, and paired slide-level metric values were used as observations in the Wilcoxon signed-rank test. Significance levels refer to the BH-adjusted pp -value: ∗ p<0.05*\ p<0.05 ; ∗∗ p<0.01**\ p<0.01 ; ∗∗∗ p<0.001***\ p<0.001 .
HyperparameterVAE-basedContrastive
Optimizer
Learning rate1×10−41\times 10^{-4}1×10−41\times 10^{-4}
Weight decay1×10−51\times 10^{-5}—
OptimizerAdam (AMSGrad)Adam (AMSGrad)
Training schedule
Batch size64256
Table 4: Training hyperparameters shared across all model configurations.
CohortHEST IDs
Colorectal CancerZEN37, ZEN40, ZEN42, ZEN43, ZEN44, ZEN47, ZEN49
Table 5: Overview of the retained samples used for model training after filtering of low quality samples and enforcing one sample per patient. HEST ID corresponds to the unique identifier of the sample in the HEST database [ 7 ] .
Figure 3: Downstream task performance for tissue niche classification ( HE→GHE\rightarrow G ). AUC prediction performance (mean over five repeated runs with different random seeds) from different gene panel sizes KK for tissue niche classification derived from scRNA-seq reference Korean cohort. Stars indicate the non-disentangled model and circles the disentangled versions. The gray arrow connects contrastive models from CLIP ( β=0\beta=0 ) to increasing disentanglement strength.
More in this topic

Cross-Modal Representation Learning

Cross-modal representation learning connects information across data types such as images, text, audio, biosignals, and sensor measurements.

See all 210 papers

Explore similar work

CardsList
Apr 4, 2026

A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction

Spatial transcriptomics (ST) enables gene expression mapping within anatomical context but remains costly and low-throughput. Hematoxylin and eosin (H&E) staining offers rich morphology yet lacks molecular resolution. We present \ours (Spatial Transcriptomics and histOlogy Representation Model), a foundation model trained on 1.2 million spatially resolved transcriptomic profiles with matched histology across 18 organs. Using a hierarchical architecture integrating morphological features, gene expression, and spatial context, STORM bridges imaging and omics through robust molecular--morphological representations. STORM enhances spatial domain discovery, producing biologically coherent tissue maps, and outperforms existing methods in predicting spatial gene expression from H&E images across 11 tumor types. The model is platform-agnostic, performing consistently across Visium, Xenium, Visium HD, and CosMx. Applied to 23 independent cohorts comprising 7,245 patients, STORM significantly improves immunotherapy response prediction and prognostication over established biomarkers, providing a scalable framework for spatially informed discovery and clinical precision medicine.

Jinxi Xiang, Siyu Hou, Yuchen Li +17

Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA · Department of Statistics and Data Science, Yale University, New Haven, CT, USA · Department of Medicine (Oncology), Stanford University School of Medicine, Stanford, CA, USA +4

Spatial TranscriptomicsGene Expression Prediction from Histopathology ImagesSpatial Gene Expression PredictionClinical Outcome Prediction
AskPDF
Apr 10, 2026

Cross-Modal Knowledge Distillation from Spatial Transcriptomics to Histology

Spatial transcriptomics provides a molecularly rich description of tissue organization, enabling unsupervised discovery of tissue niches -- spatially coherent regions of distinct cell-type composition and function that are relevant to both biological research and clinical interpretation. However, spatial transcriptomics remains costly and scarce, while H&E histology is abundant but carries a less granular signal. We propose to leverage paired spatial transcriptomics and H&E data to transfer transcriptomics-derived niche structure to a histology-only model via cross-modal distillation. Across multiple tissue types and disease contexts, the distilled model achieves substantially higher agreement with transcriptomics-derived niche structure than unsupervised morphology-based baselines trained on identical image features, and recovers biologically meaningful neighborhood composition as confirmed by cell-type analysis. The resulting framework leverages paired spatial transcriptomic and H&E data during training, and can then be applied to held-out tissue regions using histology alone, without any transcriptomic input at inference time.

Arbel Hizmi, Artemii Bakulin, Shai Bagon +1

Weizmann Institute of Science · Reichman University

Computational PathologyCross-Modal Knowledge DistillationSpatial Transcriptomics
AskPDF
Jul 15, 2026

LATTICE: Graph Self-Supervised Learning for Multimodal Spatial Omics Integration

Spatially resolved omics studies increasingly combine transcriptomic and epigenomic assays, yet downstream analysis is often still performed using single-modality pipelines. We present LATTICE (Latent Alignment of Tissue-level and Transcriptomic Information for Cross-modal Embedding), a graph-based self-supervised framework that learns spot-level representations from harmonized multimodal features. LATTICE integrates five aligned modality blocks per Visium spot: Visium RNA, scMultiome RNA, scMultiome ATAC, spatial ATAC, and spatial CUT&Tag. These modalities capture spatial transcriptomic measurements, single-cell inferred regulatory activity, and in situ chromatin and histone states within a unified lattice representation. LATTICE constructs a spatial neighborhood graph and trains a TransformerConv encoder using masked reconstruction, cross-modal alignment, and spatial smoothness objectives. On a private 11-sample melanoma cohort from an anonymized clinical collaborator comprising 54{,}912 total spots, LATTICE demonstrated stable optimization behavior, reproducible embeddings across analysis seeds, and complete multimodal integration across all samples. Adding scMultiome RNA to Visium RNA alone substantially improved concordance with Space Ranger clusters across 11 runs (adjusted Rand index [ARI] +0.157, normalized mutual information [NMI] +0.143, and spatial contiguity +0.174). Additional modalities further improved spatial contiguity and multimodal utility score (MUS), although they sometimes reduced agreement with RNA-derived reference labels, likely because the learned embeddings captured chromatin and regulatory structure beyond transcriptomic similarity alone. These results position LATTICE as a practical and empirically grounded framework for multimodal spatial omics integration, while also highlighting the need for stronger supervision and broader external benchmarking.

Jagan Mohan Reddy Dwarampudi, Veena Kochat, Suresh Satpati +2

University of Houston · MD Anderson Cancer Center

Spatial TranscriptomicsGraph SSLCross-Modal AlignmentMultimodal EmbeddingGenomicsMulti-Omics Data Integration
AskPDF