Cross-Modal Contrastive Learning for the Retrieval of Immunotherapy-Associated Molecular Signatures from Histopathology

Sigrid Vila-Bagaria; Mar Teixidó; Miquel Piñol; Felip Vilardell; Robert Montal; Veronica Vilaplana

cs.LGOct 4, 2026

Cross-Modal Contrastive Learning for the Retrieval of Immunotherapy-Associated Molecular Signatures from Histopathology

Authors: Sigrid Vila-Bagaria, Mar Teixidó, Miquel Piñol, Felip Vilardell, Robert Montal, Veronica Vilaplana

Organizations: Signal Theory and Communications Department (TSC) Universitat Politècnica de Catalunya - BarcelonaTech (UPC) · Cancer Biomarkers Research Group (GReBiC) - IRB Lleida

Abstract

Gastric Adenocarcinoma is a leading cause of cancer mortality. Although "Inflamed/Non-Inflamed" subtypes have been proposed to predict immunotherapy response, their identification relies on a costly 10-gene RNA signature. We propose a Cross-modal Contrastive Multiple Instance Learning (CCMIL) framework for cross-modal retrieval, imputing these molecular signatures directly from standard Hematoxylin & Eosin (H&E) slides. By leveraging a supervised contrastive objective, CCMIL aligns visual morphological patterns with molecular phenotypes into a shared latent space. This establishes an interpretable search-by-case retrieval engine, enabling pathologists to query a whole slide image to surface transcriptomically coherent neighbors and approximate RNA signatures without genomic sequencing at inference. Our results demonstrate that this retrieval-first approach captures the continuous phenotypic spectrum of tumor inflammation and yields clinically interpretable attention heatmaps. Furthermore, the learned representation also supports competitive downstream classification, providing a practical molecular pre-screening strategy.

Figures & tables

Panel of Figure 1 — Figure 1 : The CCMIL framework. (a) Dual-stream architecture aligning WSI and RNA features during training to enable inference from H&E alone. (b) The composite objective aligns modalities and separates “Inflamed/Non-Inflamed” cases in the joint latent space.

Retrieval
		Retrieval (Image $\to$ RNA)			k-NN Classification
Model	Q $\rightarrow$ Tgt	R@1	R@5	MRR	Acc.	F1
Baseline	Te $\rightarrow$ Tr	–	–	.71 $\pm$ .106	.57 $\pm$ .129	.53 $\pm$ .109
Baseline	Te $\rightarrow$ Te	.67 $\pm$ .115	.89 $\pm$ .086	.78 $\pm$ .067	.67 $\pm$ .107	.65 $\pm$ .074
CCMIL	Te $\rightarrow$ Tr	–	–	.75 $\pm$ 0.049	.68 $\pm$ .080	.61 $\pm$ .138
CCMIL	Te $\rightarrow$ Te	.72 $\pm$ .085	.92 $\pm$ .051	.80 $\pm$ .067	.71 $\pm$ .155	.67 $\pm$ .133

Table 1 : Quantitative performance comparison of the proposed framework against state-of-the-art baselines. All values are reported as mean

\pm

standard deviation across 5-fold cross-validation.

Components			Classification			Retrieval
THCA	SupConL	ClassL	Acc.	AUC	F1	R@1	R@5
✓	✓	✗	.54 $\pm$ .092	.45 $\pm$ .156	.66 $\pm$ .069	.69 $\pm$ .068	.88 $\pm$ .068
✓	✗	✓	.68 $\pm$ .103	.68 $\pm$ .073	.72 $\pm$ .076	.63 $\pm$ .119	.84 $\pm$ .068
✗	✓	✓	.72 $\pm$ .106	.74 $\pm$ .084	.74 $\pm$ .080	.60 $\pm$ .069	.84 $\pm$ .107
✓	✓	✓	.72 $\pm$ .157	.75 $\pm$ .135	.76 $\pm$ .112	.72 $\pm$ .085	.92 $\pm$ .051

Table 2 : Ablation study of the framework evaluating the Tumour Histology Conditioned Attention (THCA) and loss combinations for Classification and Retrieval.

Panel of Figure 2 — Figure 2 : Evaluation of the proposed framework. (a) Latent space visualization via UMAP demonstrating a structured joint representation. (b) Mean RNA Cosine Similarity between the query ground truth and retrieved cases across retrieval rank $k$ . CSLS + Reciprocal strategy remains closest to the Oracle (retrieval in the RNA space) upper bound.

Panel of Figure 3 — Figure 3 : Qualitative evaluation of CCMIL. (a) CCMIL generates dense, highly localized attention hotspots compared to the baseline models. (b) The joint latent space captures a continuous inflammatory gradient, successfully retrieving transcriptomically coherent neighbors.

Explore similar work

Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction
Jun 29, 2026Dominik Winter, Dominik Vonficht, Loïc Le Bescond +6Digital Pathology Biomarker
AstraZeneca Computational Pathology and Biomarkers, Munich, Germany · AstraZeneca, Early Oncology and Translation Medicine, Cambridge, UK
Zero-Cost Virtual RNA: Approximating Immunotherapy Signatures via Cross-Modal WSI Retrieval
Aug 1, 2026Sigrid Vila-Bagaria, Mar Teixidó, Miquel Piñol +3Transcriptomics Biomarker
Image Processing Group, Universitat Politècnica de Catalunya, Spain · Research group of Cancer Biomarkers & Oncological Pathology, IRB Lleida, Spain
Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining
May 12, 2026Yucheng Xing, Pei Liu, Jingying Ma +6Computational Pathology Pathology Foundation Models
National University of Singapore, Singapore · Hunan University, China · Peking Union Medical College Hospital (PUMCH), China +1

Jun 29, 2026, eess.IVJ/K move · Enter open · S save

Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction

Dominik Winter, Dominik Vonficht, Loïc Le Bescond, Christian Gebbe+5

AstraZeneca Computational Pathology and Biomarkers, Munich, Germany · AstraZeneca, Early Oncology and Translation Medicine, Cambridge, UK

Digital Pathology Biomarker Multimodal Alignment

Open paper AskPDF arXiv

H&E-stained whole-slide images offer cohort-scale availability and rich spatial context but lack molecular specificity, whereas bulk RNA-seq provides transcriptome-wide resolution at high cost with limited archival availability. We show that training a lightweight alignment module atop frozen histopathology and RNA-Seq foundation models enables open-vocabulary molecular prompting -- querying H&E slides with gene-set signatures to predict pathway activity without sequencing or end-to-end retraining. Using contrastive learning on a multi-cancer cohort (N=1,720), we achieve a 25-fold improvement in retrieval over baseline methods. Systematic analysis reveals a graduated predictability spectrum: morphologically grounded programs (cell-cycle programs, immune-related) are most reliably predicted (R^2>0.5), while predicting pathways with no morphological footprint remains challenging as expected. We validate clinical utility on the POSEIDON clinical trial: H&E-predicted squamous cell carcinoma scores recapitulate NSCLC subtype identity and predicted IFN-gamma mirror PD-L1 tumor-cell expression groups. Furthermore, genesets describing immune activation and fibrosis predict known tumor microenvironment archetypes from histology alone. We further validate generalization of our approach across unseen cohorts and demonstrate data-efficient domain adaptation, establishing a slide-native framework for molecular analysis on H&E images.

Abstract

Explore similar work

Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction

Zero-Cost Virtual RNA: Approximating Immunotherapy Signatures via Cross-Modal WSI Retrieval

Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining