Cross-Modal Knowledge Distillation from Spatial Transcriptomics to Histology

Arbel Hizmi; Artemii Bakulin; Shai Bagon; Nir Yosef

cs.CVApr 10, 2026

Cross-Modal Knowledge Distillation from Spatial Transcriptomics to Histology

Authors: Arbel Hizmi, Artemii Bakulin, Shai Bagon, Nir Yosef

Organizations: Weizmann Institute of Science · Reichman University

Abstract

Spatial transcriptomics provides a molecularly rich description of tissue organization, enabling unsupervised discovery of tissue niches -- spatially coherent regions of distinct cell-type composition and function that are relevant to both biological research and clinical interpretation. However, spatial transcriptomics remains costly and scarce, while H&E histology is abundant but carries a less granular signal. We propose to leverage paired spatial transcriptomics and H&E data to transfer transcriptomics-derived niche structure to a histology-only model via cross-modal distillation. Across multiple tissue types and disease contexts, the distilled model achieves substantially higher agreement with transcriptomics-derived niche structure than unsupervised morphology-based baselines trained on identical image features, and recovers biologically meaningful neighborhood composition as confirmed by cell-type analysis. The resulting framework leverages paired spatial transcriptomic and H&E data during training, and can then be applied to held-out tissue regions using histology alone, without any transcriptomic input at inference time.

Figures & tables

	$K=10$		$K=20$
Method	ARI	NMI	ARI	NMI
Histology NOLAN	$0.283\pm 0.12$	$0.383\pm 0.10$	$0.234\pm 0.08$	$0.403\pm 0.09$
Histology Leiden	$0.191\pm 0.12$	$0.288\pm 0.13$	$0.176\pm 0.08$	$0.327\pm 0.11$
Ours	$0.615\pm 0.11$	$0.603\pm 0.10$	$0.500\pm 0.10$	$0.579\pm 0.08$

Table 1: Agreement with the spatial-transcriptomics teacher niche assignments. We report mean

\pm

standard deviation of ARI and NMI, computed between each histology-based method and the teacher across all datasets, for

K\in\{10,20\}

niche categories. “Ours” denotes the distilled histology student. Higher values indicate stronger agreement with the teacher-defined niche partition; best results are in bold .

Figure 2: Representative H&E slide of ovarian cancer. Top: Input image with hierarchical views showing a cellular neighborhood from the tissue and a single-cell crop (the model’s basic unit). Bottom: Each tissue is divided into four horizontal continuous strips, where the second-from-top strip is the held-out test data, and the remainder for training. Red bands indicate an exclusion buffer at the test region boundaries. All cells whose centers fall within buffers are discarded and are not used for either training or testing. The buffer size is set to 112 pixels (30.6 $\mu$ m), which is half of our experimental cell-center crop size of $224\times 224$ pixels. This design assures that train and test crops are strictly spatially disjoint.

Figure 3: Niche segmentation of healthy human colon into 20 distinct niches: In our supervised distillation framework, the image-based student model (ours) learns to replicate the teacher’s expression-derived segmentation using histology features alone. Each color corresponds to a distinct niche. For example, in the teacher and student results, purple patches mark B-cell follicles, warm tones (orange, pink) delineate stromal compartments, and layered greens and blues along the tissue edge reflect epithelial zonation. The student closely reproduces the teacher’s spatial organization, preserving both large-scale tissue compartments and fine-grained zonal boundaries. While B-cell follicles are recovered by all methods, subtler structures such as epithelial zonation and stromal layering are resolved only by the teacher and student, highlighting the benefit of transcriptomic supervision. Histology-NOLAN baseline partially recovers coarse compartments but exhibits substantial spatial noise, while Histology-Leiden baseline produces large, spatially uniform blocks that fail to capture the tissue’s internal organization.

$K=10$
Dataset	Ours	Histology NOLAN	Histology Leiden
Human ovary	0.0052	0.0557	0.0750
Human pancreas	0.0042	0.0626	0.1572
Human breast	0.0024	0.0323	0.0781
$K=20$
Human ovary	0.0101	0.0958	0.0851

Table 2: Jensen–Shannon divergence (JSD) between the per-niche cell-type compositions of the teacher and each method

K\in\{10,20\}

. We compute per-niche cell-type distributions from the published 10x cell-type annotations (evaluation only), align niches to the teacher via one-to-one matching that minimizes JSD, and report the teacher-size-weighted mean JSD. Lower is better; best results are in bold .

	Macro-F1
Method	$K=10$	$K=20$
Histology NOLAN	0.297	0.361
Ours	0.543	0.401
Teacher	0.489	0.388

Table 3: Pathology-label probing on the Xenium human ovarian cancer dataset using partial pathology annotations (restricted to Tumor 1, Tumor 2, and Tumor 3; unlabeled cells excluded). We train an SVM to predict pathology labels from method-specific niche assignments, evaluated separately for

K=10

and

K=20

niches. Macro-F1 is reported. Best result in each

K

setting is shown in bold .

Figure 4: Niche segmentation in human cervix cancer into 20 distinct niches : Our model (student) closely replicates the teacher’s expression-derived segmentation from histology alone, with both models recovering fine-grained tissue structure including tumor epithelial regions , stromal infiltration , and the precise boundaries of invasive carcinoma nests . Histology-NOLAN and Histology-Leiden produce substantially coarser segmentation, failing to resolve the spatial heterogeneity captured by Teacher and Student.

Computational Pathology

Computational pathology uses algorithms to analyze digitized tissue images for diagnosis, prognosis, biomarker discovery, and clinical decision support.

See all 279 papers

A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction
Apr 4, 2026Jinxi Xiang, Siyu Hou, Yuchen Li +17Spatial Transcriptomics Gene Expression Prediction from Histopathology Images
Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA · Department of Statistics and Data Science, Yale University, New Haven, CT, USA · Department of Medicine (Oncology), Stanford University School of Medicine, Stanford, CA, USA +4
Path2ST: Hierarchical Cell-Tissue Grounded Cross-Modal Translation for Spatial Transcriptomics
Aug 11, 2026Ruochen Liu, Wei LouSpatial Gene Expression Prediction Gene Expression Prediction from Histopathology Images
Faculty of Science and Engineering University of Liverpool Liverpool, United Kingdom · College of Mathematical Medicine Zhejiang Normal University Jinhua, China
HiST: A Hierarchical Sparse Transformer for Cross-Modal Spatial Transcriptomics Modeling
Jun 12, 2026Weiyi Wu, Xinwen Xu, Xingjian Diao +4Computational Pathology Spatial Transcriptomics
Dartmouth College · Mass General Hospital · New Jersey Institute of Technology +1

Apr 4, 2026J/K move · Enter open · S save

A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction

Jinxi Xiang, Siyu Hou, Yuchen Li, Ryan Quinton+16

Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA · Department of Statistics and Data Science, Yale University, New Haven, CT, USA · Department of Medicine (Oncology), Stanford University School of Medicine, Stanford, CA, USA+4

Spatial Transcriptomics Gene Expression Prediction from Histopathology Images Spatial Gene Expression Prediction Clinical Outcome Prediction

Open paper AskPDF arXiv

Spatial transcriptomics (ST) enables gene expression mapping within anatomical context but remains costly and low-throughput. Hematoxylin and eosin (H&E) staining offers rich morphology yet lacks molecular resolution. We present \ours (Spatial Transcriptomics and histOlogy Representation Model), a foundation model trained on 1.2 million spatially resolved transcriptomic profiles with matched histology across 18 organs. Using a hierarchical architecture integrating morphological features, gene expression, and spatial context, STORM bridges imaging and omics through robust molecular--morphological representations. STORM enhances spatial domain discovery, producing biologically coherent tissue maps, and outperforms existing methods in predicting spatial gene expression from H&E images across 11 tumor types. The model is platform-agnostic, performing consistently across Visium, Xenium, Visium HD, and CosMx. Applied to 23 independent cohorts comprising 7,245 patients, STORM significantly improves immunotherapy response prediction and prognostication over established biomarkers, providing a scalable framework for spatially informed discovery and clinical precision medicine.

Abstract

Computational Pathology

Explore similar work

A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction

Path2ST: Hierarchical Cell-Tissue Grounded Cross-Modal Translation for Spatial Transcriptomics

HiST: A Hierarchical Sparse Transformer for Cross-Modal Spatial Transcriptomics Modeling