Concept-based explanation of gene expression prediction from H&E images
Authors: Amos Muench, Jonathan Thielmann, Reduan Achtibat, Maximilian Dreyer, Philip Bischoff, Caroline Forsythe, Hamidreza Parand, Thomas Walter, +4 more
Organizations: Institute of Pathology – Charité Universitätsmedizin, Berlin, Germany · Fraunhofer Heinrich-Hertz-Institute, Berlin, Germany · Center for Computational Biology (CBIO), Mines Paris, PSL University, Paris, France · Institut Curie, PSL University, Paris, France · German Cancer Consortium (DKTK), Partner Site Berlin, and German Cancer Research Center (DKFZ), Heidelberg, Germany · Technological University Dublin, Dublin, Ireland · Technische Universität Berlin, Berlin, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
Recent advances in pathology foundation models have enabled accurate prediction of spatial transcriptomics (ST) from routine H&E images. However, existing explainability methods for vision transformer (ViT)-based models are largely limited to local heatmaps and do not reveal how morphological concepts contribute to ST predictions. Here, we introduce an explainable framework that combines relevance propagation and concept discovery to link transcriptional programs to tissue morphology. We developed a ViT-based framework for virtual ST from H&E images that combines ViT-aware layer-wise relevance propagation with relaxed archetypal TopK sparse autoencoder-based concept discovery. This approach provides both local explanations and global insights into the morphological patterns associated with transcriptional programs. We applied the framework to colorectal cancer ST data from the HEST-1k cohort and evaluated its generalizability in TCGA COAD. Our architecture accurately predicts clinically relevant ST signatures and accompanying molecular phenotypes. Measured and predicted gene expression profiles reveal substantial spatial heterogeneity of the colorectal cancer subtypes iCMS2 and iCMS3 across a large number of samples. Spatially resolved and aggregated iCMS classification achieve weighted F1 scores of 0.872 and 0.819 (0.770 in TCGA COAD), respectively, and both stratify patient outcome. Beyond prediction, our framework establishes a relevance-based concept atlas linking molecular phenotypes to histopathological representations. Comparison of activation- with relevance-derived concepts demonstrates that relevances provide a more direct link between tissue morphology and downstream predictions. We establish a general strategy for concept-based explanation of spatial prediction, and our framework is readily applicable to a broad range of ViT-based pathology models.
Figures & tables
Figure 1: A scRNA-seq analysis by [ 25 ] revealed three states within epithelial cells in CRC, comprising malignant iCMS2, iCMS3, and normal cells, which are characterized by a corresponding set of GE signatures. To determine immune and stromal compartments, we retrieved marker genes from [ 20 ] . B . Based on the malignant epithelial signatures, multiple classifiers for bulk RNA-seq have been developed [ 44 ] , allowing for stratifying larger cohorts. To extend this classification to spot-level ST data, we developed a hierarchical argmax classifier based on GE signature. iCMS2 has a more favourable outcome compared to iCMS3. C We reanalyzed ST data from [ 24 ] , both on a spot-level and on aggregated measurements using the kNN nearest classifier, detecting highly prevalent intra-sample heterogeneity between both iCMS2 and iCMS3 on the spot-level, while the bulk-level mostly reflected the major population.
Figure 2: A For prediction, we split the 10X Visium HEST-1k COAD [ 24 ] dataset into train, validation, and test sets. We developed a framework for GE prediction from H&E tiles based on a fine-tuned UNI-2h [ 8 ] encoder, followed by patch attention [ 42 ] . Per-gene MLP heads predict stroma, immune, and iCMS signatures, preserving intra-signature variance. B To explain the predictions, we apply LRP per signature [ 5 ] . As ViTs are encoding entangled concepts, we employ an RA-SAE [ 15 ] to disentangle relevances into interpretable concepts. Next, we conduct Leiden clustering on the PCA-transformed relevances to derive monosemantic prototypes as global explanations. For these prototypes, we localize concepts in the input images.
Figure 3: Accurate prediction of GE and cell type identity is correlated with properties of spatial GE . A Predictability is correlated with properties of ST data. To determine predictability, Pearson correlation between predicted and measured GE was computed. This predictability is positively correlated with spatial variability, the occurrence of non-zero GE, and GE variance. B Our framework accurately predicts clinically relevant signatures, achieving median Pearson Correlations between measurements between 0.46 and 0.68 across different signatures, after filtering for genes with a PCC ≥0.3 . C Examplary H&E image with pathologist annotation. D Comparing tissue identity revealed moderate weighted F1 scores for all cell types and high performance for the epithelial subset. E Bulk-level classification of aggregated spatial predictions was highly congruent with aggregated measurements. F Spatial overlap for hierarchical argmax classification into tissue classes. G Predictions and measurements for representative genes showed relatively high spatial congruence.
Figure 4: Virtual ST on TCGA reveals iCMS heterogeneity and outcome stratification . A For external validation, we retrieved H&E, bulk RNA-seq, and clinical data of the TCGA COAD cohort ( n=440 , [ 33 ] ) and applied our H&E prediction framework and subsequent classification. B Comparing predicted with measured bulk-level iCMS classification demonstrated relatively high weighted F1 scores. Predicted spot-level classification aligned well with H&E-detectable tissue classes. C Our methods allows to deconvolute iCMS identity on a tissue basis, revealing iCMS heterogeneity that can partially reclassify bulk-level iCMS signals. Bulk-level displays aggregated predictions classified with kNN nearest mode. D Both measured and predicted bulk-level iCMS status can stratify overall survival. Interestingly, the proportion of iCMS3 spots was able to separate outcome as well, but the proportion of immune compartments could not stratify outcome. Stars indicate significance: *** - p-value ≤0.001 , ** - p-value ≤0.01 , * - p-value ≤0.05 .
Figure 5
Figure 6: TCGA COAD prototype detection on RA-SAE relevances. A UMAP embedding of relevances of all signatures of the virtual ST TCGA COAD dataset at the hidden dimension of the RA-SAE. Prototype tiles are highlighted and a density map of the same embedding highlights high coverage by multiple patients. B Prototypes per signature. C iCMS2 and D iCMS3 prototypes and concepts, and the fraction of patients contributing to the prototype are displayed.
Figure 7: Activation-based concepts in TCGA-COAD. A UMAP embedding of activations of all signatures of the virtual ST TCGA COAD dataset at the hidden dimension of the RA-SAE. Prototype tiles are highlighted and a density map of the same embedding highlights high coverage by multiple patients. B Prototypes per signature. C iCMS2 and D iCMS3 prototypes and concepts, and the fraction of patients contributing to the prototype are displayed.
Figure 8: Difference between RA-SAE concept usage by relevances and activations . A UMAP on dictionary, concept overlap between topk with k=100 selected concepts. Colored by normalized importance rank for Z and R. Both are highly correlated regarding their importance rank (Spearman = 0.942) and overlap moderately (Jaccard index for the top-100 = 0.587). B Concept usage breadth for R and Z by signature. C Joint and disjoint concepts between relevance and activations. We selected from the top-100 concepts for each tissue class concepts with maximal importance and visualized them on tiles selected by max Z. Thereby, we investigate #tissue classes × {joint, Z only, R only} different concepts.
Structure within Z and R codes
Concept usage
Structure within Z and R activated Concepts
Conn. ↑
Neg. Int. ↓
Imp. ℓ1
Imp. max
Breadth
nact
Intra ∣cosD∣↑
Max ∣cos∣↑
St. Rank ↓
Eff. Rank ↓
Z
0.745
1.10×106
619.0
4.331
0.405
3241
0.091
0.576
11.83
63.87
R
0.745
2.60×105
310.1
4.567
0.193
1545
0.090
0.655
13.40
62.04
Table 1: Activation ( Z ) vs. relevance ( R ) on the RA-SAE dictionary of all TCGA COAD tiles ( N=80176 , H=8000 , top- K=100 ), summing relevances of all outputs. Agreement: top- K Jaccard 0.587 , Spearman importance 0.942 Pearson importance 0.887 .
Table S2 : RA-SAE hyperparameter grid search on UNI-2h patch attention-pooled embeddings vs. TopK baseline. All combinations of K∈{16,32} , δ∈{1,25,35} , nC∈{8,16,32}×103 . Across all runs, best and second best per metric are marked; the best R 2 among RA-SAE runs is additionally marked by a framed row. One diverged run ( † ) is excluded from ranking. The hidden dimension was fixed at d=8000 . Z - SAE activations. OOD - out-of-distribution score. Stability (Hungarian aligned average cosine similarity) and maximum cosine similarity were computed for dictionaries of four experiments.
Figure 11
Supplemental Figure S2 : Entangled concepts in ViT encoder output. Example from iCMS2 upregulated signature.
Supplemental Figure S3 : HEST COAD ST data: Relevances of all signatures and colored by their respective prototypes.
Supplemental Figure S4 : HEST COAD ST data: PCX of stroma, immune and normal epithelium.
Supplemental Figure S5 : TCGA COAD: Relevances of all signatures and colored by their respective prototypes. Inlets show patient density across prototypes.
Supplemental Figure S6 : TCGA COAD: Relevance-based concepts of stroma, immune, and normal epithelial with their accompanying prototypes.
Supplemental Figure S7 : Gene-specific concepts based on activations and relevances. A Activation-based concepts for a single gene ( MYC ). iCMS2 prototypes are displayed, as MYC is part of the iCMS2 signature and concepts are selected by maximal correlation with the predicted GE of MYC . Concepts were cropped to patch-token activations. B Correlation of all activation- and relevance-based concepts with MYC log1p gene expression. C Relevance-based concepts for ( MYC ) explanations. Crops to concepts using LRP to input images.
Spatial transcriptomics (ST) enables gene expression mapping within anatomical context but remains costly and low-throughput. Hematoxylin and eosin (H&E) staining offers rich morphology yet lacks molecular resolution. We present \ours (Spatial Transcriptomics and histOlogy Representation Model), a foundation model trained on 1.2 million spatially resolved transcriptomic profiles with matched histology across 18 organs. Using a hierarchical architecture integrating morphological features, gene expression, and spatial context, STORM bridges imaging and omics through robust molecular--morphological representations. STORM enhances spatial domain discovery, producing biologically coherent tissue maps, and outperforms existing methods in predicting spatial gene expression from H&E images across 11 tumor types. The model is platform-agnostic, performing consistently across Visium, Xenium, Visium HD, and CosMx. Applied to 23 independent cohorts comprising 7,245 patients, STORM significantly improves immunotherapy response prediction and prognostication over established biomarkers, providing a scalable framework for spatially informed discovery and clinical precision medicine.
Jinxi Xiang, Siyu Hou, Yuchen Li +17
Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA · Department of Statistics and Data Science, Yale University, New Haven, CT, USA · Department of Medicine (Oncology), Stanford University School of Medicine, Stanford, CA, USA +4
Predicting gene expression from H&E-stained histology images offers a scalable alternative to costly spatial transcriptomics, yet most existing methods operate at the spot level, where signals from multiple cells are aggregated and critical cellular heterogeneity is obscured. Extending this paradigm to single-cell resolution is non-trivial. Naively applying pathology foundation models faces a scale mismatch: their patch-level representations mix multiple cells, whereas per-cell cropping or resizing distorts morphology and removes local context. Conversely, segmentation-based models without strong pretrained visual encoders often lack the morphological representation capacity needed for accurate molecular prediction and inherit errors from imperfect cell boundary masks. Here, we present CELLO, an efficient end-to-end framework that performs a single pathology foundation model forward pass per image and uses grid sampling to extract location-specific features for all cells simultaneously. We further introduce a distance-decay cross-attention module that refines each cell representation using spatially biased local morphological context. Using 52 public Xenium-H&E pairs from HEST-1k that span 12 organs and approximately 10 million cells, CELLO improves the average predictive accuracy over the evaluated baselines while reducing the mean whole-slide inference time compared to DeepSpot2Cell, a 14.0x speed-up on average that excludes upstream cell segmentation. Our work establishes a scalable foundation for single-cell gene expression prediction from H&E images.
Zijun Gao, Chunbin Gu, Jinxi Xiang +2
The Chinese University of Hong Kong · Stanford University School of Medicine
Predicting spatial gene expression from hematoxylin and eosin (H&E)-stained images offers a cost-effective alternative to spatial transcriptomics (ST). However, existing methods treat H&E images as generic visual inputs and ignore their intrinsic biological hierarchy, where spatially organized cell types collectively form functional tissue microenvironments that govern local gene expression programs. To bridge this gap, we formulate H&E-to-ST prediction as a cross-modal semantic translation task and propose Path2ST, a hierarchically grounded autoregressive framework featuring three key components: (i) a Hierarchical Cell-Tissue Conditioning mechanism that fuses explicit and implicit cellular features with tissue-level semantic representations to construct hierarchical conditioning signals; (ii) a Scale-Adaptive Autoregressive Generation process over a hierarchical semantic vocabulary, enabling coarse-to-fine, biologically consistent expression synthesis; and (iii) SpectraLoss, a full-spectrum objective that jointly enforces ordinal fidelity, models transcriptional bursts, and aligns semantic structures with cell types. Extensive experiments on three datasets demonstrate state-of-the-art performance, validating that Path2ST generates highly accurate and spatially coherent transcriptomic profiles. The related code is released at https://github.com/RuochenLiu23/Path2ST.
Ruochen Liu, Wei Lou
Faculty of Science and Engineering University of Liverpool Liverpool, United Kingdom · College of Mathematical Medicine Zhejiang Normal University Jinhua, China