MIST: Multi-Scale Spatial Context for Gene Expression Prediction from Histology

Koushik Howlader; Tirtho Roy; Md Tauhidul Islam; Wei Le

cs.CVOct 13, 2026

MIST: Multi-Scale Spatial Context for Gene Expression Prediction from Histology

Authors: Koushik Howlader, Tirtho Roy, Md Tauhidul Islam, Wei Le

Organizations: Iowa State University, Iowa, USA · Stanford University, CA, USA

Abstract

Predicting spatial gene expression from standard H&E images could extend molecular analysis to tissues without spatial transcriptomic measurements. A standard Transformer applies self-attention over spots but ignores where each spot sits in the tissue, whereas a spot's molecular state depends on tissue context at multiple spatial scales. Building on this insight, we introduce MIST, a Multi-context Spatial Transformer that augments the self-attention backbone with two context pathways applied in every layer: a coordinate-invariant local pathway that attends over spatial k-nearest-neighbor spots, and a slide pathway that broadcasts a whole-slide summary to every spot; the model predicts all spots in a single forward pass. Because coordinates enter only through pairwise distances, the local pathway is invariant to translations, rotations, and reflections when image features are fixed. In intra-cohort evaluation across six human organ cohorts from HEST-1k, MIST improves mean Pearson correlation over the strongest evaluated baseline by 9.7%, 11.6%, and 11.4% for 10-, 50-, and 100-gene panels, respectively, with the lowest average MSE and MAE. MIST further improves leave-one-organ-out transfer by 6.0% over the next strongest baseline, attains the best expression-derived domain recovery (highest mean ARI and NMI at every clustering resolution, and higher NMI than the closest baseline on 84% of held-out slides), and reproduces spatial expression patterns of selected cancer-marker genes. Together, these results show that adding local and slide context to a vanilla Transformer backbone is an effective approach to spatial gene expression prediction from histology.

Figures & tables

Figure 1: MIST overview. (a) Frozen UNI and/or CONCH embeddings represent spot-centered H&E patches, while spot coordinates define spatial $k$ NN neighborhoods. (b) Each Transformer layer augments the self-attention backbone with two pathways: local spatial-neighbor attention and an attention-pooled slide summary broadcast to all spots. The final spot representations are mapped to predicted gene expression.

Panel	Model	IDC	CCRCC	LUNG	PAAD	PRAD	SKCM	PCC Avg. $\uparrow$	MSE Avg. $\downarrow$	MAE Avg. $\downarrow$
HVG	ST-Net	$0.551{\pm}0.01$	$0.163{\pm}0.02$	$0.521{\pm}0.01$	$0.366{\pm}0.03$	$0.287{\pm}0.02$	$0.661{\pm}0.01$	$0.425$	$1.995$	$0.969$
	Hist2ST	$0.305{\pm}0.11$	$0.180{\pm}0.03$	$0.430{\pm}0.05$	$0.248{\pm}0.04$	$0.323{\pm}0.01$	$0.523{\pm}0.00$	$0.335$	$1.561$	$0.890$
	BLEEP	$0.519{\pm}0.00$	$0.175{\pm}0.02$	$0.565{\pm}0.01$	$0.412{\pm}0.03$	$0.346{\pm}0.01$	$0.650{\pm}0.00$	$0.444$	$1.318$	$0.819$
	STEM	$0.434{\pm}0.00$	$0.118{\pm}0.01$	$0.407{\pm}0.00$	$0.233{\pm}0.02$	$0.207{\pm}0.02$	$0.461{\pm}0.02$	$0.310$	$1.747$	$0.920$
	MIST (ours)	$0.581{\pm}0.01$	$0.252{\pm}0.01$	$0.589{\pm}0.01$	$0.437{\pm}0.03$	$0.410{\pm}0.01$	$0.707{\pm}0.01$	$0.496$	$1.233$	$0.794$
DEG	ST-Net	$0.697{\pm}0.01$	$0.312{\pm}0.01$	$0.566{\pm}0.01$	$0.425{\pm}0.03$	$0.413{\pm}0.00$	$0.752{\pm}0.01$	$0.528$	$3.336$	$1.457$

Table 1: Performance across gene-selection strategies (50-gene panels). Dataset PCC is mean

\pm

SD over three seeds; final columns are six-cohort mean PCC, MSE, and MAE. Best per strategy in bold ;

\uparrow

higher and

\downarrow

lower is better.

Method	PCC $\uparrow$
ST-Net He et al. (2020) (NBE’20)	$0.351_{0.02}$
Hist2ST Zeng et al. (2022) (BIB’22)	$0.556_{0.01}$
BLEEP Xie et al. (2023) (NeurIPS’23)	$0.587_{0.00}$
STEM Zhu et al. (2025) (ICLR’25)	$0.273_{0.00}$
MIST (ours)	$0.599_{0.01}$

Table 2: Cross-organ POOLED (six-organ subset). 5-fold slide-level holdout; mean

{}_{\text{std}}

over three seeds; best in bold .

Model	IDC	CCRCC	LUNG	PAAD	PRAD	SKCM	PCC Avg. $\uparrow$	MSE Avg. $\downarrow$	MAE Avg. $\downarrow$
ST-Net	$0.332{\pm}0.02$	$0.074{\pm}0.00$	$0.535{\pm}0.07$	$0.382{\pm}0.00$	$0.078{\pm}0.01$	$0.500{\pm}0.09$	$0.317$	$1.912$	$0.984$
Hist2ST	$0.350{\pm}0.00$	$0.064{\pm}0.01$	$0.581{\pm}0.00$	$0.214{\pm}0.17$	$0.062{\pm}0.01$	$0.694{\pm}0.02$	$0.328$	$1.356$	$0.876$
BLEEP	$0.401{\pm}0.04$	$0.058{\pm}0.00$	$0.630{\pm}0.00$	$0.497{\pm}0.03$	$0.091{\pm}0.00$	$0.708{\pm}0.01$	$0.398$	$1.410$	$0.847$
STEM	$0.353{\pm}0.00$	$0.055{\pm}0.02$	$0.543{\pm}0.02$	$0.393{\pm}0.02$	$0.041{\pm}0.02$	$0.492{\pm}0.07$	$0.313$	$1.413$	$0.814$
MIST (ours)	$0.425{\pm}0.03$	$0.098{\pm}0.01$	$0.687{\pm}0.02$	$0.499{\pm}0.03$	$0.105{\pm}0.01$	$0.720{\pm}0.03$	$0.422$	$1.288$	$0.849$

Table 3: Leave-one-organ-out (LOOO) performance. Per-organ PCC, mean

\pm

std over three seeds; final columns average PCC/MSE/MAE across the six held-out organs.

Figure 2: Context-stream ablation: self-attention is the vanilla Transformer backbone; local and slide are the added levers. (a) Context Streams. (b) Neighborhood.

	CCRCC (HVG; six folds)		PRAD (HVG; two folds)		Pooled (14 genes; five folds)
Coordinates	PCC $\uparrow$	$\Delta$ PCC	PCC $\uparrow$	$\Delta$ PCC	PCC $\uparrow$	$\Delta$ PCC
Original	$0.25158{\pm}0.01041$	—	$0.41020{\pm}0.00730$	—	$0.59875{\pm}0.00671$	—
Translation	$0.25157{\pm}0.01027$	$-0.000014$	$0.41025{\pm}0.00732$	$+0.000058$	$0.59871{\pm}0.00668$	$-0.000036$
Rotation	$0.25175{\pm}0.01053$	$+0.000166$	$0.41026{\pm}0.00729$	$+0.000066$	$0.59876{\pm}0.00670$	$+0.000010$
Reflection	$0.25158{\pm}0.01041$	$0$	$0.41020{\pm}0.00730$	$0$	$0.59875{\pm}0.00671$	$0$
Spot permutation	$0.25154{\pm}0.01029$	$-0.000035$	$0.41022{\pm}0.00730$	$+0.000026$	$0.59870{\pm}0.00674$	$-0.000049$

Table 4: Coordinate-transform check on held-out slides. PCC (mean

\pm

std over three seeds, image features fixed);

\Delta

PCC relative to original coordinates. The pooled column uses a different gene panel.

Figure 3: Spatial-domain recovery across all methods. Mean ARI (left) and NMI (right) (KMeans, $k{=}6$ ; predicted vs. measured domains); robustness to $k$ is in Appendix I .

Figure 4: Qualitative spatial expression maps for SKCM slide TENX117 (two marker genes, rows); per-gene PCC shown, MIST in orange.

Figure 5: Case study: per-gene recovery of canonical cancer markers. Per-gene Pearson (200-gene panel, mean over folds and three seeds); darker is higher.

Figure 6: Spatial recovery of cancer markers. Measured vs. MIST -predicted expression on held-out melanoma (top) and breast (bottom) slides (dark = low, bright = high; per-panel $r$ shown).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Cohort	Organ	Samples	Spots	Folds
IDC	Breast	4	44,974	4
CCRCC	Kidney	24	74,220	6
LUNG	Lung	2	7,505	2
PAAD	Pancreas	3	9,793	3
PRAD	Prostate	23	62,710	2
SKCM	Skin	2	5,716	2

Appendix

Table 5: HEST-1k cohort statistics. Cohorts used in the reported experiments; spots are counted after barcode filtering and folds denote per-cohort train/test splits.

Outcome	Comparison	Sample size	Reported result
PCC@50	MIST vs. BLEEP	$n=18$ paired runs	gain $0.051$ ; 95% bootstrap CI $[0.041,0.062]$ ; $p=7.6\times 10^{-6}$
Per-gene PCC@50	MIST vs. BLEEP	$n=300$ genes (descriptive)	MIST higher on 87%
LOOO PCC	MIST vs. BLEEP	$n=18$ organ $\times$ seed; also $n=6$ cohorts	$p=0.018$ ( $n{=}18$ ); exact $p=0.031$ , all six positive ( $n{=}6$ )
Spatial ARI	MIST vs. BLEEP	$n=58$ slides	MIST wins 60%; $p=0.03$
Spatial NMI	MIST vs. BLEEP	$n=58$ slides	MIST wins 84%; $p=1.5\times 10^{-8}$

Appendix

Table 6: Summary of paired statistical tests. All tests are two-sided Wilcoxon signed-rank tests unless stated otherwise;

p

-values are reported as available from the matched evaluation.

Panel size	Model	IDC	CCRCC	LUNG	PAAD	PRAD	SKCM	PCC Avg. $\uparrow$	MSE Avg. $\downarrow$	MAE Avg. $\downarrow$
PCC@10	ST-Net	$0.666{\pm}0.01$	$0.231{\pm}0.01$	$0.702{\pm}0.01$	$0.511{\pm}0.04$	$0.442{\pm}0.03$	$0.728{\pm}0.00$	$0.547$	$6.087$	$1.908$
	Hist2ST	$0.636{\pm}0.01$	$0.273{\pm}0.02$	$0.519{\pm}0.02$	$0.377{\pm}0.07$	$0.459{\pm}0.02$	$0.471{\pm}0.04$	$0.456$	$3.297$	$1.374$
	BLEEP	$0.635{\pm}0.02$	$0.231{\pm}0.02$	$0.735{\pm}0.00$	$0.536{\pm}0.03$	$0.434{\pm}0.03$	$0.697{\pm}0.01$	$0.545$	$2.859$	$1.243$
	STEM	$0.591{\pm}0.02$	$0.170{\pm}0.00$	$0.616{\pm}0.01$	$0.393{\pm}0.04$	$0.325{\pm}0.03$	$0.598{\pm}0.01$	$0.449$	$3.181$	$1.310$
	MIST (ours)	$0.708{\pm}0.00$	$0.322{\pm}0.01$	$0.759{\pm}0.00$	$0.519{\pm}0.05$	$0.524{\pm}0.02$	$0.769{\pm}0.01$	$0.600$	$2.620$	$1.211$
PCC@100	ST-Net	$0.611{\pm}0.01$	$0.221{\pm}0.01$	$0.595{\pm}0.00$	$0.355{\pm}0.04$	$0.327{\pm}0.01$	$0.696{\pm}0.01$	$0.468$	$3.320$	$1.354$

Appendix

Table 7: Performance across HVG panel sizes (10 and 100 genes). Dataset PCC is mean

\pm

SD over three seeds; final columns are six-cohort mean PCC, MSE, and MAE. The 50-gene HVG results are in Table 1 .

Encoder	Cohort	ST-Net	Hist2ST	BLEEP	STEM	MIST (ours)
UNI (1024-d)	IDC Breast	$0.538{\pm}0.008$	$0.250{\pm}0.067$	$0.508{\pm}0.005$	$0.436{\pm}0.005$	$0.569{\pm}0.009$
	CCRCC Kidney	$0.172{\pm}0.010$	$0.214{\pm}0.015$	$0.184{\pm}0.007$	$0.116{\pm}0.002$	$0.264{\pm}0.010$
	LUNG Lung	$0.482{\pm}0.012$	$0.378{\pm}0.097$	$0.559{\pm}0.007$	$0.393{\pm}0.009$	$0.590{\pm}0.005$
	PAAD Pancreas	$0.353{\pm}0.022$	$0.243{\pm}0.061$	$0.393{\pm}0.035$	$0.227{\pm}0.017$	$0.392{\pm}0.040$
	PRAD Prostate	$0.251{\pm}0.008$	$0.322{\pm}0.020$	$0.343{\pm}0.009$	$0.190{\pm}0.010$	$0.394{\pm}0.003$
	SKCM Skin	$0.632{\pm}0.010$	$0.510{\pm}0.015$	$0.639{\pm}0.015$	$0.471{\pm}0.031$	$0.694{\pm}0.001$

Appendix

Table 8: Encoder ablation: intra-cohort PCC@50 per encoder. UNI-only (1024-d), CONCH-only (512-d), and concatenated UNI+CONCH (1536-d); mean

{}_{\text{std}}

over three seeds; best per row in bold .

Figure 7: Encoder-averaged intra-cohort PCC@50 per model. Six-cohort mean for each encoder (UNI, CONCH, UNI+CONCH); same values as Table 8 . MIST is highest under every encoder, and UNI+CONCH is best for it.

Figure 8: Per-gene recovery of canonical cancer markers across four cohorts. Per-gene Pearson on the 200-gene panel (melanoma/SKCM, breast/IDC, kidney/CCRCC, prostate/PRAD), mean over folds and three seeds; darker is higher. STEM values for CCRCC and PRAD are computed from the folds available at submission time.

	ARI $\uparrow$				NMI $\uparrow$
Method	$k{=}4$	$k{=}6$	$k{=}8$	$k{=}10$	$k{=}4$	$k{=}6$	$k{=}8$	$k{=}10$
ST-Net	$0.070$	$0.065$	$0.055$	$0.049$	$0.085$	$0.091$	$0.092$	$0.095$
Hist2ST	$0.079$	$0.073$	$0.062$	$0.055$	$0.084$	$0.087$	$0.089$	$0.091$
BLEEP	$0.090$	$0.081$	$0.075$	$0.070$	$0.088$	$0.092$	$0.095$	$0.099$
STEM	$0.034$	$0.031$	$0.027$	$0.025$	$0.028$	$0.028$	$0.030$	$0.032$
MIST (ours)	$0.106$	$0.091$	$0.080$	$0.072$	$0.107$	$0.110$	$0.111$	$0.113$

Appendix

Table 9: Domain recovery vs. number of clusters

k

. Mean ARI and NMI over

n{=}58

held-out slides (three seeds); best per column in bold .

Explore similar work

Jul 10, 2026

COAST: Context-Aware Differential Learning for Gene Expression Prediction in Spatial Transcriptomics

Spatial transcriptomics enables profiling of spatial gene expression but is limited by high cost and low throughput, motivating prediction from H&E histopathology images. Existing context-aware methods mainly supervise absolute expression, while relative expression relationships between spots are rarely used explicitly. We propose COAST, a context-aware differential learning framework for spatial gene expression prediction. COAST conditions the local and global context features with type-specific modulation and aggregates the target and context spot tokens using a Transformer encoder to capture both fine-grained local patterns and slide-level structure. It is trained with a joint objective that combines absolute expression regression with signed differential regression between the target and context spots. Experiments on multiple spatial transcriptomics datasets show consistent improvements in correlation- and distribution-based metrics, demonstrating the effectiveness of context-aware differential learning for histology-based spatial gene expression prediction.

Keunho Byeon, Sunhong Park, Jeewoo Lim +1

School of Electrical Engineering, Korea University, Seoul 02841, Republic of Korea

Spatial Transcriptomics Spatial Gene Expression Prediction Gene Expression Prediction from Histopathology Images

Ask PDF

Jun 12, 2026

HiST: A Hierarchical Sparse Transformer for Cross-Modal Spatial Transcriptomics Modeling

Spatial transcriptomics (ST) links gene expression with tissue morphology but remains expensive and low-throughput, motivating surrogates that infer expression from routine histology. Whole-slide H&E-to-ST inference pairs a gigapixel image with gene measurements at a sparse, irregular set of locations, making multiscale modeling challenging without incurring dense-grid overhead or quadratic token mixing. We propose HiST, a hierarchical sparse transformer that treats measured locations as a lattice-indexed sparse field and builds a dyadic encoder--decoder directly on the active tissue footprint. HiST combines sparse window attention for local geometric correspondence with resolution-changing operators for rapid multiscale context integration. For a fixed window size, the dominant runtime and memory scale with the number of observed locations rather than the dense slide area. To mitigate slide-specific acquisition variation, HiST adds a bottlenecked global conditioning pathway via a slide calibration token that summarizes slide-level context and conditions local representations. On a multi-organ benchmark spanning diverse tissues and acquisition sources, HiST improves predictive performance over recent baselines while reducing runtime and peak memory.

Weiyi Wu, Xinwen Xu, Xingjian Diao +4

Dartmouth College · Mass General Hospital · New Jersey Institute of Technology +1

Computational Pathology Spatial Transcriptomics Cross-Modal Representation Learning Transformer Hierarchical Representation Learning Spatial Gene Expression Prediction+1

Ask PDF

Jul 27, 2026

HistoGPA: A Context-Conditioned Gene-Prior Attention Framework for Histology-Based Spatial Gene Expression Prediction

Predicting spatial gene expression from routine hematoxylin and eosin (H&E) images provides a practical complement to experimental spatial transcriptomics. Existing approaches focus on local or multi-scale visual features and often treat pretrained gene representations as fixed priors, although the interpretation of local morphology and the relevance of gene priors depend on tissue context. We propose HistoGPA, a context-conditioned gene-prior attention framework that uses a shared slide-level representation in two parallel pathways: one modulates local morphological features, whereas the other conditions pretrained gene embeddings and retrieves gene-prior information through cross-attention. This design enables each spatial location to retrieve context-adapted gene-prior information using its local morphology, position, and slide context. Across ten cancer types in HEST-1k, HistoGPA achieves the highest macro-averaged gene-wise Pearson correlation coefficient among the compared methods under the same evaluation protocol for both the top-50 and top-1,500 highly variable gene sets. Additional analyses show that HistoGPA better recovers the spatial expression patterns of cancer-associated genes and yields greater agreement between clusters derived independently from predicted and ground-truth expression profiles. Together, these findings motivate a context-dependent view of histology-to-expression prediction, in which local morphological representations and gene priors are jointly adapted to the broader tissue context.

Ziang Liu, Xinhai Chen, Yigui Feng +3

Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology · College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China

Computational Pathology Spatial Transcriptomics Spatial Gene Expression Prediction Genomics Gene Expression Prediction from Histopathology Images

Ask PDF