Towards Scalable Context-Aware Single-Cell Spatial Transcriptomics Prediction from Histology Images

Zijun Gao; Chunbin Gu; Jinxi Xiang; Xiangde Luo; Pheng-Ann Heng

cs.CVSep 29, 2026

Towards Scalable Context-Aware Single-Cell Spatial Transcriptomics Prediction from Histology Images

Authors: Zijun Gao, Chunbin Gu, Jinxi Xiang, Xiangde Luo, Pheng-Ann Heng

Organizations: The Chinese University of Hong Kong · Stanford University School of Medicine

Abstract

Predicting gene expression from H&E-stained histology images offers a scalable alternative to costly spatial transcriptomics, yet most existing methods operate at the spot level, where signals from multiple cells are aggregated and critical cellular heterogeneity is obscured. Extending this paradigm to single-cell resolution is non-trivial. Naively applying pathology foundation models faces a scale mismatch: their patch-level representations mix multiple cells, whereas per-cell cropping or resizing distorts morphology and removes local context. Conversely, segmentation-based models without strong pretrained visual encoders often lack the morphological representation capacity needed for accurate molecular prediction and inherit errors from imperfect cell boundary masks. Here, we present CELLO, an efficient end-to-end framework that performs a single pathology foundation model forward pass per image and uses grid sampling to extract location-specific features for all cells simultaneously. We further introduce a distance-decay cross-attention module that refines each cell representation using spatially biased local morphological context. Using 52 public Xenium-H&E pairs from HEST-1k that span 12 organs and approximately 10 million cells, CELLO improves the average predictive accuracy over the evaluated baselines while reducing the mean whole-slide inference time compared to DeepSpot2Cell, a 14.0x speed-up on average that excludes upstream cell segmentation. Our work establishes a scalable foundation for single-cell gene expression prediction from H&E images.

Figures & tables

Figure 1: (A) Task definition: predicting single-cell spatial gene expression from H&E images with cell locations. (B) PFM-based naive cropping methods such as DeepSpot2Cell require $N$ per-cell forward passes and distort local morphology. (C) Cell mask based UNet3+ lacks strong pretrained visual representations. (D) CELLO performs a single PFM forward pass and uses grid sampling to extract initial feature for all cells simultaneously. A context-aware cross-attention module with 2D RoPE and Gaussian distance-decay bias captures local cellular context and cell–cell interactions.

Figure 2: Given the 2D visual token map from a single PFM forward pass, CELLO first queries a location-specific cell feature by bilinear interpolation. The cell query is then refined by distance-decay cross-attention over patch tokens, where spatially nearby tokens receive a larger prior. This produces a context-aware cell embedding that preserves precise cell-location information while incorporating local morphological context.

In-distribution (ID)					Out-of-distribution (OOD)
Tissue	Model	HVG-10	HVG-50	HVG-200	Tissue	Model	HVG-10	HVG-50	HVG-200
Lung	UNet3+	0.4293	0.2177	0.0824	Kidney	UNet3+	0.1828	0.1189	0.0367
	DeepSpot2Cell	0.4082	0.2264	0.0763		DeepSpot2Cell	0.3303	0.1803	0.0610
	Grid	0.5481	0.3730	0.1455		Grid	0.4728	0.3147	0.1235
	CELLO	0.5537	0.3827	0.1492		CELLO	0.5515	0.2743	0.1078
Breast	UNet3+	0.3078	0.1910	0.0546	Heart	UNet3+	0.2141	0.1209	0.0430

Table 1: HVG results on the 10X-Xenium-52 benchmark. ID tissues are shown on the left and OOD tissues on the right. Higher PCC values are better. The best results within each tissue are highlighted in bold .

Figure 3: Ablation and efficiency analysis of CELLO. (A) Performance comparison of different vision encoders on ID settings. Columns show, from left to right: MPG-10, HVG-10, SVG-10. (B) Effect of CLS-token fusion. Mean $\Delta$ PCC is computed as PCC (with cls) - PCC (w/o cls), averaged across different vision encoders. (C) Total inference time over the 12 test slides, split into data loading, image-encoder forward pass, and cell features with gene head. Cell segmentation (CellViT-SAM-H), required by all methods, is shown separately. CELLO is 14.0 $\times$ faster than DeepSpot2Cell.

Figure 4: Visualization of spatial gene expression predicted from H&E. Rows show, from top to bottom: H&E image, measured gene expression, CELLO predictions, and UNet3+ predictions. Compared with UNet3+, CELLO produced substantially higher correlations with ground-truth gene expression.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Organ	Health condition	# samples	Sample IDs
Train	Bowel	Cancer	3	TENX111 , TENX139 , TENX147
Train	Bowel	Healthy	1	TENX114
Train	Breast	Cancer	4	NCBI783 , TENX96 , TENX97 , TENX98
Train	Kidney	Cancer	1	TENX105
Train	Liver	Healthy	1	TENX121
Train	Lung	Cancer	1	TENX141

Appendix

Table 2: Dataset split composition of the 10X-Xenium-52. Sample IDs are grouped by data split, organ type, and health condition.

Split	Samples	Mean	Min	Median	Max
Train	36	365	280	343	480
Val	4	364	313	360	422
Test	12	414	313	400	480

Appendix

Table 3: Measured genes per sample, excluding Xenium control probes.

Split	Organ	Condition	Sample	Genes	Cells	Patches
Train	Bowel	Cancer	TENX111	425	587,016	13,816
Train	Bowel	Cancer	TENX139	480	388,015	8,858
Train	Bowel	Cancer	TENX147	422	275,875	9,997
Train	Bowel	Healthy	TENX114	325	270,788	37,902
Train	Breast	Cancer	NCBI783	280	141,829	8,177
Train	Breast	Cancer	TENX96	380	365,592	24,657

Appendix

Table 4: Per-sample statistics of the 10X-Xenium-52 (genes exclude Xenium control probes).

Figure 5: Distribution of the 10X-Xenium-52. (A) Number of samples per organ. (B) Total number of cells and image patches across organs. (C) Per-sample cell-count distribution on a log scale, with points denoting individual samples and vertical bars denoting organ-level medians. (D) Disease-state composition across organs, including cancer, diseased, and healthy samples.

In-distribution (ID)								Out-of-distribution (OOD)
Tissue	Model	M-10	M-50	M-200	S-10	S-50	S-200	Tissue	Model	M-10	M-50	M-200	S-10	S-50	S-200
Lung	UNet3+	0.4530	0.2470	0.1322	0.4530	0.2470	0.1225	Kidney	UNet3+	0.2065	0.1356	0.0697	0.2065	0.1348	0.0479
	DeepSpot2Cell	0.4232	0.2580	0.1387	0.4232	0.2580	0.1279		DeepSpot2Cell	0.3617	0.1975	0.0871	0.3617	0.1970	0.0717
	Grid	0.5492	0.3956	0.2390	0.5492	0.3956	0.2234		Grid	0.4879	0.3431	0.1730	0.4879	0.3431	0.1483
	CELLO	0.5538	0.4060	0.2450	0.5538	0.4060	0.2286		CELLO	0.5648	0.3838	0.2085	0.5648	0.3753	0.1610
Breast	UNet3+	0.3126	0.2138	0.1048	0.3126	0.2138	0.0778	Heart	UNet3+	0.2166	0.1255	0.0600	0.2166	0.1255	0.0451

Appendix

Table 5: MPG and SVG results on the 10X-Xenium-52 benchmark. ID tissues are shown on the left and OOD tissues on the right. Higher PCC values are better. The best results within each tissue are highlighted in bold .

Figure 6: Full gene panel performance comparison between DeepSpot2Cell, UNet3+ and CELLO. (A) Full gene panel performance of ID setting. (B) Full gene panel performance of OOD setting.

Lung
Model	MPG			HVG			SVG
Model	PCC-10 $\uparrow$	PCC-50 $\uparrow$	PCC-200 $\uparrow$	PCC-10 $\uparrow$	PCC-50 $\uparrow$	PCC-200 $\uparrow$	PCC-10 $\uparrow$	PCC-50 $\uparrow$	PCC-200 $\uparrow$
UNI	0.5451	0.3969	0.2410	0.5451	0.3753	0.1492	0.5451	0.3969	0.2253
UNI_cls	0.5441	0.3931	0.2368	0.5441	0.3714	0.1437	0.5441	0.3931	0.2204
H0	0.5533	0.4053	0.2457	0.5533	0.3795	0.1485	0.5533	0.4053	0.2292
H0_cls	0.4892	0.3734	0.2213	0.4880	0.3427	0.1249	0.4892	0.3734	0.1960

Appendix

Table 6: Ablation results on the 10X-Xenium-52 ID setting evaluated by MPG, HVG, and SVG metrics. We compare different pathology foundation models as vision encoders and investigate whether incorporating the CLS token to provide additional global contextual information improves performance. Higher values of PCC-10, PCC-50, and PCC-200 indicate better results. The best results are highlighted in bold .

	ID			OOD
Setting	10	50	200	10	50	200
$\lambda=0$	0.5598	0.3882	0.1752	0.4314	0.2574	0.0975
$\lambda=0.1$	0.5565	0.3866	0.1732	0.4178	0.2473	0.0933
$\lambda=0.25$	0.5533	0.3823	0.1683	0.4147	0.2484	0.0922
$\lambda=0.5$	0.5447	0.3750	0.1665	0.4018	0.2349	0.0850
$\lambda=1.0$	0.5614	0.3808	0.1692	0.4298	0.2553	0.0950

Appendix

Table 7: Loss weight

\lambda

and distance-decay bias (50-epoch runs, HVG PCC).

Kidney
Model	MPG			HVG			SVG
Model	PCC-10 $\uparrow$	PCC-50 $\uparrow$	PCC-200 $\uparrow$	PCC-10 $\uparrow$	PCC-50 $\uparrow$	PCC-200 $\uparrow$	PCC-10 $\uparrow$	PCC-50 $\uparrow$	PCC-200 $\uparrow$
UNI	0.5638	0.3600	0.1783	0.5220	0.3229	0.1269	0.5638	0.3600	0.1545
UNI_cls	0.5155	0.3419	0.1737	0.4850	0.3070	0.1249	0.5155	0.3419	0.1515
H0	0.5041	0.3605	0.1843	0.5041	0.3404	0.1368	0.5041	0.3604	0.1592
H0_cls	0.4927	0.3275	0.1628	0.4750	0.3045	0.1127	0.4927	0.3275	0.1378

Appendix

Table 8: Ablation results on the 10X-Xenium-52 OOD setting evaluated by MPG, HVG, and SVG metrics. We compare different pathology foundation models as vision encoders and investigate whether incorporating the CLS token to provide additional global contextual information improves performance. Higher values of PCC-10, PCC-50, and PCC-200 indicate better results. The best results are highlighted in bold .

	ID			OOD
Method	10	50	200	10	50	200
Per-cell crop	0.4981	0.3126	0.1258	0.3184	0.1691	0.0535
CELLO	0.5822	0.3988	0.1629	0.4526	0.2656	0.0972

Appendix

Table 9: Per-cell crop baseline vs. CELLO (HVG PCC, single-cell evaluation).

Sample	Organ (subtype)	Cells	MPG	HVG	SVG
TENX94	Breast (ILC)	356,730	0.700 / 0.580 / 0.358	0.700 / 0.575 / 0.331	0.700 / 0.580 / 0.350
TENX190	Lung (LUAD)	278,349	0.580 / 0.436 / 0.234	0.578 / 0.421 / 0.198	0.580 / 0.436 / 0.220
TENX199	Breast (IDC)	149,995	0.454 / 0.314 / 0.125	0.454 / 0.314 / 0.125	0.454 / 0.314 / 0.125
TENX189	Lung (LUAD)	276,951	0.518 / 0.358 / 0.219	0.485 / 0.285 / 0.089	0.516 / 0.350 / 0.140
TENX158	Skin (SKCM)	111,193	0.604 / 0.401 / 0.199	0.470 / 0.196 / 0.051	0.604 / 0.390 / 0.119

Appendix

Table 10: External validation on five additional HEST-1k Xenium samples (PCC at

k

= 10 / 50 / 200).

Perturbation	Level	Cells	Mean PCC	Change
None	–	43,572	0.1348	–
Centroid jitter (std)	5 px	43,572	0.1346	$-0.2\%$
	10 px	43,572	0.1336	$-0.9\%$
	20 px	43,572	0.1294	$-4.1\%$
	40 px	43,572	0.1206	$-10.6\%$
	80 px	43,572	0.1114	$-17.4\%$

Appendix

Table 11: Sensitivity of CELLO to cell localization errors on 3,000 test patches.

Explore similar work

VOICE: A Vision-Omics Foundation Model Integrating Direct and Retrieval-Based Prediction of In-situ Single-Cell Gene Expression
Aug 8, 2026Xin Luo, Yicheng Tao, Haoxuan Zeng +6Spatial Transcriptomics Single-Cell Rna
University of Michigan Ann Arbor, Michigan, USA · Weill Cornell Medicine New York, New York, USA
GC-MoE: Genomics-Guided Cell-Type-Specific Mixture of Experts for Histology-Based Single-Cell Spatial Transcriptomics
Jun 1, 2026Kaito Shiku, Ahtisham Fazeel Abbasi, Ryoma Bise +4Spatial Transcriptomics Cells
Kyushu University, Japan · German Research Center for Artificial Intelligence (DFKI GmbH), Germany · RPTU University Kaiserslautern-Landau, Germany +3
HiST: A Hierarchical Sparse Transformer for Cross-Modal Spatial Transcriptomics Modeling
Jun 12, 2026Weiyi Wu, Xinwen Xu, Xingjian Diao +4Spatial Transcriptomics Spatio-Temporal Transformers
Dartmouth College · Mass General Hospital · New Jersey Institute of Technology +1

Aug 8, 2026, cs.CVJ/K move · Enter open · S save

VOICE: A Vision-Omics Foundation Model Integrating Direct and Retrieval-Based Prediction of In-situ Single-Cell Gene Expression

Xin Luo, Yicheng Tao, Haoxuan Zeng, Suyuan Wang+5

University of Michigan Ann Arbor, Michigan, USA · Weill Cornell Medicine New York, New York, USA

Spatial Transcriptomics Single-Cell Rna Multi-Omic Integration Genome-Wide Association Study Multimodal Foundation Model

Open paper AskPDF arXiv

Spatial transcriptomics can resolve gene expression at single-cell resolution, but it is costly, limited to targeted panels of a few hundred to a few thousand genes, and applicable to only a small number of samples. H&E imaging, by contrast, is cheap and collected routinely at scale. This makes predicting single-cell expression directly from morphology a practical way to bring molecular analysis to large tissue archives. We therefore present VOICE, a multimodal foundation model that predicts single-cell gene expression from H&E images using paired Xenium data. VOICE first aligns cell centered H&E morphology from a pathology foundation model with single-cell expression embeddings from a transcriptome foundation model, trained using contrastive learning over 23 million cells. Next it predicts expression through two branches. One branch directly regresses expression from morphology. The other branch retrieves measured expression from similar reference cells, recovering genes that do not have morphological signal. Because genes vary in morphological predictability, VOICE fuses the two branches with a per-gene weight. After training, VOICE generalizes to heldout patients, slides, and partially overlapping gene panels from Xenium, and it consistently outperforms prior single-cell expression prediction methods on seven metrics.

Abstract

Explore similar work

VOICE: A Vision-Omics Foundation Model Integrating Direct and Retrieval-Based Prediction of In-situ Single-Cell Gene Expression

GC-MoE: Genomics-Guided Cell-Type-Specific Mixture of Experts for Histology-Based Single-Cell Spatial Transcriptomics

HiST: A Hierarchical Sparse Transformer for Cross-Modal Spatial Transcriptomics Modeling