GATE-ST: Gene-Aware Text-image Encoder for Spatial Transcriptomics

Lucas Ni; Jian Luo; Wentao Huang; Chao Chen

cs.AISep 30, 2026

GATE-ST: Gene-Aware Text-image Encoder for Spatial Transcriptomics

Authors: Lucas Ni, Jian Luo, Wentao Huang, Chao Chen

Organizations: Ridge High School Basking Ridge, United States · Stony Brook University Stony Brook, United States

Abstract

Spatial transcriptomics enables spatially resolved gene expression analysis from slide-level images while preserving morphological features, providing valuable information for studying disease mechanisms and developing treatments. However, spatial gene expression profiling typically requires expensive and time-consuming tests. While existing image-based prediction optimizations mostly revolve around including positional embeddings and further image-based changes, text-based optimizations remain relatively unexplored. We present GATE-ST, which incorporates text-based inputs into image-based spatial gene expression predictions. With this approach, generated text descriptions of genes are utilized to better spatial transcriptomics prediction results. Gene summaries are put through a text encoder, generating embeddings that integrate with image embeddings through cross-attention layers to align with morphological features. We demonstrate the effectiveness of such text inputs by benchmarking performance against random gene embeddings and multiple other image-text fusion architectures, and show that GATE-ST outperforms these alternatives. Our results demonstrate the effectiveness of GATE-ST in pathology imaging, which may greatly reduce the time and cost of accurate spatial transcriptomic predictions, proving the potential of text-guided spatial gene expression prediction.

Figures & tables

Fig. 1: Overview of the model architecture.

Hyperparameter	UNI	CONCH
Batch size	32	32
Learning rate	$3\times 10^{-4}$	$3\times 10^{-4}$
Maximum epochs	200	200
Prediction-head dropout	0.4	0.4
LoRA rank ( $r$ )	8	8
LoRA scaling ( $\alpha$ )	16	16

TABLE I: Final hyperparameter configurations for models using UNI and CONCH image encoders.

Methods	CONCH [ 15 ]		UNI [ 14 ]
Methods	MSE $\downarrow$	Pearson $\uparrow$	MSE $\downarrow$	Pearson $\uparrow$
Pure MLP	0.8180	0.5910	0.6731	0.6635
Direct Feature Concatenation	0.8286	0.5857	0.6968	0.6516
Embedding-Image Similarity Based Predictions	0.8217	0.5892	0.6822	0.6589
Cross-attention Fusion	0.8009	0.5996	0.6798	0.6601
Cross-attention with image-residual MLP	0.7867	0.6066	0.6785	0.6608

TABLE II: Comparison of gene-expression prediction architectures using CONCH and UNI image encoders.

Fig. 2: Comparison Heatmaps of the IGKC Gene Expression.

Model Variant	Text Embeddings	Image Residual	LoRA	CONCH		UNI
Model Variant	Text Embeddings	Image Residual	LoRA	MSE $\downarrow$	Pearson $\uparrow$	MSE $\downarrow$	Pearson $\uparrow$
Random Gene Embeddings	–	✓	✓	0.8191	0.5905	0.7338	0.6311
Cross-Attention	✓	–	–	0.8009	0.5996	0.6798	0.6601
+ Image Residual	✓	✓	–	0.7867	0.6066	0.6785	0.6608
Full Model	✓	✓	✓	0.7280	0.6360	0.6774	0.6613

TABLE III: Ablation study of individual components in GATE-ST using CONCH and UNI image encoders.

Fig. 3: Cross-Attention Layers Fine Tuning

Fig. 4: Learning Rate Fine Tuning

Explore similar work

Contrastive and Adaptive Multi-modal Masked Autoencoder for Spatial Transcriptomics
Jun 19, 2026Joohyeok Kim, Taejin Jeong, Jinyeong Kim +1Spatial Transcriptomics 3D Masked Autoencoders
Department of Artificial Intelligence, Yonsei University, Seoul, Republic of Korea · Department of Computer Science, Yonsei University, Seoul, Republic of Korea · Yonsei University College of Medicine, Seoul, Republic of Korea
Hyperbolic Contrastive Learning with Entailment for Spatial Transcriptomics
Sep 14, 2026Daniela Vega, Paula Cárdenas, Hannah Ceballos +2Spatial Transcriptomics Spatio-Temporal Transformers
Center for Research and Formation in Artificial Intelligence Universidad de los Andes, Colombia
Spatial Transcriptomics as Images for Large-Scale Pretraining
Mar 13, 2026Yishun Zhu, Jiaxin Qi, Jian Wang +2Spatial Transcriptomics Spatio-Temporal Transformers
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of the Chinese Academy of Sciences, Zhejiang, China

Jun 19, 2026, cs.CVJ/K move · Enter open · S save

Contrastive and Adaptive Multi-modal Masked Autoencoder for Spatial Transcriptomics

Joohyeok Kim, Taejin Jeong, Jinyeong Kim, Seong Jae Hwang

Department of Artificial Intelligence, Yonsei University, Seoul, Republic of Korea · Department of Computer Science, Yonsei University, Seoul, Republic of Korea · Yonsei University College of Medicine, Seoul, Republic of Korea

Spatial Transcriptomics 3D Masked Autoencoders Autoencoder Architectures

Open paper AskPDF arXiv

The high cost of spatial transcriptomics (ST) has driven extensive studies into predicting gene expression directly from H&E histology images. However, this prediction task faces an inherent limitation, as tissue morphology alone provides insufficient information to fully resolve underlying gene expression. To address this limitation, a recent study leverages partial gene expression to guide the prediction process alongside histology images. Building on this paradigm, we approach the prediction task as a spatial imputation problem, employing a Masked Autoencoder (MAE) to utilize a small fraction of gene expression as genetic anchors for inferring whole-slide gene expression profiles. Specifically, we propose a bio-saliency score and a learning-to-rank strategy to adaptively identify the most informative spots within the tissue. Based on these identified spots, our framework selects contiguous regions as genetic anchors to ensure suitability for real-world ST profiling hardware. To effectively leverage these anchors, we design a cross-modal joint encoder that integrates visual and genetic modalities. By aligning the selected anchors with their corresponding visual features via contrastive learning, the encoder generates robust joint representations to accurately predict gene expression across the whole slide. Notably, our framework consistently surpasses existing methods in both histology-only prediction and spatial imputation, achieving superior accuracy even without genetic anchors and further excelling with as little as 10% transcriptomic coverage. Our code is available at https://github.com/Kyyle2114/CAMMST.

Abstract

Explore similar work

Contrastive and Adaptive Multi-modal Masked Autoencoder for Spatial Transcriptomics

Hyperbolic Contrastive Learning with Entailment for Spatial Transcriptomics

Spatial Transcriptomics as Images for Large-Scale Pretraining