Extending Sequence Length is Not All You Need: Effective Integration of Multimodal Signals for Gene Expression Prediction

Zhao Yang; Yi Duan; Jiwei Zhu; Ying Ba; Chuan Cao; Bing Su

cs.LGFeb 25, 2026

Extending Sequence Length is Not All You Need: Effective Integration of Multimodal Signals for Gene Expression Prediction

Authors: Zhao Yang, Yi Duan, Jiwei Zhu, Ying Ba, Chuan Cao, Bing Su

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Zhongguancun Academy, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE

Abstract

Gene expression prediction, which predicts mRNA expression levels from DNA sequences, presents significant challenges. Previous works often focus on extending input sequence length to locate distal enhancers, which may influence target genes from hundreds of kilobases away. Our work first reveals that for current models, long sequence modeling can decrease performance. Even carefully designed algorithms only mitigate the performance degradation caused by long sequences. Instead, we find that proximal multimodal epigenomic signals near target genes prove more essential. Hence we focus on how to better integrate these signals, which has been overlooked. We find that different signal types serve distinct biological roles, with some directly marking active regulatory elements while others reflect background chromatin patterns that may introduce confounding effects. Simple concatenation may lead models to develop spurious associations with these background patterns. To address this challenge, we propose Prism, a framework that learns multiple combinations of high-dimensional epigenomic features to represent distinct background chromatin states and uses backdoor adjustment to mitigate confounding effects. Our experimental results demonstrate that proper modeling of multimodal epigenomic signals achieves state-of-the-art performance using only short sequences for gene expression prediction.

Figures & tables

Figure 1: (a) Long-range regulatory interactions through chromatin looping. (b) Current long-sequence models suffer from technical limitations. (c) Multimodal epigenomic signals provide cell-type specific regulatory information. (d) Performance of Seq2Exp ( Su et al., 2025 ) and Caduceus ( Schiff et al., 2024 ) with varying input sequence lengths. (e) Different signals show varying contributions. (f) Performance degradation when specific signals are removed during testing from a model trained with all signals.

Figure 2: Shortening input length at test time.

Figure 3: The SCM.

Figure 4: Architecture of Prism. Epigenomic signals $S$ are processed by two encoders: a signal encoder $g_{\theta}$ extracts high-dimension epigenomic features $H$ , while a confounder encoder $g_{\omega}$ learns $n$ distinct weights representing the confounder $C$ . A final predictor $h_{\phi}$ uses these weighted features along with the DNA sequence $X$ to make a prediction.

	K562			GM12878
	MSE $\downarrow$	MAE $\downarrow$	Pearson $\uparrow$	MSE $\downarrow$	MAE $\downarrow$	Pearson $\uparrow$
Enformer	0.2920 $\pm$ 0.0050	0.4056 $\pm$ 0.0040	0.7961 $\pm$ 0.0019	0.2889 $\pm$ 0.0009	0.4185 $\pm$ 0.0013	0.8327 $\pm$ 0.0025
HyenaDNA	0.2265 $\pm$ 0.0013	0.3497 $\pm$ 0.0012	0.8425 $\pm$ 0.0008	0.2217 $\pm$ 0.0018	0.3562 $\pm$ 0.0012	0.8729 $\pm$ 0.0010
Mamba	0.2241 $\pm$ 0.0027	0.3416 $\pm$ 0.0026	0.8412 $\pm$ 0.0021	0.2145 $\pm$ 0.0021	0.3446 $\pm$ 0.0022	0.8788 $\pm$ 0.0011
Caduceus	0.2197 $\pm$ 0.0038	0.3327 $\pm$ 0.0070	0.8475 $\pm$ 0.0014	0.2124 $\pm$ 0.0037	0.3436 $\pm$ 0.0031	0.8819 $\pm$ 0.0009
EPInformer	0.2140 $\pm$ 0.0042	0.3291 $\pm$ 0.0031	0.8473 $\pm$ 0.0017	0.1975 $\pm$ 0.0031	0.3246 $\pm$ 0.0025	0.8907 $\pm$ 0.0011

Table 1: Performance on Gene Expression CAGE Prediction with Standard Deviation for Both Cell Types.

$n$	MSE $\downarrow$	MAE $\downarrow$	Pearson $\uparrow$
0	0.1863 $\pm$ 0.0035	0.3092 $\pm$ 0.0050	0.8713 $\pm$ 0.0023
1	0.1891 $\pm$ 0.0047	0.3084 $\pm$ 0.0039	0.8676 $\pm$ 0.0032
2	0.1789 $\pm$ 0.0041	0.3000 $\pm$ 0.0058	0.8751 $\pm$ 0.0036
3	0.1818 $\pm$ 0.0091	0.3018 $\pm$ 0.0090	0.8739 $\pm$ 0.0031
4	0.1762 $\pm$ 0.0071	0.2961 $\pm$ 0.0070	0.8780 $\pm$ 0.0028
5	0.1788 $\pm$ 0.0062	0.2996 $\pm$ 0.0071	0.8752 $\pm$ 0.0030

Table 2: Hyperparameter sensitivity analysis for Prism on the K562 cell line. We evaluate the model’s performance while varying (a) the number of background states

n

, (b) the intervention loss weight

\alpha

, and (c) the diversity loss weight

\beta

Figure 5: Visualization of learned confounder weights ( $a_{1},a_{2}$ ) for three sampled genes.

Model	Trainable Parameters
Caduceus	574K
Seq2Exp	1.1M
Prism	585K

Table 3: Parameter comparison between models.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Input Length	MSE $\downarrow$	MAE $\downarrow$	Pearson $\uparrow$
200000	0.1856	0.3054	0.8723
10000	0.1855	0.3074	0.8751
8000	0.1864	0.3082	0.8747
3000	0.1943	0.3134	0.8698
2500	0.1996	0.3174	0.8674
2100	0.2301	0.3471	0.8603

Appendix

Table 4: Performance of Seq2Exp ( Su et al., 2025 ) when testing with shortened input sequences on the K562 cell line.

Length	MAE	MSE	Pearson
100	0.3394	0.2233	0.8441
500	0.3096	0.1879	0.8744
2000	0.3150	0.1971	0.8678
5000	0.3098	0.1949	0.8703
10000	0.3088	0.1897	0.8719

Appendix

Table 5: Performance comparison with varying input lengths (left: Seq2Exp ( Su et al., 2025 ) , right: Caduceus ( Schiff et al., 2024 ) )

Configuration	MSE	MAE	Pearson r
No signals	0.2163	0.3325	0.8485
+H3K27ac	0.1873	0.3080	0.8628
+DNase	0.2089	0.3227	0.8497
+Hi-C	0.2135	0.3264	0.8530
All signals	0.1886	0.3079	0.8652

Appendix

Table 6: Caduceus performance with different epigenomic signal configurations

Condition	MAE	MSE	Pearson
Drop H3K27ac	0.5653	0.6115	0.6964
Drop DNase	0.3890	0.2962	0.8189
Drop Hi-C	0.3548	0.2280	0.8467
Baseline (all signals)	0.3078	0.1886	0.8652

Appendix

Table 7: Performance degradation from signal removal (trained with all signals)

Statistic	K562	GM12878
% of genes with promoter-interacting 1	98.9%	99.3%
% of genes with long-range ( $>$ 50kb) promoter 1	98.9%	99.3%
% of genes linked to a distal enhancer via ABC model 2	100.0%	100.0%
Median number of long-range partners per gene 1	199,899	199,899
Median distance of long-range interactions (kb) 1	49,707.0	49,939.0

Appendix

Table 8: Statistical Summary of Hi-C Long-Range Interactions in K562 and GM12878 Cell Lines

Figure 6: A representative genomic locus (Entrez ID: ENSG00000080561) where DNase and Hi-C signals are broadly active, but H3K27ac shows no enrichment. Despite strong chromatin accessibility and spatial contacts, gene expression remains low (0.6021). This supports the hypothesis that such pervasive background signals (quantified in Table 8 ) act as confounders rather than causal regulators.

Hyperparameters	Values
Layers of backbone	4
Hidden dimensions	128
Max training steps	50000
Batch size	8
Learning rate	5e-4
Scheduler strategy	CosineLR with Linear Warmup

Appendix

Table 9: Hyperparameter values following Seq2Exp ( Su et al., 2025 ) .

Model	Metric	From Scratch	Pre-trained
Enformer	MSE $\downarrow$	0.2920 $\pm$ 0.0050	0.2913 ( $\downarrow$ 0.0007) $\pm$ 0.0209
Enformer	Pearson $\uparrow$	0.7961 $\pm$ 0.0019	0.7983 ( $\uparrow$ 0.0022) $\pm$ 0.0229
Caduceus w/signals (2k input)	MSE $\downarrow$	0.1863 $\pm$ 0.0035	0.1858 ( $\downarrow$ 0.0005) $\pm$ 0.0082
Caduceus w/signals (2k input)	Pearson $\uparrow$	0.8713 $\pm$ 0.0023	0.8759 ( $\uparrow$ 0.0046) $\pm$ 0.0042
Caduceus w/signals (200k input)	MSE $\downarrow$	0.1959 $\pm$ 0.0036	0.1897 ( $\downarrow$ 0.0062) $\pm$ 0.0026
Caduceus w/signals (200k input)	Pearson $\uparrow$	0.8630 $\pm$ 0.0008	0.8743 ( $\uparrow$ 0.0113) $\pm$ 0.0030

Appendix

Table 10: The effect of pre-training

Signal	MSE $\downarrow$	MAE $\downarrow$	Pearson $\uparrow$
No Signal	0.2215 $\pm$ 0.0086	0.3342 $\pm$ 0.0081	0.8502 $\pm$ 0.0026
H3K27ac	0.1986 $\pm$ 0.0059	0.3179 $\pm$ 0.0054	0.8645 $\pm$ 0.0037
DNase-seq	0.2207 $\pm$ 0.0060	0.3342 $\pm$ 0.0085	0.8530 $\pm$ 0.0037
Hi-C	0.2202 $\pm$ 0.0045	0.3330 $\pm$ 0.0064	0.8489 $\pm$ 0.0039
H3K4me3	0.1801 $\pm$ 0.0079	0.3084 $\pm$ 0.0099	0.8781 $\pm$ 0.0018
ChIA-PET	0.2262 $\pm$ 0.0062	0.3387 $\pm$ 0.0060	0.8422 $\pm$ 0.0059

Appendix

Table 11: Single Signal Input (Caduceus w/signals (2k input))

Model Configuration	MSE $\downarrow$	MAE $\downarrow$	Pearson $\uparrow$
Caduceus w/signals
(initial 3 signals)	0.1863 $\pm$ 0.0035	0.3092 $\pm$ 0.0050	0.8713 $\pm$ 0.0023
Caduceus w/signals
(initial 3 signals + DNase footprint)	0.1870 $\pm$ 0.0059	0.3092 $\pm$ 0.0058	0.8703 $\pm$ 0.0026
Caduceus w/signals
(initial 3 signals + H3K4me3)	0.1789 $\pm$ 0.0122	0.3067 $\pm$ 0.0117	0.8804 $\pm$ 0.0101

Appendix

Table 12: Effect of Compositional Signal Input (Models with 2k input)

Dataset	Model	MSE ↓	MAE ↓	Pearson ↑
K562	Seq2Exp (cell-specific)	0.1856 ± 0.0032	0.3054 ± 0.0024	0.8723 ± 0.0012
K562	Prism (cell-specific)	0.1789 ± 0.0041	0.3000 ± 0.0058	0.8751 ± 0.0036
K562	Prism (mixed-training)	0.1875 ± 0.0085	0.3084 ± 0.0077	0.8662 ± 0.0049
GM12878	Seq2Exp (cell-specific)	0.1873 ± 0.0044	0.3137 ± 0.0028	0.8951 ± 0.0038
GM12878	Prism (cell-specific)	0.1759 ± 0.0054	0.3054 ± 0.0048	0.9016 ± 0.0024
GM12878	Prism (mixed-training)	0.1759 ± 0.0038	0.3027 ± 0.0041	0.9012 ± 0.0032

Appendix

Table 13: Performance of mixed-training

Input Length	Caduceus w/signal	Seq2Exp-soft	Prism ( $\alpha$ = 1)	Prism ( $\alpha$ = 2)
100	0.8488 ± 0.0042	0.8492 ± 0.0064	0.8493 ± 0.0056	-
500	0.8719 ± 0.0043	0.8694 ± 0.0051	0.8726 ± 0.0045	-
2000	0.8713 ± 0.0023	0.8643 ± 0.0088	0.8751 ± 0.0036	-
5000	0.8662 ± 0.0035	0.8675 ± 0.0035	0.8690 ± 0.0037	-
10000	0.8614 ± 0.0059	0.8699 ± 0.0032	0.8661 ± 0.0027	0.8699 ± 0.0023

Appendix

Table 14: Pearson across different input lengths on K562

Method	MSE ↓	MAE ↓	Pearson ↑
Caduceus w/signal (2k input)	0.2751 ± 0.0104	0.3929 ± 0.0103	0.6681 ± 0.0137
Seq2Exp-soft (our reproduction)	0.2784 ± 0.0064	0.3957 ± 0.0045	0.6595 ± 0.0089
Prism	0.2642 ± 0.0060	0.3817 ± 0.0044	0.6844 ± 0.0078

Appendix

Table 15: Performance on H1 cell line

Model	MSE ↓	MAE ↓	Pearson ↑
Caduceus w/ Dropout (rate = 0.9)	0.1874 ± 0.0074	0.3062 ± 0.0064	0.8702 ± 0.0026
Caduceus w/ Dropout (rate = 0.7)	0.2059 ± 0.0075	0.3199 ± 0.0010	0.8625 ± 0.0041
Caduceus w/ Dropout (rate = 0.5)	0.2248 ± 0.0115	0.3428 ± 0.0119	0.8446 ± 0.0109
Prism w/ Learned Weights (rate $\approx$ 0.35)	0.1834 ± 0.0092	0.3032 ± 0.0083	0.8745 ± 0.0061

Appendix

Table 16: Comparison of dropout and learned weight methods

Genomics

Genomics uses computational and experimental methods to study genomes, genetic variation, gene regulation, and their links to traits and disease.

See all 124 papers

FLAG: Foundation model representation with Latent diffusion Alignment via Graph for spatial gene expression prediction
May 18, 2026Qi Si, Penglei Wang, Yushuai Wu +5Spatial Transcriptomics Spatial Gene Expression Prediction
Shanghai Academy of Artificial Intelligence for Science, Shanghai, China. · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China.
MEDNA-DFM: A Dual-View FiLM-MoE Model for Explainable DNA Methylation Prediction
Feb 26, 2026Yi He, Yina Cao, Jixiu Zhai +3Genomics Sequence Modeling
Cuiying Honors College, Lanzhou University, Lanzhou, Gansu, China · School of Mathematics and Statistics, Lanzhou University, Lanzhou, Gansu, China · Cuiying Honors College, Lanzhou University, Lanzhou, Gansu, China. +7
Biological Reasoning-Informed Regression for Interpretable Regulatory DNA Activity Prediction
Jun 6, 2026Yi Duan, Zhao Yang, Jiwei Zhu +3Interpretable ML
Gaoling School of Artificial Intelligence · Renmin University of China · Beijing, China +1

May 18, 2026J/K move · Enter open · S save

FLAG: Foundation model representation with Latent diffusion Alignment via Graph for spatial gene expression prediction

Qi Si, Penglei Wang, Yushuai Wu, Yifeng Jiao+4

Shanghai Academy of Artificial Intelligence for Science, Shanghai, China. · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China.

Spatial Transcriptomics Spatial Gene Expression Prediction Genomics Gene Expression Prediction from Histopathology Images Diffusion Models

Open paper AskPDF arXiv

Predicting spatial gene expression from routine H&E enables large-scale molecular profiling, yet current models treat this as isolated pointwise tasks, thereby overlooking essential biological structures like gene coordination and spatial distribution. To preserve these relationships, we introduce FLAG, a diffusion-based framework that redefines this task as structured distribution modeling. At the same time, we identify the critical Gene Dimension Curse, where joint modeling gene expression and their spatial interactions fail in high-dimensional spaces, and FLAG solves this challenge by integrating a spatial graph encoder for topological consistency and utilizing Gene Foundation Model (GFM) alignment for gene-gene fidelity in the generation process. To rigorously assess model performance, we propose a set of novel structural evaluation metrics, including Gene Structural Correlation (GSC) and Spatial Structural Correlation (SSC). Our experiments demonstrate that FLAG is highly competitive in traditional accuracy (PCC/MSE) while achieving significantly enhanced structural fidelity in capturing both gene-gene and gene-spatial relationships. The code is available at https://github.com/darkflash03/FLAG.

Abstract

Genomics

Explore similar work

FLAG: Foundation model representation with Latent diffusion Alignment via Graph for spatial gene expression prediction

MEDNA-DFM: A Dual-View FiLM-MoE Model for Explainable DNA Methylation Prediction

Biological Reasoning-Informed Regression for Interpretable Regulatory DNA Activity Prediction