Extending Sequence Length is Not All You Need: Effective Integration of Multimodal Signals for Gene Expression Prediction
Authors: Zhao Yang, Yi Duan, Jiwei Zhu, Ying Ba, Chuan Cao, Bing Su
Organizations: Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Zhongguancun Academy, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE
Gene expression prediction, which predicts mRNA expression levels from DNA sequences, presents significant challenges. Previous works often focus on extending input sequence length to locate distal enhancers, which may influence target genes from hundreds of kilobases away. Our work first reveals that for current models, long sequence modeling can decrease performance. Even carefully designed algorithms only mitigate the performance degradation caused by long sequences. Instead, we find that proximal multimodal epigenomic signals near target genes prove more essential. Hence we focus on how to better integrate these signals, which has been overlooked. We find that different signal types serve distinct biological roles, with some directly marking active regulatory elements while others reflect background chromatin patterns that may introduce confounding effects. Simple concatenation may lead models to develop spurious associations with these background patterns. To address this challenge, we propose Prism, a framework that learns multiple combinations of high-dimensional epigenomic features to represent distinct background chromatin states and uses backdoor adjustment to mitigate confounding effects. Our experimental results demonstrate that proper modeling of multimodal epigenomic signals achieves state-of-the-art performance using only short sequences for gene expression prediction.
Figures & tables
Figure 1: (a) Long-range regulatory interactions through chromatin looping. (b) Current long-sequence models suffer from technical limitations. (c) Multimodal epigenomic signals provide cell-type specific regulatory information. (d) Performance of Seq2Exp ( Su et al., 2025 ) and Caduceus ( Schiff et al., 2024 ) with varying input sequence lengths. (e) Different signals show varying contributions. (f) Performance degradation when specific signals are removed during testing from a model trained with all signals.
Figure 2: Shortening input length at test time.
Figure 3: The SCM.
Figure 4: Architecture of Prism. Epigenomic signals S are processed by two encoders: a signal encoder gθ extracts high-dimension epigenomic features H , while a confounder encoder gω learns n distinct weights representing the confounder C . A final predictor hϕ uses these weighted features along with the DNA sequence X to make a prediction.
K562
GM12878
MSE ↓
MAE ↓
Pearson ↑
MSE ↓
MAE ↓
Pearson ↑
Enformer
0.2920 ± 0.0050
0.4056 ± 0.0040
0.7961 ± 0.0019
0.2889 ± 0.0009
0.4185 ± 0.0013
0.8327 ± 0.0025
HyenaDNA
0.2265 ± 0.0013
0.3497 ± 0.0012
0.8425 ± 0.0008
0.2217 ± 0.0018
0.3562 ± 0.0012
0.8729 ± 0.0010
Mamba
0.2241 ± 0.0027
0.3416 ± 0.0026
0.8412 ± 0.0021
0.2145 ± 0.0021
0.3446 ± 0.0022
0.8788 ± 0.0011
Caduceus
0.2197 ± 0.0038
0.3327 ± 0.0070
0.8475 ± 0.0014
0.2124 ± 0.0037
0.3436 ± 0.0031
0.8819 ± 0.0009
EPInformer
0.2140 ± 0.0042
0.3291 ± 0.0031
0.8473 ± 0.0017
0.1975 ± 0.0031
0.3246 ± 0.0025
0.8907 ± 0.0011
Table 1: Performance on Gene Expression CAGE Prediction with Standard Deviation for Both Cell Types.
n
MSE ↓
MAE ↓
Pearson ↑
0
0.1863 ± 0.0035
0.3092 ± 0.0050
0.8713 ± 0.0023
1
0.1891 ± 0.0047
0.3084 ± 0.0039
0.8676 ± 0.0032
2
0.1789 ± 0.0041
0.3000 ± 0.0058
0.8751 ± 0.0036
3
0.1818 ± 0.0091
0.3018 ± 0.0090
0.8739 ± 0.0031
4
0.1762 ± 0.0071
0.2961 ± 0.0070
0.8780 ± 0.0028
5
0.1788 ± 0.0062
0.2996 ± 0.0071
0.8752 ± 0.0030
Table 2: Hyperparameter sensitivity analysis for Prism on the K562 cell line. We evaluate the model’s performance while varying (a) the number of background states n , (b) the intervention loss weight α , and (c) the diversity loss weight β .
Figure 5: Visualization of learned confounder weights ( a1,a2 ) for three sampled genes.
Model
Trainable Parameters
Caduceus
574K
Seq2Exp
1.1M
Prism
585K
Table 3: Parameter comparison between models.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Input Length
MSE ↓
MAE ↓
Pearson ↑
200000
0.1856
0.3054
0.8723
10000
0.1855
0.3074
0.8751
8000
0.1864
0.3082
0.8747
3000
0.1943
0.3134
0.8698
2500
0.1996
0.3174
0.8674
2100
0.2301
0.3471
0.8603
Appendix
Table 4: Performance of Seq2Exp ( Su et al., 2025 ) when testing with shortened input sequences on the K562 cell line.
Length
MAE
MSE
Pearson
100
0.3394
0.2233
0.8441
500
0.3096
0.1879
0.8744
2000
0.3150
0.1971
0.8678
5000
0.3098
0.1949
0.8703
10000
0.3088
0.1897
0.8719
Appendix
Table 5: Performance comparison with varying input lengths (left: Seq2Exp ( Su et al., 2025 ) , right: Caduceus ( Schiff et al., 2024 ) )
Configuration
MSE
MAE
Pearson r
No signals
0.2163
0.3325
0.8485
+H3K27ac
0.1873
0.3080
0.8628
+DNase
0.2089
0.3227
0.8497
+Hi-C
0.2135
0.3264
0.8530
All signals
0.1886
0.3079
0.8652
Appendix
Table 6: Caduceus performance with different epigenomic signal configurations
Condition
MAE
MSE
Pearson
Drop H3K27ac
0.5653
0.6115
0.6964
Drop DNase
0.3890
0.2962
0.8189
Drop Hi-C
0.3548
0.2280
0.8467
Baseline (all signals)
0.3078
0.1886
0.8652
Appendix
Table 7: Performance degradation from signal removal (trained with all signals)
Statistic
K562
GM12878
% of genes with promoter-interacting 1
98.9%
99.3%
% of genes with long-range ( > 50kb) promoter 1
98.9%
99.3%
% of genes linked to a distal enhancer via ABC model 2
100.0%
100.0%
Median number of long-range partners per gene 1
199,899
199,899
Median distance of long-range interactions (kb) 1
49,707.0
49,939.0
Appendix
Table 8: Statistical Summary of Hi-C Long-Range Interactions in K562 and GM12878 Cell Lines
Figure 6: A representative genomic locus (Entrez ID: ENSG00000080561) where DNase and Hi-C signals are broadly active, but H3K27ac shows no enrichment. Despite strong chromatin accessibility and spatial contacts, gene expression remains low (0.6021). This supports the hypothesis that such pervasive background signals (quantified in Table 8 ) act as confounders rather than causal regulators.
Hyperparameters
Values
Layers of backbone
4
Hidden dimensions
128
Max training steps
50000
Batch size
8
Learning rate
5e-4
Scheduler strategy
CosineLR with Linear Warmup
Appendix
Table 9: Hyperparameter values following Seq2Exp ( Su et al., 2025 ) .
Model
Metric
From Scratch
Pre-trained
Enformer
MSE ↓
0.2920 ± 0.0050
0.2913 ( ↓ 0.0007) ± 0.0209
Pearson ↑
0.7961 ± 0.0019
0.7983 ( ↑ 0.0022) ± 0.0229
Caduceus w/signals (2k input)
MSE ↓
0.1863 ± 0.0035
0.1858 ( ↓ 0.0005) ± 0.0082
Pearson ↑
0.8713 ± 0.0023
0.8759 ( ↑ 0.0046) ± 0.0042
Caduceus w/signals (200k input)
MSE ↓
0.1959 ± 0.0036
0.1897 ( ↓ 0.0062) ± 0.0026
Pearson ↑
0.8630 ± 0.0008
0.8743 ( ↑ 0.0113) ± 0.0030
Appendix
Table 10: The effect of pre-training
Signal
MSE ↓
MAE ↓
Pearson ↑
No Signal
0.2215 ± 0.0086
0.3342 ± 0.0081
0.8502 ± 0.0026
H3K27ac
0.1986 ± 0.0059
0.3179 ± 0.0054
0.8645 ± 0.0037
DNase-seq
0.2207 ± 0.0060
0.3342 ± 0.0085
0.8530 ± 0.0037
Hi-C
0.2202 ± 0.0045
0.3330 ± 0.0064
0.8489 ± 0.0039
H3K4me3
0.1801 ± 0.0079
0.3084 ± 0.0099
0.8781 ± 0.0018
ChIA-PET
0.2262 ± 0.0062
0.3387 ± 0.0060
0.8422 ± 0.0059
Appendix
Table 11: Single Signal Input (Caduceus w/signals (2k input))
Model Configuration
MSE ↓
MAE ↓
Pearson ↑
Caduceus w/signals
(initial 3 signals)
0.1863 ± 0.0035
0.3092 ± 0.0050
0.8713 ± 0.0023
Caduceus w/signals
(initial 3 signals + DNase footprint)
0.1870 ± 0.0059
0.3092 ± 0.0058
0.8703 ± 0.0026
Caduceus w/signals
(initial 3 signals + H3K4me3)
0.1789 ± 0.0122
0.3067 ± 0.0117
0.8804 ± 0.0101
Appendix
Table 12: Effect of Compositional Signal Input (Models with 2k input)
Dataset
Model
MSE ↓
MAE ↓
Pearson ↑
K562
Seq2Exp (cell-specific)
0.1856 ± 0.0032
0.3054 ± 0.0024
0.8723 ± 0.0012
K562
Prism (cell-specific)
0.1789 ± 0.0041
0.3000 ± 0.0058
0.8751 ± 0.0036
K562
Prism (mixed-training)
0.1875 ± 0.0085
0.3084 ± 0.0077
0.8662 ± 0.0049
GM12878
Seq2Exp (cell-specific)
0.1873 ± 0.0044
0.3137 ± 0.0028
0.8951 ± 0.0038
GM12878
Prism (cell-specific)
0.1759 ± 0.0054
0.3054 ± 0.0048
0.9016 ± 0.0024
GM12878
Prism (mixed-training)
0.1759 ± 0.0038
0.3027 ± 0.0041
0.9012 ± 0.0032
Appendix
Table 13: Performance of mixed-training
Input Length
Caduceus w/signal
Seq2Exp-soft
Prism ( α = 1)
Prism ( α = 2)
100
0.8488 ± 0.0042
0.8492 ± 0.0064
0.8493 ± 0.0056
-
500
0.8719 ± 0.0043
0.8694 ± 0.0051
0.8726 ± 0.0045
-
2000
0.8713 ± 0.0023
0.8643 ± 0.0088
0.8751 ± 0.0036
-
5000
0.8662 ± 0.0035
0.8675 ± 0.0035
0.8690 ± 0.0037
-
10000
0.8614 ± 0.0059
0.8699 ± 0.0032
0.8661 ± 0.0027
0.8699 ± 0.0023
Appendix
Table 14: Pearson across different input lengths on K562
Method
MSE ↓
MAE ↓
Pearson ↑
Caduceus w/signal (2k input)
0.2751 ± 0.0104
0.3929 ± 0.0103
0.6681 ± 0.0137
Seq2Exp-soft (our reproduction)
0.2784 ± 0.0064
0.3957 ± 0.0045
0.6595 ± 0.0089
Prism
0.2642 ± 0.0060
0.3817 ± 0.0044
0.6844 ± 0.0078
Appendix
Table 15: Performance on H1 cell line
Model
MSE ↓
MAE ↓
Pearson ↑
Caduceus w/ Dropout (rate = 0.9)
0.1874 ± 0.0074
0.3062 ± 0.0064
0.8702 ± 0.0026
Caduceus w/ Dropout (rate = 0.7)
0.2059 ± 0.0075
0.3199 ± 0.0010
0.8625 ± 0.0041
Caduceus w/ Dropout (rate = 0.5)
0.2248 ± 0.0115
0.3428 ± 0.0119
0.8446 ± 0.0109
Prism w/ Learned Weights (rate ≈ 0.35)
0.1834 ± 0.0092
0.3032 ± 0.0083
0.8745 ± 0.0061
Appendix
Table 16: Comparison of dropout and learned weight methods
Predicting spatial gene expression from routine H&E enables large-scale molecular profiling, yet current models treat this as isolated pointwise tasks, thereby overlooking essential biological structures like gene coordination and spatial distribution. To preserve these relationships, we introduce FLAG, a diffusion-based framework that redefines this task as structured distribution modeling. At the same time, we identify the critical Gene Dimension Curse, where joint modeling gene expression and their spatial interactions fail in high-dimensional spaces, and FLAG solves this challenge by integrating a spatial graph encoder for topological consistency and utilizing Gene Foundation Model (GFM) alignment for gene-gene fidelity in the generation process. To rigorously assess model performance, we propose a set of novel structural evaluation metrics, including Gene Structural Correlation (GSC) and Spatial Structural Correlation (SSC). Our experiments demonstrate that FLAG is highly competitive in traditional accuracy (PCC/MSE) while achieving significantly enhanced structural fidelity in capturing both gene-gene and gene-spatial relationships. The code is available at https://github.com/darkflash03/FLAG.
Qi Si, Penglei Wang, Yushuai Wu +5
Shanghai Academy of Artificial Intelligence for Science, Shanghai, China. · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China.
Accurate computational identification of DNA methylation is essential for understanding epigenetic regulation. Although deep learning excels in this binary classification task, its "black-box" nature impedes biological insight. We address this by introducing a high-performance model MEDNA-DFM, alongside mechanism-inspired signal purification algorithms. Our investigation demonstrates that MEDNA-DFM effectively captures conserved methylation patterns, achieving robust distinction across diverse species. Validation on external independent datasets confirms that the model's generalization is driven by conserved intrinsic motifs (e.g., GC content) rather than phylogenetic proximity. Furthermore, applying our developed algorithms extracted motifs with significantly higher reliability than prior studies. Finally, empirical evidence from a Drosophila 6mA case study prompted us to propose a "sequence-structure synergy" hypothesis, suggesting that the GAGG core motif and an upstream A-tract element function cooperatively. We further validated this hypothesis via in silico mutagenesis, confirming that the ablation of either or both elements significantly degrades the model's recognition capabilities. This work provides a powerful tool for methylation prediction and demonstrates how explainable deep learning can drive both methodological innovation and the generation of biological hypotheses.
Yi He, Yina Cao, Jixiu Zhai +3
Cuiying Honors College, Lanzhou University, Lanzhou, Gansu, China · School of Mathematics and Statistics, Lanzhou University, Lanzhou, Gansu, China · Cuiying Honors College, Lanzhou University, Lanzhou, Gansu, China. +7
DNA cis-regulatory elements (CREs) such as enhancers control gene expression levels. Accurately predicting regulatory activity from DNA sequences is valuable but challenging, as it requires understanding complex biological regulatory processes. Existing methods typically regress activity scores from sequences in a black-box manner, limiting both interpretability and regression performance. Meanwhile, large language models (LLMs) benefit from explicit reasoning processes, yet directly applying LLMs to raw DNA sequences performs poorly. In this paper, we bridge this gap by introducing R3LM, a framework that teaches LLMs reasoning-informed regression on regulatory DNA through structured biological knowledge. Specifically, we design a biologically grounded data format that structures DNA's regulatory information for improved LLM understanding, and construct CRE-ReasonBench, the first dataset that associates DNA sequences and activity scores with mechanistic reasoning traces. Through two-stage training that first teaches LLMs reasoning over structured biological information then performs regression, R3LM achieves state-of-the-art performance on enhancer prediction across three cell types, outperforming both LLMs with raw sequence input and specialized DNA models while providing interpretable mechanistic explanations. We expect R3LM as an interpretable reward model that can effectively assist biologists in CRE design. Code is available at https://github.com/DuanYi516/R3LM.
Yi Duan, Zhao Yang, Jiwei Zhu +3
Gaoling School of Artificial Intelligence · Renmin University of China · Beijing, China +1