RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

Ziyuan Wang; Bohao Tang; Fei Zhang; Shuo Han; Pengfei Liu

cs.LGAug 24, 2026

RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

Authors: Ziyuan Wang, Bohao Tang, Fei Zhang, Shuo Han, Pengfei Liu

Organizations: Shanghai Innovation Institute · Center for Excellence in Molecular Cell Science, CAS · University of Chinese Academy of Sciences · Generative Artificial Intelligence Research Lab ·

Abstract

Full-length RNAs, particularly messenger RNAs, often exceed the context lengths used to pretrain existing RNA foundation models, limiting complete-transcript modeling at single-nucleotide resolution. We present RIBOSPAN, a 1.61-billion-parameter bidirectional RNA foundation model natively pretrained with context lengths up to 10,240 nt. RIBOSPAN combines dense bidirectional self-attention, single-nucleotide tokenization, and attention-isolated sequence packing to enable high-resolution modeling of complete long RNAs. Native 10K pretraining preserves strong reconstruction at 10,240 tokens and, in a controlled long-context benchmark, maintains strong contextual responsiveness and context-specific representation separation while keeping perturbation-induced changes highly localized. Inference-time YaRN scaling recovers much of the contextual organization lost by direct short-context extrapolation, but induces substantially greater distal representation diffusion. Frozen RNA-type evaluations show that RIBOSPAN learns state-of-the-art RNA representations, with a particularly clear advantage on long RNAs. Across downstream biological benchmarks, RIBOSPAN emerges as the strongest encoder-only RNA foundation model, achieving state-of-the-art performance in both full-transcript biological property prediction and zero-shot mutation-fitness modeling. Building on the same backbone, we develop a multidimensionally conditioned discrete-diffusion framework for full-length mRNA generation and redesign, including synonymous-codon diffusion for protein-preserving CDS optimization. Together, RIBOSPAN establishes a powerful long-context foundation for transferable RNA representation learning, biological prediction, and full-transcript mRNA design.

Figures & tables

Setting	Configuration	Setting	Configuration
Transformer Layers	32	Model Dimension	2,048
FFN Intermediate size	5,440	Attention Heads	32
Activation	SwiGLU	Normalization	LayerNorm
Position Encoding	RoPE (rotary dim = 64)	Head Dimension	64
Vocabulary Size	16	Token Unit	nucleotide
Native Context Length	10,240 tokens	Parameters	1.61B

Table 1: Backbone configuration of RiboSpan .

Figure 1: Architecture and downstream use of RiboSpan . Single-nucleotide RNA sequences are encoded by 32 bidirectional Transformer layers into transferable nucleotide-level representations for downstream RNA modeling.

Figure 2: Sequence-length distributions and RNA-class composition of the pretraining corpus.

RNA Class	Training	Validation	Test
rRNA	30,748,160	10,000	10,000
mRNA	18,316,110	10,000	10,000
tRNA	6,135,163	10,000	10,000
miscRNA	3,966,721	10,000	10,000
lncRNA	3,818,804	10,000	10,000
others	940,161	5,000	5,000

Table 2: RNA-class composition of the pretraining corpus and held-out sets.

Model Variant	Native Context Length	Masking Rate	Training Stage
RiboSpan -10K-15	10,240	15%	Pretraining
RiboSpan -10K-40	10,240	40%	Continued pretraining from 10K-15
RiboSpan -1K-15	1,024	15%	Pretraining
RiboSpan -1K-40	1,024	40%	Continued pretraining from 1K-15

Table 3: RiboSpan pretraining variants.

	15% Masking		40% Masking
Model	1,024 tokens	10,240 tokens	1,024 tokens	10,240 tokens
AIDO.RNA-CDS ( Zou et al., 2024 )	1.08489	1.15072	1.13499	1.19024
RiboSpan -1K-15	0.67434	0.98153	0.93860	1.27385
RiboSpan -1K-40	0.68036	1.02138	0.77438	1.06730
RiboSpan -10K-15	0.76147	0.72417	0.99496	0.91122
RiboSpan -10K-40	0.75660	0.72519	0.88394	0.80033

Table 4: Masked language modeling loss on the mRNA subset of the pretraining test split. Bold and underlined values in result tables indicate the best and second-best performance, respectively.

	15% Masking		40% Masking
Model	1,024 tokens	10,240 tokens	1,024 tokens	10,240 tokens
AIDO.RNA-CDS ( Zou et al., 2024 )	0.91953	0.91576	0.79203	0.77791
RiboSpan -1K-15	0.94973	0.93083	0.83343	0.77168
RiboSpan -1K-40	0.94602	0.92589	0.86076	0.80264
RiboSpan -10K-15	0.94349	0.94763	0.82015	0.83769
RiboSpan -10K-40	0.94163	0.94517	0.84094	0.85887

Table 5: Global nucleotide reconstruction accuracy on the mRNA subset of the pretraining test split. Bold and underlined values in result tables indicate the best and second-best performance, respectively.

Model	Setting	$\Delta\mathit{CS}\,\uparrow$	$C_{\mathrm{cross}}\,\downarrow$	$D_{\mathrm{distal}}$
HydraRNA ( Li et al., 2025 )	Direct	0.313846	0.521605	0.000667
AIDO.RNA-CDS ( Zou et al., 2024 )	Base	0.208178	0.660208	0.004666
AIDO.RNA-CDS ( Zou et al., 2024 )	YaRN	0.446120	0.317370	0.025390
RiboSpan -1K-15	Base	0.221320	0.698757	0.006626
RiboSpan -1K-15	YaRN	0.451068	0.334152	0.016084
RiboSpan -1K-40	Base	0.186200	0.744354	0.004569

Table 6: Final-layer long-context representation metrics at 10,240 nt. Bold and underlined values in result tables indicate the best and second-best performance, respectively.

Figure 3: Length sweep of final-layer long-context representation metrics. $\Delta\mathit{CS}$ , $C_{\mathrm{cross}}$ , and $D_{\mathrm{distal}}$ are shown across input lengths; error bars denote 95% bootstrap confidence intervals.

Model	Metric	Overall Biotype	Functional	Regulatory	Long RNA	Rfam
RNA-FM ( Chen et al., 2022 )	Acc.	0.865033	0.954608	0.953270	0.809924	0.990485
RNA-FM ( Chen et al., 2022 )	Purity	0.819314	0.939623	0.941489	0.727898	0.986791
RiNALMo ( Penić et al., 2025 )	Acc.	0.857395	0.944081	0.952534	0.755867	0.985000
RiNALMo ( Penić et al., 2025 )	Purity	0.810520	0.927728	0.936635	0.673561	0.978312
AIDO.RNA-CDS ( Zou et al., 2024 )	Acc.	0.805125	0.954296	0.870781	0.841973	0.952242
AIDO.RNA-CDS ( Zou et al., 2024 )	Purity	0.735243	0.935233	0.813092	0.770362	0.936439

Table 7: RNA-type representation quality measured by leave-one-out 10-NN accuracy and neighborhood purity. Bold and underlined values in result tables indicate the best and second-best performance, respectively.

Figure 4: t-SNE visualization of frozen RNA sequence representations across Overall Biotype, Functional, Regulatory, Long RNA ( $>1{,}024$ nt), and Rfam label spaces.

Model	Model Type	Spearman $r$	AUROC	MCC
RNA-FM ( Chen et al., 2022 )	Encoder-only (MLM)	0.1674	0.5874	0.1279
RiNALMo ( Penić et al., 2025 )		0.2080	0.6051	0.1512
AIDO.RNA-CDS ( Zou et al., 2024 )		0.1997	0.6024	0.1618
HydraRNA ( Li et al., 2025 )		0.1921	0.5989	0.1508
RiboSpan -1K-15	Encoder-only (MLM)	0.2409	0.6225	0.1828
RiboSpan -1K-40		0.2524	0.6280	0.1904

Table 8: Zero-shot mutation-fitness performance on RNAGym. Bold and underlined values in the results indicate the best and second-best performance among encoder-only models , respectively; Evo2-7B results reported by RNAGym ( Arora et al., 2025 ) are included as an external decoder-only reference.

Model	HL	MRL	TE	eCLIP	GO	VEP
Metric	Pearson $r$	Pearson $r$	Pearson $r$	AUPRC	AUPRC	AUPRC
RNA-FM ( Chen et al., 2022 )	0.4660	0.2928	0.6045	0.3507	0.3337	0.2835
RiNALMo ( Penić et al., 2025 )	0.4671	0.2802	0.6112	0.3613	0.3435	0.3253
AIDO.RNA-CDS ( Zou et al., 2024 )	0.5613	0.3711	0.6769	0.4057	0.4449	0.3300
HydraRNA ( Li et al., 2025 )	0.6228	0.4605	0.7505	0.4611	0.4907	0.3376
RiboSpan -1K-15	0.6248	0.4637	0.7247	0.4374	0.4689	0.3480

Table 9: Frozen full-transcript linear-probe performance on mRNABench. Bold and underlined values indicate the best and second-best performance, respectively.

Figure 5: Overview of the RiboSpan -based conditional discrete-diffusion framework for full-length mRNA generation and redesign. Diffusion timesteps and design conditions are incorporated through AdaLN-Zero-conditioned blocks to guide nucleotide denoising, with synonymous-codon constraints for CDS-preserving optimization.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Source	Sequences	Nucleotide Tokens
Training	RNAcentral v26.0	49,250,975	34,111,085,735
Training	Ensembl 115 & Ensembl Genomes 62	18,316,110	51,603,244,706
Validation	Combined	90,000	77,781,035
Test	Combined	90,000	76,893,030

Appendix

Table 10: Pretraining corpus statistics by split and source.

Criterion	Requirement
Annotation	Protein-coding transcript annotation and, when available, protein-coding gene annotation; neither is annotated as a pseudogene.
Completeness	CDS, start codon, stop codon, 5 ′ UTR, and 3 ′ UTR annotations are present.
Coding consistency	CDS length is positive and divisible by 3, with consistent start- and stop-codon annotations.
Sequence quality	$\geq 20$ nt, $\leq 5\%$ N, and no homopolymer $>50$ nt.

Appendix

Table 11: Filtering criteria for Ensembl protein-coding transcripts.

Setting	15% MLM	40% MLM Continuation
Epochs	6	2
Global Batch Size	2,048 sequences	2,048 sequences
Optimizer	AdamW	AdamW
Peak LR	$5\times 10^{-5}$	$1\times 10^{-5}$
Min LR	$1\times 10^{-5}$	$1\times 10^{-6}$
LR Scheduler	cosine	cosine

Appendix

Table 12: Hyperparameters for RiboSpan pretraining.

Model	1,024	2,048	4,096	8,192
Panel A. Additional Context Separation ( $\Delta\mathit{CS}\,\uparrow$ )
HydraRNA	0.034261	0.132366	0.258054	0.260502
AIDO-CDS	0.042053	0.209758	0.385709	0.295581
AIDO-CDS-YaRN	0.041869	0.200108	0.375539	0.406480
1K-15	0.061452	0.145217	0.327177	0.234125
1K-15-YaRN	0.061353	0.155104	0.325023	0.374779

Appendix

Table 13: Long-context representation metrics at 1,024–8,192 nt.

Model	$r_{i}\geq 0.25$	$r_{i}\geq 0.50$	$r_{i}\geq 0.75$
HydraRNA	0.000636	0.000632	0.000667
AIDO-CDS	0.004527	0.004939	0.004666
AIDO-CDS-YaRN	0.030155	0.027632	0.025390
1K-15	0.006011	0.006095	0.006626
1K-15-YaRN	0.018542	0.016600	0.016084
1K-40	0.004245	0.004344	0.004569

Appendix

Table 14: Sensitivity of Relative-distal Diffusion to the normalized-distance threshold at 10,240 nt.

Model A	Model B	$\Delta$ (B $-$ A)	95% CI	$d_{z}$	BH $q$
Panel A. Additional Context Separation ( $\Delta\mathit{CS}\uparrow$ )
AIDO-CDS	AIDO-CDS-YaRN	0.237942	$[0.221491,\,0.252612]$	9.011	$<0.002$
AIDO-CDS-YaRN	10K-15	-0.040159	$[-0.055810,\,-0.025636]$	-1.472	$<0.002$
AIDO-CDS-YaRN	HydraRNA	-0.132275	$[-0.149472,\,-0.115740]$	-4.645	$<0.002$
HydraRNA	10K-15	0.092116	$[0.067625,\,0.116443]$	2.199	$<0.002$
1K-15	1K-15-YaRN	0.229748	$[0.202010,\,0.257162]$	4.966	$<0.002$

Appendix

Table 15: Paired model comparisons at 10,240 nt.

Model	Layers	Hidden Dim.	Context Length (tokens)
RNA-FM ( Chen et al., 2022 )	12	640	1,024
RiNALMo ( Penić et al., 2025 )	33	1,280	1,024
AIDO.RNA-CDS ( Zou et al., 2024 )	32	2,048	1,024
HydraRNA ( Li et al., 2025 )	12	1,024	10,240
RiboSpan -10K-15 / 10K-40	32	2,048	10,240

Appendix

Table 16: Model configurations used in the RNA-type representation benchmark.

RNA Type	Test-set $n$	Long RNA $n$	Functional Class
lncRNA	10,000	4,191	Regulatory
mRNA	10,000	8,771	Coding
miscRNA	10,000	1,113	–
rRNA	10,000	2,031	Housekeeping
tRNA	10,000	–	Housekeeping
SRP RNA	5,000	–	–

Appendix

Table 17: RNA-type composition across Overall, Functional, and Long RNA evaluations.

Class	$n$	RNA-FM	RiNALMo	AIDO-CDS	HydraRNA	10K-15	10K-40
Coding	10,000	0.870860	0.832980	0.872060	0.939680	0.930560	0.932130
Housekeeping	20,997	0.988627	0.989289	0.963104	0.987146	0.989822	0.990008
Regulatory	29,895	0.928205	0.916183	0.936789	0.964854	0.972514	0.972574

Appendix

Table 18: Class-level neighborhood purity in the Functional evaluation. Bold and underlined values in result tables indicate the best and second-best performance, respectively.

Model	Regulatory $\rightarrow$ Coding	Coding $\rightarrow$ Regulatory	Total
RNA-FM	1,770	732	2,502
RiNALMo	2,241	880	3,121
AIDO-CDS	1,019	871	1,890
HydraRNA	771	352	1,123
10K-15	516	427	943
10K-40	519	415	934

Appendix

Table 19: Major confusion counts in the Functional evaluation. Bold and underlined values in result tables indicate the best and second-best performance, respectively.

RNA Type	$n$	RNA-FM	RiNALMo	AIDO-CDS	HydraRNA	10K-15	10K-40
Antisense RNA	56	0.033929	0.008929	0.028571	0.039286	0.057143	0.051786
lncRNA	4,191	0.622429	0.489167	0.685898	0.763803	0.782582	0.784586
mRNA	8,771	0.838627	0.802679	0.862171	0.916133	0.917649	0.918915
miscRNA	1,113	0.352022	0.336208	0.584097	0.681491	0.727853	0.723270
ncRNA	83	0.375904	0.295181	0.271084	0.309639	0.292771	0.302410
rRNA	2,031	0.941310	0.949926	0.948104	0.966420	0.947070	0.945987

Appendix

Table 20: Class-level neighborhood purity in the Long RNA evaluation. Bold and underlined values in result tables indicate the best and second-best performance, respectively.

Model	mRNA $\rightarrow$ lncRNA	lncRNA $\rightarrow$ mRNA	miscRNA $\rightarrow$ mRNA	miscRNA $\rightarrow$ lncRNA	Total
RNA-FM	619	766	619	129	2,133
RiNALMo	784	1,495	630	124	3,033
AIDO-CDS	686	452	264	156	1,558
HydraRNA	289	291	243	72	895
10K-15	345	210	199	83	837
10K-40	330	203	204	79	816

Appendix

Table 21: Major confusion counts in the Long RNA evaluation. Bold and underlined values in result tables indicate the best and second-best performance, respectively.

RNA Type	Description	Assays	Mutants	Single	Multiple	WT Length (nt)
Aptamer	Target binding ability	2	3,069	446	2,623	70–82
Ribozyme	Cleavage and splicing	26	758,333	4,118	754,215	45–425
Transfer RNA (tRNA)	Stability and growth	3	95,202	434	94,768	72–105
Messenger RNA (mRNA)	Splicing ability	2	5,722	330	5,392	51–63
Messenger RNA (mRNA)	Coding mRNA fitness	37	254,969	40,468	214,501	144–5,592
Total		70	1,117,295	45,796	1,071,499	45–5,592

Appendix

Table 22: Composition of the RNAGym mutation-fitness evaluation.

Model	Metric	mRNA		tRNA	Aptamer	Ribozyme
Model	Metric	Splicing	Fitness
RNA-FM	Spearman $r$	0.2488	0.0582	0.3857	0.0150	0.1292
	AUROC	0.6320	0.5331	0.6920	0.5156	0.5646
	MCC	0.1963	0.0479	0.2902	0.0072	0.0982
RiNALMo	Spearman $r$	0.3557	0.0630	0.4581	0.0463	0.1167
	AUROC	0.6881	0.5312	0.7247	0.5231	0.5582

Appendix

Table 23: RNAGym performance by RNA category. Bold and underlined values indicate the best and second-best performance among the evaluated encoder-only models, respectively.

Family	Task Type	Unique Subtasks	Mean Length (nt)	Max Length (nt)	Metric
HL	Regression	2	3,549	12,288	Pearson $r$
MRL	Regression	1	2,620	12,275	Pearson $r$
TE	Regression	2	3,723	17,497	Pearson $r$
eCLIP	Binary classification	40	3,370	205,012	AUPRC
GO	Multilabel classification	3	3,522	12,258	micro-AUPRC
VEP	Binary classification	2	3,659	28,227	AUPRC

Appendix

Table 24: Task composition of the mRNABench full-transcript evaluation.

Long-Context Modeling

Long-context modeling studies how AI systems process, retrieve, and reason over sequences that exceed ordinary context windows.

See all 136 papers

Scaling RNA Foundation Models through Iterative Refinement
Oct 9, 2026Pengan Chen, Liang Hong, Xinyuan Liu +6Iterative Refinement
Department of Computer Science and Engineering, The Chinese University of Hong Kong
RiboSphere: Learning Unified and Efficient Representations of RNA Structures
Mar 20, 2026Zhou Zhang, Hanqun Cao, Cheng Tan +3Molecular Representation Learning Representation Learning
State Key Laboratory for Novel Software Technology at Nanjing University, School of Computer Science, Nanjing University · The Chinese University of Hong Kong · Stanford University
Fair splits flip the leaderboard: CHANRG reveals limited generalization in RNA secondary-structure prediction
Mar 20, 2026Zhiyuan Chen, Zhenfeng Deng, Pan Deng +4OOD Generalization
Department of Electrical and Computer Engineering, The University of Hong Kong, Pok Fu Lam, Hong Kong Island, Hong Kong SAR, China. · AI4Science Center, Shanghai Artificial Intelligence Laboratory, 129 Longwen Road, Xuhui, 200232, Shanghai, China. · Department of Pathology, Stanford Medicine, 265 Campus Drive, Stanford, 94305, Califorina, US. +7

Oct 9, 2026J/K move · Enter open · S save

Scaling RNA Foundation Models through Iterative Refinement

Pengan Chen, Liang Hong, Xinyuan Liu, Han Huang+5

Department of Computer Science and Engineering, The Chinese University of Hong Kong

Iterative Refinement

Open paper AskPDF arXiv

Biological function in RNA is encoded across scales, from local sequence motifs to long-range structural dependencies. Conventional RNA foundation models couple parameter capacity and inference depth within a fixed architecture, prescribing a fixed computation graph for each inference. Here we present Eco-RNA, an 80M-parameter RNA foundation model that makes inference depth adjustable through learned iterative refinement. Its shared, input-conditioned Transformer core refines sequence representations through repeated application. The iteration count is selected after training and controls effective depth at fixed model parameters. We train across recurrent depths with masked reconstruction, state regularization, and a one-sided loss that anchors deeper reconstruction to a shallow reference. On the 70-assay RNAGym fitness benchmark, increasing refinement from one to four iterations raises aggregate absolute Spearman correlation from 0.242 to 0.286, with the gain retained through eight iterations. At four iterations, Eco-RNA surpasses the current SOTA Evo 2 40B on all three aggregate fitness metrics despite 500x fewer parameters. Its iteration configurations advance the compute-performance Pareto frontier among the evaluated models. It also outperforms the evaluated baselines in non-coding RNA family classification, RNA secondary-structure prediction, and chemical probing profile prediction. These results establish internal inference computation as an effective scaling dimension for RNA foundation models.

Abstract

Long-Context Modeling

Explore similar work

Scaling RNA Foundation Models through Iterative Refinement

RiboSphere: Learning Unified and Efficient Representations of RNA Structures

Fair splits flip the leaderboard: CHANRG reveals limited generalization in RNA secondary-structure prediction