RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts

Frederikke Isa Marin; Panagiotis Antoniadis; Dionysia Danai Brilli; Andreas Bjerregaard; Rachael DeVries; Yan Li; Ole Winther; Wouter Boomsma

cs.LGOct 8, 2026

RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts

Authors: Frederikke Isa Marin, Panagiotis Antoniadis, Dionysia Danai Brilli, Andreas Bjerregaard, Rachael DeVries, Yan Li, Ole Winther, Wouter Boomsma

Organizations: University of Copenhagen · Novo Nordisk A/S · Technical University of Denmark

Abstract

The genome holds the blueprint that governs the biological properties of the cell. Consequently, advancing our knowledge of genomic function is crucial both for a broader understanding of biology and for continued biomedical advances. The success of large language models on natural language and protein sequences has motivated similar efforts on genomic data. However, standard genomic language models (gLMs) often require extremely large computational resources and still fall behind traditional methods on some downstream tasks. Recently, MSA-based pretraining has been proposed as an efficient alternative, but existing models are limited to short input contexts, restricting their use to short-range tasks, such as variant effect prediction. In this work, we present RAGenome, the first retrieval-based gLM that scales pretraining to longer contexts (100 $\times$ longer than existing MSA-based gLMs), allowing it to capture both across-species evolutionary relationships and within-species longer-range interactions. Trained on whole-genome alignments from 100 vertebrates, RAGenome substantially improves the long-range capabilities of MSA-based gLMs, raising gene finding performance from 0.45 to 0.60, while remaining competitive on purely evolutionary-based tasks like prioritizing pathogenic variants. RAGenome provides competitive gLM performance at a fraction of the training cost, unifying evolutionary modeling and long-range capabilities within a single, flexible, scalable framework. Code is available at https://github.com/PanosAntoniadis/RAGenome.

Figures & tables

Figure 1: Overview of the retrieval mechanism of RAGenome. For each query sequence, we retrieve aligned homologous sequences from a WGA and remove gap tokens, preserving the original alignment column of each retained token as its positional embedding. We then concatenate all sequences, add taxonomy embeddings to distinguish species, and restrict retrieval to a fixed budget $B$ using the phylogenetic tree. Tokens are then masked per clade, and the resulting sequence is passed to the RAGenome Transformer, which predicts the masked query tokens.

Figure 2: Total number of input tokens as a function of context length

L

, comparing MSA-based methods ( Ye et al., 2026 ) that keep gap tokens against our proposed retrieval-based approach, across three genome coverage levels. The shaded region indicates the reduction in tokens achieved by removing gap tokens. The percentage of gaps in each panel is the total fraction of tokens that are gaps at that coverage level.

Figure 3: RAGenome Transformer alternates between two types of attention layers. Left: Intra-species local attention restricts each position to attend only within a local window neighborhood in the same species. Right: Inter-species cross-attention allows the query species (blue) to attend over all the retrieved sequences from the other species. For illustration, local attention windows are shown as non-overlapping blocks.

Figure 4: Performance of RAGenome on gene finding (left) and variant effect prediction (right) as a function of training compute. The colored regions indicate the pretraining stages presented in Section 4.4 . Gene finding performance improves gradually within each stage but increases sharply at each context-extension transition, indicating that longer input context is crucial for gene annotation. Variant effect prediction remains flat, reflecting that it relies primarily on local evolutionary conservation signals captured from the WGA.

Model	Parameters	Max length	MSA	$\sim$ GPU Hours	Gene finding	VEP
GPN-MSA	86M	128	✓	20	0.43	0.90
GPN-Star (V)	200M	128	✓	1,000	0.45	0.92
HyenaDNA	7M	1M		5,376	0.33	0.51
DNABERT-2 1	117M	10,000		12,000	0.41	–
GENA-LM 1, 2	336M	4,500		–	0.50	–
NT-MS	2.5B	5,994		215,000	0.66	0.57

Table 1: Results on Gene Finding from BEND ( Marin et al., 2024 ) and Variant Effect Prediction (VEP) from ClinVar ( Landrum et al., 2014 ) . Each model is evaluated at its native context length. MSA indicates whether the model uses a whole-genome alignment. For models present in the original BEND paper (HyenaDNA, DNABERT-2, GENA-LM and NT-MS), we evaluated them again ourselves, with only minor differences from the original numbers.

Figure 5: Per-class confusion matrices for RAGenome, NT-MS, and Evo2 on the gene finding task ( Marin et al., 2024 ) , normalized by true class. RAGenome outperforms NT-MS at identifying donor (D) and acceptor (A) splice sites on both the forward (F) and reverse (R) strand, while NT-MS is better at classifying introns (I). Evo2 outperforms both models on most classes given its much larger scale in model size, data, and compute.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: UMAP projection of the learned taxonomy embeddings for the 100 vertebrate species of the WGA.

Architecture
Parameters	168M
Hidden dimension	1024
Attention heads	16
Dimension per head	64
Transformer Layers	15 (12 local attention, 3 cross-attention)
Local attention ( $w$ )	512

Appendix

Table 2: Hyperparameters of RAGenome.

	Stage 1	Stage 2	Stage 3	Stage 4
Context length $L$	1,024	1,024	4,096	13,312
Genome coverage	top 5%	top 40%	top 40%	full
Retrieval budget $B$	24,000	24,000	80,000	80,000
Effective batch size	256	512	512	512
GPU hours	864	1,216	6,272	7,680

Appendix

Table 3: Hyperparameters of pretraining strategy.

Train context	Inference context	Gene finding (MCC)
1,024	1,024	0.47
4,096	1,024	0.51
4,096	4,096	0.54
13,312	1,024	0.53
	4,096	0.57
	13,312	0.60

Appendix

Table 4: Gene finding performance of RAGenome across different pretraining stages, evaluated at different context lengths.

Figure 7: Performance of RAGenome on gene finding (MCC, left) and variant effect prediction (AUROC, right), evaluated at different retrieval budgets

B

at inference time.

Explore similar work

Wisteria: A Unified Multi-Scale Feature Learning Framework for DNA Language Model
May 7, 2026Weihua Wang, Haoji Li, Feilong Bao +2Representation Learning Language Modeling
College of Computer Science, Inner Mongolia University, Hohhot, Inner Mongolia, 010021, China · National and Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, Hohhot, Inner Mongolia, 010021, China · Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology, Hohhot, Inner Mongolia, 010021, China +1
DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks
Jun 29, 2026Romain Karpinsky, Julien Mozziconacci, Mickaël DelceyLanguage Model Pretraining Fine-Tuning
CNRS, LORIA · Nancy, 54000, France · CNRS +1
GenoMorph: Pathway-Grounded Genomic Disease Reasoning via Adaptive Latent Computation
Sep 28, 2026Tanmoy Kanti Halder, Akash Ghosh, Arijit Roy +1LLM Reasoning with Graphs Genomics
Department of Computer Science and Engineering, Indian Institute of Technology Patna, Bihta, 801106, Bihar, India · Prasannadeb Women’s College, India

May 7, 2026, cs.AIJ/K move · Enter open · S save

Wisteria: A Unified Multi-Scale Feature Learning Framework for DNA Language Model

Weihua Wang, Haoji Li, Feilong Bao, Lei Yang+1

College of Computer Science, Inner Mongolia University, Hohhot, Inner Mongolia, 010021, China · National and Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, Hohhot, Inner Mongolia, 010021, China · Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology, Hohhot, Inner Mongolia, 010021, China+1

Representation Learning Language Modeling Genomics Sequence Modeling

Open paper AskPDF arXiv

DNA language model aims to decipher the regulatory grammar and semantic of genomes by capturing long range dependencies in DNA sequences. Existing methods emphasize long range token interactions but often ignore the interplay between local motifs and global dependencies. In this paper, we propose Wisteria, a genomic language model that integrates multi scale feature learning within a unified framework for DNA sequence. Specifically, Wisteria augments the Mamba based architecture with gated dilated convolutions to capture local motifs and regulatory patterns, while gated multilayer perceptrons refine global dependencies. We further introduce a Fourier based attention mechanism to support frequency domain modeling, periodic extension and length generalization. Across four experimental settings with both short and long range dependencies, Wisteria demonstrates strong performance on downstream benchmarks against competitive DNA language model baselines. These results indicate that Wisteria effectively unifies local and global dependency modeling for multi scale genomic sequence analysis.

Abstract

Explore similar work

Wisteria: A Unified Multi-Scale Feature Learning Framework for DNA Language Model

DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks

GenoMorph: Pathway-Grounded Genomic Disease Reasoning via Adaptive Latent Computation