JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

Ariel Larey; Elay Dahan; Amit Bleiweiss; Raizy Kellerman; Guy Leib; Omri Nayshool; Dan Ofer; Tal Zinger

cs.AIFeb 19, 2026

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

Authors: Ariel Larey, Elay Dahan, Amit Bleiweiss, Raizy Kellerman, Guy Leib, Omri Nayshool, Dan Ofer, Tal Zinger, +10 more

Organizations: Applied AI Architecture, NVIDIA, Israel. · Worldwide Field Ops, NVIDIA, Israel. · Developer Programs, NVIDIA, Israel. · Cancer Research Center and Wohl Institute of Translational Medicine, Sheba Medical Center, Tel Hashomer, Israel. · Windreich Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, USA.

Abstract

Genomic Foundation Models (GFMs) typically rely on Masked Language Modeling (MLM) or Next-Token Prediction (NTP) to learn the "Laws of Nature". While effective at capturing local syntax, these generative paradigms prioritize token-level reconstruction over high-level functional context. We introduce JEPA-DNA, a model-agnostic continual training framework that integrates a Joint-Embedding Predictive Architecture (JEPA) with traditional generative objectives. By supervising global sequence embeddings in a latent space, JEPA-DNA forces models to predict the functional representations of masked genomic segments, shifting the learning signal from token recovery to semantic alignment. We evaluate JEPA-DNA on 17 diverse genomic benchmark tasks, demonstrating consistent gains in linear probing and zero-shot performance regardless of the underlying GFM architecture or generative objective. Our framework establishes a new state-of-the-art for GFMs, surpassing the best existing models by bridging generative precision with latent semantic grounding. Through extensive ablation studies, we further characterize the synergistic interplay between generative and latent objectives. Our code is publicly available at https://github.com/NVIDIA-Digital-Bio/JEPA-DNA.

Figures & tables

Task	Baseline			JEPA-DNA (Ours)			Gain
	MCC	AUROC	AUPRC	MCC	AUROC	AUPRC	% AUROC
DNABERT-2 (Transformer + MLM + CLS; Multi-Species DS)
GUE TF Binding	0.429	0.788	0.773	0.515	0.839	0.828	+6.47%
GUE Promoter	0.607	0.884	0.877	0.717	0.928	0.913	+4.98%
GUE Splice Site	0.000	0.653	0.452	0.139	0.705	0.500	+7.96%
VB Coding Path.	0.002	0.624	0.513	0.078	0.610	0.509	-2.24%

Table 1: Linear probing performance across diverse GFM backbones, objectives, aggregation strategies, and self-supervised datasets (DS). Bold values indicating the top performance within each backbone group.

Task	Len.	Best Baseline GFM	Best JEPA-DNA (Ours)
TF Binding	100	0.837	0.849
Promoter	300	0.907	0.937
Splice Site	400	0.713	0.733
Coding Pathogenicity	1024	0.668	0.675
Expression Effect	1024	0.674	0.689
Common vs. Rare	1024	0.589	0.510

Table 2: Linear probing comparison between the top-performing state-of-the-art GFMs and their corresponding JEPA-DNA augmented counterparts. Bold values indicate the overall state-of-the-art.

Task	DNABERT-2		JEPA-DNA (Ours)		Gain
	AUROC	AUPRC	AUROC	AUPRC	% AUROC
BEND Expression Effect	0.543	0.084	0.574	0.091	+5.71%
BEND Disease Variant	0.680	0.141	0.762	0.262	+12.06%
TraitGym Complex	0.517	0.106	0.503	0.100	-2.71%
TraitGym Mendelian	0.515	0.103	0.540	0.110	+4.85%
Songlab ClinVar	0.540	0.566	0.527	0.557	-2.41%

Table 3: Zero-shot performance of JEPA-DNA using the DNABERT-2 backbone compared to the original DNABERT-2 baseline. Bold values denote superior zero-shot performance.

Configuration	Components				AUROC			Avg. Gain
	MLM	JEPA	VICReg	Sched. Mask	Promoter	Splice	Express.	% AUROC
Baseline (DNABERT-2)	$\times$	$\times$	$\times$	$\times$	0.884	0.653	0.543	—
+MLM	✓	$\times$	$\times$	$\times$	0.929	0.674	0.532	+2.06%
+MLM+JEPA	✓	✓	$\times$	$\times$	0.927	0.669	0.529	+1.58%
+MLM+JEPA+SM	✓	✓	$\times$	✓	0.925	0.687	0.519	+1.81%
+MLM+JEPA+VR	✓	✓	✓	$\times$	0.928	0.689	0.565	+4.79%

Table 4: Evaluation of incremental impact of continual training with key components of the JEPA-DNA paradigm. Promoter, Splice, and Express denote GUE Promoter, GUE Splice Site, and BEND Expression Effect tasks, respectively. Bold values indicate the optimal configuration.

Predictor	GUE Promoter		GUE Splice Site		BEND Expression		Avg. Gain
Architecture	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC	% AUROC
Baseline	0.884	0.877	0.653	0.452	0.543	0.084	—
MLP	0.900	0.889	0.648	0.450	0.564	0.088	+1.64%
2 layers	0.928	0.911	0.691	0.485	0.561	0.087	+4.70%
4 layers (Ours)	0.928	0.913	0.705	0.500	0.574	0.091	+6.22%
6 layers	0.928	0.910	0.655	0.451	0.555	0.086	+2.50%

Table 5: Evaluation of predictor head’s architecture and depth. Bold values denote the best-performing configuration.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Task	DNABERT-2 (Baseline)		JEPA-DNA (Ours)		Gain
	AUROC	AUPRC	AUROC	AUPRC	% AUROC
GUE
TF Binding	0.787 $\pm$ .001	0.773 $\pm$ .001	0.839 $\pm$ .001	0.828 $\pm$ .001	+6.61%
Promoter	0.885 $\pm$ .001	0.877 $\pm$ .001	0.928 $\pm$ .000	0.913 $\pm$ .000	+4.86%
Splice Site	0.653 $\pm$ .001	0.451 $\pm$ .002	0.704 $\pm$ .002	0.500 $\pm$ .003	+7.81%
VB

Appendix

Table 6: Impact of JEPA-DNA on Representation Stability and Performance. We compare the DNABERT-2 baseline against its JEPA-DNA enhanced counterpart using linear probing across five random seeds. We report the mean AUROC and AUPRC with standard deviations. JEPA-DNA consistently improves both performance and stability (lower variance) across the majority of tasks, with relative AUROC gains of up to 7.8%. Bold values indicate the superior mean performance.

Task	Baseline		JEPA-DNA (Ours)		Gain
	AUROC	AUPRC	AUROC	AUPRC	% AUROC
DNABERT-2 (Transformer + MLM + CLS; Multi-Species DS)
BEND Expression Effect	0.543	0.084	0.574	0.091	+5.71%
BEND Disease Variant	0.680	0.141	0.762	0.262	+12.06%
TraitGym Complex	0.517	0.106	0.503	0.100	-2.71%
TraitGym Mendelian	0.515	0.103	0.540	0.110	+4.85%

Appendix

Table 7: We evaluate the zero-shot capabilities of JEPA-DNA augmented models against their original baselines. Bold values denote superior zero-shot performance within each backbone group.

Task	DNABERT	DNABERT-2	NTv3 (8M)	NTv3 (100M)	EVO 2 (1B)	HyenaDNA	Caduceus
GUE TF Binding	0.812	0.788	0.792	0.837	0.630	0.698	0.651
GUE Promoter	0.836	0.884	0.904	0.907	0.815	0.686	0.693
GUE Splice Site	0.574	0.653	0.640	0.713	0.657	0.506	0.418
VB Coding Pathogenicity	0.613	0.624	0.595	0.668	0.659	0.515	0.586
VB Expression Effect	0.666	0.674	0.617	0.642	0.628	0.580	0.632
VB Common vs. Rare	0.589	0.505	0.466	0.502	0.525	0.503	0.501

Appendix

Table 8: Performance comparison of supervised tasks for genomic foundation models. Multiple backbone models are evaluated under a linear probing setup on diverse genomic prediction benchmarks to identify the current state of the art for each task under these settings.

Optimizer	Promoter		Splice Site		Expression		Avg. Gain
	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC	% AUROC
Baseline	0.884	0.877	0.653	0.452	0.543	0.084	—
Adam [ kingma2014adam ]	0.776	0.777	0.523	0.351	0.518	0.080	-12.24%
AdamW [ loshchilov2017decoupled ]	0.799	0.795	0.589	0.388	0.576	0.091	-4.45%
AdaBelief [ zhuang2020adabelief ]	0.901	0.894	0.662	0.454	0.545	0.086	+1.22%
SGDM [ bottou2010large ]	0.928	0.913	0.705	0.500	0.574	0.091	+6.22%

Appendix

Table 9: Optimizer Sensitivity in JEPA-DNA Continual Pre-training. We evaluate the impact of different optimization algorithms on the stability and performance of JEPA-DNA using DNABERT-2 as the backbone. Bold values indicate peak performance.

Paradigm	Token	Global	Sched.	Promoter		Splice Site		Expression		Avg. Gain
	Type	JEPA	Mask	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC	% AUROC
MLM Only
MLM	MLM	$\times$	$\times$	0.929	0.914	0.674	0.467	0.532	0.084	+2.06%
MLM	MLM	$\times$	$\checkmark$	0.927	0.911	0.669	0.463	0.530	0.084	+1.67%
JEPA Only
JEPA	JEPA	$\checkmark$	$\checkmark$	0.919	0.906	0.680	0.474	0.562	0.089	+3.86%

Appendix

Table 10: Comparative analysis of training paradigms. All JEPA variants include the VICReg loss. When the token type is JEPA, the JEPA objective is applied to the representations of individual masked tokens, analogous to the I-JEPA formulation. Bold values indicate the optimal configuration.

Learning Paradigm	Promoter		Splice Site		Expression		Avg. Gain
	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC	% AUROC
DNASimCLR [ yang2024dnasimclr ]	0.922	0.912	0.600	0.407	0.549	0.085	—
JEPA-DNA (Ours)	0.928	0.913	0.705	0.500	0.574	0.091	+7.57%

Appendix

Table 11: DNASimCLR vs. JEPA-DNA across genomic tasks.

Figure 2: JEPA-DNA Embedding visualization colored by Ensembl annotations.

Figure 3: The JEPA-DNA architecture using a CLS-based representation, as implemented with DNABERT-2.

Explore similar work

VANDAM: Viewing a nucleotide sequence with DNA molecular priors
Sep 30, 2026Jeremy Levy, Ariel Larey, Yury Nahshan +18Self-Supervised Pre-Training Genomics
Applied AI Architecture, NVIDIA, Israel. · Worldwide Field Ops, NVIDIA, Israel. · Developer Programs, NVIDIA, Israel. +3
LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling
Jun 3, 2026Daria Ledneva, Denis KuznetsovHierarchical Representation Learning Genomics
Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia.
Entropy, Disagreement, and the Limits of Foundation Models in Genomics
Apr 5, 2026Maxime Rochkoulets, Lovro Vrček, Mile ŠikićGenomics Self-Supervised Pre-Training
Genome Institute of Singapore, A*STAR, Singapore · KU Leuven, Belgium · Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia

Sep 30, 2026J/K move · Enter open · S save

VANDAM: Viewing a nucleotide sequence with DNA molecular priors

Jeremy Levy, Ariel Larey, Yury Nahshan, Raizy Kellerman+17

Applied AI Architecture, NVIDIA, Israel. · Worldwide Field Ops, NVIDIA, Israel. · Developer Programs, NVIDIA, Israel.+3

Self-Supervised Pre-Training Genomics Sequence Modeling Molecular Representation Learning

Open paper AskPDF arXiv

Contemporary Genomic Foundation Models (GFMs) rely on a DNA-as-a-string paradigm that employs masked token prediction objectives for pretraining. However, this abstraction does not explicitly model the biochemical, structural, and physical properties essential to biological function. Many molecular properties can be estimated from sequence using established biophysical models, so their utility lies not in providing an independent modality, but in introducing priors that training objectives can explicitly exploit. We introduce VANDAM, a framework that extends the training of GFMs with DNA molecular priors. In self-supervised training, VANDAM predicts regional molecular properties from pooled representations. When functional labels are available and can reward retaining molecular priors, local features are additionally injected at the input. VANDAM consistently improves downstream performance across four architecture families and nine held-out genomic tasks by complementing token-based objectives. Probing experiments further demonstrate that the use of molecular priors generalizes to other unseen molecular properties.

Abstract

Explore similar work

VANDAM: Viewing a nucleotide sequence with DNA molecular priors

LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling

Entropy, Disagreement, and the Limits of Foundation Models in Genomics