q-bio.GNMar 6, 2026

How Private Are DNA Embeddings? Inverting Foundation Model Representations of Genomic Sequences

Authors: Sofiane Ouaari, Jules Kreuer, Nico Pfeifer

Organizations: Methods in Medical Informatics, Department of Computer Science, University of Tuebingen, Germany · Institute for Bioinformatics and Medical Informatics (IBMI), University of Tuebingen, Germany

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

DNA foundation models have become transformative tools in bioinformatics and healthcare applications. Trained on vast genomic datasets, these models can be used to generate sequence embeddings, dense vector representations that capture complex genomic information. These embeddings are increasingly being shared via Embeddings-as-a-Service (EaaS) frameworks to facilitate downstream tasks, while supposedly protecting the privacy of the underlying raw sequences. However, as this practice becomes more prevalent, the security of these representations is being called into question. This study evaluates the resilience of DNA foundation models to model inversion attacks, whereby adversaries attempt to reconstruct sensitive training data from model outputs. In our study, the model's output for reconstructing the DNA sequence is a zero-shot embedding, which is then fed to a decoder. We evaluated the privacy of three DNA foundation models: DNABERT-2, Evo 2, and Nucleotide Transformer v2 (NTv2). Our results show that per-token embeddings allow near-perfect sequence reconstruction across all models. For mean-pooled embeddings, reconstruction quality degrades as sequence length increases, though it remains substantially above random baselines. Evo 2 and NTv2 prove to be most vulnerable, especially for shorter sequences with reconstruction similarities > 90%, while DNABERT-2's BPE tokenization provides the greatest resilience. We found that the correlation between embedding similarity and sequence similarity was a key predictor of reconstruction success. Our findings emphasize the urgent need for privacy-aware design in genomic foundation models prior to their widespread deployment in EaaS settings. Training code, model weights and evaluation pipeline are released on: https://github.com/not-a-feature/DNA-Embedding-Inversion.

Figures & tables

Fig. 1: Overall Pipeline of the model inversion attack scenario on DNA Foundation Models shared embedding
Fig. 2: Left: Collision embedding analysis for mean embeddings of sequences of length l=100l=100 . The figure shows the normalised Euclidean distances of all pairwise combinations of a random subsample of 2,0002{,}000 unique sequences (see Appendix D for all sequence lengths). Right: Per-position reconstruction accuracy for per-token embeddings at sequence length l=100l=100 .
Fig. 3: Mean-pooled reconstruction performance across sequence lengths for the encoder-only architecture: (a) Levenshtein similarity and (b) nucleotide accuracy.
Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 4: Token count vs. sequence length for the three foundation models. Evo 2 (char-level) produces exactly ll tokens, NTv2 (single nt and 6-mer) follows a fixed compression ratio, and DNABERT-2 (BPE) exhibits variable, content-dependent tokenisation. Shaded regions indicate ±1\pm 1 standard deviation computed.
Fig. 5: Collision analysis: pairwise normalised Euclidean distance distributions for mean-pooled embeddings across all evaluated sequence lengths.
Fig. 6: UMAP projections of mean-pooled DNABERT-2 embeddings across all sequence lengths.
Fig. 7: UMAP projections of mean-pooled NTv2 embeddings across all sequence lengths.
Fig. 8: UMAP projections of mean-pooled Evo 2 embeddings across all sequence lengths.
Fig. 9: Euclidean distance vs. sequence similarity correlation for DNABERT-2 across all sequence lengths.
Fig. 10: Euclidean distance vs. sequence similarity correlation for NTv2 across all sequence lengths.
Fig. 11: Euclidean distance vs. sequence similarity correlation for Evo 2 across all sequence lengths.
ModelSimilarity10152025303540455060708090100
DNABERT-2Cosine0.01540.01540.02890.02890.03450.03450.04000.04000.04330.04330.04790.04790.06800.06800.06690.06690.06830.06830.07830.07830.08520.08520.09440.09440.12110.12110.11270.1127
Euclidean0.02240.02240.02830.02830.03840.03840.03990.03990.03810.03810.04540.04540.07170.07170.05710.05710.04600.04600.06700.06700.06710.06710.08800.08800.10630.10630.10050.1005
Evo 2Cosine0.1484\mathbf{0.1484}0.3775\mathbf{0.3775}0.4405\mathbf{0.4405}0.3260\mathbf{0.3260}0.2660\mathbf{0.2660}0.2307\mathbf{0.2307}0.2108\mathbf{0.2108}0.2096\mathbf{0.2096}0.1976\mathbf{0.1976}0.1847\mathbf{0.1847}0.17600.17600.16620.16620.17360.17360.16800.1680
Euclidean0.07030.07030.2986\mathbf{0.2986}0.4354\mathbf{0.4354}0.3176\mathbf{0.3176}0.2538\mathbf{0.2538}0.2164\mathbf{0.2164}0.19190.19190.1890\mathbf{0.1890}0.1730\mathbf{0.1730}0.15620.15620.14740.14740.13250.13250.14100.14100.13550.1355
NTv2Cosine0.09910.09910.09180.09180.08160.08160.09150.09150.06350.06350.13110.13110.18230.18230.14140.14140.13530.13530.14460.14460.1796\mathbf{0.1796}0.1747\mathbf{0.1747}0.1857\mathbf{0.1857}0.2226\mathbf{0.2226}
Euclidean0.1008\mathbf{0.1008}0.09500.09500.08690.08690.09840.09840.07170.07170.14870.14870.1992\mathbf{0.1992}0.15420.15420.14930.14930.1590\mathbf{0.1590}0.1896\mathbf{0.1896}0.1879\mathbf{0.1879}0.2012\mathbf{0.2012}0.2310\mathbf{0.2310}
Appendix
TABLE I: Spearman correlation between embedding similarity (cosine / Euclidean) and sequence similarity (Levenshtein) for different models and sequence lengths.
ModelKey HyperparametersApprox. Parameters
Encoderd=128d=128 , h=8h=8 , L=6L=6 , dff=1024d_{ff}=1024 , dropout =0.1=0.1≈12\approx 12 M
Decoderd=128d=128 , h=8h=8 , L=6L=6 , dff=1024d_{ff}=1024 , dropout =0.1=0.1≈12\approx 12 M
ResNetd=256d=256 , 4 blocks, k=5k=5≈22\approx 22 M
MLP (per-token)[256,256,128][256,256,128] , dropout =0.2=0.2≈0.3\approx 0.3 M
Nearest Neighbour non-parametric
Appendix
TABLE II: Inversion model architectures and approximate parameter counts (excluding input projection, which varies by foundation model embedding dimension).
Figure 14
Fig. 14: Mean-pooled reconstruction evaluation across sequence lengths. Left column: Levenshtein similarity; right column: nucleotide accuracy. Top row: DNABERT-2, middle row: NTv2, bottom row: Evo 2.
Figure 16
MethodMetric10152025303540455060708090100
DecoderLevenshtein0.45±0.190.45\pm 0.190.44±0.150.44\pm 0.150.45±0.110.45\pm 0.110.46±0.100.46\pm 0.100.45±0.100.45\pm 0.100.46±0.090.46\pm 0.090.46±0.080.46\pm 0.080.46±0.080.46\pm 0.080.46±0.080.46\pm 0.080.47±0.070.47\pm 0.070.47±0.070.47\pm 0.070.46±0.070.46\pm 0.070.46±0.060.46\pm 0.060.46±0.060.46\pm 0.06
Accuracy0.37±0.210.37\pm 0.210.33±0.160.33\pm 0.160.33±0.130.33\pm 0.130.32±0.120.32\pm 0.120.31±0.110.31\pm 0.110.31±0.100.31\pm 0.100.31±0.090.31\pm 0.090.31±0.090.31\pm 0.090.30±0.090.30\pm 0.090.30±0.080.30\pm 0.080.30±0.080.30\pm 0.080.29±0.070.29\pm 0.070.29±0.070.29\pm 0.070.29±0.070.29\pm 0.07
ResNetLevenshtein0.44±0.170.44\pm 0.170.44±0.120.44\pm 0.120.45±0.100.45\pm 0.100.46±0.090.46\pm 0.090.45±0.080.45\pm 0.080.46±0.080.46\pm 0.080.46±0.080.46\pm 0.080.46±0.070.46\pm 0.070.46±0.070.46\pm 0.070.46±0.060.46\pm 0.060.47±0.060.47\pm 0.060.46±0.060.46\pm 0.060.46±0.050.46\pm 0.050.47±0.050.47\pm 0.05
Accuracy0.34±0.200.34\pm 0.200.32±0.150.32\pm 0.150.31±0.120.31\pm 0.120.31±0.110.31\pm 0.110.30±0.100.30\pm 0.100.30±0.090.30\pm 0.090.30±0.090.30\pm 0.090.30±0.080.30\pm 0.080.30±0.080.30\pm 0.080.29±0.070.29\pm 0.070.29±0.070.29\pm 0.070.29±0.070.29\pm 0.070.29±0.060.29\pm 0.060.28±0.060.28\pm 0.06
EncoderLevenshtein0.46±0.190.46\pm 0.190.45±0.140.45\pm 0.140.46±0.110.46\pm 0.110.47±0.100.47\pm 0.100.47±0.090.47\pm 0.090.47±0.080.47\pm 0.080.47±0.080.47\pm 0.080.47±0.080.47\pm 0.080.48±0.070.48\pm 0.070.47±0.070.47\pm 0.070.48±0.060.48\pm 0.060.47±0.060.47\pm 0.060.47±0.060.47\pm 0.060.47±0.050.47\pm 0.05
Accuracy0.37±0.220.37\pm 0.220.34±0.160.34\pm 0.160.33±0.130.33\pm 0.130.32±0.120.32\pm 0.120.32±0.110.32\pm 0.110.31±0.100.31\pm 0.100.31±0.090.31\pm 0.090.31±0.090.31\pm 0.090.31±0.080.31\pm 0.080.30±0.080.30\pm 0.080.30±0.070.30\pm 0.070.29±0.070.29\pm 0.070.29±0.060.29\pm 0.060.29±0.060.29\pm 0.06
Appendix
TABLE III: Reconstruction results for DNABERT-2 embeddings.
MethodMetric10152025303540455060708090100
DecoderLevenshtein0.53±0.130.53\pm 0.130.96±0.050.96\pm 0.050.89±0.080.89\pm 0.080.82±0.080.82\pm 0.080.73±0.080.73\pm 0.080.67±0.080.67\pm 0.080.63±0.070.63\pm 0.070.56±0.070.56\pm 0.070.53±0.070.53\pm 0.070.50±0.060.50\pm 0.060.47±0.060.47\pm 0.060.45±0.060.45\pm 0.060.44±0.060.44\pm 0.060.43±0.050.43\pm 0.05
Accuracy0.51±0.140.51\pm 0.140.96±0.050.96\pm 0.050.87±0.100.87\pm 0.100.80±0.100.80\pm 0.100.70±0.100.70\pm 0.100.62±0.090.62\pm 0.090.58±0.090.58\pm 0.090.53±0.080.53\pm 0.080.50±0.080.50\pm 0.080.47±0.070.47\pm 0.070.44±0.060.44\pm 0.060.43±0.060.43\pm 0.060.42±0.060.42\pm 0.060.41±0.060.41\pm 0.06
ResNetLevenshtein0.55±0.120.55\pm 0.120.91±0.080.91\pm 0.080.86±0.080.86\pm 0.080.97±0.060.97\pm 0.060.91±0.090.91\pm 0.090.82±0.090.82\pm 0.090.71±0.080.71\pm 0.080.65±0.070.65\pm 0.070.62±0.060.62\pm 0.060.56±0.060.56\pm 0.060.52±0.050.52\pm 0.050.49±0.050.49\pm 0.050.46±0.050.46\pm 0.050.47±0.050.47\pm 0.05
Accuracy0.52±0.130.52\pm 0.130.90±0.080.90\pm 0.080.84±0.100.84\pm 0.100.96±0.080.96\pm 0.080.88±0.120.88\pm 0.120.77±0.130.77\pm 0.130.64±0.110.64\pm 0.110.57±0.100.57\pm 0.100.54±0.090.54\pm 0.090.49±0.080.49\pm 0.080.46±0.070.46\pm 0.070.44±0.070.44\pm 0.070.42±0.060.42\pm 0.060.41±0.060.41\pm 0.06
EncoderLevenshtein0.58±0.120.58\pm 0.120.98±0.040.98\pm 0.040.99±0.030.99\pm 0.030.97±0.070.97\pm 0.070.92±0.090.92\pm 0.090.85±0.100.85\pm 0.100.76±0.090.76\pm 0.090.71±0.080.71\pm 0.080.66±0.070.66\pm 0.070.59±0.060.59\pm 0.060.53±0.050.53\pm 0.050.50±0.050.50\pm 0.050.48±0.050.48\pm 0.050.46±0.050.46\pm 0.05
Accuracy0.56±0.130.56\pm 0.130.98±0.040.98\pm 0.040.99±0.040.99\pm 0.040.96±0.090.96\pm 0.090.89±0.120.89\pm 0.120.80±0.140.80\pm 0.140.70±0.120.70\pm 0.120.63±0.110.63\pm 0.110.58±0.100.58\pm 0.100.52±0.090.52\pm 0.090.47±0.070.47\pm 0.070.45±0.070.45\pm 0.070.43±0.060.43\pm 0.060.42±0.060.42\pm 0.06
Appendix
TABLE IV: Reconstruction results for Evo 2 embeddings.
MethodMetric10152025303540455060708090100
DecoderLevenshtein0.88±0.120.88\pm 0.120.87±0.100.87\pm 0.100.82±0.110.82\pm 0.110.81±0.100.81\pm 0.100.77±0.100.77\pm 0.100.65±0.080.65\pm 0.080.64±0.080.64\pm 0.080.64±0.080.64\pm 0.080.62±0.070.62\pm 0.070.60±0.070.60\pm 0.070.57±0.060.57\pm 0.060.56±0.060.56\pm 0.060.53±0.060.53\pm 0.060.53±0.060.53\pm 0.06
Accuracy0.88±0.130.88\pm 0.130.86±0.120.86\pm 0.120.81±0.130.81\pm 0.130.79±0.120.79\pm 0.120.75±0.120.75\pm 0.120.59±0.100.59\pm 0.100.57±0.100.57\pm 0.100.57±0.100.57\pm 0.100.54±0.100.54\pm 0.100.52±0.100.52\pm 0.100.46±0.080.46\pm 0.080.45±0.080.45\pm 0.080.41±0.080.41\pm 0.080.41±0.070.41\pm 0.07
ResNetLevenshtein0.87±0.120.87\pm 0.120.86±0.100.86\pm 0.100.80±0.110.80\pm 0.110.78±0.100.78\pm 0.100.73±0.090.73\pm 0.090.64±0.080.64\pm 0.080.63±0.080.63\pm 0.080.63±0.080.63\pm 0.080.62±0.070.62\pm 0.070.59±0.080.59\pm 0.080.59±0.070.59\pm 0.070.58±0.070.58\pm 0.070.55±0.070.55\pm 0.070.56±0.070.56\pm 0.07
Accuracy0.86±0.140.86\pm 0.140.85±0.120.85\pm 0.120.78±0.120.78\pm 0.120.75±0.120.75\pm 0.120.70±0.110.70\pm 0.110.58±0.100.58\pm 0.100.57±0.100.57\pm 0.100.56±0.100.56\pm 0.100.54±0.100.54\pm 0.100.49±0.100.49\pm 0.100.48±0.100.48\pm 0.100.46±0.100.46\pm 0.100.42±0.090.42\pm 0.090.43±0.100.43\pm 0.10
EncoderLevenshtein0.90±0.110.90\pm 0.110.88±0.100.88\pm 0.100.86±0.110.86\pm 0.110.85±0.100.85\pm 0.100.80±0.100.80\pm 0.100.69±0.090.69\pm 0.090.66±0.080.66\pm 0.080.66±0.080.66\pm 0.080.65±0.080.65\pm 0.080.60±0.070.60\pm 0.070.60±0.070.60\pm 0.070.59±0.060.59\pm 0.060.55±0.060.55\pm 0.060.57±0.060.57\pm 0.06
Accuracy0.89±0.120.89\pm 0.120.87±0.110.87\pm 0.110.85±0.120.85\pm 0.120.83±0.120.83\pm 0.120.78±0.110.78\pm 0.110.64±0.110.64\pm 0.110.60±0.110.60\pm 0.110.60±0.110.60\pm 0.110.58±0.100.58\pm 0.100.52±0.100.52\pm 0.100.50±0.090.50\pm 0.090.48±0.090.48\pm 0.090.43±0.080.43\pm 0.080.44±0.080.44\pm 0.08
Appendix
TABLE V: Reconstruction results for NTv2 embeddings.
Fig. 17: Reconstruction performance of mean embeddings for sequences of length l=10 to 100l=10\text{ to }100 for (a) DNABERT-2, (b) Evo 2, (c) NTv2 and the random baseline (gray).

Explore similar work

CardsList
Oct 11, 2026

What DNA Foundation Models Learn Beyond Sequence Composition

Do genomic foundation models (FMs) provide predictive information beyond simple sequence composition? We evaluate frozen representations from five DNA FMs against normalized k-mer frequencies across 57 genomic classification datasets and two exploratory gene-expression regression tasks. Simple k-mer features remain competitive with most models, whereas NTv3 shows a consistent advantage; the ranking is robust to globally fixed and multi-scale k-mer baselines and alternative downstream probes. To test incremental predictive utility directly, we compare global 6-mer features with their concatenation with each FM embedding under a matched Random Forest protocol. NTv3 improves 46 of 57 datasets (mean ΔΔMCC = +0.072), while Caduceus provides smaller gains and the other models do not improve mean MCC. We separately decompose embeddings into a component linearly predictable from 6-mer frequencies and a ridge residual, finding that linear recoverability does not reliably indicate downstream utility. Exact 6-mer-preserving shuffles further reveal task-relevant sequence-order sensitivity, most pronounced for NTv3, although order sensitivity alone does not guarantee predictive improvements. Our results establish compositional features as essential controls and offer complementary diagnostics for evaluating the added value of frozen genomic representations, without making claims about fine-tuning capacity.

Vincenzo Yuto Civale, Andrew David Bagdanov, Alberto Magi

University of Florence

Molecular Representation LearningGenomicsSequence Modeling
AskPDF
Apr 5, 2026

Entropy, Disagreement, and the Limits of Foundation Models in Genomics

Foundation models in genomics have shown mixed success compared to their counterparts in natural language processing. Yet, the reasons for their limited effectiveness remain poorly understood. In this work, we investigate the role of entropy as a fundamental factor limiting the capacities of such models to learn from their training data and develop foundational capabilities. We train ensembles of models on text and DNA sequences and analyze their predictions, static embeddings, and empirical Fisher information flow. We show that the high entropy of genomic sequences -- from the point of view of unseen token prediction -- leads to near-uniform output distributions, disagreement across models, and unstable static embeddings, even for models that are matched in architecture, training and data. We then demonstrate that models trained on DNA concentrate Fisher information in embedding layers, seemingly failing to exploit inter-token relationships. Our results suggest that self-supervised training from sequences alone may not be applicable to genomic data, calling into question the assumptions underlying current methodologies for training genomic foundation models.

Maxime Rochkoulets, Lovro Vrček, Mile Šikić

Genome Institute of Singapore, A*STAR, Singapore · KU Leuven, Belgium · Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia

GenomicsSelf-Supervised Pre-TrainingSequence ModelingMolecular Representation Learning
AskPDF
Feb 19, 2026

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

Genomic Foundation Models (GFMs) typically rely on Masked Language Modeling (MLM) or Next-Token Prediction (NTP) to learn the "Laws of Nature". While effective at capturing local syntax, these generative paradigms prioritize token-level reconstruction over high-level functional context. We introduce JEPA-DNA, a model-agnostic continual training framework that integrates a Joint-Embedding Predictive Architecture (JEPA) with traditional generative objectives. By supervising global sequence embeddings in a latent space, JEPA-DNA forces models to predict the functional representations of masked genomic segments, shifting the learning signal from token recovery to semantic alignment. We evaluate JEPA-DNA on 17 diverse genomic benchmark tasks, demonstrating consistent gains in linear probing and zero-shot performance regardless of the underlying GFM architecture or generative objective. Our framework establishes a new state-of-the-art for GFMs, surpassing the best existing models by bridging generative precision with latent semantic grounding. Through extensive ablation studies, we further characterize the synergistic interplay between generative and latent objectives. Our code is publicly available at https://github.com/NVIDIA-Digital-Bio/JEPA-DNA.

Ariel Larey, Elay Dahan, Amit Bleiweiss +15

Applied AI Architecture, NVIDIA, Israel. · Worldwide Field Ops, NVIDIA, Israel. · Developer Programs, NVIDIA, Israel. +2

Molecular Representation LearningGenomicsJoint-Embedding Predictive Architecture
AskPDF