What DNA Foundation Models Learn Beyond Sequence Composition

Vincenzo Yuto Civale; Andrew David Bagdanov; Alberto Magi

cs.LGOct 11, 2026

What DNA Foundation Models Learn Beyond Sequence Composition

Authors: Vincenzo Yuto Civale, Andrew David Bagdanov, Alberto Magi

Organizations: University of Florence

Abstract

Do genomic foundation models (FMs) provide predictive information beyond simple sequence composition? We evaluate frozen representations from five DNA FMs against normalized k-mer frequencies across 57 genomic classification datasets and two exploratory gene-expression regression tasks. Simple k-mer features remain competitive with most models, whereas NTv3 shows a consistent advantage; the ranking is robust to globally fixed and multi-scale k-mer baselines and alternative downstream probes. To test incremental predictive utility directly, we compare global 6-mer features with their concatenation with each FM embedding under a matched Random Forest protocol. NTv3 improves 46 of 57 datasets (mean $Δ$ MCC = +0.072), while Caduceus provides smaller gains and the other models do not improve mean MCC. We separately decompose embeddings into a component linearly predictable from 6-mer frequencies and a ridge residual, finding that linear recoverability does not reliably indicate downstream utility. Exact 6-mer-preserving shuffles further reveal task-relevant sequence-order sensitivity, most pronounced for NTv3, although order sensitivity alone does not guarantee predictive improvements. Our results establish compositional features as essential controls and offer complementary diagnostics for evaluating the added value of frozen genomic representations, without making claims about fine-tuning capacity.

Figures & tables

Panel of Figure 1 — Figure 1: Left: Schematic illustration of $k$ -mer frequency extraction (shown for $k=3$ for visual clarity) and frozen FM embedding extraction. Right: Dataset-level wins and losses across 57 classification datasets against a globally fixed $k=6$ compositional baseline.

Model	Fixed $k=4$	Fixed $k=5$	Fixed $k=6$	Best- $k$ oracle
NTv3-650M	16	16	13	20
HyenaDNA	48	45	40	49
DNABERT-2	52	53	44	54
Caduceus-Ph	27	30	26	32
Evo2-1B	53	52	52	53

Table 1: Datasets (of 57) where the indicated compositional baseline outperforms the standalone frozen FM embedding. Best-

k

denotes the retrospective task-specific oracle. These are loss counts for the FM , not independent test-set significance estimates.

Figure 2: Paired per-dataset MCC for each frozen embedding against the fixed 6-mer Random Forest baseline. Each of the 57 datasets appears once per model, colored by benchmark category. The dashed diagonal marks equal MCC; model-level win/loss/tie counts refer to full-precision paired results. The corresponding best-

k

oracle scatter is retained in the appendix.

Model	Mean $\Delta$ MCC	Median $\Delta$ MCC	Improved	Worse / tied
NTv3-650M	$+0.072$	$+0.055$	46/57	11 / 0
Caduceus-Ph	$+0.022$	$+0.012$	35/57	22 / 0
HyenaDNA	$-0.016$	$-0.017$	18/57	39 / 0
DNABERT-2	$-0.034$	$-0.042$	18/57	36 / 3
Evo2-1B	$-0.075$	$-0.065$	8/57	48 / 1

Table 2: Incremental utility in the T6 protocol, using exact 6-mer frequencies and frozen, mean-pooled FM embeddings. Differences are test MCC of [6-mer; FM] minus test MCC of 6-mer alone. Positive/negative/tie counts are based on unrounded dataset-level differences.

Figure 3: T6 incremental predictive utility across 57 paired datasets per model, measured as

\mathrm{MCC}([6\mathrm{-mer},\mathrm{FM}])-\mathrm{MCC}(6\mathrm{-mer})

on the held-out test set. Data use full-precision checkpoint metrics; reported summaries are rounded to three decimals. Paired differences correspond to the same dataset and fixed 6-mer reference. This plot uses the T6 Random Forest pipeline rather than the E8 rebuttal pipeline.

Figure 4: Dataset-paired E3 MCC differences: full minus projection, residual minus fixed 6-mer, full minus fixed 6-mer, and projection minus fixed 6-mer (an explicit fourth diagnostic contrast). The component classifiers use an E3 fixed 500-tree Random Forest without CV or scaling, whereas the standalone full and 6-mer values come from the original RF benchmark. Thus this plot is a descriptive mixed-protocol diagnostic, not the matched E4 comparison. Evo2 includes 56 common datasets because one residual record is missing; the other models include 57. Medians and sign counts are computed before rounding.

Figure 5: MCC change (original minus shuffled) under exact preservation of global 6-mer counts, for full embeddings, ridge residuals, ridge-predicted components, and 6-mer controls, using a linear downstream probe (training-fitted scaler, fixed

C=1

) and ridge

\alpha=1

. Three shuffle seeds are averaged within each of 57 datasets, and results are shown separately for frozen and retrained probes. The 6-mer input is invariant; small changes of the retrained predicted component arise from refitting rather than changed composition. No Caduceus or Evo2 perturbation results are claimed. Bars report the median dataset-level MCC decrease across 57 datasets after averaging three shuffle seeds within each dataset. Mean decreases are reported separately in Section 4.4 and Appendix A.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Mean $\Delta$ MCC	Median $\Delta$ MCC	Improved datasets
NTv3	$+0.071$	$+0.053$	45/57
Caduceus	$+0.022$	$+0.010$	35/57
HyenaDNA	$-0.016$	$-0.023$	18/57
DNABERT-2	$-0.035$	$-0.042$	18/57
Evo2	$-0.076$	$-0.056$	8/57

Appendix

Table 3: E8 paired incremental MCC versus fixed global 6-mer frequencies; numbers are not pooled with the T6 results. All models have 57 paired datasets.

Model	Random Forest	L2 linear	MLP
NTv3-650M	$+0.061$	$+0.096$	$+0.069$
HyenaDNA	$-0.052$	$-0.009$	$-0.016$
DNABERT-2	$-0.088$	$-0.078$	$-0.089$
Caduceus-Ph	$-0.006$	$+0.012$	$+0.003$
Evo2-1B	$-0.222$	$-0.260$	$-0.260$

Appendix

Table 4: Mean paired MCC difference for standalone FM embedding minus fixed 6-mer features (positive favors FM). Results from the rebuttal; these are not the retrospective best-

k

contrasts in Table 9 .

Model	Mean	Max	Mean+max	CLS	Last	Attention
NTv3	.602	.565	.601	–	–	.441
HyenaDNA	.497	.411	.490	–	.331	.491
DNABERT-2	.428	.381	.423	.463	–	.428
Caduceus	.518	.465	.522	–	.403	.517
Evo2	.245	.295	.279	–	–	.245

Appendix

Table 5: Mean MCC under alternative token aggregation choices, as reported in the rebuttal. CLS and last-token columns are only available for the indicated models; these means must not be interpreted as matched 57-task comparisons when coverage differs.

Figure 6: Distribution of held-out average per-dimension ridge

R^{2}

values for the fixed 6-mer predictor of each FM embedding. This measures linear predictability from the specified compositional basis; it does not quantify genuinely non-compositional information or establish downstream relevance.

Model	Residual 6-mer	Residual multi- $k$
NTv3	.487	.486
HyenaDNA	.269	.341
DNABERT-2	.297	.270
Caduceus	.336	.373
Evo2	.162	.102

Appendix

Table 6: Median standalone ridge-residual MCC with a fixed 6-mer mapping versus a concatenated

k=4,5,6

mapping. The residual is conditional on the mapping and is not a pure non-compositional feature.

Protocol	NTv3	HyenaDNA	DNABERT-2	NTv3–DNABERT-2 [95% CI]
Frozen probe	.143	.063	.024	.119 [.083, .160]
Retrained probe	.079	.030	.013	.066 [.044, .090]

Appendix

Table 7: Mean MCC decreases under exact 6-mer-preserving shuffles;

\Delta

is original minus shuffled performance. Bootstrap confidence intervals for the NTv3–DNABERT-2 difference are as reported in the rebuttal.

Model	Spearman $\rho$	BH $p$
NTv3	.235	.390
HyenaDNA	-.106	.721
DNABERT-2	.143	.721
Caduceus	-.049	.895
Evo2	-.001	.996

Appendix

Table 8: Across 57 datasets, dataset-level Spearman correlation between sequence length and FM advantage over fixed 6-mer features, with Benjamini–Hochberg adjusted

p

values. No association passes the specified multiple-test control.

Figure 7: Original classification comparison against the retrospectively chosen best

k

on each dataset. This figure is retained for sensitivity comparison, not as the principal fixed-

k

result.

Model	Mean $\Delta$	Median $\Delta$	Wins / Losses	Wilcoxon $p$ (FDR)	Mixed-effects $p$
NTv3-650M	$+0.027$	$+0.024$	36 / 20	$0.007$	$0.081$
HyenaDNA-160k	$-0.087$	$-0.074$	8 / 49	$1.90\times 10^{-8}$	$1.44\times 10^{-8}$
DNABERT-2-117M	$-0.108$	$-0.095$	3 / 54	$8.43\times 10^{-10}$	$1.35\times 10^{-12}$
Caduceus-Ph	$-0.042$	$-0.013$	25 / 32	$0.021$	$0.006$
Evo2-1B	$-0.266$	$-0.277$	4 / 53	$5.39\times 10^{-10}$	$1.62\times 10^{-67}$

Appendix

Table 9: Statistical comparison of FM vs. best k-mer MCC across 57 classification datasets.

\Delta=\text{MCC(FM)}-\text{MCC(best k-mer)}

. Wilcoxon

p

-values are FDR-corrected across five comparisons using the Benjamini–Hochberg procedure. Mixed-effects estimates share the same

\Delta

point estimate by construction; the

p

-value reflects significance after absorbing dataset-level variability via random intercepts.

Figure 8: Linear probe (LP) MCC vs. Random Forest (RF) MCC across 57 classification datasets for five foundation models and the best k-mer baseline, coloured by biological category. Points above the diagonal favour LP; points below favour RF. NTv3 and k-mers show no systematic advantage for either classifier. HyenaDNA and DNABERT-2 are significantly better under RF, while Caduceus benefits from LP.

Model	LP $>$ RF	RF $>$ LP	Mean $\Delta$	Median $\Delta$	$p$ -value
NTv3-650M	25	14	$+0.009$	$+0.006$	$0.094$
HyenaDNA	10	37	$-0.036$	$-0.032$	$<0.001^{***}$
DNABERT-2	4	43	$-0.041$	$-0.036$	$<0.001^{***}$
Caduceus	40	12	$+0.023$	$+0.023$	$0.002^{**}$
Evo2-1B	21	26	$-0.027$	$-0.006$	$0.121$
Best k-mer	24	21	$-0.004$	$+0.004$	$0.908$

Appendix

Table 10: Random Forest vs. linear probe comparison across 57 datasets.

\Delta=\text{LP}-\text{RF}

MCC. Positive values favour LP; negative values favour RF.

p

-values from two-sided Wilcoxon signed-rank test.

Figure 9: Distribution of Random Forest classification MCC across

k\in\{4,5,6\}

for each biological task category, using the historical standalone

k

-mer benchmark results. Each box summarizes the distribution of dataset-level MCC scores within a category; the central line indicates the median, the box spans the interquartile range, and points outside the whiskers denote outliers. The optimal

k

varies across tasks and categories. These descriptive results are distinct from the T6 incremental-utility experiment, which uses a globally fixed

k=6

reference and four-fold cross-validation for hyperparameter selection.

Figure 10: Historical canonical-versus-standard

k

-mer comparison across the classification datasets. This pooled illustration does not replace the stratified analysis: reverse-complement canonicalization is biologically appropriate only for the prespecified strand-invariant tasks.

#	Dataset	Best $k$ -mer	NTv3	HyenaDNA	DNABERT-2	Caduceus	Evo2
1	EMP/Yeast_H3	0.673	0.734	0.653	0.443	0.572	0.516
2	EMP/Yeast_H3K14ac	0.383	0.536	0.544	0.352	0.288	0.124
3	EMP/Yeast_H3K36me3	0.426	0.591	0.594	0.419	0.356	0.137
4	EMP/Yeast_H4	0.716	0.812	0.827	0.717	0.645	0.363
5	EMP/Yeast_H3K79me3	0.567	0.628	0.656	0.550	0.481	0.221
6	EMP/Yeast_H3K9ac	0.425	0.507	0.538	0.410	0.314	0.303

Appendix

Table 11: Per-dataset MCC across 57 classification benchmarks. Underline : best

k

-mer baseline. Bold : overall best per dataset. Category abbreviations: gb = genomic_benchmark, iPro = iPro-WAEL.

Figure 11: Mean classification MCC versus estimated GFLOPs per 500-bp sequence (logarithmic cost axis). The fixed 6-mer feature vector is included at

1.9\times 10^{-5}

in the supplied cost data, although

k

-mer counting is not naturally a floating-point-operation workload. Evo2 is omitted because no verified corresponding 500-bp cost estimate is available. Costs are externally supplied estimates, rather than newly measured runtime or GPU throughput.

Explore similar work

Entropy, Disagreement, and the Limits of Foundation Models in Genomics
Apr 5, 2026Maxime Rochkoulets, Lovro Vrček, Mile ŠikićGenomics Self-Supervised Pre-Training
Genome Institute of Singapore, A*STAR, Singapore · KU Leuven, Belgium · Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia
VANDAM: Viewing a nucleotide sequence with DNA molecular priors
Sep 30, 2026Jeremy Levy, Ariel Larey, Yury Nahshan +18Self-Supervised Pre-Training Genomics
Applied AI Architecture, NVIDIA, Israel. · Worldwide Field Ops, NVIDIA, Israel. · Developer Programs, NVIDIA, Israel. +3
How Private Are DNA Embeddings? Inverting Foundation Model Representations of Genomic Sequences
Mar 6, 2026Sofiane Ouaari, Jules Kreuer, Nico PfeiferModel Inversion Attacks Genomics
Methods in Medical Informatics, Department of Computer Science, University of Tuebingen, Germany · Institute for Bioinformatics and Medical Informatics (IBMI), University of Tuebingen, Germany

Apr 5, 2026J/K move · Enter open · S save

Entropy, Disagreement, and the Limits of Foundation Models in Genomics

Maxime Rochkoulets, Lovro Vrček, Mile Šikić

Genome Institute of Singapore, A*STAR, Singapore · KU Leuven, Belgium · Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia

Genomics Self-Supervised Pre-Training Sequence Modeling Molecular Representation Learning

Open paper AskPDF arXiv

Foundation models in genomics have shown mixed success compared to their counterparts in natural language processing. Yet, the reasons for their limited effectiveness remain poorly understood. In this work, we investigate the role of entropy as a fundamental factor limiting the capacities of such models to learn from their training data and develop foundational capabilities. We train ensembles of models on text and DNA sequences and analyze their predictions, static embeddings, and empirical Fisher information flow. We show that the high entropy of genomic sequences -- from the point of view of unseen token prediction -- leads to near-uniform output distributions, disagreement across models, and unstable static embeddings, even for models that are matched in architecture, training and data. We then demonstrate that models trained on DNA concentrate Fisher information in embedding layers, seemingly failing to exploit inter-token relationships. Our results suggest that self-supervised training from sequences alone may not be applicable to genomic data, calling into question the assumptions underlying current methodologies for training genomic foundation models.

Abstract

Explore similar work

Entropy, Disagreement, and the Limits of Foundation Models in Genomics

VANDAM: Viewing a nucleotide sequence with DNA molecular priors

How Private Are DNA Embeddings? Inverting Foundation Model Representations of Genomic Sequences