Entropy, Disagreement, and the Limits of Foundation Models in Genomics

Maxime Rochkoulets; Lovro Vrček; Mile Šikić

cs.LGApr 5, 2026

Entropy, Disagreement, and the Limits of Foundation Models in Genomics

Authors: Maxime Rochkoulets, Lovro Vrček, Mile Šikić

Organizations: Genome Institute of Singapore, A*STAR, Singapore · KU Leuven, Belgium · Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia

Abstract

Foundation models in genomics have shown mixed success compared to their counterparts in natural language processing. Yet, the reasons for their limited effectiveness remain poorly understood. In this work, we investigate the role of entropy as a fundamental factor limiting the capacities of such models to learn from their training data and develop foundational capabilities. We train ensembles of models on text and DNA sequences and analyze their predictions, static embeddings, and empirical Fisher information flow. We show that the high entropy of genomic sequences -- from the point of view of unseen token prediction -- leads to near-uniform output distributions, disagreement across models, and unstable static embeddings, even for models that are matched in architecture, training and data. We then demonstrate that models trained on DNA concentrate Fisher information in embedding layers, seemingly failing to exploit inter-token relationships. Our results suggest that self-supervised training from sequences alone may not be applicable to genomic data, calling into question the assumptions underlying current methodologies for training genomic foundation models.

Figures & tables

Figure 1: Jensen-Shannon distance

\mathbb{E}_{{\textnormal{x}}\sim\mathcal{D}}\left[d_{\mathrm{JS}}(P_{i},P_{j})\right]

between models of the same ensemble as a function of top-

p

mass kept. Values computed over

100\,000

samples unseen during training.

	Top-10 Overlap $\uparrow$	Local Spearman $\uparrow$	Procrustes Cosine $\uparrow$	Procrustes Disparity $\downarrow$
Text	0.54	0.68	0.71	0.49
DNA (BPE)	0.39	0.48	0.61	0.62
DNA ( $k$ -mer)	0.41	0.47	0.59	0.66

Table 1: Different measures of agreement in static word embedding space.

Figure 2: Normalized layer-wise aggregation of empirical Fisher information in text and DNA models. Estimate computed over

100\,000

samples unseen during training.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Top-1	Top-3	Top-5	Top-10	Top-20	Top-50	Top-100	Top-1000	Top-4000
Text	0.75	0.60	0.56	0.54	0.52	0.50	0.49	0.52	0.97
DNA (BPE)	0.42	0.38	0.38	0.39	0.40	0.41	0.41	0.52	0.97
DNA (k-mer)	0.37	0.36	0.38	0.41	0.42	0.41	0.40	0.54	0.96

Appendix

Table 2: Additional results for top-

k

overlap.

	$k=3$	$k=5$	$k=10$	$k=20$	$k=50$	$k=100$	$k=1000$	$k=4000$
Text	0.68	0.68	0.68	0.67	0.65	0.65	0.61	0.73
DNA (BPE)	0.39	0.44	0.48	0.51	0.54	0.55	0.57	0.76
DNA (k-mer)	0.34	0.40	0.47	0.53	0.57	0.56	0.58	0.76

Appendix

Table 3: Additional results for Spearman nearest-neighbor correlation.

Figure 3: Detailed normalized empirical Fisher information content per model, for each layer. Estimate computed over

100\,000

samples unseen during training.

Explore similar work

What DNA Foundation Models Learn Beyond Sequence Composition
Oct 11, 2026Vincenzo Yuto Civale, Andrew David Bagdanov, Alberto MagiMolecular Representation Learning Genomics
University of Florence
GENEB: Why Genomic Models Are Hard to Compare
Jun 3, 2026Daria Ledneva, Mikhail Nuridinov, Denis KuznetsovBenchmark Design Genomics
Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia · Moscow State Institute of Steel and Alloys, Moscow, Russia
JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures
Feb 19, 2026Ariel Larey, Elay Dahan, Amit Bleiweiss +15Molecular Representation Learning Genomics
Applied AI Architecture, NVIDIA, Israel. · Worldwide Field Ops, NVIDIA, Israel. · Developer Programs, NVIDIA, Israel. +2

Oct 11, 2026J/K move · Enter open · S save

What DNA Foundation Models Learn Beyond Sequence Composition

Vincenzo Yuto Civale, Andrew David Bagdanov, Alberto Magi

University of Florence

Molecular Representation Learning Genomics Sequence Modeling

Open paper AskPDF arXiv

Do genomic foundation models (FMs) provide predictive information beyond simple sequence composition? We evaluate frozen representations from five DNA FMs against normalized k-mer frequencies across 57 genomic classification datasets and two exploratory gene-expression regression tasks. Simple k-mer features remain competitive with most models, whereas NTv3 shows a consistent advantage; the ranking is robust to globally fixed and multi-scale k-mer baselines and alternative downstream probes. To test incremental predictive utility directly, we compare global 6-mer features with their concatenation with each FM embedding under a matched Random Forest protocol. NTv3 improves 46 of 57 datasets (mean

Δ

MCC = +0.072), while Caduceus provides smaller gains and the other models do not improve mean MCC. We separately decompose embeddings into a component linearly predictable from 6-mer frequencies and a ridge residual, finding that linear recoverability does not reliably indicate downstream utility. Exact 6-mer-preserving shuffles further reveal task-relevant sequence-order sensitivity, most pronounced for NTv3, although order sensitivity alone does not guarantee predictive improvements. Our results establish compositional features as essential controls and offer complementary diagnostics for evaluating the added value of frozen genomic representations, without making claims about fine-tuning capacity.

Abstract

Explore similar work

What DNA Foundation Models Learn Beyond Sequence Composition

GENEB: Why Genomic Models Are Hard to Compare

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures