When Genomic Masking Priors Fail to Transfer: Strong Variant Prediction, Weak Functional Generation

Susu Hu; Preetam Gattogi; Jens Lehmann; Sahar Vahdati; Stefanie Speidel; Julien Vibert

cs.LGSep 4, 2026

When Genomic Masking Priors Fail to Transfer: Strong Variant Prediction, Weak Functional Generation

Authors: Susu Hu, Preetam Gattogi, Jens Lehmann, Sahar Vahdati, Stefanie Speidel, Julien Vibert

Organizations: Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Dresden, Germany · Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden, Germany · Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Dresden, Germany · German Cancer Research Center (DKFZ), Heidelberg, Germany · ScaDS.AI – Dresden University of Technology · TUD Dresden University of Technology, Germany · Amazon (work done outside of Amazon), Dresden, Germany · Leibniz Information Centre for Science and Technology, Hannover, Germany · Drug Development Department, Gustave Roussy, Villejuif, France · Université Paris-Saclay, Gustave Roussy, Inserm U981, IHU PRISM, Villejuif, France · Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany

Abstract

Bidirectional discrete diffusion model appears naturally suited to genomic modeling because it can reconstruct missing sequence from both flanks. We developed GenDA (Genomic Density-optimized Absorbing Diffusion) under the additional hypothesis that entropy-guided span placement would concentrate reconstruction pressure on compositionally complex regions, improving both downstream variant-effect prediction and functional sequence generation. Our results only partially support this premise. After supervised fine-tuning, the 202M-parameter GenDA model reaches a pooled ClinVar SNV AUROC of 0.774, exceeding a similarly scaled autoregressive model by 0.103. However, a matched random-span variant reaches 0.777, providing no evidence that entropy guidance causes the ClinVar improvement. More unexpectedly, GenDA fails a zero-shot functional inpainting stress test: across promoters, enhancers, exon boundaries, and intron boundaries, it does not consistently outperform a control that shuffles the native gap while exactly preserving 3-mer composition. Failure is already present for 50--500-bp gaps, although enhancer degradation worsens at longer gaps. Diagnostics identify several boundary conditions: entropy measures local sequence complexity rather than functional importance; 1-mer tokenization limits physical context; training spans are capped at 300 bp; and high absolute AlphaGenome fidelity can coexist with negative control-normalized restoration. These results show that strong fine-tuned variant prediction, a plausible corruption prior, and functional generation are distinct claims that require separate validation.

Figures & tables

Figure 1: GenDA density-guided spans compared with independent uniform masking and Random Span.

Region	Evo 2	AR Llama	GenDA w/o Curriculum	GenDA
Enhancer	0.747	0.662	0.704	0.746
Exon	0.699	0.665	0.722	0.757
Intron	0.719	0.663	0.739	0.773
Promoter	0.768	0.694	0.730	0.778
Overall	0.733	0.671	0.743	0.774

Table 1: ClinVar SNV AUROC. GenDA w/o Curriculum uses the same masking policy without the progressive corruption curriculum. Evo 2 is evaluated zero-shot; AR Llama and both GenDA configurations undergo supervised fine-tuning. Bold denotes the best result in each row.

Model	Promoter CAGE	Enhancer H3K27ac	Exon usage	Intron usage
Evo 2	$-13.6_{\scriptscriptstyle[-16.8,-9.0]}$	$-23.0_{\scriptscriptstyle[-37.5,-12.6]}$	$-2.8_{\scriptscriptstyle[-12.1,-0.1]}$	$0.0_{\scriptscriptstyle[-1.6,0.0]}$
AR Llama	$-14.8_{\scriptscriptstyle[-18.7,-12.1]}$	$-31.1_{\scriptscriptstyle[-37.9,-25.2]}$	$-6.2_{\scriptscriptstyle[-15.7,-0.2]}$	$0.0_{\scriptscriptstyle[-4.7,0.0]}$
D3LM	$-41.5_{\scriptscriptstyle[-73.9,-20.3]}$	$-84.1_{\scriptscriptstyle[-118.1,-60.8]}$	$-12.0_{\scriptscriptstyle[-28.8,0.1]}$	$-2.2_{\scriptscriptstyle[-8.3,0.1]}$
GenDA	$-37.9_{\scriptscriptstyle[-65.9,-27.2]}$	$-130.8_{\scriptscriptstyle[-206.6,-62.5]}$	$-17.0_{\scriptscriptstyle[-22.9,0.1]}$	$0.1_{\scriptscriptstyle[-0.9,0.2]}$
Eligible sites	43/50	25/50	47/50	45/50

Table 2: AlphaGenome functional inpainting across gap lengths. Entries are median site-level restoration gain averaged across evaluated gap lengths, with 95% bootstrap confidence intervals shown as subscripts. Higher is better; zero denotes parity with the 3-mer-preserving corrupted control.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Collapse under exclusive independent uniform 1-mer masking. Nucleotide frequencies across unconditional denoising steps. The model rapidly concentrates on a single nucleotide despite using the same backbone, data, curriculum, and computational budget as GenDA.

Region	Random Span			GenDA			GenDA-large
	P1	P2	P3	P1	P2	P3	P1	P2	P3
Enhancer	0.737	0.741	0.747	0.697	0.716	0.746	0.740	0.757	0.768
Exon	0.758	0.764	0.758	0.727	0.740	0.757	0.746	0.757	0.762
Intron	0.759	0.777	0.778	0.723	0.739	0.773	0.765	0.777	0.787
Promoter	0.765	0.775	0.785	0.720	0.757	0.778	0.760	0.784	0.787
Overall	0.770	0.779	0.777	0.735	0.750	0.774	0.765	0.776	0.786

Appendix

Table 3: Region-specific ClinVar SNV AUROC across masking strategies, model sizes, and successive curriculum stages. P1–P3 are sequential checkpoints rather than compute-matched standalone models. GenDA-large uses twice the depth of GenDA. Higher is better.

Model	Promoter CAGE	Enhancer H3K27ac	Exon usage	Intron usage
Random Span	$-56.8_{\scriptscriptstyle[-130.7,-34.1]}$	$\mathbf{-58.4}_{\scriptscriptstyle[-89.4,-33.8]}$	$-19.3_{\scriptscriptstyle[-42.9,0.3]}$	$-0.2_{\scriptscriptstyle[-10.1,0.3]}$
GenDA	$\mathbf{-37.9}_{\scriptscriptstyle[-65.9,-27.2]}$	$-130.8_{\scriptscriptstyle[-206.6,-62.5]}$	$\mathbf{-17.0}_{\scriptscriptstyle[-22.9,0.1]}$	$\mathbf{0.1}_{\scriptscriptstyle[-0.9,0.2]}$

Appendix

Table 4: AlphaGenome inpainting ablation comparing Random Span with GenDA. Entries are median restoration gain, with 95% bootstrap confidence intervals shown as subscripts. Higher is better.

Model	Promoter CAGE	Enhancer H3K27ac	Exon usage	Intron usage
GenDA	$-37.9_{\scriptscriptstyle[-65.9,-27.2]}$	$-130.8_{\scriptscriptstyle[-206.6,-62.5]}$	$-17.0_{\scriptscriptstyle[-22.9,0.1]}$	$0.1_{\scriptscriptstyle[-0.9,0.2]}$
GenDA+BioCheck	$-69.4_{\scriptscriptstyle[-122.0,-40.2]}$	$-42.6_{\scriptscriptstyle[-53.7,-36.1]}$	$-17.6_{\scriptscriptstyle[-27.7,0.4]}$	$0.1_{\scriptscriptstyle[-0.2,0.3]}$

Appendix

Table 5: Exploratory BioCheck ablation. Entries are median AlphaGenome restoration gain, with 95% bootstrap confidence intervals shown as subscripts. Higher is better.

Model	Gap range	Promoter CAGE RG $\uparrow$	Enhancer H3K27ac RG $\uparrow$	Exon usage RG $\uparrow$	Intron usage RG $\uparrow$
Evo 2	Short	$-23.6_{\;[-33.1,-14.9]}$	$-30.0_{\;[-39.1,-13.3]}$	$-3.5_{\;[-11.9,-0.2]}$	$-0.0_{\;[-0.4,0.0]}$
	Medium	$-3.4_{\;[-10.2,0.2]}$	$-25.9_{\;[-33.5,-18.2]}$	$-0.1_{\;[-5.1,0.0]}$	$-0.0_{\;[-1.8,0.0]}$
	Long	$-8.4_{\;[-14.6,-3.8]}$	$-15.3_{\;[-23.6,-8.6]}$	$0.0_{\;[-4.0,0.0]}$	$0.0_{\;[0.0,0.0]}$
AR Llama	Short	$-29.4_{\;[-33.5,-22.9]}$	$-38.0_{\;[-53.4,-31.4]}$	$-8.8_{\;[-25.0,-0.2]}$	$-0.0_{\;[-3.6,0.0]}$
	Medium	$-7.0_{\;[-12.5,-3.9]}$	$-43.3_{\;[-50.8,-32.2]}$	$-0.3_{\;[-7.1,-0.0]}$	$0.0_{\;[-0.1,0.0]}$
	Long	$-7.3_{\;[-14.7,-4.9]}$	$-28.0_{\;[-46.2,-21.4]}$	$-0.0_{\;[-2.2,0.0]}$	$0.0_{\;[0.0,0.1]}$

Appendix

Table 6: AlphaGenome restoration gain stratified by gap range. Entries report median site-level RG with 95% site-bootstrap confidence intervals. Short: 50–500 bp; medium: 800–1,500 bp; long: 2,000–3,500 bp. Random Span is the GenDA masking ablation from Appendix A.3 .

Model	Promoter CAGE	Enhancer H3K27ac	Exon usage	Intron usage
Evo 2	$67.9_{\;[56.8,73.5]}$	$95.7_{\;[95.3,96.1]}$	$35.5_{\;[20.1,67.7]}$	$46.8_{\;[7.5,56.2]}$
AR Llama	$69.1_{\;[57.0,72.8]}$	$95.2_{\;[94.8,95.5]}$	$34.6_{\;[20.0,67.6]}$	$47.5_{\;[7.6,57.2]}$
D3LM	$55.7_{\;[51.9,58.9]}$	$92.8_{\;[92.4,93.2]}$	$36.8_{\;[24.3,63.3]}$	$44.8_{\;[8.8,56.6]}$
GenDA	$49.6_{\;[44.7,54.8]}$	$90.1_{\;[88.6,91.9]}$	$38.6_{\;[23.6,62.8]}$	$47.7_{\;[10.0,56.4]}$
GenDA+BioCheck	$42.6_{\;[39.2,44.4]}$	$95.1_{\;[94.6,95.7]}$	$39.7_{\;[26.8,58.6]}$	$45.9_{\;[12.9,57.5]}$

Appendix

Table 7: Absolute AlphaGenome restoration fidelity across all gap lengths. Entries report median gap-averaged fidelity with 95% site-bootstrap confidence intervals. Higher values indicate closer agreement with the native AlphaGenome prediction. High enhancer fidelity contrasts with negative control-normalized restoration gain in Table 2 .

Category	Train	Validation	Test	Total	% of Total
Exon	869,036	104,948	105,939	1,079,923	33.7%
Enhancer	768,982	96,123	96,122	961,227	30.0%
Intron	455,092	56,435	57,393	568,920	17.8%
Repeat	408,598	51,075	51,074	510,747	16.0%
Promoter	62,837	7,859	7,856	78,552	2.5%
Total	2,564,545	316,440	318,384	3,199,369	100.0%

Appendix

Table 8: Global Dataset Split Summary. The dataset comprises

\sim

3.2 million genomic anchors partitioned via a strict gene-aware split to prevent structural leakage.

Category	Min	P25	Median	Mean	P75	Max	Fits $\leq 4096$
Promoter	2,576	4,000	4,000	4,000	4,000	4,000	100.0%
Enhancer	150	208	287	273	340	350	100.0%
Exon	1	102	175	405	380	347,300	99.4%
Repeat	500	596	781	1,287	1,248	500,000	96.7%
Intron	200	1,002	2,898	11,335	9,745	1,240,120	58.0%
Global	1	201	344	2,544	965	1,240,120	91.8%

Appendix

Table 9: Processed feature lengths and context fit. Lengths describe the biological anchor before cropping or addition of flanking context. Most promoters, enhancers, and exons fit within the 4,096-bp model window, whereas many introns do not.

Masked Diffusion Models

Masked diffusion models generate sequences by repeatedly predicting and revising masked tokens, using bidirectional context for parallel sampling.

See all 173 papers

Feb 19, 2026

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

Genomic Foundation Models (GFMs) typically rely on Masked Language Modeling (MLM) or Next-Token Prediction (NTP) to learn the "Laws of Nature". While effective at capturing local syntax, these generative paradigms prioritize token-level reconstruction over high-level functional context. We introduce JEPA-DNA, a model-agnostic continual training framework that integrates a Joint-Embedding Predictive Architecture (JEPA) with traditional generative objectives. By supervising global sequence embeddings in a latent space, JEPA-DNA forces models to predict the functional representations of masked genomic segments, shifting the learning signal from token recovery to semantic alignment. We evaluate JEPA-DNA on 17 diverse genomic benchmark tasks, demonstrating consistent gains in linear probing and zero-shot performance regardless of the underlying GFM architecture or generative objective, and establishing a new state-of-the-art for GFMs under the linear-probing protocol. Through extensive ablation studies, we further characterize the synergistic interplay between generative and latent objectives. Our code is provided in the supplementary material.

Ariel Larey, Elay Dahan, Amit Bleiweiss +15

Applied AI Architecture, NVIDIA, Israel. · Worldwide Field Ops, NVIDIA, Israel. · Developer Programs, NVIDIA, Israel. +2

Genomic Foundation Models Molecular Representation Learning Genomics Joint-Embedding Predictive Architecture

Ask PDF

Mar 2, 2026

D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation

Early DNA foundation models adopted BERT-style training, achieving good performance on DNA understanding tasks but lacking generative capabilities. Recent autoregressive models enable DNA generation, but employ left-to-right causal modeling that is suboptimal for DNA where regulatory relationships are inherently bidirectional. We present D3LM (Discrete DNA Diffusion Language Model), which unifies bidirectional representation learning and DNA generation through masked diffusion. D3LM directly adopts the Nucleotide Transformer (NT) v2 architecture but reformulates the training objective as masked diffusion in discrete DNA space, enabling both bidirectional understanding and generation capabilities within a single model. Compared to NT v2 of the same size, D3LM achieves improved performance on understanding tasks. Notably, on regulatory element generation, D3LM achieves an SFID of 10.92, closely approaching real DNA sequences (7.85) and substantially outperforming the previous best result of 29.16 from autoregressive models. Our work suggests diffusion language models as a promising paradigm for unified DNA foundation models. We further present the first systematic study of masked diffusion models in the DNA domain, investigating practical design choices such as tokenization schemes and sampling strategies, thereby providing empirical insights and a solid foundation for future research. D3LM has been released at https://huggingface.co/collections/Hengchang-Liu/d3lm.

Zhao Yang, Hengchang Liu, Chuan Cao +1

Gaoling School of Artificial Intelligence, Renmin University of China · Zhongguancun Academy

Molecular Foundation Models Genomic Foundation Models Language Modeling Molecular Generation Diffusion Language Models Masked Diffusion Models+1

Ask PDF

Apr 5, 2026

Entropy, Disagreement, and the Limits of Foundation Models in Genomics

Foundation models in genomics have shown mixed success compared to their counterparts in natural language processing. Yet, the reasons for their limited effectiveness remain poorly understood. In this work, we investigate the role of entropy as a fundamental factor limiting the capacities of such models to learn from their training data and develop foundational capabilities. We train ensembles of models on text and DNA sequences and analyze their predictions, static embeddings, and empirical Fisher information flow. We show that the high entropy of genomic sequences -- from the point of view of unseen token prediction -- leads to near-uniform output distributions, disagreement across models, and unstable static embeddings, even for models that are matched in architecture, training and data. We then demonstrate that models trained on DNA concentrate Fisher information in embedding layers, seemingly failing to exploit inter-token relationships. Our results suggest that self-supervised training from sequences alone may not be applicable to genomic data, calling into question the assumptions underlying current methodologies for training genomic foundation models.

Maxime Rochkoulets, Lovro Vrček, Mile Šikić

Genome Institute of Singapore, A*STAR, Singapore · KU Leuven, Belgium · Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia

Genomic Foundation Models Genomics Self-Supervised Pre-Training Sequence Modeling Molecular Representation Learning

Ask PDF

Abstract

Masked Diffusion Models

Explore similar work

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation

Entropy, Disagreement, and the Limits of Foundation Models in Genomics