When Genomic Masking Priors Fail to Transfer: Strong Variant Prediction, Weak Functional Generation

Susu Hu; Preetam Gattogi; Jens Lehmann; Sahar Vahdati; Stefanie Speidel; Julien Vibert

cs.LGSep 4, 2026

When Genomic Masking Priors Fail to Transfer: Strong Variant Prediction, Weak Functional Generation

Authors: Susu Hu, Preetam Gattogi, Jens Lehmann, Sahar Vahdati, Stefanie Speidel, Julien Vibert

Organizations: Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Dresden, Germany · Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden, Germany · Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Dresden, Germany · German Cancer Research Center (DKFZ), Heidelberg, Germany · ScaDS.AI – Dresden University of Technology · TUD Dresden University of Technology, Germany · Amazon (work done outside of Amazon), Dresden, Germany · Leibniz Information Centre for Science and Technology, Hannover, Germany · Drug Development Department, Gustave Roussy, Villejuif, France · Université Paris-Saclay, Gustave Roussy, Inserm U981, IHU PRISM, Villejuif, France · Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany

Abstract

Bidirectional discrete diffusion model appears naturally suited to genomic modeling because it can reconstruct missing sequence from both flanks. We developed GenDA (Genomic Density-optimized Absorbing Diffusion) under the additional hypothesis that entropy-guided span placement would concentrate reconstruction pressure on compositionally complex regions, improving both downstream variant-effect prediction and functional sequence generation. Our results only partially support this premise. After supervised fine-tuning, the 202M-parameter GenDA model reaches a pooled ClinVar SNV AUROC of 0.774, exceeding a similarly scaled autoregressive model by 0.103. However, a matched random-span variant reaches 0.777, providing no evidence that entropy guidance causes the ClinVar improvement. More unexpectedly, GenDA fails a zero-shot functional inpainting stress test: across promoters, enhancers, exon boundaries, and intron boundaries, it does not consistently outperform a control that shuffles the native gap while exactly preserving 3-mer composition. Failure is already present for 50--500-bp gaps, although enhancer degradation worsens at longer gaps. Diagnostics identify several boundary conditions: entropy measures local sequence complexity rather than functional importance; 1-mer tokenization limits physical context; training spans are capped at 300 bp; and high absolute AlphaGenome fidelity can coexist with negative control-normalized restoration. These results show that strong fine-tuned variant prediction, a plausible corruption prior, and functional generation are distinct claims that require separate validation.

Figures & tables

Figure 1: GenDA density-guided spans compared with independent uniform masking and Random Span.

Region	Evo 2	AR Llama	GenDA w/o Curriculum	GenDA
Enhancer	0.747	0.662	0.704	0.746
Exon	0.699	0.665	0.722	0.757
Intron	0.719	0.663	0.739	0.773
Promoter	0.768	0.694	0.730	0.778
Overall	0.733	0.671	0.743	0.774

Table 1: ClinVar SNV AUROC. GenDA w/o Curriculum uses the same masking policy without the progressive corruption curriculum. Evo 2 is evaluated zero-shot; AR Llama and both GenDA configurations undergo supervised fine-tuning. Bold denotes the best result in each row.

Model	Promoter CAGE	Enhancer H3K27ac	Exon usage	Intron usage
Evo 2	$-13.6_{\scriptscriptstyle[-16.8,-9.0]}$	$-23.0_{\scriptscriptstyle[-37.5,-12.6]}$	$-2.8_{\scriptscriptstyle[-12.1,-0.1]}$	$0.0_{\scriptscriptstyle[-1.6,0.0]}$
AR Llama	$-14.8_{\scriptscriptstyle[-18.7,-12.1]}$	$-31.1_{\scriptscriptstyle[-37.9,-25.2]}$	$-6.2_{\scriptscriptstyle[-15.7,-0.2]}$	$0.0_{\scriptscriptstyle[-4.7,0.0]}$
D3LM	$-41.5_{\scriptscriptstyle[-73.9,-20.3]}$	$-84.1_{\scriptscriptstyle[-118.1,-60.8]}$	$-12.0_{\scriptscriptstyle[-28.8,0.1]}$	$-2.2_{\scriptscriptstyle[-8.3,0.1]}$
GenDA	$-37.9_{\scriptscriptstyle[-65.9,-27.2]}$	$-130.8_{\scriptscriptstyle[-206.6,-62.5]}$	$-17.0_{\scriptscriptstyle[-22.9,0.1]}$	$0.1_{\scriptscriptstyle[-0.9,0.2]}$
Eligible sites	43/50	25/50	47/50	45/50

Table 2: AlphaGenome functional inpainting across gap lengths. Entries are median site-level restoration gain averaged across evaluated gap lengths, with 95% bootstrap confidence intervals shown as subscripts. Higher is better; zero denotes parity with the 3-mer-preserving corrupted control.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Collapse under exclusive independent uniform 1-mer masking. Nucleotide frequencies across unconditional denoising steps. The model rapidly concentrates on a single nucleotide despite using the same backbone, data, curriculum, and computational budget as GenDA.

Region	Random Span			GenDA			GenDA-large
	P1	P2	P3	P1	P2	P3	P1	P2	P3
Enhancer	0.737	0.741	0.747	0.697	0.716	0.746	0.740	0.757	0.768
Exon	0.758	0.764	0.758	0.727	0.740	0.757	0.746	0.757	0.762
Intron	0.759	0.777	0.778	0.723	0.739	0.773	0.765	0.777	0.787
Promoter	0.765	0.775	0.785	0.720	0.757	0.778	0.760	0.784	0.787
Overall	0.770	0.779	0.777	0.735	0.750	0.774	0.765	0.776	0.786

Appendix

Table 3: Region-specific ClinVar SNV AUROC across masking strategies, model sizes, and successive curriculum stages. P1–P3 are sequential checkpoints rather than compute-matched standalone models. GenDA-large uses twice the depth of GenDA. Higher is better.

Model	Promoter CAGE	Enhancer H3K27ac	Exon usage	Intron usage
Random Span	$-56.8_{\scriptscriptstyle[-130.7,-34.1]}$	$\mathbf{-58.4}_{\scriptscriptstyle[-89.4,-33.8]}$	$-19.3_{\scriptscriptstyle[-42.9,0.3]}$	$-0.2_{\scriptscriptstyle[-10.1,0.3]}$
GenDA	$\mathbf{-37.9}_{\scriptscriptstyle[-65.9,-27.2]}$	$-130.8_{\scriptscriptstyle[-206.6,-62.5]}$	$\mathbf{-17.0}_{\scriptscriptstyle[-22.9,0.1]}$	$\mathbf{0.1}_{\scriptscriptstyle[-0.9,0.2]}$

Appendix

Table 4: AlphaGenome inpainting ablation comparing Random Span with GenDA. Entries are median restoration gain, with 95% bootstrap confidence intervals shown as subscripts. Higher is better.

Model	Promoter CAGE	Enhancer H3K27ac	Exon usage	Intron usage
GenDA	$-37.9_{\scriptscriptstyle[-65.9,-27.2]}$	$-130.8_{\scriptscriptstyle[-206.6,-62.5]}$	$-17.0_{\scriptscriptstyle[-22.9,0.1]}$	$0.1_{\scriptscriptstyle[-0.9,0.2]}$
GenDA+BioCheck	$-69.4_{\scriptscriptstyle[-122.0,-40.2]}$	$-42.6_{\scriptscriptstyle[-53.7,-36.1]}$	$-17.6_{\scriptscriptstyle[-27.7,0.4]}$	$0.1_{\scriptscriptstyle[-0.2,0.3]}$

Appendix

Table 5: Exploratory BioCheck ablation. Entries are median AlphaGenome restoration gain, with 95% bootstrap confidence intervals shown as subscripts. Higher is better.

Model	Gap range	Promoter CAGE RG $\uparrow$	Enhancer H3K27ac RG $\uparrow$	Exon usage RG $\uparrow$	Intron usage RG $\uparrow$
Evo 2	Short	$-23.6_{\;[-33.1,-14.9]}$	$-30.0_{\;[-39.1,-13.3]}$	$-3.5_{\;[-11.9,-0.2]}$	$-0.0_{\;[-0.4,0.0]}$
	Medium	$-3.4_{\;[-10.2,0.2]}$	$-25.9_{\;[-33.5,-18.2]}$	$-0.1_{\;[-5.1,0.0]}$	$-0.0_{\;[-1.8,0.0]}$
	Long	$-8.4_{\;[-14.6,-3.8]}$	$-15.3_{\;[-23.6,-8.6]}$	$0.0_{\;[-4.0,0.0]}$	$0.0_{\;[0.0,0.0]}$
AR Llama	Short	$-29.4_{\;[-33.5,-22.9]}$	$-38.0_{\;[-53.4,-31.4]}$	$-8.8_{\;[-25.0,-0.2]}$	$-0.0_{\;[-3.6,0.0]}$
	Medium	$-7.0_{\;[-12.5,-3.9]}$	$-43.3_{\;[-50.8,-32.2]}$	$-0.3_{\;[-7.1,-0.0]}$	$0.0_{\;[-0.1,0.0]}$
	Long	$-7.3_{\;[-14.7,-4.9]}$	$-28.0_{\;[-46.2,-21.4]}$	$-0.0_{\;[-2.2,0.0]}$	$0.0_{\;[0.0,0.1]}$

Appendix

Table 6: AlphaGenome restoration gain stratified by gap range. Entries report median site-level RG with 95% site-bootstrap confidence intervals. Short: 50–500 bp; medium: 800–1,500 bp; long: 2,000–3,500 bp. Random Span is the GenDA masking ablation from Appendix A.3 .

Model	Promoter CAGE	Enhancer H3K27ac	Exon usage	Intron usage
Evo 2	$67.9_{\;[56.8,73.5]}$	$95.7_{\;[95.3,96.1]}$	$35.5_{\;[20.1,67.7]}$	$46.8_{\;[7.5,56.2]}$
AR Llama	$69.1_{\;[57.0,72.8]}$	$95.2_{\;[94.8,95.5]}$	$34.6_{\;[20.0,67.6]}$	$47.5_{\;[7.6,57.2]}$
D3LM	$55.7_{\;[51.9,58.9]}$	$92.8_{\;[92.4,93.2]}$	$36.8_{\;[24.3,63.3]}$	$44.8_{\;[8.8,56.6]}$
GenDA	$49.6_{\;[44.7,54.8]}$	$90.1_{\;[88.6,91.9]}$	$38.6_{\;[23.6,62.8]}$	$47.7_{\;[10.0,56.4]}$
GenDA+BioCheck	$42.6_{\;[39.2,44.4]}$	$95.1_{\;[94.6,95.7]}$	$39.7_{\;[26.8,58.6]}$	$45.9_{\;[12.9,57.5]}$

Appendix

Table 7: Absolute AlphaGenome restoration fidelity across all gap lengths. Entries report median gap-averaged fidelity with 95% site-bootstrap confidence intervals. Higher values indicate closer agreement with the native AlphaGenome prediction. High enhancer fidelity contrasts with negative control-normalized restoration gain in Table 2 .

Category	Train	Validation	Test	Total	% of Total
Exon	869,036	104,948	105,939	1,079,923	33.7%
Enhancer	768,982	96,123	96,122	961,227	30.0%
Intron	455,092	56,435	57,393	568,920	17.8%
Repeat	408,598	51,075	51,074	510,747	16.0%
Promoter	62,837	7,859	7,856	78,552	2.5%
Total	2,564,545	316,440	318,384	3,199,369	100.0%

Appendix

Table 8: Global Dataset Split Summary. The dataset comprises

\sim

3.2 million genomic anchors partitioned via a strict gene-aware split to prevent structural leakage.

Category	Min	P25	Median	Mean	P75	Max	Fits $\leq 4096$
Promoter	2,576	4,000	4,000	4,000	4,000	4,000	100.0%
Enhancer	150	208	287	273	340	350	100.0%
Exon	1	102	175	405	380	347,300	99.4%
Repeat	500	596	781	1,287	1,248	500,000	96.7%
Intron	200	1,002	2,898	11,335	9,745	1,240,120	58.0%
Global	1	201	344	2,544	965	1,240,120	91.8%

Appendix

Table 9: Processed feature lengths and context fit. Lengths describe the biological anchor before cropping or addition of flanking context. Most promoters, enhancers, and exons fit within the 4,096-bp model window, whereas many introns do not.

Masked Diffusion Models

Masked diffusion models generate sequences by repeatedly predicting and revising masked tokens, using bidirectional context for parallel sampling.

See all 173 papers

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures
Feb 19, 2026Ariel Larey, Elay Dahan, Amit Bleiweiss +15Genomic Foundation Models Molecular Representation Learning
Applied AI Architecture, NVIDIA, Israel. · Worldwide Field Ops, NVIDIA, Israel. · Developer Programs, NVIDIA, Israel. +2
D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation
Mar 2, 2026Zhao Yang, Hengchang Liu, Chuan Cao +1Molecular Foundation Models Genomic Foundation Models
Gaoling School of Artificial Intelligence, Renmin University of China · Zhongguancun Academy
Entropy, Disagreement, and the Limits of Foundation Models in Genomics
Apr 5, 2026Maxime Rochkoulets, Lovro Vrček, Mile ŠikićGenomic Foundation Models Genomics
Genome Institute of Singapore, A*STAR, Singapore · KU Leuven, Belgium · Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia

Feb 19, 2026J/K move · Enter open · S save

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

Ariel Larey, Elay Dahan, Amit Bleiweiss, Raizy Kellerman+14

Applied AI Architecture, NVIDIA, Israel. · Worldwide Field Ops, NVIDIA, Israel. · Developer Programs, NVIDIA, Israel.+2

Genomic Foundation Models Molecular Representation Learning Genomics Joint-Embedding Predictive Architecture

Open paper AskPDF arXiv

Genomic Foundation Models (GFMs) typically rely on Masked Language Modeling (MLM) or Next-Token Prediction (NTP) to learn the "Laws of Nature". While effective at capturing local syntax, these generative paradigms prioritize token-level reconstruction over high-level functional context. We introduce JEPA-DNA, a model-agnostic continual training framework that integrates a Joint-Embedding Predictive Architecture (JEPA) with traditional generative objectives. By supervising global sequence embeddings in a latent space, JEPA-DNA forces models to predict the functional representations of masked genomic segments, shifting the learning signal from token recovery to semantic alignment. We evaluate JEPA-DNA on 17 diverse genomic benchmark tasks, demonstrating consistent gains in linear probing and zero-shot performance regardless of the underlying GFM architecture or generative objective, and establishing a new state-of-the-art for GFMs under the linear-probing protocol. Through extensive ablation studies, we further characterize the synergistic interplay between generative and latent objectives. Our code is provided in the supplementary material.

Abstract

Masked Diffusion Models

Explore similar work

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation

Entropy, Disagreement, and the Limits of Foundation Models in Genomics