Early DNA foundation models adopted BERT-style training, achieving good performance on DNA understanding tasks but lacking generative capabilities. Recent autoregressive models enable DNA generation, but employ left-to-right causal modeling that is suboptimal for DNA where regulatory relationships are inherently bidirectional. We present D3LM (Discrete DNA Diffusion Language Model), which unifies bidirectional representation learning and DNA generation through masked diffusion. D3LM directly adopts the Nucleotide Transformer (NT) v2 architecture but reformulates the training objective as masked diffusion in discrete DNA space, enabling both bidirectional understanding and generation capabilities within a single model. Compared to NT v2 of the same size, D3LM achieves improved performance on understanding tasks. Notably, on regulatory element generation, D3LM achieves an SFID of 10.92, closely approaching real DNA sequences (7.85) and substantially outperforming the previous best result of 29.16 from autoregressive models. Our work suggests diffusion language models as a promising paradigm for unified DNA foundation models. We further present the first systematic study of masked diffusion models in the DNA domain, investigating practical design choices such as tokenization schemes and sampling strategies, thereby providing empirical insights and a solid foundation for future research. D3LM has been released at https://huggingface.co/collections/Hengchang-Liu/d3lm.
Figures & tables
Figure 1: Comparison of DNA modeling approaches. (a) Enhancers regulate promoters from either upstream or downstream positions, demonstrating bidirectional regulatory relationships in DNA. (b) BERT-style models use bidirectional attention but employ fixed masking ratios and lack generative capabilities. (c) Autoregressive models generate sequences left-to-right but cannot adjust earlier positions once generated, making it difficult to satisfy global constraints. (d) D3LM combines bidirectional modeling with generation through masked diffusion with variable masking ratios, enabling iterative refinement of all positions simultaneously.
Figure 2: D3LM framework overview. (a) Training with variable masking ratios sampled from a uniform distribution. The model learns to predict masked tokens using bidirectional attention. (b) Iterative generation starting from fully masked sequences, progressively unmasking tokens through repeated sampling. (c) Fine-tuning on downstream genomic tasks using frozen or trainable encoder with task-specific heads for promoter classification, histone modification, and splice site prediction.
Sequence Length = 256
Sequence Length = 2048
Model
Div ( ↑ )
Nov ( ↑ )
GC
SFID ( ↓ )
Div ( ↑ )
Nov ( ↑ )
GC
SFID ( ↓ )
Truth
189.0
170.0
1.10
31.94
1529
1470
1.06
7.85
Random
192.0
176.0
1.02
148.20
1537
1491
1.00
122.97
HyenaDNA
192.0
176.0
1.05
122.79
1533
1479
1.05
29.16
Evo
191.0
170.0
1.51
123.06
1533
1471
0.86
1359.98
DPLM
191.0
175.0
0.91
124.96
1531
1485
0.85
95.34
Table 1: Evaluation results of unconditional generation on different sequence lengths.
Task
DNABERT-2
Enformer
HyenaDNA
NT-MSv2 (50M)
D3LM (50M)
D3LM-r (50M)
H2AFZ
0.49 ( ± 0.013)
0.522 ( ± 0.019)
0.467 ( ± 0.012)
0.501 ( ± 0.012)
0.501 ( ± 0.020)
0.451 ( ± 0.017)
H3K27ac
0.491 ( ± 0.01)
0.52 ( ± 0.015)
0.421 ( ± 0.01)
0.483 ( ± 0.012)
0.497 ( ± 0.019)
0.381 ( ± 0.017)
H3K27me3
0.599 ( ± 0.01)
0.552 ( ± 0.007)
0.55 ( ± 0.009)
0.595 ( ± 0.004)
0.590 ( ± 0.029)
0.547 ( ± 0.014)
H3K36me3
0.637 ( ± 0.007)
0.567 ( ± 0.017)
0.553 ( ± 0.011)
0.607 ( ± 0.006)
0.602 ( ± 0.019)
0.524 ( ± 0.009)
H3K4me1
0.49 ( ± 0.008)
0.504 ( ± 0.021)
0.423 ( ± 0.016)
0.478 ( ± 0.012)
0.479 ( ± 0.016)
0.402 ( ± 0.020)
H3K4me2
0.558 ( ± 0.013)
0.626 ( ± 0.015)
0.515 ( ± 0.018)
0.549 ( ± 0.014)
0.537 ( ± 0.024)
0.488 ( ± 0.014)
Table 2: Performance on downstream NT understanding tasks (MCC ± Std).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Analysis of Model Properties. (a) Illustration of the scaling law behavior. (b) Visualization of the tokenization process.
We present a parameter-efficient Diffusion Transformer (DiT) for generating 200bp cell-type-specific regulatory DNA sequences. By replacing the U-Net backbone of DNA-Diffusion with a transformer denoiser equipped with a 2D CNN input encoder, our model matches the U-Net's best validation loss in 13 epochs (60× fewer) and converges 39% lower, while reducing memorization from 5.3% to 1.7% of generated sequences aligning to training data via BLAT. Ablations show the CNN encoder is essential: without it, validation loss increases 70% regardless of positional embedding choice. We further apply DDPO finetuning using Enformer as a reward model, achieving a 38× improvement in predicted regulatory activity. Cross-validation against DRAKES on an independent prediction task confirms that improvements reflect genuine regulatory signal rather than reward model overfitting.
Jonathan Liu, Kia Ghods
Department of Computer Science Princeton University
Diffusion Language Models (DLMs) generate text by iteratively denoising a masked sequence, independently predicting multiple tokens at each step. This conditional independence discards inter-token dependencies and degrades coherence-an issue that parallels the multi-modality problem in Non-Autoregressive Translation (NAT). Drawing on the Directed Acyclic Transformer (DAT), which tackles this problem in NAT via a Directed Acyclic Graph (DAG), we propose DA-DLM, a model that adapts DAG-based dependency modeling to DLMs' iterative setting through a position-oriented DAG design. The position-oriented DAG binds node groups to fixed output positions so that tokens fixed in earlier steps anchor neighboring predictions via learned transitions, and evolves with denoising to focus on remaining uncertainty as anchors accumulate. On language modeling, open-ended generation, and summarization, DA-DLM consistently outperforms Block Diffusion, especially under fewer denoising steps, and matches autoregressive models while preserving the parallel generation advantage. Our code is publicly available at https://github.com/jipy0222/DA-DLM.
Pengyu Ji, Zichen Zhang, Xiang Hu +1
‡School of Information Science and Technology, ShanghaiTech University · §Shanghai Engineering Research Center of Intelligent Vision and Imaging · ¶Tencent
Genomic foundation models increasingly adopt large language model architectures, yet almost universally rely on fixed tokenization schemes such as k-mers, BPE, or single nucleotides, which impose arbitrary sequence boundaries that may obscure biologically relevant structure. We present LDARNet, a 120M-parameter hierarchical genomic foundation model that adapts H-Net-style dynamic chunking from autoregressive generation to masked language modeling, combining BiMamba-2 state-space layers with local attention, bidirectional routing, and a ratio-based regularizer to induce adaptive token boundaries without supervision. Fine-tuned on 27 tasks from the Nucleotide Transformer and Genomic Benchmarks suites, LDARNet achieves 11/18 wins among compact models (<300M parameters) and state-of-the-art results on 5 histone modification tasks, outperforming models up to 20× larger. A FLOPs-matched controlled experiment isolates learned routing as the source of these gains: learned boundaries beat fixed-grid boundaries by up to 14 percentage points on histone tasks at identical compute. Nucleotide-resolution analysis further shows that the learned boundaries align with canonical promoter motifs and splice junctions without supervision, providing a biological interpretation for adaptive tokenization in genomic foundation models.
Daria Ledneva, Denis Kuznetsov
Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia.