D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation

Zhao Yang; Hengchang Liu; Chuan Cao; Bing Su

cs.LGMar 2, 2026

D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation

Authors: Zhao Yang, Hengchang Liu, Chuan Cao, Bing Su

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · Zhongguancun Academy

Abstract

Early DNA foundation models adopted BERT-style training, achieving good performance on DNA understanding tasks but lacking generative capabilities. Recent autoregressive models enable DNA generation, but employ left-to-right causal modeling that is suboptimal for DNA where regulatory relationships are inherently bidirectional. We present D3LM (Discrete DNA Diffusion Language Model), which unifies bidirectional representation learning and DNA generation through masked diffusion. D3LM directly adopts the Nucleotide Transformer (NT) v2 architecture but reformulates the training objective as masked diffusion in discrete DNA space, enabling both bidirectional understanding and generation capabilities within a single model. Compared to NT v2 of the same size, D3LM achieves improved performance on understanding tasks. Notably, on regulatory element generation, D3LM achieves an SFID of 10.92, closely approaching real DNA sequences (7.85) and substantially outperforming the previous best result of 29.16 from autoregressive models. Our work suggests diffusion language models as a promising paradigm for unified DNA foundation models. We further present the first systematic study of masked diffusion models in the DNA domain, investigating practical design choices such as tokenization schemes and sampling strategies, thereby providing empirical insights and a solid foundation for future research. D3LM has been released at https://huggingface.co/collections/Hengchang-Liu/d3lm.

Figures & tables

Figure 1: Comparison of DNA modeling approaches. (a) Enhancers regulate promoters from either upstream or downstream positions, demonstrating bidirectional regulatory relationships in DNA. (b) BERT-style models use bidirectional attention but employ fixed masking ratios and lack generative capabilities. (c) Autoregressive models generate sequences left-to-right but cannot adjust earlier positions once generated, making it difficult to satisfy global constraints. (d) D3LM combines bidirectional modeling with generation through masked diffusion with variable masking ratios, enabling iterative refinement of all positions simultaneously.

Figure 2: D3LM framework overview. (a) Training with variable masking ratios sampled from a uniform distribution. The model learns to predict masked tokens using bidirectional attention. (b) Iterative generation starting from fully masked sequences, progressively unmasking tokens through repeated sampling. (c) Fine-tuning on downstream genomic tasks using frozen or trainable encoder with task-specific heads for promoter classification, histone modification, and splice site prediction.

	Sequence Length = 256				Sequence Length = 2048
Model	Div ( $\uparrow$ )	Nov ( $\uparrow$ )	GC	SFID ( $\downarrow$ )	Div ( $\uparrow$ )	Nov ( $\uparrow$ )	GC	SFID ( $\downarrow$ )
Truth	189.0	170.0	1.10	31.94	1529	1470	1.06	7.85
Random	192.0	176.0	1.02	148.20	1537	1491	1.00	122.97
HyenaDNA	192.0	176.0	1.05	122.79	1533	1479	1.05	29.16
Evo	191.0	170.0	1.51	123.06	1533	1471	0.86	1359.98
DPLM	191.0	175.0	0.91	124.96	1531	1485	0.85	95.34

Table 1: Evaluation results of unconditional generation on different sequence lengths.

Task	DNABERT-2	Enformer	HyenaDNA	NT-MSv2 (50M)	D3LM (50M)	D3LM-r (50M)
H2AFZ	0.49 ( $\pm$ 0.013)	0.522 ( $\pm$ 0.019)	0.467 ( $\pm$ 0.012)	0.501 ( $\pm$ 0.012)	0.501 ( $\pm$ 0.020)	0.451 ( $\pm$ 0.017)
H3K27ac	0.491 ( $\pm$ 0.01)	0.52 ( $\pm$ 0.015)	0.421 ( $\pm$ 0.01)	0.483 ( $\pm$ 0.012)	0.497 ( $\pm$ 0.019)	0.381 ( $\pm$ 0.017)
H3K27me3	0.599 ( $\pm$ 0.01)	0.552 ( $\pm$ 0.007)	0.55 ( $\pm$ 0.009)	0.595 ( $\pm$ 0.004)	0.590 ( $\pm$ 0.029)	0.547 ( $\pm$ 0.014)
H3K36me3	0.637 ( $\pm$ 0.007)	0.567 ( $\pm$ 0.017)	0.553 ( $\pm$ 0.011)	0.607 ( $\pm$ 0.006)	0.602 ( $\pm$ 0.019)	0.524 ( $\pm$ 0.009)
H3K4me1	0.49 ( $\pm$ 0.008)	0.504 ( $\pm$ 0.021)	0.423 ( $\pm$ 0.016)	0.478 ( $\pm$ 0.012)	0.479 ( $\pm$ 0.016)	0.402 ( $\pm$ 0.020)
H3K4me2	0.558 ( $\pm$ 0.013)	0.626 ( $\pm$ 0.015)	0.515 ( $\pm$ 0.018)	0.549 ( $\pm$ 0.014)	0.537 ( $\pm$ 0.024)	0.488 ( $\pm$ 0.014)

Table 2: Performance on downstream NT understanding tasks (MCC

\pm

Std).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Analysis of Model Properties. (a) Illustration of the scaling law behavior. (b) Visualization of the tokenization process.

strategy	Div ( $\uparrow$ )	Nov ( $\uparrow$ )	GC	SFID ( $\downarrow$ )
Truth	1529	1470	1.06	7.85
Random	1537	1491	1.00	122.97
P2	1662.0	1451.0	12.7	3847.58
MaskGit	1531.0	1469.0	1.06	10.93
Entropy	1530.0	1472.0	1.07	10.99
Topk-M	1529.0	1468.0	1.05	11.05

Appendix

Table 3: Abaltion study of sampling strategy.

Steps ( $T$ )	Div ( $\uparrow$ )	Nov ( $\uparrow$ )	GC	SFID ( $\downarrow$ )
Truth	1529	1470	1.06	7.85
Random	1537	1491	1.00	122.97
1	1536	1488	1.04	62.40
5	1534	1482	1.04	28.96
10	1532	1478	1.04	17.90
50	1528	1471	1.07	10.92

Appendix

Table 4: Ablation study of denoising steps.

Temp ( $\tau$ )	Div ( $\uparrow$ )	Nov ( $\uparrow$ )	GC	SFID ( $\downarrow$ )
Truth	1529	1470	1.06	7.85
Random	1537	1491	1.00	122.97
0.9	1513.0	1452.0	1.11	38.50
1.0	1524.0	1464.0	1.08	14.31
1.1	1528	1471	1.07	10.92
1.2	1532.0	1475.0	1.05	14.74

Appendix

Table 5: Ablation study of sampling temperature.

Explore similar work

Continuous Diffusion Transformers for Designing Synthetic Regulatory Elements
Mar 11, 2026Jonathan Liu, Kia GhodsMolecular Generation Diffusion Transformer
Department of Computer Science Princeton University
DA-DLM: Explicitly Modeling Token Dependencies in Diffusion Language Models
Sep 14, 2026Pengyu Ji, Zichen Zhang, Xiang Hu +1Non-Autoregressive Text Generation Diffusion Language Model Decoding
‡School of Information Science and Technology, ShanghaiTech University · §Shanghai Engineering Research Center of Intelligent Vision and Imaging · ¶Tencent
LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling
Jun 3, 2026Daria Ledneva, Denis KuznetsovHierarchical Representation Learning Genomics
Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia.

Mar 11, 2026J/K move · Enter open · S save

Continuous Diffusion Transformers for Designing Synthetic Regulatory Elements

Jonathan Liu, Kia Ghods

Department of Computer Science Princeton University

Molecular Generation Diffusion Transformer Conditional Diffusion Models Diffusion Model Fine-Tuning

Open paper AskPDF arXiv

We present a parameter-efficient Diffusion Transformer (DiT) for generating 200bp cell-type-specific regulatory DNA sequences. By replacing the U-Net backbone of DNA-Diffusion with a transformer denoiser equipped with a 2D CNN input encoder, our model matches the U-Net's best validation loss in 13 epochs (60

\times

fewer) and converges 39% lower, while reducing memorization from 5.3% to 1.7% of generated sequences aligning to training data via BLAT. Ablations show the CNN encoder is essential: without it, validation loss increases 70% regardless of positional embedding choice. We further apply DDPO finetuning using Enformer as a reward model, achieving a 38

\times

improvement in predicted regulatory activity. Cross-validation against DRAKES on an independent prediction task confirms that improvements reflect genuine regulatory signal rather than reward model overfitting.

Abstract

Explore similar work

Continuous Diffusion Transformers for Designing Synthetic Regulatory Elements

DA-DLM: Explicitly Modeling Token Dependencies in Diffusion Language Models

LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling