We present a parameter-efficient Diffusion Transformer (DiT) for generating 200bp cell-type-specific regulatory DNA sequences. By replacing the U-Net backbone of DNA-Diffusion with a transformer denoiser equipped with a 2D CNN input encoder, our model matches the U-Net's best validation loss in 13 epochs (60× fewer) and converges 39% lower, while reducing memorization from 5.3% to 1.7% of generated sequences aligning to training data via BLAT. Ablations show the CNN encoder is essential: without it, validation loss increases 70% regardless of positional embedding choice. We further apply DDPO finetuning using Enformer as a reward model, achieving a 38× improvement in predicted regulatory activity. Cross-validation against DRAKES on an independent prediction task confirms that improvements reflect genuine regulatory signal rather than reward model overfitting.
Figures & tables
Figure 1: Loss Curve Comparison of the U-Net and our DiT.
Figure 2: Memorization and Modeling Analysis. (a) Blat Memorization Analysis counting the unique 20-bp BLAT matches across Training, Test, Generated, and Random sequences of DNA. (b) JS Distance comparing the distances between distributions of DNA in our generated DNA and the endogenous DNA sequences.
Model
Mode
GM12878
HepG2
K562
hESCT0 (DNAse)
DNA-Diffusion
In-Situ
0.53399
0.06957
0.05587
–
CNN-DiT
In-Situ
0.19025
0.04854
0.59428
0.20012
CNN-DiT
Ex-Situ
0.07316
0.04656
0.03551
0.03540
CNN-DiT-DDPO
In-Situ
4.19501
4.11424
4.76197
1.86090
CNN-DiT-DDPO
Ex-Situ
1.16401
1.17669
0.40109
0.28727
Table 1: Cell-type-specific median activity scores predicted by Enformer of generated 200 bp regulatory sequences across models and training modes. The hESCT0 predictions use the DNAse activity. Bolded values are best for their category. In-situ denotes the predictions when the 200bp sequence is embedded into the GATA1 enhancer DNA. The Ex-Situ condition predicts the activity score using only the 200bp sequence surrounded by filler tokens ( [0.25,0.25,0.25,0.25] )
Figure 3: Distribution of Enformer-predicted In-Situ Predicted Activity from 250 generated sequences. Black crosses denote the median predictions of the pre-trained model.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Cell Line
ENCODE Accession
hESCT0
ENCLB449ZZZ
K562
ENCLB843GMH
HepG2
ENCLB029COU
GM12878
ENCLB441ZZZ
Appendix
Table 2: Summary of Cell Lines and ENCODE Accession Numbers
Parameter
U-Net
DiT CNN2D
Backbone
2D Conv
Transformer
Input encoding
Direct 4×200
CNN2D (5x4 kernel)
Positional embedding
–
Learned
Hidden dim
200
320
Depth / Layers
3 (mults [1,2,4] )
6
Attention heads
4
8 (dim_head = 48)
Appendix
Table 3: Architecture and training configuration comparison between the U-Net baseline (DNA-Diffusion) and our DiT CNN2D model. U-Net values are from the pretrained checkpoint; entries marked “–” were not reported.
Early DNA foundation models adopted BERT-style training, achieving good performance on DNA understanding tasks but lacking generative capabilities. Recent autoregressive models enable DNA generation, but employ left-to-right causal modeling that is suboptimal for DNA where regulatory relationships are inherently bidirectional. We present D3LM (Discrete DNA Diffusion Language Model), which unifies bidirectional representation learning and DNA generation through masked diffusion. D3LM directly adopts the Nucleotide Transformer (NT) v2 architecture but reformulates the training objective as masked diffusion in discrete DNA space, enabling both bidirectional understanding and generation capabilities within a single model. Compared to NT v2 of the same size, D3LM achieves improved performance on understanding tasks. Notably, on regulatory element generation, D3LM achieves an SFID of 10.92, closely approaching real DNA sequences (7.85) and substantially outperforming the previous best result of 29.16 from autoregressive models. Our work suggests diffusion language models as a promising paradigm for unified DNA foundation models. We further present the first systematic study of masked diffusion models in the DNA domain, investigating practical design choices such as tokenization schemes and sampling strategies, thereby providing empirical insights and a solid foundation for future research. D3LM has been released at https://huggingface.co/collections/Hengchang-Liu/d3lm.
Zhao Yang, Hengchang Liu, Chuan Cao +1
Gaoling School of Artificial Intelligence, Renmin University of China · Zhongguancun Academy
Motivation: Interpretability is not optional in biology: understanding gene regulation requires models whose learned structure can be directly interrogated, not merely accurate predictors whose internals resist mapping onto regulatory relationships. We ask whether an architecture mirroring the central dogma yields attention and gradient maps that recover known regulatory elements and networks in inspectable form. Results: Central Dogma Transformer II (CDT-II) mirrors the central dogma in its architecture -- DNA self-attention, RNA self-attention, and DNA-to-RNA cross-attention -- requiring only genomic embeddings and raw per-cell expression. On K562 CRISPR interference (CRISPRi) data with five genes held out entirely, CDT-II predicts perturbation effects (per-gene mean r = 0.84), recovers the GFI1B regulatory network (6.6-fold enrichment, P = 3.5 x 10^-17), and concentrates cross-attention on ENCODE regulatory elements including CTCF sites (mean 7.67x across 28 target genes, P < 0.001). Gradient attribution predicts consequences of perturbing therapeutic targets (mean r = 0.82). For TFRC, target of the anti-TfR1 antibody PPMX-T003, it identifies erythrocyte-structure, iron-dependent DNA-synthesis and oxidative-stress genes, matching anemia and ferroptosis reported clinically and preclinically -- without clinical data as input. CDT-II acts as an AI microscope, surfacing clinically relevant regulatory structure from perturbation experiments alone. Availability: Source code is available at https://github.com/nobusama/CDT2. Pre-computed embeddings, training data, and model weights are available at https://huggingface.co/datasets/nobusama17/CDT2-data.
DNA cis-regulatory elements (CREs) such as enhancers control gene expression levels. Accurately predicting regulatory activity from DNA sequences is valuable but challenging, as it requires understanding complex biological regulatory processes. Existing methods typically regress activity scores from sequences in a black-box manner, limiting both interpretability and regression performance. Meanwhile, large language models (LLMs) benefit from explicit reasoning processes, yet directly applying LLMs to raw DNA sequences performs poorly. In this paper, we bridge this gap by introducing R3LM, a framework that teaches LLMs reasoning-informed regression on regulatory DNA through structured biological knowledge. Specifically, we design a biologically grounded data format that structures DNA's regulatory information for improved LLM understanding, and construct CRE-ReasonBench, the first dataset that associates DNA sequences and activity scores with mechanistic reasoning traces. Through two-stage training that first teaches LLMs reasoning over structured biological information then performs regression, R3LM achieves state-of-the-art performance on enhancer prediction across three cell types, outperforming both LLMs with raw sequence input and specialized DNA models while providing interpretable mechanistic explanations. We expect R3LM as an interpretable reward model that can effectively assist biologists in CRE design. Code is available at https://github.com/DuanYi516/R3LM.
Yi Duan, Zhao Yang, Jiwei Zhu +3
Gaoling School of Artificial Intelligence · Renmin University of China · Beijing, China +1