Tokens serve as the basic units of representation in DNA language models (DNALMs), yet their design remains underexplored. Unlike natural language, DNA lacks inherent token boundaries or predefined compositional rules, making tokenization a fundamental modeling decision rather than a naturally specified one. While existing approaches like byte-pair encoding (BPE) excel at capturing token structures that reflect human-generated linguistic regularities, DNA is organized by biological function and evolutionary constraint rather than linguistic convention. We argue that DNA tokenization should prioritize functional sequence patterns like regulatory motifs-short, recurring segments under evolutionary constraint and typically preserved across species. We incorporate evolutionary information directly into the tokenization process through EvoLen, a tokenizer that combines evolutionary stratification with length-aware decoding to better preserve motif-scale functional sequence units. EvoLen uses cross-species evolutionary signals to group DNA sequences, trains separate BPE tokenizers on each group, merges the resulting vocabularies via a rule prioritizing preserved patterns, and applies length-aware decoding with dynamic programming. Through controlled experiments, EvoLen improves the preservation of functional sequence patterns, differentiation across genomic contexts, and alignment with evolutionary constraint, while matching or outperforming standard BPE across diverse DNALM benchmarks. These results demonstrate that tokenization introduces a critical inductive bias and that incorporating evolutionary information yields more biologically meaningful and interpretable sequence representations. Code, pretrained and fine-tuned checkpoints, and tokenizer files are available at https://github.com/HN020719/EvoLen and https://huggingface.co/EvoLenTokenizer.
Figures & tables
Figure 1: EvoLen tokenization pipeline. A) Partition the genome into conserved, neutral, and accelerated regions using phyloP scores and train a category-specific BPE tokenizer on each sequence pool. B) Merge the resulting vocabularies using a conservation-prioritized rule. C) Apply length-aware scoring to the merged vocabulary and dynamic programming to ensure optimal, non-overlapping segmentation. This enables EvoLen to preserve more functional motifs as intact tokens than BPE, enhancing representation coherence.
Figure 2: (A) EvoLen increases the fraction of motifs preserved as single tokens across all vocabulary sizes (P1). (B) Increased Jensen–Shannon distances at vocabulary size 5,120 demonstrate that EvoLen produces more distinct token-length distributions between promoters, enhancers, and exons than BPE (P2). (C) Relative gains in mean phyloP scores signify enhanced alignment with evolutionary conservation (P3).
Figure 3: Token enrichment (mean log2 fold-change) relative to neutral intronic background, crossed by genomic region and conservation category(P4).
GUE
GBM
NT
Steps
EMP
Mou
P3
PC
Spl
TF
HR
ME
Inv
His
Enh
Pro
Spl
cCRE
ATAC
Tokenizer Comparison
DNAbert2
200k
46.2
53.6
76.1
59.1
73.5
59.0
68.7
48.6
69.9
54.5
46.8
72.9
69.6
22.0
8.8
NT
200k
40.4
46.2
79.1
63.4
75.3
53.9
62.4
50.5
67.8
52.2
46.8
73.4
93.9
21.3
9.8
Grover
200k
48.2
60.8
76.3
62.3
79.9
62.4
72.0
47.7
71.4
54.8
47.9
73.5
79.6
23.9
14.5
Baseline Comparison
Table 1: Average MCC (%) across downstream benchmarks. Bold = best; underline = second-best, computed across all seven model rows (including both step counts of Base and EvoLen). Δ % = relative improvement of EvoLen over baseline BPE. Steps = number of pretraining steps. Fine-tuning hyperparameters are selected per task. Column abbreviations — GUE: EMP = EMP (Yeast), Mou = Mouse, P3 = Promoter-300, PC = Promoter-Core, Spl = Splice, TF = TF binding; GBM: HR = Human regulatory, ME = Mouse enhancers, Inv = Invertebrates; NT: His = Histone marks, Enh = Enhancers, Pro = Promoters, Spl = Splice; cCRE = Multi-SCREEN (Human cCRE); ATAC = snATAC-seq (Human/Mouse brain).
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Motif preservation diagnostics across vocabulary sizes: perfect match rate, fragmentation, token coverage fraction, and tokenization consistency. EvoLen improves exact motif preservation while maintaining comparable performance on other metrics.
Figure 5: Motif fragmentation examples at vocabulary size 5,120. EvoLen preserves motifs as intact tokens, whereas BPE splits them across token boundaries.
Tokenizer
AvgTok/Motif
PerfectMatch%
ExactVocab%
AvgTokenFrac
Baseline BPE 5120
2.776
1.659
1.659
0.389
EvoLen 5120
2.804
2.114
2.114
0.389
Baseline BPE 4096
2.842
1.525
1.525
0.380
EvoLen 4096
2.849
1.793
1.793
0.382
Baseline BPE 3072
2.914
1.391
1.391
0.371
EvoLen 3072
2.934
1.445
1.445
0.371
Appendix
Table 2: Motif preservation summary across tokenizer variants and vocabulary sizes.
Method
Vocab
Region
MeanPhyloP
% > 0
MeanVar
Tokens
EvoLen
2048
conserved
0.558
100.0
0.283
2,042
EvoLen
2048
neutral
0.080
87.1
0.348
2,042
EvoLen
2048
accelerated
− 0.273
0.3
0.635
2,042
Baseline
2048
conserved
0.549
100.0
0.283
2,035
Baseline
2048
neutral
0.072
84.7
0.347
2,042
Baseline
2048
accelerated
− 0.271
0.9
0.633
2,042
Appendix
Table 3: PhyloP summary by conservation category and vocabulary size, including other tokenization strategies at their native vocabulary sizes.
Method
Vocab
Region
Pct1–2
Pct3–5
Pct6–8
Pct9+
Baseline
3072
promoter
4.48
63.72
30.91
0.90
Baseline
3072
enhancer
4.12
62.00
33.23
0.66
Baseline
3072
exon
4.01
62.25
32.91
0.84
Baseline
3072
intron
2.15
61.01
35.32
1.52
EvoLen
3072
promoter
26.59
23.71
49.03
0.67
EvoLen
3072
enhancer
27.79
18.85
52.81
0.55
Appendix
Table 4: Token length-bin composition (%) across four genomic regions at vocabulary sizes 3,072 and 5,120.
RegionA
RegionB
Baseline
EvoLen
promoter
enhancer
0.001
0.001
promoter
exon
0.000
0.001
promoter
intron
0.005
0.004
enhancer
exon
0.000
0.000
enhancer
intron
0.005
0.002
exon
intron
0.004
0.001
Appendix
Table 5: Pairwise Jensen–Shannon divergence between region token-length distributions at vocabulary size 5,120.
Model
Vocab
SigTokens
SigProm
SigEnh
PromShare%
EnhMeanLen
EnhMeanGC
Baseline
2048
1,736
456
1,280
26.27
6.21
0.337
EvoLen
2048
1,852
380
1,472
20.52
6.31
0.332
Baseline
3072
2,449
598
1,851
24.42
6.51
0.345
EvoLen
3072
2,746
517
2,229
18.83
6.65
0.336
Baseline
4096
3,062
755
2,307
24.66
6.69
0.347
EvoLen
4096
3,546
685
2,861
19.32
6.86
0.340
Appendix
Table 6: Promoter versus enhancer token enrichment diagnostics.
Model
Vocab
SigTokens
SigExon
SigIntron
BestExon
BestIntron
Baseline
2048
1,350
470
880
A
TATT
EvoLen
2048
1,532
558
974
C
T
Baseline
3072
1,754
603
1,151
A
TATT
EvoLen
3072
2,026
720
1,306
C
T
Baseline
4096
2,066
714
1,352
A
TATT
EvoLen
4096
2,442
857
1,585
C
T
Appendix
Table 7: Exon versus intron token enrichment summary.
Tokenizer
Mean phyloP
% > 0
Mean Var
Tok/Motif
Perfect%
References
Baseline BPE
0.082
85.0
0.357
2.78
1.66
EvoLen
0.097
88.7
0.361
2.80
2.11
Ablations
No Partition
0.085
85.9
0.359
2.78
1.66
No Priority
0.071
81.0
0.374
3.08
1.69
Appendix
Table 8: Ablation structural analysis (vocab size 5,120). Bold : best ablation per metric.
Benchmark
Task Group
EvoLen
w/o Len
w/o Part
w/o Pri
GUE
EMP (Yeast)
46.8
45.5
45.7
43.7
GUE
Mouse
58.8
50.2
55.8
51.5
GUE
Promoter-300
78.5
76.7
75.0
77.3
GUE
Promoter-Core
63.6
61.1
63.1
59.2
GUE
Splice
75.2
71.6
69.0
68.9
GUE
TF binding
61.0
58.5
59.0
54.5
Appendix
Table 9: Ablation downstream performance: average MCC (%) comparing full EvoLen against variants without length encoding (w/o Len), without partition (w/o Part), and without priority (w/o Pri). Bold = best; underline = second best.
#
Benchmark
Task
Base
EvoLen
1
GBM
demo_coding_vs_intergenomic_seqs
82.2
81.4
2
GBM
demo_human_or_worm
91.8
92.1
3
GBM
drosophila_enhancers_stark
55.3
54.9
4
GBM
dummy_mouse_enhancers_ensembl
58.1
63.8
5
GBM
human_enhancers_cohn
50.2
49.7
6
GBM
human_enhancers_ensembl
77.5
78.3
Appendix
Table 10: Per-task MCC (%) for baseline vs EvoLen comparison.
#
Benchmark
Task
Base
EvoLen
1
GBM
demo_coding_vs_intergenomic_seqs
91.1
90.7
2
GBM
demo_human_or_worm
95.9
96.1
3
GBM
drosophila_enhancers_stark
77.8
77.6
4
GBM
dummy_mouse_enhancers_ensembl
79.3
81.8
5
GBM
human_enhancers_cohn
75.1
74.9
6
GBM
human_enhancers_ensembl
88.7
89.1
Appendix
Table 11: Per-task ACC (%) for baseline vs EvoLen comparison.
#
Benchmark
Task
Base
EvoLen
1
GBM
demo_coding_vs_intergenomic_seqs
91.1
90.7
2
GBM
demo_human_or_worm
95.9
96.1
3
GBM
drosophila_enhancers_stark
77.3
77.4
4
GBM
dummy_mouse_enhancers_ensembl
79.1
81.7
5
GBM
human_enhancers_cohn
75.1
74.9
6
GBM
human_enhancers_ensembl
88.6
89.0
Appendix
Table 12: Per-task F1 (%) for baseline vs EvoLen comparison.
#
Benchmark
Task
Base
EvoLen
1
GBM
demo_coding_vs_intergenomic_seqs
91.1
90.7
2
GBM
demo_human_or_worm
95.9
96.1
3
GBM
drosophila_enhancers_stark
78.2
77.5
4
GBM
dummy_mouse_enhancers_ensembl
79.0
81.7
5
GBM
human_enhancers_cohn
75.1
74.9
6
GBM
human_enhancers_ensembl
88.9
89.3
Appendix
Table 13: Per-task Precision (%) for baseline vs EvoLen comparison.
#
Benchmark
Task
Base
EvoLen
1
GBM
demo_coding_vs_intergenomic_seqs
91.1
90.7
2
GBM
demo_human_or_worm
95.9
96.1
3
GBM
drosophila_enhancers_stark
77.1
77.3
4
GBM
dummy_mouse_enhancers_ensembl
79.1
82.2
5
GBM
human_enhancers_cohn
75.1
74.9
6
GBM
human_enhancers_ensembl
88.6
89.0
Appendix
Table 14: Per-task Recall (%) for baseline vs EvoLen comparison.
#
Benchmark
Task
DNAbert2
NT
Grover
1
GBM
demo_coding_vs_intergenomic_seqs
80.5
77.9
83.1
2
GBM
demo_human_or_worm
91.6
87.7
92.8
3
GBM
drosophila_enhancers_stark
37.5
37.9
38.4
4
GBM
dummy_mouse_enhancers_ensembl
48.6
50.5
47.7
5
GBM
human_enhancers_cohn
45.9
43.5
47.4
6
GBM
human_enhancers_ensembl
78.7
66.9
81.1
Appendix
Table 15: Per-task MCC (%) for tokenizer-strategy comparison.
#
Benchmark
Task
w/o Len
w/o Part
w/o Pri
1
GBM
demo_coding_vs_intergenomic_seqs
80.9
81.1
80.2
2
GBM
demo_human_or_worm
90.5
91.7
91.2
3
GBM
drosophila_enhancers_stark
52.4
49.2
41.8
4
GBM
dummy_mouse_enhancers_ensembl
46.3
47.0
59.2
5
GBM
human_enhancers_cohn
48.7
46.5
47.3
6
GBM
human_enhancers_ensembl
76.9
77.3
75.4
Appendix
Table 16: Per-task MCC (%) for ablation variants.
#
Benchmark
Task
DNAbert2
NT
Grover
1
GBM
demo_coding_vs_intergenomic_seqs
90.2
89.0
91.5
2
GBM
demo_human_or_worm
95.8
93.8
96.4
3
GBM
drosophila_enhancers_stark
69.1
69.1
69.5
4
GBM
dummy_mouse_enhancers_ensembl
71.9
75.2
72.7
5
GBM
human_enhancers_cohn
72.8
71.7
73.6
6
GBM
human_enhancers_ensembl
89.3
83.4
90.5
Appendix
Table 17: Per-task Acc (%) for tokenizer-strategy comparison.
#
Benchmark
Task
w/o Len
w/o Part
w/o Pri
1
GBM
demo_coding_vs_intergenomic_seqs
90.4
90.6
90.1
2
GBM
demo_human_or_worm
95.2
95.8
95.6
3
GBM
drosophila_enhancers_stark
76.2
74.7
71.0
4
GBM
dummy_mouse_enhancers_ensembl
73.6
71.9
78.5
5
GBM
human_enhancers_cohn
74.4
73.2
73.6
6
GBM
human_enhancers_ensembl
88.4
88.6
87.6
Appendix
Table 18: Per-task Acc (%) for ablation variants.
#
Benchmark
Task
DNAbert2
NT
Grover
1
GBM
demo_coding_vs_intergenomic_seqs
90.2
89.0
91.5
2
GBM
demo_human_or_worm
95.8
93.8
96.4
3
GBM
drosophila_enhancers_stark
68.5
68.9
68.8
4
GBM
dummy_mouse_enhancers_ensembl
71.8
75.1
72.7
5
GBM
human_enhancers_cohn
72.8
71.7
73.6
6
GBM
human_enhancers_ensembl
89.3
83.4
90.5
Appendix
Table 19: Per-task F1 (%) for tokenizer-strategy comparison.
#
Benchmark
Task
w/o Len
w/o Part
w/o Pri
1
GBM
demo_coding_vs_intergenomic_seqs
90.4
90.6
90.1
2
GBM
demo_human_or_worm
95.2
95.8
95.6
3
GBM
drosophila_enhancers_stark
76.1
74.6
70.9
4
GBM
dummy_mouse_enhancers_ensembl
73.1
71.9
78.5
5
GBM
human_enhancers_cohn
74.4
73.2
73.6
6
GBM
human_enhancers_ensembl
88.4
88.6
87.6
Appendix
Table 20: Per-task F1 (%) for ablation variants.
#
Benchmark
Task
DNAbert2
NT
Grover
1
GBM
demo_coding_vs_intergenomic_seqs
90.3
89.0
91.5
2
GBM
demo_human_or_worm
95.8
93.8
96.4
3
GBM
drosophila_enhancers_stark
69.1
68.9
69.7
4
GBM
dummy_mouse_enhancers_ensembl
74.8
75.1
73.9
5
GBM
human_enhancers_cohn
73.1
71.8
73.8
6
GBM
human_enhancers_ensembl
89.4
83.5
90.6
Appendix
Table 21: Per-task Precision (%) for tokenizer-strategy comparison.
#
Benchmark
Task
w/o Len
w/o Part
w/o Pri
1
GBM
demo_coding_vs_intergenomic_seqs
90.4
90.6
90.1
2
GBM
demo_human_or_worm
95.3
95.8
95.6
3
GBM
drosophila_enhancers_stark
76.1
74.6
70.9
4
GBM
dummy_mouse_enhancers_ensembl
73.1
73.7
79.5
5
GBM
human_enhancers_cohn
74.4
73.3
73.7
6
GBM
human_enhancers_ensembl
88.5
88.8
87.7
Appendix
Table 22: Per-task Precision (%) for ablation variants.
#
Benchmark
Task
DNAbert2
NT
Grover
1
GBM
demo_coding_vs_intergenomic_seqs
90.2
89.0
91.5
2
GBM
demo_human_or_worm
95.8
93.8
96.4
3
GBM
drosophila_enhancers_stark
68.5
69.0
68.7
4
GBM
dummy_mouse_enhancers_ensembl
73.8
75.4
73.9
5
GBM
human_enhancers_cohn
72.8
71.7
73.6
6
GBM
human_enhancers_ensembl
89.3
83.3
90.5
Appendix
Table 23: Per-task Recall (%) for tokenizer-strategy comparison.
#
Benchmark
Task
w/o Len
w/o Part
w/o Pri
1
GBM
demo_coding_vs_intergenomic_seqs
90.4
90.6
90.1
2
GBM
demo_human_or_worm
95.2
95.8
95.6
3
GBM
drosophila_enhancers_stark
76.2
74.6
71.0
4
GBM
dummy_mouse_enhancers_ensembl
73.1
73.3
79.6
5
GBM
human_enhancers_cohn
74.3
73.2
73.6
6
GBM
human_enhancers_ensembl
88.4
88.6
87.6
Appendix
Table 24: Per-task Recall (%) for ablation variants.
#
Benchmark
Task
Base
EvoLen
Δ
1
GBM
demo_coding_vs_intergenomic_seqs
82.1
81.8
-0.2
2
GBM
demo_human_or_worm
92.3
92.7
+0.4
3
GBM
drosophila_enhancers_stark
54.3
56.4
+2.1
4
GBM
dummy_mouse_enhancers_ensembl
53.8
64.2
+10.4
5
GBM
human_enhancers_cohn
45.3
50.1
+4.8
6
GBM
human_enhancers_ensembl
79.4
79.3
-0.1
Appendix
Table 25: Per-task MCC (%) for the compute-matched comparison at 200k pretraining steps. Both models use identical fine-tuning hyperparameters per task; the only difference is the tokenizer. EvoLen improves on 43 of 56 tasks.
Window (bp)
Windows
SD
Lag-1
Agreement%
κ
5
49,791,284
0.360
0.256
89.12
0.293
10
24,895,642
0.285
0.363
90.74
0.399
25
9,958,256
0.224
0.521
93.45
0.562
50
4,979,128
0.195
0.608
95.67
0.703
100
2,489,564
0.175
0.614
—
—
200
1,244,782
0.157
0.569
95.70
0.699
Appendix
Table 26: PhyloP window-size diagnostics on chromosome 1 (upper) and motif boundary-crossing probability by bin size (lower).
Vocab
Con–Neu
Acc–Con
Acc–Neu
2048
0.579
0.324
0.389
3072
0.523
0.325
0.359
4096
0.547
0.330
0.364
5120
0.564
0.336
0.375
Appendix
Table 27: Pairwise vocabulary Jaccard overlap (upper) and per-pool compression and entropy (lower) for the three category-specific BPEs.
Protein Language Models (PLMs) have made remarkable progress following scaling laws established in natural language processing across sequence- and structure-based tasks, yet the potential of tokenization remains underexploited. Unlike human language, proteins preserve structure despite extensive sequence variation a property standard tokenization strategies fundamentally fail to capture. We introduce ZEST (Zoned Encoding of Sequence Traits), an evolution-informed vocabulary derived from conserved regions of multiple sequence alignments. ZEST allows embedding domain-level biological priors directly at the tokenization stage rather than learning them implicitly through scale. ZEST natively compresses sequences to an average token length of 4 residues, enabling our model to process 4,000 residues within a standard 1024-token context window. Building on this, we present LEMON (Layered Extraction of Molecular Ordering from Nature), a compact 200M-parameter sequence-based model for detection of remote homology between protein sequences trained on a single H100 GPU for one week. Despite its modest size, LEMON outperforms state-of-the-art models ranging from 600M to 3B parameters. Our results demonstrate that evolution-informed tokenization can substitute for massive parameter scaling, opening a new direction for efficient, biologically-grounded protein representation learning. All code, model weights, and results are publicly available under the MIT license.
Biswajit Banerjee, Claudia Alvarez Carreno, Anton S. Petrov
Georgia Institute of Technology · University College London
Early DNA foundation models adopted BERT-style training, achieving good performance on DNA understanding tasks but lacking generative capabilities. Recent autoregressive models enable DNA generation, but employ left-to-right causal modeling that is suboptimal for DNA where regulatory relationships are inherently bidirectional. We present D3LM (Discrete DNA Diffusion Language Model), which unifies bidirectional representation learning and DNA generation through masked diffusion. D3LM directly adopts the Nucleotide Transformer (NT) v2 architecture but reformulates the training objective as masked diffusion in discrete DNA space, enabling both bidirectional understanding and generation capabilities within a single model. Compared to NT v2 of the same size, D3LM achieves improved performance on understanding tasks. Notably, on regulatory element generation, D3LM achieves an SFID of 10.92, closely approaching real DNA sequences (7.85) and substantially outperforming the previous best result of 29.16 from autoregressive models. Our work suggests diffusion language models as a promising paradigm for unified DNA foundation models. We further present the first systematic study of masked diffusion models in the DNA domain, investigating practical design choices such as tokenization schemes and sampling strategies, thereby providing empirical insights and a solid foundation for future research. D3LM has been released at https://huggingface.co/collections/Hengchang-Liu/d3lm.
Zhao Yang, Hengchang Liu, Chuan Cao +1
Gaoling School of Artificial Intelligence, Renmin University of China · Zhongguancun Academy
Recent breakthroughs in foundation models and Large Language Models (LLMs) have introduced new opportunities for studying and decoding genomic sequences. Several state-of-the-art approaches, such as DNABERT2, rely on transformer-based architectures, while others, such as ConvNova, still build upon more conventional convolutional models. However, systematic benchmark comparisons across these methods remain scarce. Given that transformer-based models require extensive and costly pretraining, it is crucial to evaluate whether their performance gains justify this overhead. Moreover, LLMs such as DNABERT2 typically rely on Byte Pair Encoding (BPE) tokenization, whose relevance for DNA sequence representation is still debated within the genomics community. In this work, we investigate three key questions: (i) do transformer-based models provide sufficient improvements on fine-tuning tasks upon heavy pretraining, (ii) what is the actual contribution of pretraining in this setting, and (iii) how does BPE tokenization impact performance on genomics-related tasks?