EvoLen: Evolution-Guided Tokenization for DNA Language Model

Nan Huang; Xiaoxiao Zhou; Junxia Cui; Mario Tapia-Pacheco; Tiffany Amariuta; Yang Li; Jingbo Shang

cs.LGApr 9, 2026

EvoLen: Evolution-Guided Tokenization for DNA Language Model

Authors: Nan Huang, Xiaoxiao Zhou, Junxia Cui, Mario Tapia-Pacheco, Tiffany Amariuta, Yang Li, Jingbo Shang

Organizations: University of California, San Diego, La Jolla, CA 92093, USA · Washington University in St. Louis, St. Louis, MO 63130, USA

Abstract

Tokens serve as the basic units of representation in DNA language models (DNALMs), yet their design remains underexplored. Unlike natural language, DNA lacks inherent token boundaries or predefined compositional rules, making tokenization a fundamental modeling decision rather than a naturally specified one. While existing approaches like byte-pair encoding (BPE) excel at capturing token structures that reflect human-generated linguistic regularities, DNA is organized by biological function and evolutionary constraint rather than linguistic convention. We argue that DNA tokenization should prioritize functional sequence patterns like regulatory motifs-short, recurring segments under evolutionary constraint and typically preserved across species. We incorporate evolutionary information directly into the tokenization process through EvoLen, a tokenizer that combines evolutionary stratification with length-aware decoding to better preserve motif-scale functional sequence units. EvoLen uses cross-species evolutionary signals to group DNA sequences, trains separate BPE tokenizers on each group, merges the resulting vocabularies via a rule prioritizing preserved patterns, and applies length-aware decoding with dynamic programming. Through controlled experiments, EvoLen improves the preservation of functional sequence patterns, differentiation across genomic contexts, and alignment with evolutionary constraint, while matching or outperforming standard BPE across diverse DNALM benchmarks. These results demonstrate that tokenization introduces a critical inductive bias and that incorporating evolutionary information yields more biologically meaningful and interpretable sequence representations. Code, pretrained and fine-tuned checkpoints, and tokenizer files are available at https://github.com/HN020719/EvoLen and https://huggingface.co/EvoLenTokenizer.

Figures & tables

Figure 1: EvoLen tokenization pipeline. A) Partition the genome into conserved, neutral, and accelerated regions using phyloP scores and train a category-specific BPE tokenizer on each sequence pool. B) Merge the resulting vocabularies using a conservation-prioritized rule. C) Apply length-aware scoring to the merged vocabulary and dynamic programming to ensure optimal, non-overlapping segmentation. This enables EvoLen to preserve more functional motifs as intact tokens than BPE, enhancing representation coherence.

Figure 2: (A) EvoLen increases the fraction of motifs preserved as single tokens across all vocabulary sizes (P1). (B) Increased Jensen–Shannon distances at vocabulary size 5,120 demonstrate that EvoLen produces more distinct token-length distributions between promoters, enhancers, and exons than BPE (P2). (C) Relative gains in mean phyloP scores signify enhanced alignment with evolutionary conservation (P3).

Figure 3: Token enrichment (mean $\log_{2}$ fold-change) relative to neutral intronic background, crossed by genomic region and conservation category(P4).

		GUE						GBM			NT
	Steps	EMP	Mou	P3	PC	Spl	TF	HR	ME	Inv	His	Enh	Pro	Spl	cCRE	ATAC
Tokenizer Comparison
DNAbert2	200k	46.2	53.6	76.1	59.1	73.5	59.0	68.7	48.6	69.9	54.5	46.8	72.9	69.6	22.0	8.8
NT	200k	40.4	46.2	79.1	63.4	75.3	53.9	62.4	50.5	67.8	52.2	46.8	73.4	93.9	21.3	9.8
Grover	200k	48.2	60.8	76.3	62.3	79.9	62.4	72.0	47.7	71.4	54.8	47.9	73.5	79.6	23.9	14.5
Baseline Comparison

Table 1: Average MCC (%) across downstream benchmarks. Bold = best; underline = second-best, computed across all seven model rows (including both step counts of Base and EvoLen).

\Delta

% = relative improvement of EvoLen over baseline BPE. Steps = number of pretraining steps. Fine-tuning hyperparameters are selected per task. Column abbreviations — GUE: EMP = EMP (Yeast), Mou = Mouse, P3 = Promoter-300, PC = Promoter-Core, Spl = Splice, TF = TF binding; GBM: HR = Human regulatory, ME = Mouse enhancers, Inv = Invertebrates; NT: His = Histone marks, Enh = Enhancers, Pro = Promoters, Spl = Splice; cCRE = Multi-SCREEN (Human cCRE); ATAC = snATAC-seq (Human/Mouse brain).

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Motif preservation diagnostics across vocabulary sizes: perfect match rate, fragmentation, token coverage fraction, and tokenization consistency. EvoLen improves exact motif preservation while maintaining comparable performance on other metrics.

Figure 5: Motif fragmentation examples at vocabulary size 5,120. EvoLen preserves motifs as intact tokens, whereas BPE splits them across token boundaries.

Tokenizer	AvgTok/Motif	PerfectMatch%	ExactVocab%	AvgTokenFrac
Baseline BPE 5120	2.776	1.659	1.659	0.389
EvoLen 5120	2.804	2.114	2.114	0.389
Baseline BPE 4096	2.842	1.525	1.525	0.380
EvoLen 4096	2.849	1.793	1.793	0.382
Baseline BPE 3072	2.914	1.391	1.391	0.371
EvoLen 3072	2.934	1.445	1.445	0.371

Appendix

Table 2: Motif preservation summary across tokenizer variants and vocabulary sizes.

Method	Vocab	Region	MeanPhyloP	% $>$ 0	MeanVar	Tokens
EvoLen	2048	conserved	0.558	100.0	0.283	2,042
EvoLen	2048	neutral	0.080	87.1	0.348	2,042
EvoLen	2048	accelerated	$-$ 0.273	0.3	0.635	2,042
Baseline	2048	conserved	0.549	100.0	0.283	2,035
Baseline	2048	neutral	0.072	84.7	0.347	2,042
Baseline	2048	accelerated	$-$ 0.271	0.9	0.633	2,042

Appendix

Table 3: PhyloP summary by conservation category and vocabulary size, including other tokenization strategies at their native vocabulary sizes.

Method	Vocab	Region	Pct1–2	Pct3–5	Pct6–8	Pct9+
Baseline	3072	promoter	4.48	63.72	30.91	0.90
Baseline	3072	enhancer	4.12	62.00	33.23	0.66
Baseline	3072	exon	4.01	62.25	32.91	0.84
Baseline	3072	intron	2.15	61.01	35.32	1.52
EvoLen	3072	promoter	26.59	23.71	49.03	0.67
EvoLen	3072	enhancer	27.79	18.85	52.81	0.55

Appendix

Table 4: Token length-bin composition (%) across four genomic regions at vocabulary sizes 3,072 and 5,120.

RegionA	RegionB	Baseline	EvoLen
promoter	enhancer	0.001	0.001
promoter	exon	0.000	0.001
promoter	intron	0.005	0.004
enhancer	exon	0.000	0.000
enhancer	intron	0.005	0.002
exon	intron	0.004	0.001

Appendix

Table 5: Pairwise Jensen–Shannon divergence between region token-length distributions at vocabulary size 5,120.

Model	Vocab	SigTokens	SigProm	SigEnh	PromShare%	EnhMeanLen	EnhMeanGC
Baseline	2048	1,736	456	1,280	26.27	6.21	0.337
EvoLen	2048	1,852	380	1,472	20.52	6.31	0.332
Baseline	3072	2,449	598	1,851	24.42	6.51	0.345
EvoLen	3072	2,746	517	2,229	18.83	6.65	0.336
Baseline	4096	3,062	755	2,307	24.66	6.69	0.347
EvoLen	4096	3,546	685	2,861	19.32	6.86	0.340

Appendix

Table 6: Promoter versus enhancer token enrichment diagnostics.

Model	Vocab	SigTokens	SigExon	SigIntron	BestExon	BestIntron
Baseline	2048	1,350	470	880	A	TATT
EvoLen	2048	1,532	558	974	C	T
Baseline	3072	1,754	603	1,151	A	TATT
EvoLen	3072	2,026	720	1,306	C	T
Baseline	4096	2,066	714	1,352	A	TATT
EvoLen	4096	2,442	857	1,585	C	T

Appendix

Table 7: Exon versus intron token enrichment summary.

References
Tokenizer	Mean phyloP	% $>$ 0	Mean Var	Tok/Motif	Perfect%
Baseline BPE	0.082	85.0	0.357	2.78	1.66
EvoLen	0.097	88.7	0.361	2.80	2.11
Ablations
No Partition	0.085	85.9	0.359	2.78	1.66
No Priority	0.071	81.0	0.374	3.08	1.69

Appendix

Table 8: Ablation structural analysis (vocab size 5,120). Bold : best ablation per metric.

Benchmark	Task Group	EvoLen	w/o Len	w/o Part	w/o Pri
GUE	EMP (Yeast)	46.8	45.5	45.7	43.7
GUE	Mouse	58.8	50.2	55.8	51.5
GUE	Promoter-300	78.5	76.7	75.0	77.3
GUE	Promoter-Core	63.6	61.1	63.1	59.2
GUE	Splice	75.2	71.6	69.0	68.9
GUE	TF binding	61.0	58.5	59.0	54.5

Appendix

Table 9: Ablation downstream performance: average MCC (%) comparing full EvoLen against variants without length encoding (w/o Len), without partition (w/o Part), and without priority (w/o Pri). Bold = best; underline = second best.

#	Benchmark	Task	Base	EvoLen
1	GBM	demo_coding_vs_intergenomic_seqs	82.2	81.4
2	GBM	demo_human_or_worm	91.8	92.1
3	GBM	drosophila_enhancers_stark	55.3	54.9
4	GBM	dummy_mouse_enhancers_ensembl	58.1	63.8
5	GBM	human_enhancers_cohn	50.2	49.7
6	GBM	human_enhancers_ensembl	77.5	78.3

Appendix

Table 10: Per-task MCC (%) for baseline vs EvoLen comparison.

#	Benchmark	Task	Base	EvoLen
1	GBM	demo_coding_vs_intergenomic_seqs	91.1	90.7
2	GBM	demo_human_or_worm	95.9	96.1
3	GBM	drosophila_enhancers_stark	77.8	77.6
4	GBM	dummy_mouse_enhancers_ensembl	79.3	81.8
5	GBM	human_enhancers_cohn	75.1	74.9
6	GBM	human_enhancers_ensembl	88.7	89.1

Appendix

Table 11: Per-task ACC (%) for baseline vs EvoLen comparison.

#	Benchmark	Task	Base	EvoLen
1	GBM	demo_coding_vs_intergenomic_seqs	91.1	90.7
2	GBM	demo_human_or_worm	95.9	96.1
3	GBM	drosophila_enhancers_stark	77.3	77.4
4	GBM	dummy_mouse_enhancers_ensembl	79.1	81.7
5	GBM	human_enhancers_cohn	75.1	74.9
6	GBM	human_enhancers_ensembl	88.6	89.0

Appendix

Table 12: Per-task F1 (%) for baseline vs EvoLen comparison.

#	Benchmark	Task	Base	EvoLen
1	GBM	demo_coding_vs_intergenomic_seqs	91.1	90.7
2	GBM	demo_human_or_worm	95.9	96.1
3	GBM	drosophila_enhancers_stark	78.2	77.5
4	GBM	dummy_mouse_enhancers_ensembl	79.0	81.7
5	GBM	human_enhancers_cohn	75.1	74.9
6	GBM	human_enhancers_ensembl	88.9	89.3

Appendix

Table 13: Per-task Precision (%) for baseline vs EvoLen comparison.

#	Benchmark	Task	Base	EvoLen
1	GBM	demo_coding_vs_intergenomic_seqs	91.1	90.7
2	GBM	demo_human_or_worm	95.9	96.1
3	GBM	drosophila_enhancers_stark	77.1	77.3
4	GBM	dummy_mouse_enhancers_ensembl	79.1	82.2
5	GBM	human_enhancers_cohn	75.1	74.9
6	GBM	human_enhancers_ensembl	88.6	89.0

Appendix

Table 14: Per-task Recall (%) for baseline vs EvoLen comparison.

#	Benchmark	Task	DNAbert2	NT	Grover
1	GBM	demo_coding_vs_intergenomic_seqs	80.5	77.9	83.1
2	GBM	demo_human_or_worm	91.6	87.7	92.8
3	GBM	drosophila_enhancers_stark	37.5	37.9	38.4
4	GBM	dummy_mouse_enhancers_ensembl	48.6	50.5	47.7
5	GBM	human_enhancers_cohn	45.9	43.5	47.4
6	GBM	human_enhancers_ensembl	78.7	66.9	81.1

Appendix

Table 15: Per-task MCC (%) for tokenizer-strategy comparison.

#	Benchmark	Task	w/o Len	w/o Part	w/o Pri
1	GBM	demo_coding_vs_intergenomic_seqs	80.9	81.1	80.2
2	GBM	demo_human_or_worm	90.5	91.7	91.2
3	GBM	drosophila_enhancers_stark	52.4	49.2	41.8
4	GBM	dummy_mouse_enhancers_ensembl	46.3	47.0	59.2
5	GBM	human_enhancers_cohn	48.7	46.5	47.3
6	GBM	human_enhancers_ensembl	76.9	77.3	75.4

Appendix

Table 16: Per-task MCC (%) for ablation variants.

#	Benchmark	Task	DNAbert2	NT	Grover
1	GBM	demo_coding_vs_intergenomic_seqs	90.2	89.0	91.5
2	GBM	demo_human_or_worm	95.8	93.8	96.4
3	GBM	drosophila_enhancers_stark	69.1	69.1	69.5
4	GBM	dummy_mouse_enhancers_ensembl	71.9	75.2	72.7
5	GBM	human_enhancers_cohn	72.8	71.7	73.6
6	GBM	human_enhancers_ensembl	89.3	83.4	90.5

Appendix

Table 17: Per-task Acc (%) for tokenizer-strategy comparison.

#	Benchmark	Task	w/o Len	w/o Part	w/o Pri
1	GBM	demo_coding_vs_intergenomic_seqs	90.4	90.6	90.1
2	GBM	demo_human_or_worm	95.2	95.8	95.6
3	GBM	drosophila_enhancers_stark	76.2	74.7	71.0
4	GBM	dummy_mouse_enhancers_ensembl	73.6	71.9	78.5
5	GBM	human_enhancers_cohn	74.4	73.2	73.6
6	GBM	human_enhancers_ensembl	88.4	88.6	87.6

Appendix

Table 18: Per-task Acc (%) for ablation variants.

#	Benchmark	Task	DNAbert2	NT	Grover
1	GBM	demo_coding_vs_intergenomic_seqs	90.2	89.0	91.5
2	GBM	demo_human_or_worm	95.8	93.8	96.4
3	GBM	drosophila_enhancers_stark	68.5	68.9	68.8
4	GBM	dummy_mouse_enhancers_ensembl	71.8	75.1	72.7
5	GBM	human_enhancers_cohn	72.8	71.7	73.6
6	GBM	human_enhancers_ensembl	89.3	83.4	90.5

Appendix

Table 19: Per-task F1 (%) for tokenizer-strategy comparison.

#	Benchmark	Task	w/o Len	w/o Part	w/o Pri
1	GBM	demo_coding_vs_intergenomic_seqs	90.4	90.6	90.1
2	GBM	demo_human_or_worm	95.2	95.8	95.6
3	GBM	drosophila_enhancers_stark	76.1	74.6	70.9
4	GBM	dummy_mouse_enhancers_ensembl	73.1	71.9	78.5
5	GBM	human_enhancers_cohn	74.4	73.2	73.6
6	GBM	human_enhancers_ensembl	88.4	88.6	87.6

Appendix

Table 20: Per-task F1 (%) for ablation variants.

#	Benchmark	Task	DNAbert2	NT	Grover
1	GBM	demo_coding_vs_intergenomic_seqs	90.3	89.0	91.5
2	GBM	demo_human_or_worm	95.8	93.8	96.4
3	GBM	drosophila_enhancers_stark	69.1	68.9	69.7
4	GBM	dummy_mouse_enhancers_ensembl	74.8	75.1	73.9
5	GBM	human_enhancers_cohn	73.1	71.8	73.8
6	GBM	human_enhancers_ensembl	89.4	83.5	90.6

Appendix

Table 21: Per-task Precision (%) for tokenizer-strategy comparison.

#	Benchmark	Task	w/o Len	w/o Part	w/o Pri
1	GBM	demo_coding_vs_intergenomic_seqs	90.4	90.6	90.1
2	GBM	demo_human_or_worm	95.3	95.8	95.6
3	GBM	drosophila_enhancers_stark	76.1	74.6	70.9
4	GBM	dummy_mouse_enhancers_ensembl	73.1	73.7	79.5
5	GBM	human_enhancers_cohn	74.4	73.3	73.7
6	GBM	human_enhancers_ensembl	88.5	88.8	87.7

Appendix

Table 22: Per-task Precision (%) for ablation variants.

#	Benchmark	Task	DNAbert2	NT	Grover
1	GBM	demo_coding_vs_intergenomic_seqs	90.2	89.0	91.5
2	GBM	demo_human_or_worm	95.8	93.8	96.4
3	GBM	drosophila_enhancers_stark	68.5	69.0	68.7
4	GBM	dummy_mouse_enhancers_ensembl	73.8	75.4	73.9
5	GBM	human_enhancers_cohn	72.8	71.7	73.6
6	GBM	human_enhancers_ensembl	89.3	83.3	90.5

Appendix

Table 23: Per-task Recall (%) for tokenizer-strategy comparison.

#	Benchmark	Task	w/o Len	w/o Part	w/o Pri
1	GBM	demo_coding_vs_intergenomic_seqs	90.4	90.6	90.1
2	GBM	demo_human_or_worm	95.2	95.8	95.6
3	GBM	drosophila_enhancers_stark	76.2	74.6	71.0
4	GBM	dummy_mouse_enhancers_ensembl	73.1	73.3	79.6
5	GBM	human_enhancers_cohn	74.3	73.2	73.6
6	GBM	human_enhancers_ensembl	88.4	88.6	87.6

Appendix

Table 24: Per-task Recall (%) for ablation variants.

#	Benchmark	Task	Base	EvoLen	$\Delta$
1	GBM	demo_coding_vs_intergenomic_seqs	82.1	81.8	-0.2
2	GBM	demo_human_or_worm	92.3	92.7	+0.4
3	GBM	drosophila_enhancers_stark	54.3	56.4	+2.1
4	GBM	dummy_mouse_enhancers_ensembl	53.8	64.2	+10.4
5	GBM	human_enhancers_cohn	45.3	50.1	+4.8
6	GBM	human_enhancers_ensembl	79.4	79.3	-0.1

Appendix

Table 25: Per-task MCC (%) for the compute-matched comparison at 200k pretraining steps. Both models use identical fine-tuning hyperparameters per task; the only difference is the tokenizer. EvoLen improves on 43 of 56 tasks.

Window (bp)	Windows	SD	Lag-1	Agreement%	$\kappa$
5	49,791,284	0.360	0.256	89.12	0.293
10	24,895,642	0.285	0.363	90.74	0.399
25	9,958,256	0.224	0.521	93.45	0.562
50	4,979,128	0.195	0.608	95.67	0.703
100	2,489,564	0.175	0.614	—	—
200	1,244,782	0.157	0.569	95.70	0.699

Appendix

Table 26: PhyloP window-size diagnostics on chromosome 1 (upper) and motif boundary-crossing probability by bin size (lower).

Vocab	Con–Neu	Acc–Con	Acc–Neu
2048	0.579	0.324	0.389
3072	0.523	0.325	0.359
4096	0.547	0.330	0.364
5120	0.564	0.336	0.375

Appendix

Table 27: Pairwise vocabulary Jaccard overlap (upper) and per-pool compression and entropy (lower) for the three category-specific BPEs.

Explore similar work

LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling
Sep 29, 2026Biswajit Banerjee, Claudia Alvarez Carreno, Anton S. PetrovProtein Language Models Protein Representation Learning
Georgia Institute of Technology · University College London
D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation
Mar 2, 2026Zhao Yang, Hengchang Liu, Chuan Cao +1Language Modeling Molecular Generation
Gaoling School of Artificial Intelligence, Renmin University of China · Zhongguancun Academy
DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks
Jun 29, 2026Romain Karpinsky, Julien Mozziconacci, Mickaël DelceyLanguage Model Pretraining Fine-Tuning
CNRS, LORIA · Nancy, 54000, France · CNRS +1

Sep 29, 2026J/K move · Enter open · S save

LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling

Biswajit Banerjee, Claudia Alvarez Carreno, Anton S. Petrov

Georgia Institute of Technology · University College London

Protein Language Models Protein Representation Learning Sequence Modeling

Open paper AskPDF arXiv

Protein Language Models (PLMs) have made remarkable progress following scaling laws established in natural language processing across sequence- and structure-based tasks, yet the potential of tokenization remains underexploited. Unlike human language, proteins preserve structure despite extensive sequence variation a property standard tokenization strategies fundamentally fail to capture. We introduce ZEST (Zoned Encoding of Sequence Traits), an evolution-informed vocabulary derived from conserved regions of multiple sequence alignments. ZEST allows embedding domain-level biological priors directly at the tokenization stage rather than learning them implicitly through scale. ZEST natively compresses sequences to an average token length of 4 residues, enabling our model to process 4,000 residues within a standard 1024-token context window. Building on this, we present LEMON (Layered Extraction of Molecular Ordering from Nature), a compact 200M-parameter sequence-based model for detection of remote homology between protein sequences trained on a single H100 GPU for one week. Despite its modest size, LEMON outperforms state-of-the-art models ranging from 600M to 3B parameters. Our results demonstrate that evolution-informed tokenization can substitute for massive parameter scaling, opening a new direction for efficient, biologically-grounded protein representation learning. All code, model weights, and results are publicly available under the MIT license.

Abstract

Explore similar work

LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling

D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation

DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks