BarcodeMAE+: Rethinking Masked Pretraining and Global Representations for DNA Barcode Foundation Models

Monireh Safari; Pablo Millan Arias; Scott C. Lowe; Lila Kari; Angel X. Chang; Graham W. Taylor

q-bio.GNSep 26, 2026

BarcodeMAE+: Rethinking Masked Pretraining and Global Representations for DNA Barcode Foundation Models

Authors: Monireh Safari, Pablo Millan Arias, Scott C. Lowe, Lila Kari, Angel X. Chang, Graham W. Taylor

Organizations: Cheriton School of Computer Science, University of Waterloo, Ontario, Canada · Vector Institute, Ontario, Canada · School of Computing Science, Simon Fraser University, British Columbia, Canada · Alberta Machine Intelligence Institute (Amii), Alberta, Canada ·

Abstract

Many DNA foundation models are pretrained by masking parts of a sequence and asking the model to reconstruct them. Standard masked pretraining exposes the encoder to special [MASK] tokens that are absent at inference, creating a mismatch between training and downstream use. The role of an explicit global sequence representation such as a [CLS] token and how it should be trained also remain poorly understood for DNA barcodes. We introduce BarcodeMAE+ and study model architecture, global [CLS] representation, and auxiliary pretraining objectives across arthropod COI (BIOSCAN-5M) and fungal ITS (UNITE+INSD) barcodes. Across both barcode regions, the encoder-decoder MAE-LM architecture outperforms its matched encoder-only counterpart in nearly all evaluated configurations, supporting MAE-LM as an effective architectural design for DNA barcode foundation models. A trained global [CLS] representation provides substantial additional gains: on BIOSCAN-5M, [CLS] accuracy increases from 47.53% without an auxiliary objective to 80.65% with cross-entropy genus classification. The best auxiliary objective is region-dependent: cross-entropy performs best on BIOSCAN-5M, whereas pairwise same-genus classification performs best on UNITE+INSD, reaching 73.19% on Yeast and 63.07% on Filamentous Fungi. BarcodeMAE+ outperforms published DNA foundation model baselines on BIOSCAN-5M and achieves the highest Yeast accuracy among the evaluated UNITE+INSD baselines using frozen encoder representations. Similarity-weighted softmax KNN voting further stabilizes accuracy as neighbourhood size increases. Overall, encoder-decoder masked pretraining and an explicitly trained global representation are strong design choices for DNA barcode foundation models, while the optimal objective for learning that representation depends on the biological domain.

Figures & tables

Figure 1: Architecture of the two encoder pretraining variants compared in this work, both operating on the same tokenized DNA sequence. Input sequences are tokenized into

k

-mers using a random offset

o\in\{0,\ldots,k-1\}

, improving robustness to mutations. All

k

-mers containing ambiguous nucleotides, N , are mapped to [UNK] . A random subset of tokens (50%) is selected for masking, and an optional learnable [CLS] token is prepended, attending to all visible sequence tokens through standard multi-head self-attention. Encoder-only model (left). Masked tokens are replaced by [MASK] embeddings, and an encoder processes the full sequence. The encoder outputs are used directly to predict the masked tokens and, when applicable, to predict taxonomic labels from the [CLS] representation. Encoder-decoder model (right). Masked tokens are removed before the encoder, which therefore processes only the observed tokens. A decoder receives the encoder representations at observed positions and [MASK] embeddings at masked positions, and reconstructs the original tokens. When an auxiliary objective is used, the [CLS] representation from encoder is passed to a taxonomic head for pairwise binary classification, triplet learning, or cross-entropy classification. Only sequences with genus-level labels contribute to the auxiliary loss,

\mathcal{L}_{\mathrm{aux}}

, while all sequences contribute to

\mathcal{L}_{\mathrm{rec}}

. The total loss is

\mathcal{L}=\mathcal{L}_{\mathrm{rec}}+\alpha\mathcal{L}_{\mathrm{aux}}

. The decoder in the encoder-decoder model, and the auxiliary head in both models are discarded after pretraining, and only the encoder is used for downstream evaluation.

#	Architecture	CLS	Aux
1	Encoder-decoder	×	–
2	Encoder-decoder	✓	–
3	Encoder-decoder	✓	Binary
4	Encoder-decoder	✓	Triplet
5	Encoder-decoder	✓	CE
6	Encoder-only	×	–

Table 1: Pretraining configurations evaluated on the barcode datasets (BIOSCAN-5M and UNITE+INSD). The Architecture column indicates whether the architecture uses an encoder-decoder MAE-LM or an encoder-only model. The CLS column indicates whether a global CLS token is prepended to the encoder input. The Aux column indicates the auxiliary taxonomic objective attached to the [CLS] token; “–” means reconstruction loss only.

Representation	Description
Tokens	Mean over DNA tokens only (excluding CLS)
CLS	Output of [CLS] token
Tokens+CLS	Mean over all encoder outputs (DNA tokens + CLS)

Table 2: Sequence representation strategies used for evaluation in this work.

Architecture	Pretraining data	Model	Training task	1-NN Acc.	Softmax $\Delta$ (pp)	BIN AMI	HM
State space	Human genome	HyenaDNA-tiny [ 20 ]	NTP	20.52	0.00	61.26	30.74
	Human genome	Caduceus-PS-1k [ 25 ]	MLM	12.84	0.00	54.92	20.81
	BIOSCAN-5M	BarcodeMamba+ [ 11 ]	NTP	29.47	0.00	66.78	40.89
Encoder-only	Multi-species DNA	DNABERT-2 [ 31 ]	MLM	16.08	0.00	30.70	21.11
	Multi-species DNA	DNABERT-S [ 32 ]	CL	19.41	+1.17 (k=20)	60.25	29.36
	Multi-species DNA	Nucleotide Transformer [ 3 ]	MLM	22.88	0.00	32.52	26.86

Table 3: Comparison with DNA foundation model baselines, evaluated zero-shot (frozen embeddings, no fine-tuning) on the unseen-species test partition of BIOSCAN-5M . Baselines are pretrained on a range of corpora. We report genus-level 1-NN accuracy (

k{=}1

, closed-world), the best softmax-voted KNN accuracy minus the 1-NN accuracy, in percentage points (Softmax

\Delta

), searched over

k\in\{1,3,5,7,10,15,20,25,50\}

and

T\in\{0.01,0.02,0.05,0.07,0.1,0.2,0.5,1.0\}

, zero-shot BIN reconstruction AMI (open-world), and their harmonic mean (HM). A value of 0.00 (shown in gray) indicates that the best softmax accuracy occurs at

k=1

, where softmax weighting reduces to 1-NN voting, so the value is identical to the 1-NN column. None of these models are trained end-to-end via full supervised training; all are evaluated on frozen embeddings. Training task abbreviations are, NTP = next-token prediction, MLM = masked language modelling, CL = contrastive learning, MLM+CE = masked language modelling with a cross-entropy genus-classification auxiliary objective. The best value is shown in bold , and the second-best value is underlined .

Architecture	Training data	Model	Training task	Yeast		Filamentous
				1-NN	Softmax $\Delta$ (pp)	1-NN	Softmax $\Delta$ (pp)
State space	Human genome	HyenaDNA-tiny [ 20 ]	NTP	35.17	0.00	37.72	0.00
	Human genome	Caduceus-PS-1k [ 25 ]	MLM	27.95	0.00	33.51	0.00
	UNITE+INSD	BarcodeMamba+ [ 10 ]	NTP	62.35	0.00	55.58	+1.46 (k=5)
	UNITE+INSD	BarcodeMamba+ (large) [ 10 ]	NTP	61.03	+1.51 (k=7)	55.49	+0.82 (k=5)
Encoder-only	Multi-species DNA	DNABERT-2 [ 31 ]	MLM	61.03	0.00	54.02	+0.13 (k=3)

Table 4: Comparison with baselines on UNITE+INSD , evaluated using frozen embeddings on the deduplicated Yeast and Filamentous query pools. We report genus-level 1-NN accuracy and the best softmax-voted KNN accuracy minus the 1-NN accuracy, in percentage points (Softmax

\Delta

), searched over

k\in\{1,3,5,7,10,15,20,25,50\}

and

T\in\{0.01,0.02,0.05,0.07,0.1,0.2,0.5,1.0\}

for each test set. A value of 0.00 (shown in gray) indicates that the best softmax accuracy occurs at

k=1

, where softmax weighting reduces to 1-NN voting, so the value is identical to the 1-NN column. Training task abbreviations are, NTP = next-token prediction, MLM = masked language modelling, CL = contrastive learning, SC = supervised classification, MLM+Binary = masked language modelling with a same-genus pairwise auxiliary objective. The best value in each 1-NN column is shown in bold , and the second-best value is underlined .

		Encoder-decoder	Encoder-only
Configuration	Representation	BIN AMI	BIN AMI
Base (no CLS)	Tokens	74.32	55.65
+CLS	CLS	54.94	51.78
+CLS+Binary	CLS	75.02	73.39
+CLS+Triplet	CLS	74.31	72.76
+CLS+CE	CLS	75.79	74.45

Table 5: Zero-shot BIN reconstruction AMI (%) on BIOSCAN-5M for all five configurations, comparing the encoder-decoder and encoder-only architectures. The best value is shown in bold , and the second-best value is underlined .

Figure 2: Genus-level 1-NN (

k{=}1

) accuracy across all architectures, auxiliary objectives, and sequence representations (Tokens, CLS, Tokens+CLS columns), for (a) BIOSCAN-5M, (b) UNITE+INSD Yeast, and (c) UNITE+INSD Filamentous. The red dash-dot line shows the randomly-initialized encoder baseline.

Figure 3: Genus-level KNN accuracy vs.

k

for each dataset’s best configuration: (a) BIOSCAN-5M (encoder-decoder, +CLS+CE, CLS representation), (b) UNITE+INSD Yeast and (c) UNITE+INSD Filamentous (encoder-decoder, +CLS+Binary, CLS representation), comparing uniform voting (dashed), softmax voting (solid,

T=0.02

), and an untrained model (dotted). Softmax voting substantially stabilizes accuracy at large

k

in all cases.

Voting	Peak acc. (%)	Mean (%)	Std. (pp)	Range (pp)
Uniform	80.65 ( $k{=}\phantom{0}1$ )	72.55	7.00	23.35
Softmax	80.65 ( $k{=}\phantom{0}1$ )	80.53	0.09	0 0.27

Table 6: Stability of the best BIOSCAN-5M configuration (encoder-decoder, +CLS+CE, CLS representation) across

k\in\{1,3,5,7,10,15,20,25,50\}

, under uniform and softmax voting. Mean is the average accuracy across all nine values of

k

tested.

Test set	Voting	Peak acc. (%)	Mean (%)	Std. (pp)	Range (pp)
Yeast	Uniform	74.90 ( $k{=}7$ )	72.05	3.10	10.45
Yeast	Softmax	76.43 ( $k{=}25$ )	75.14	1.12	0 3.24
Filamentous	Uniform	63.36 ( $k{=}7$ )	62.50	0.98	0 3.32
Filamentous	Softmax	65.21 ( $k{=}50$ )	64.07	0.71	0 2.17

Table 7: Stability of the best UNITE+INSD configuration (encoder-decoder, +CLS+Binary, CLS representation) on the deduplicated Yeast and Filamentous query pools across

k\in\{1,3,5,7,10,15,20,25,50\}

under uniform and softmax voting. Mean is the average accuracy across all nine values of

k

tested.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: BIOSCAN-5M hierarchical taxonomic distribution. Each ring is one taxonomic rank, from phylum (innermost) through class, order, family, to genus (outermost); wedge angular width is proportional to the number of specimens at that taxon. Since BIOSCAN-5M covers only arthropods, the innermost ring is a single wedge (phylum Arthropoda); colour and the legend denote class, the coarsest rank with more than one value in this dataset, with descendant ranks shaded progressively lighter shades of their class’s colour moving outward.

Panel of Figure 5 — Figure 5: UNITE+INSD hierarchical taxonomic distribution. As in Figure 4 , each ring is one taxonomic rank, from phylum (innermost) through class, order, family, to genus (outermost), with wedge angular width proportional to the number of specimens at that taxon. Colour and the legend denote phylum here (rather than class, as for BIOSCAN-5M), since UNITE+INSD spans multiple fungal phyla rather than a single one; descendant ranks are shaded progressively lighter shades of their phylum’s colour moving outward. The two query sets are the deduplicated, genus-level query populations actually used for evaluation ( Table 8 ), not the raw released test sets.

Test set	Metric	Training set	Original test set	Our test set
Yeast (Test 1)	Sequences	5,230,185	4,247	526
	Unique species	14,742	1,126	476
	Unique genera	3,695	203	122
Filamentous (Test 2)	Sequences	5,230,185	10,873	3,136
	Unique species	14,742	5,415	2,348
	Unique genera	3,695	1,642	827

Appendix

Table 8: UNITE+INSD for the Yeast and Filamentous test sets. Training set is the 5.23M-sequence pretraining corpus. Our test set is the deduplicated genus-level query set used for evaluation: exact-sequence duplicates of the training set are removed, the remaining specimens are restricted to those with a resolved species label that does not appear in the training set at all (unseen species).

		Encoder-decoder		Encoder-only
Aux	Repr	1-NN Acc. (%)	ZSC AMI (%)	1-NN Acc. (%)	ZSC AMI (%)
Binary	Tokens	65.14	74.67	48.85	55.28
Binary	CLS	71.23	74.86	55.77	71.39
Binary	Tokens+CLS	65.37	74.35	48.91	55.52
Triplet (hard)	Tokens	64.08	74.30	50.53	55.02
Triplet (hard)	CLS	57.77	73.33	58.57	72.59

Appendix

Table 9: BIOSCAN-5M results with family-level auxiliary labels. Genus-level 1-NN accuracy (%) and zero-shot BIN reconstruction AMI (%), for Binary, Triplet, and CE. Bold marks the best value in each column; underline marks the second-best.

		Encoder-decoder		Encoder-only
Aux	Repr	1-NN Acc. (%)	ZSC AMI (%)	1-NN Acc. (%)	ZSC AMI (%)
Binary	Tokens	68.90	74.93	60.45	57.46
Binary	CLS	72.56	75.27	62.93	73.38
Binary	Tokens+CLS	68.99	75.29	60.51	57.64
Triplet (hard)	Tokens	72.56	74.82	63.43	56.50
Triplet (hard)	CLS	68.82	74.39	71.82	74.01

Appendix

Table 10: BIOSCAN-5M results with BIN auxiliary labels. Same evaluation task as Table 9 . Bold marks the best value in each column; underline marks the second-best.

		Encoder-decoder		Encoder-only
Aux	Repr	Yeast 1-NN (%)	Filamentous 1-NN (%)	Yeast 1-NN (%)	Filamentous 1-NN (%)
Binary	Tokens	59.89	52.17	56.27	52.10
Binary	CLS	71.29	60.52	63.69	55.96
Binary	Tokens+CLS	60.27	52.58	57.03	52.20
Triplet (hard)	Tokens	58.37	51.66	55.51	50.96
Triplet (hard)	CLS	67.11	57.21	64.64	55.13

Appendix

Table 11: UNITE+INSD results with family-level auxiliary labels. Genus-level 1-NN accuracy (%) on the deduplicated Yeast and Filamentous query pools, for auxiliary pairwise/triplet/CE labels drawn from family rank. Bold marks the best value in each column; underline marks the second-best.

Figure 6: BIOSCAN-5M genus-level KNN accuracy vs.

k

under softmax voting, for the best configuration (encoder-decoder, +CLS+CE, CLS representation), at each tested temperature, plus uniform voting (black dotted) for reference.

T=0.02

is highlighted in red; lower temperatures are closer to winner-take-all voting, higher temperatures (warmer colours) are closer to uniform voting, and the

T=1.0

curve is visibly converging toward the uniform-voting curve as

k

grows.

$T$	Peak acc. (%)	Mean (%)	Std. (pp)	Range (pp)
0.01	80.64 ( $k{=}\phantom{0}1$ )	80.61	0.06	0.19
0.02 (default)	80.65 ( $k{=}\phantom{0}1$ )	80.53	0.09	0.27
0.05	80.65 ( $k{=}\phantom{0}1$ )	79.83	0.72	2.27
0.07 (DINOv2 default)	80.65 ( $k{=}\phantom{0}1$ )	79.41	1.02	3.29
0.1	80.65 ( $k{=}\phantom{0}1$ )	78.58	1.83	5.86
0.2	80.65 ( $k{=}\phantom{0}1$ )	76.30	3.52	12.01

Appendix

Table 12: BIOSCAN-5M softmax-voting stability statistics for each tested temperature, across

k\in\{1,3,5,7,10,15,20,25,50\}

. Mean is the average accuracy across all nine values of

k

tested.

T=0.02

is the value adopted throughout the main text;

T=0.07

is DINOv2’s original default, included for comparison. Bold marks the best (highest for Peak/Mean, lowest for Std./Range) value in each column; underline marks the second-best.

Figure 7: UNITE+INSD genus-level KNN accuracy vs.

k

under softmax voting, for the best configuration (encoder-decoder, +CLS+Binary, CLS representation), at each tested temperature, plus uniform voting (black dotted) for reference, on the deduplicated Yeast (left) and Filamentous (right) query pools.

T=0.02

(the value adopted throughout the main text) is highlighted in red.

Test set	$T$	Peak acc. (%)	Mean (%)	Std. (pp)	Range (pp)
Yeast (Test 1)	0.01	75.48 ( $k{=}50$ )	74.50	0.78	2.28
	0.02 (default)	76.43 ( $k{=}25$ )	75.14	1.12	3.24
	0.05	76.24 ( $k{=}7$ )	75.12	1.16	3.04
	0.07 (DINOv2 default)	76.05 ( $k{=}7$ )	74.57	1.14	3.81
	0.1	75.86 ( $k{=}7$ )	74.04	1.35	5.13
	0.2	75.48 ( $k{=}7$ )	73.28	1.90	7.03

Appendix

Table 13: UNITE+INSD softmax-voting stability statistics for each tested temperature, for the best configuration (encoder-decoder, +CLS+Binary, CLS representation) on the deduplicated Yeast and Filamentous query pools, across

k\in\{1,3,5,7,10,15,20,25,50\}

. Mean is the average accuracy across all nine values of

k

tested.

T=0.02

(matches the corresponding main-text table’s softmax rows) is the value adopted throughout the main text;

T=0.07

is DINOv2’s original default, included for comparison. Bold marks the best (highest for Peak/Mean, lowest for Std./Range) value in each column, per test set; underline marks the second-best.

Dataset	Test set	Model	Best $T$	Best $k$	Accuracy (%)
BIOSCAN-5M	—	DNABERT-S [ 32 ]	0.01	20	20.58
UNITE+INSD	Yeast	BarcodeMamba+ (large) [ 10 ]	0.01	7	62.54
UNITE+INSD	Filamentous	BarcodeMamba+ [ 10 ]	0.01	5	57.04
UNITE+INSD	Filamentous	BarcodeMamba+ (large) [ 10 ]	0.01	5	56.31
UNITE+INSD	Filamentous	DNABERT-2 [ 31 ]	0.01	3	54.15
UNITE+INSD	Yeast	DNABERT-S [ 32 ]	0.01	5	50.95

Appendix

Table 14: Best temperature and

k

for external baselines where softmax voting at

k>1

improves over

k{=}1

Architecture	Training data	Model	Training task	Yeast 1-NN (%)	Filamentous 1-NN (%)
State space	Human genome	HyenaDNA-tiny [ 20 ]	NTP	90.87	67.47
	Human genome	Caduceus-PS-1k [ 25 ]	MLM	89.83	64.27
	UNITE+INSD	BarcodeMamba+ [ 10 ]	NTP	94.62	78.51
	UNITE+INSD	BarcodeMamba+ (large) [ 10 ]	NTP	94.46	78.87
Encoder-only	Multi-species DNA	DNABERT-2 [ 31 ]	MLM	94.43	77.90
	Multi-species DNA	DNABERT-S [ 32 ]	CL	93.03	76.94

Appendix

Table 15: Comparison with baselines on UNITE+INSD ’s original (non-deduplicated) published Yeast and Filamentous test sets, evaluated using frozen embeddings. We report genus-level 1-NN accuracy for each test set. Training task abbreviations are, NTP = next-token prediction, MLM = masked language modelling, CL = contrastive learning, SC = supervised classification, MLM+Binary = masked language modelling with a same-genus pairwise auxiliary objective. Bold marks the best (highest) value in each column; underline marks the second-best.

		Encoder-decoder	Encoder-only
Configuration	Representation	BIN AMI	BIN AMI
(no CLS)	Tokens	74.32	55.65
+CLS	Tokens	74.67	54.40
+CLS	CLS	54.94	51.78
+CLS	Tokens+CLS	74.79	54.40
+CLS+Binary	Tokens	74.72	55.79

Appendix

Table 16: Zero-shot BIN reconstruction AMI (%) on BIOSCAN-5M for all representations of every model configuration, including both Triplet mining strategies. Bold marks the best (highest) value; underline marks the second-best.

		Encoder-decoder	Encoder-only
Configuration	Representation	1-NN Acc. (%)	1-NN Acc. (%)
(no CLS)	Tokens	66.73	51.27
+CLS	Tokens	68.76	55.39
+CLS	CLS	47.53	47.53
+CLS	Tokens+CLS	68.52	55.42
+CLS+Binary	Tokens	69.05	58.19

Appendix

Table 17: Genus-level 1-NN accuracy (%) on BIOSCAN-5M for every configuration and representation, including randomly-initialized encoder baseline. Bold marks the best value in each column; underline marks the second-best.

		Yeast		Filamentous
Configuration	Representation	Encoder-decoder	Encoder-only	Encoder-decoder	Encoder-only
(no CLS)	Tokens	54.94	49.81	49.11	48.85
+CLS	Tokens	63.50	61.79	56.63	55.93
+CLS	CLS	67.87	66.73	59.95	56.28
+CLS	Tokens+CLS	64.26	62.17	56.86	56.09
+CLS+Binary	Tokens	66.54	65.02	57.27	56.76

Appendix

Table 18: Genus-level 1-NN accuracy (%) on UNITE+INSD (deduplicated Yeast and Filamentous query pools) for every configuration and representation including the randomly-initialized encoder baseline. Bold marks the best value in each column; underline marks the second-best.

$J$	Repr	1-NN Acc. (%)	ZSC AMI (%)
3	Tokens	62.54	73.50
3	CLS	40.43	72.19
3	Tokens+CLS	62.40	73.52
6	Tokens	58.86	72.63
6	CLS	69.38	74.53
6	Tokens+CLS	58.57	72.72

Appendix

Table 19: BIOSCAN-5M Jumbo CLS token supplemental results (MLP expansion

1\times

). Genus-level 1-NN accuracy (%) and zero-shot BIN reconstruction AMI (%), for the encoder-decoder, +CLS+CE configuration. Bold marks the best (highest) value in each column; underline marks the second-best.

$J$	Repr	Yeast 1-NN (%)	Filamentous 1-NN (%)
3	Tokens	47.53	43.91
3	CLS	63.12	54.43
3	Tokens+CLS	48.29	44.58
6	Tokens	39.16	37.69
6	CLS	65.02	56.25
6	Tokens+CLS	43.35	41.45

Appendix

Table 20: UNITE+INSD Jumbo CLS token supplemental results (MLP expansion

1\times

). Genus-level 1-NN accuracy (%) on the deduplicated Yeast and Filamentous query pools, for the encoder-decoder, +CLS+Binary configuration. Bold marks the best (highest) value in each column; underline marks the second-best.

		Uniform				Softmax
Arch	Configuration	Peak (%)	Mean (%)	Std. (pp)	Range (pp)	Peak (%)	Mean (%)	Std. (pp)	Range (pp)
Encoder-decoder	(no CLS)	66.73 ( $k{=}1$ )	60.51	4.87	17.14	66.73 ( $k{=}1$ )	62.96	1.97	0 6.75
Encoder-decoder	+CLS	47.53 ( $k{=}1$ )	41.72	4.22	14.26	47.53 ( $k{=}1$ )	43.79	2.65	0 8.25
Encoder-decoder	+CLS+Binary	74.76 ( $k{=}\phantom{0}1$ )	66.61	6.29	21.38	74.76 ( $k{=}\phantom{0}1$ )	74.02	0.74	0 2.33
Encoder-decoder	+CLS+Triplet	74.85 ( $k{=}\phantom{0}1$ )	63.47	7.65	24.41	74.85 ( $k{=}\phantom{0}1$ )	70.00	3.49	10.34
Encoder-decoder	+CLS+CE	80.65 ( $k{=}\phantom{0}1$ )	72.55	7.00	23.35	80.65 ( $k{=}\phantom{0}1$ )	80.53	0.09	0 0.27

Appendix

Table 21: BIOSCAN-5M stability statistics for all ten model configurations, under uniform and softmax voting, across

k\in\{1,3,5,7,10,15,20,25,50\}

. Mean is the average accuracy across all nine values of

k

tested. Bold marks the best Peak/Mean accuracy in each column; underline marks the second-best.

			Uniform				Softmax
Test set	Arch	Configuration	Peak (%)	Mean (%)	Std. (pp)	Range (pp)	Peak (%)	Mean (%)	Std. (pp)	Range (pp)
Yeast	Encoder-decoder	(no CLS)	56.84 ( $k{=}3$ )	55.13	1.22	0 3.42	57.79 ( $k{=}\phantom{0}5$ )	54.24	2.83	10.07
	Encoder-decoder	+CLS	69.39 ( $k{=}3$ )	64.98	4.62	14.07	71.10 ( $k{=}\phantom{0}5$ )	68.27	2.48	0 8.55
	Encoder-decoder	+CLS+Binary	74.90 ( $k{=}7$ )	72.05	3.10	10.45	76.43 ( $k{=}25$ )	75.14	1.12	0 3.24
	Encoder-decoder	+CLS+Triplet	73.00 ( $k{=}3$ )	70.78	2.22	0 7.60	73.57 ( $k{=}\phantom{0}3$ )	72.09	1.07	0 3.99
	Encoder-decoder	+CLS+CE	73.57 ( $k{=}7$ )	72.57	1.13	0 2.85	75.29 ( $k{=}10$ )	73.68	1.35	0 4.57

Appendix

Table 22: UNITE+INSD stability statistics for all ten model configurations, on the deduplicated Yeast and Filamentous query pools, under uniform and softmax voting. Mean is the average accuracy across the

k

values tested. Bold marks the best Peak/Mean accuracy in each column; underline marks the second-best.

Molecular Representation Learning

Molecular representation learning encodes chemical structures and context for prediction, generation, and discovery across chemistry and biology.

See all 207 papers

Apr 17, 2026

In Search of Lost DNA Sequence Pretraining

DNA sequence encoding is fundamental to gene function prediction, protein synthesis, and diverse downstream biological tasks. Despite the substantial progress achieved by large-scale DNA sequence pretraining, existing studies have overwhelmingly emphasized pretraining scale and custom downstream evaluation datasets, while neglecting some essential components of the pretraining paradigm. In this paper, we reveal three critical yet heretofore overlooked problems in DNA pretraining: inappropriate downstream datasets, inherent flaws in the neighbor-masking strategy, and the lack of detailed discussion on vocabulary. Therefore, we undertake comprehensive investigations and propose principled guidelines, including selection criteria for evaluation datasets, guiding task design, and in-depth vocabulary analysis. Extensive experiments validate the significance of our identified problems and support the rationale behind our recommendations. Finally, we introduce a standardized testbed that enables reproducible and rigorous benchmarking of DNA pretraining methods to advance the development of genomic foundation models.

Zhijiang Tang, Jiaxin Qi, Yan Cui +3

Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Zhejiang, China

Language Model Pretraining Masked Language Modeling Benchmark Design

Ask PDF

Feb 19, 2026

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

Genomic Foundation Models (GFMs) typically rely on Masked Language Modeling (MLM) or Next-Token Prediction (NTP) to learn the "Laws of Nature". While effective at capturing local syntax, these generative paradigms prioritize token-level reconstruction over high-level functional context. We introduce JEPA-DNA, a model-agnostic continual training framework that integrates a Joint-Embedding Predictive Architecture (JEPA) with traditional generative objectives. By supervising global sequence embeddings in a latent space, JEPA-DNA forces models to predict the functional representations of masked genomic segments, shifting the learning signal from token recovery to semantic alignment. We evaluate JEPA-DNA on 17 diverse genomic benchmark tasks, demonstrating consistent gains in linear probing and zero-shot performance regardless of the underlying GFM architecture or generative objective, and establishing a new state-of-the-art for GFMs under the linear-probing protocol. Through extensive ablation studies, we further characterize the synergistic interplay between generative and latent objectives. Our code is provided in the supplementary material.

Ariel Larey, Elay Dahan, Amit Bleiweiss +15

Applied AI Architecture, NVIDIA, Israel. · Worldwide Field Ops, NVIDIA, Israel. · Developer Programs, NVIDIA, Israel. +2

Genomic Foundation Models Molecular Representation Learning Genomics Joint-Embedding Predictive Architecture

Ask PDF

Mar 2, 2026

D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation

Early DNA foundation models adopted BERT-style training, achieving good performance on DNA understanding tasks but lacking generative capabilities. Recent autoregressive models enable DNA generation, but employ left-to-right causal modeling that is suboptimal for DNA where regulatory relationships are inherently bidirectional. We present D3LM (Discrete DNA Diffusion Language Model), which unifies bidirectional representation learning and DNA generation through masked diffusion. D3LM directly adopts the Nucleotide Transformer (NT) v2 architecture but reformulates the training objective as masked diffusion in discrete DNA space, enabling both bidirectional understanding and generation capabilities within a single model. Compared to NT v2 of the same size, D3LM achieves improved performance on understanding tasks. Notably, on regulatory element generation, D3LM achieves an SFID of 10.92, closely approaching real DNA sequences (7.85) and substantially outperforming the previous best result of 29.16 from autoregressive models. Our work suggests diffusion language models as a promising paradigm for unified DNA foundation models. We further present the first systematic study of masked diffusion models in the DNA domain, investigating practical design choices such as tokenization schemes and sampling strategies, thereby providing empirical insights and a solid foundation for future research. D3LM has been released at https://huggingface.co/collections/Hengchang-Liu/d3lm.

Zhao Yang, Hengchang Liu, Chuan Cao +1

Gaoling School of Artificial Intelligence, Renmin University of China · Zhongguancun Academy

Molecular Foundation Models Genomic Foundation Models Language Modeling Molecular Generation Diffusion Language Models Masked Diffusion Models+1

Ask PDF

Abstract

Molecular Representation Learning

Explore similar work

In Search of Lost DNA Sequence Pretraining

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation