BEND 2.0: Benchmarking the Capabilities Needed to Design DNA
Authors: Andreas Bjerregaard, Rachael DeVries, Dionysia Danai Brilli, Panagiotis Antoniadis, Yan Li, Anders Krogh, Wouter Boomsma, Ole Winther, Frederikke Isa Marin
Organizations: University of Copenhagen · Novo Nordisk A/S · Technical University of Denmark
Genomic foundation models are increasingly used to propose DNA sequences and edits. But successful design requires capabilities that current benchmarks do not measure. Here we introduce BEND 2.0 (BENchmark for DNA language models), which tests how well models predict the effects of DNA changes and whether these predictions can guide sequence design. Across eight tasks built on five experimental datasets, we find that model choice and how representations are accessed both affect performance. Linear readouts fitted to frozen representations can match or exceed models' native outputs, and functional post-training substantially improves the models' capabilities. Linear readouts transfer well across experimental assays and genes, suggesting that models capture functional patterns beyond the data used to fit individual predictors. At the same time, strong prediction of average effects can conceal weaker prediction of context-specific effects, and accurate prediction of combined mutation effects does not imply accurate prediction of their interactions. In design tasks, model predictions can reduce the number of candidates needed to find a desired response, but still leave substantial room for better design choices, particularly for context-specific and multi-cell objectives. Our benchmark reveals that genomic models now display signs of general utility but have yet to saturate on performance in even simple tasks.
Figures & tables
Model
Fig. 1 A Prior loss bits ↓
Fig. 2 A Splicing effects ρ↑
Fig. 3 B Cell contrast ρ↑
Fig. 12 B Tissue ID % ↑
Fig. 4 A Background interaction ρ↑
Fig. 5 A Single-edit rank percentile ↓
Fig. 5 C Cell selectivity U↑
Fig. 6 A Multi-cell gain log 2 ↑
Base composition
1.974
—
—
—
—
—
—
—
Evo2 20B
1.388
0.367
0.041
21.6
0.050
43.5
- 0.350
0.275
CENO 1B
1.549
0.295
0.036
0 9.8
0.010
48.9
- 0.318
- 0.581
GENERATORv2
1.711
0.388
0.066
17.6
0.046
44.7
- 0.256
0.234
DNABERT-2
—
0.198
0.076
0 5.9
0.020
48.6
- 0.302
0.233
NTv3 100M pre
1.537
0.349
0.076
0 7.8
0.010
47.3
- 0.207
0.339
Table 1: The BEND 2.0 benchmark. Metrics across categories collected in one overview table. The referenced figures and their corresponding sections provide further exploration. Deeper shading indicates better performance within each column. Results use embedding-fitted linear heads, except for prior loss (native sequence logits) and Enformer/indented rows (native functional outputs). ‡ : linear head fitted to native outputs. † : Experimental references use experimental subsets but can be naive, e.g., Background interaction uses a simple transfer from HepG2 to K562 (see Appendix A.1 ).
Figure 1: Sequence-prior performance depends on the representation extraction. A: masked loss on 512 windows. NTv3 uses fixed 12% additional masking ( † ), averaging losses over two draws. GENERATORv2 uses temperature calibration fitted on separate data ( ‡ ). B: additional masking substantially improves post-trained NTv3 sequence loss. C: calibration improves GENERATORv2’s predicted probabilities without changing base-prediction accuracy. Whiskers show 95% block-bootstrap intervals. Colors denote model/baseline category.
Figure 2: Functional post-training supports transfer across assays and genes. A: mutation-induced changes in exon inclusion on OpenSplice, with complete genes held out of readout fitting (466 genes). B: transfer of OpenSplice-trained predictors to 4,618 COMPASS variants in 88 genes excluded from source training, without COMPASS fitting or model selection. Scores average within-gene Spearman correlations. Experimental references compare repeat measurements. Native heads ( † ) have splicing-related outputs. Gray hollow points show random network weights while whiskers are 95% gene-bootstrap intervals (Appendix C ).
Figure 3: Predicting cell-specific effects. A/B: mean mutation effect and HepG2–K562 difference across 128 Siraj regulatory elements. Experimental references compare groups of repeat measurements (127 elements); they are not ceilings. † marks native outputs; unmarked NTv3 entries use fitted readouts. Whiskers show available 95% element-bootstrap intervals.
Figure 4: Accurate combined effects can conceal weak interaction prediction. A: difference in mutation effects between two backgrounds differing at one base. B: single, double and non-additive mutation effects in 110 COMPASS genes. C: Interaction prediction by distance between mutations. HepG2–K562 transfer compares measured interactions across cells (A/C). The experimental reference compares repeat measurements of interactions (B). Neither is a ceiling. † marks native outputs, and whiskers show 95% bootstrap intervals over regulatory elements (A/C) or genes (B).
Figure 5: Selecting edits for a desired response and cell specificity. A/B: rank of the first selected edit and success at matching a reference-edit response as testing budget increases (331 targets, 110 regulatory elements). C: target-cell expression increase versus absolute other-cell change across 127 loci, varying the penalty on unwanted effects. Open markers ignore the other cell ( λ=0 ), and filled markers penalize it equally ( λ=1 ). Experimental references in A–C select edits on one group of repeat measurements and evaluate on the other. Models may outperform these references. D: task settings. † marks native outputs in A, and whiskers show 95% element-bootstrap intervals.
Figure 6: Selecting DNA with activity in desired cells and little activity elsewhere. Models choose ten enhancers for each of sixteen requested expression patterns. A: gain in utility over random selection among sequences originally designed for the same ON/OFF pattern, averaged across patterns. Utility rewards the weakest ON-cell activity and penalizes the strongest OFF-cell activity above the negative-control level. Dots show the two repeat measurements. The experimental reference selects sequences using one measurement and evaluates them using the other, then swaps them. B: example across cell types, showing ten enhancers chosen by AlphaGenome having selective activity in HepG2 and K562.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Sequence prediction across repeat content. Reduction in base loss relative to the tuned Markov model. Higher is better. Black bars show paired 95% bootstrap confidence intervals; numbers give window counts.
Figure 8: Base prediction depends on the masking pattern. A–B: Base loss with 144 additional masked or replaced bases. Distant changes leave 256 bases on each side of the target untouched. C–D: Change in loss when replacing the transformer representation, comparing versions with and without extra masks. Representations come from the same DNA in C and unrelated DNA with similar GC content in D. Skip features stay fixed. Blue denotes pre-trained and teal post-trained models. Whiskers show bootstrap intervals over genomic blocks (A–C) or sequence pairs (D).
Figure 9: Predicting alleles in natural haplotypes. A: base log loss at 512 SNP sites, with REF and ALT carriers equally weighted. Uniform prediction gives two bits. B: change in ALT/REF log odds between carrier backgrounds for 396 pairs with different surrounding DNA. Positive values indicate a shift towards ALT. NTv3 ( † ) averages base probabilities from eight masked queries; causal models use uncalibrated probabilities. Whiskers show 95% confidence intervals from resampling the 39 genomic blocks.
Figure 10: OpenSplice prediction with fewer assay labels. Larger training subsets include the smaller ones. Lines show the mean across three runs with different training samples and seeds while the shading shows their range. The training-set sizes count only OpenSplice examples. NTv3 post was previously trained on functional genomic data.
Figure 11: Predicting editing efficiency from surrounding DNA. Spearman correlations for four editing systems in the K562 TRIP dataset. All models receive the same 8-kb genomic sequence; AlphaGenome receives additional N-padding on both sides to meet its 16-kb input requirement. Results are from five chromosome-held-out folds. Whiskers show pointwise 95% confidence intervals from resampling chromosomes.
Figure 12: Predicting expression differences and identifying tissues. A: correlation between predicted and measured expression differences within genes. Each tissue is paired with the tissue where measured expression differs most. B: tissue identification by matching predicted and measured expression profiles across genes (chance is 1/51 ). Experimental references compare donor groups. † marks native outputs. Whiskers show available 95% confidence intervals from resampling genes.
Figure 13: Personal DNA helps at familiar genes. Expression-prediction correlations for new people at genes seen during training (left) or unseen gene regions (right). Δ gives the gain over shuffled genotypes. Intervals are obtained by resampling genes.
Figure 14: Reducing effects on the other allelic background. Within the same cell type, we compare the target response and the absolute response on the other allelic background for mutations selected by AlphaGenome at 114 Siraj loci. The control matches the number of edits and mean target response. Results are shown separately for repeats 1–2 and 3–5 while the same mutation is evaluated on both backgrounds. Whiskers show unadjusted 95% confidence intervals.
Figure 15: Considering unwanted activity improves sequence selection. Arrows run from selection based only on the weakest ON-cell prediction (ticks) to selection that also penalizes the strongest OFF-cell prediction (dots). Both are scored using Equation 6 , averaged over ten sequences, sixteen expression patterns and both RNA measurements. The experimental reference selects using one measurement and evaluates using the other.
Model
Prior loss [2pt] bits ↓
Splicing effects [2pt] ρ↑
Cell contrast [2pt] ρ↑
Tissue ID [2pt] % ↑
Base composition
[1.953, 1.997]
—
—
—
Evo2 20B
[1.250, 1.538]
[0.352, 0.381]
[0.022, 0.060]
21.6
CENO 1B
[1.385, 1.736]
[0.282, 0.309]
[0.018, 0.053]
[5.9, 19.6]
GENERATORv2
[1.652, 1.775]
[0.374, 0.402]
[0.050, 0.083]
[9.8, 21.6]
DNABERT-2
—
[0.188, 0.210]
[0.066, 0.087]
[2.0, 9.8]
NTv3 100M pre
[1.454, 1.620]
[0.335, 0.363]
[0.063, 0.089]
[5.9, 15.7]
Appendix
Table 5: Uncertainty of the main results. Pointwise 95% confidence intervals; multi-cell gain shows ranges across the two RNA measurements. Single values are point estimates. Shading and notation follow Table 1 ; —: not reported.
Large Language Models (LLMs) are increasingly adopted as conversational assistants in genomics, where they are mainly used to reason over biological knowledge, annotations, and analysis outputs through natural language interfaces. However, existing benchmarks either focus on specialized DNA models trained for sequence prediction or evaluate biological knowledge using text-only questions, leaving the behavior of general-purpose LLMs when directly exposed to raw genome sequences underexplored. We introduce GenomeQA, a benchmark designed to provide a controlled evaluation setting for general-purpose LLMs on sequence-based genome inference tasks. GenomeQA comprises 5,200 samples drawn from multiple biological databases, with sequence lengths ranging from 6 to 1,000 base pairs (bp), spanning six task families: Enhancer and Promoter Identification, Splice Site Identification, Taxonomic Classification, Histone Mark Prediction, Transcription Factor Binding Site Prediction, and TF Motif Prediction. Across six frontier LLMs, we find that models consistently outperform random baselines and can exploit local sequence signals such as GC content and short motifs, while performance degrades on tasks that require more indirect or multi-step inference over sequence patterns. GenomeQA establishes a diagnostic benchmark for studying and improving the use of general-purpose LLMs on raw genomic sequences.
Weicai Long, Yusen Hou, Junning Feng +4
Hong Kong University of Science and Technology (Guangzhou) · The University of Hong Kong · Peking University
Contemporary Genomic Foundation Models (GFMs) rely on a DNA-as-a-string paradigm that employs masked token prediction objectives for pretraining. However, this abstraction does not explicitly model the biochemical, structural, and physical properties essential to biological function. Many molecular properties can be estimated from sequence using established biophysical models, so their utility lies not in providing an independent modality, but in introducing priors that training objectives can explicitly exploit. We introduce VANDAM, a framework that extends the training of GFMs with DNA molecular priors. In self-supervised training, VANDAM predicts regional molecular properties from pooled representations. When functional labels are available and can reward retaining molecular priors, local features are additionally injected at the input. VANDAM consistently improves downstream performance across four architecture families and nine held-out genomic tasks by complementing token-based objectives. Probing experiments further demonstrate that the use of molecular priors generalizes to other unseen molecular properties.
Jeremy Levy, Ariel Larey, Yury Nahshan +18
Applied AI Architecture, NVIDIA, Israel. · Worldwide Field Ops, NVIDIA, Israel. · Developer Programs, NVIDIA, Israel. +3
Progress in genomic foundation models is difficult to assess due to fragmented benchmarks, incompatible evaluation protocols, and task-specific reporting. As a result, claims of superiority or generality across models are often not directly comparable. We introduce GENEB, a large-scale diagnostic benchmark that evaluates frozen representations from 40 genomic foundation models across 100 tasks spanning 13 functional categories under a unified probing-based protocol, including few-shot regimes. GENEB enables controlled comparison across model scale, architecture, tokenization, and pretraining data while explicitly exposing task-level trade-offs. Our analysis shows that aggregate leaderboards are unstable: model rankings vary sharply across task categories, scale provides only modest and inconsistent gains, and architectural and pretraining alignment frequently outweigh parameter count. These results highlight limitations of current evaluation practices and position GENEB as a reference framework for principled comparison and category-aware model selection in genomic machine learning.
Daria Ledneva, Mikhail Nuridinov, Denis Kuznetsov
Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia · Moscow State Institute of Steel and Alloys, Moscow, Russia