q-bio.GNNov 2, 2026
SaveBEND 2.0: Benchmarking the Capabilities Needed to Design DNA
Organizations: University of Copenhagen · Novo Nordisk A/S · Technical University of Denmark
Abstract
Genomic foundation models are increasingly used to propose DNA sequences and edits. But successful design requires capabilities that current benchmarks do not measure. Here we introduce BEND 2.0 (BENchmark for DNA language models), which tests how well models predict the effects of DNA changes and whether these predictions can guide sequence design. Across eight tasks built on five experimental datasets, we find that model choice and how representations are accessed both affect performance. Linear readouts fitted to frozen representations can match or exceed models' native outputs, and functional post-training substantially improves the models' capabilities. Linear readouts transfer well across experimental assays and genes, suggesting that models capture functional patterns beyond the data used to fit individual predictors. At the same time, strong prediction of average effects can conceal weaker prediction of context-specific effects, and accurate prediction of combined mutation effects does not imply accurate prediction of their interactions. In design tasks, model predictions can reduce the number of candidates needed to find a desired response, but still leave substantial room for better design choices, particularly for context-specific and multi-cell objectives. Our benchmark reveals that genomic models now display signs of general utility but have yet to saturate on performance in even simple tasks.
Figures & tables
| Model | Fig. 1 A Prior loss bits | Fig. 2 A Splicing effects | Fig. 3 B Cell contrast | Fig. 12 B Tissue ID % | Fig. 4 A Background interaction | Fig. 5 A Single-edit rank percentile | Fig. 5 C Cell selectivity | Fig. 6 A Multi-cell gain log 2 |
| Base composition | 1.974 | — | — | — | — | — | — | — |
| Evo2 20B | 1.388 | 0.367 | 0.041 | 21.6 | 0.050 | 43.5 | - 0.350 | 0.275 |
| CENO 1B | 1.549 | 0.295 | 0.036 | 0 9.8 | 0.010 | 48.9 | - 0.318 | - 0.581 |
| GENERATORv2 | 1.711 | 0.388 | 0.066 | 17.6 | 0.046 | 44.7 | - 0.256 | 0.234 |
| DNABERT-2 | — | 0.198 | 0.076 | 0 5.9 | 0.020 | 48.6 | - 0.302 | 0.233 |
| NTv3 100M pre | 1.537 | 0.349 | 0.076 | 0 7.8 | 0.010 | 47.3 | - 0.207 | 0.339 |
Table 1: The BEND 2.0 benchmark. Metrics across categories collected in one overview table. The referenced figures and their corresponding sections provide further exploration. Deeper shading indicates better performance within each column. Results use embedding-fitted linear heads, except for prior loss (native sequence logits) and Enformer/indented rows (native functional outputs). : linear head fitted to native outputs. : Experimental references use experimental subsets but can be naive, e.g., Background interaction uses a simple transfer from HepG2 to K562 (see Appendix A.1 ).
Figure 1: Sequence-prior performance depends on the representation extraction. A: masked loss on 512 windows. NTv3 uses fixed 12% additional masking ( ), averaging losses over two draws. GENERATORv2 uses temperature calibration fitted on separate data ( ). B: additional masking substantially improves post-trained NTv3 sequence loss. C: calibration improves GENERATORv2’s predicted probabilities without changing base-prediction accuracy. Whiskers show 95% block-bootstrap intervals. Colors denote model/baseline category.
Figure 2: Functional post-training supports transfer across assays and genes. A: mutation-induced changes in exon inclusion on OpenSplice, with complete genes held out of readout fitting (466 genes). B: transfer of OpenSplice-trained predictors to 4,618 COMPASS variants in 88 genes excluded from source training, without COMPASS fitting or model selection. Scores average within-gene Spearman correlations. Experimental references compare repeat measurements. Native heads ( ) have splicing-related outputs. Gray hollow points show random network weights while whiskers are 95% gene-bootstrap intervals (Appendix C ).
Figure 3: Predicting cell-specific effects. A/B: mean mutation effect and HepG2–K562 difference across 128 Siraj regulatory elements. Experimental references compare groups of repeat measurements (127 elements); they are not ceilings. marks native outputs; unmarked NTv3 entries use fitted readouts. Whiskers show available 95% element-bootstrap intervals.
Figure 4: Accurate combined effects can conceal weak interaction prediction. A: difference in mutation effects between two backgrounds differing at one base. B: single, double and non-additive mutation effects in 110 COMPASS genes. C: Interaction prediction by distance between mutations. HepG2–K562 transfer compares measured interactions across cells (A/C). The experimental reference compares repeat measurements of interactions (B). Neither is a ceiling. marks native outputs, and whiskers show 95% bootstrap intervals over regulatory elements (A/C) or genes (B).
Figure 5: Selecting edits for a desired response and cell specificity. A/B: rank of the first selected edit and success at matching a reference-edit response as testing budget increases (331 targets, 110 regulatory elements). C: target-cell expression increase versus absolute other-cell change across 127 loci, varying the penalty on unwanted effects. Open markers ignore the other cell ( ), and filled markers penalize it equally ( ). Experimental references in A–C select edits on one group of repeat measurements and evaluate on the other. Models may outperform these references. D: task settings. marks native outputs in A, and whiskers show 95% element-bootstrap intervals.
Figure 6: Selecting DNA with activity in desired cells and little activity elsewhere. Models choose ten enhancers for each of sixteen requested expression patterns. A: gain in utility over random selection among sequences originally designed for the same ON/OFF pattern, averaged across patterns. Utility rewards the weakest ON-cell activity and penalizes the strongest OFF-cell activity above the negative-control level. Dots show the two repeat measurements. The experimental reference selects sequences using one measurement and evaluates them using the other, then swaps them. B: example across cell types, showing ten enhancers chosen by AlphaGenome having selective activity in HepG2 and K562.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Sequence prediction across repeat content. Reduction in base loss relative to the tuned Markov model. Higher is better. Black bars show paired 95% bootstrap confidence intervals; numbers give window counts.
Figure 8: Base prediction depends on the masking pattern. A–B: Base loss with 144 additional masked or replaced bases. Distant changes leave 256 bases on each side of the target untouched. C–D: Change in loss when replacing the transformer representation, comparing versions with and without extra masks. Representations come from the same DNA in C and unrelated DNA with similar GC content in D. Skip features stay fixed. Blue denotes pre-trained and teal post-trained models. Whiskers show bootstrap intervals over genomic blocks (A–C) or sequence pairs (D).
Figure 9: Predicting alleles in natural haplotypes. A: base log loss at 512 SNP sites, with REF and ALT carriers equally weighted. Uniform prediction gives two bits. B: change in ALT/REF log odds between carrier backgrounds for 396 pairs with different surrounding DNA. Positive values indicate a shift towards ALT. NTv3 ( ) averages base probabilities from eight masked queries; causal models use uncalibrated probabilities. Whiskers show 95% confidence intervals from resampling the 39 genomic blocks.
Figure 10: OpenSplice prediction with fewer assay labels. Larger training subsets include the smaller ones. Lines show the mean across three runs with different training samples and seeds while the shading shows their range. The training-set sizes count only OpenSplice examples. NTv3 post was previously trained on functional genomic data.
Figure 11: Predicting editing efficiency from surrounding DNA. Spearman correlations for four editing systems in the K562 TRIP dataset. All models receive the same 8-kb genomic sequence; AlphaGenome receives additional N-padding on both sides to meet its 16-kb input requirement. Results are from five chromosome-held-out folds. Whiskers show pointwise 95% confidence intervals from resampling chromosomes.
Figure 12: Predicting expression differences and identifying tissues. A: correlation between predicted and measured expression differences within genes. Each tissue is paired with the tissue where measured expression differs most. B: tissue identification by matching predicted and measured expression profiles across genes (chance is ). Experimental references compare donor groups. marks native outputs. Whiskers show available 95% confidence intervals from resampling genes.
Figure 13: Personal DNA helps at familiar genes. Expression-prediction correlations for new people at genes seen during training (left) or unseen gene regions (right). gives the gain over shuffled genotypes. Intervals are obtained by resampling genes.
Figure 14: Reducing effects on the other allelic background. Within the same cell type, we compare the target response and the absolute response on the other allelic background for mutations selected by AlphaGenome at 114 Siraj loci. The control matches the number of edits and mean target response. Results are shown separately for repeats 1–2 and 3–5 while the same mutation is evaluated on both backgrounds. Whiskers show unadjusted 95% confidence intervals.
Figure 15: Considering unwanted activity improves sequence selection. Arrows run from selection based only on the weakest ON-cell prediction (ticks) to selection that also penalizes the strongest OFF-cell prediction (dots). Both are scored using Equation 6 , averaged over ten sequences, sixteen expression patterns and both RNA measurements. The experimental reference selects using one measurement and evaluates using the other.
| Model | Prior loss [2pt] bits | Splicing effects [2pt] | Cell contrast [2pt] | Tissue ID [2pt] % |
| Base composition | [1.953, 1.997] | — | — | — |
| Evo2 20B | [1.250, 1.538] | [0.352, 0.381] | [0.022, 0.060] | 21.6 |
| CENO 1B | [1.385, 1.736] | [0.282, 0.309] | [0.018, 0.053] | [5.9, 19.6] |
| GENERATORv2 | [1.652, 1.775] | [0.374, 0.402] | [0.050, 0.083] | [9.8, 21.6] |
| DNABERT-2 | — | [0.188, 0.210] | [0.066, 0.087] | [2.0, 9.8] |
| NTv3 100M pre | [1.454, 1.620] | [0.335, 0.363] | [0.063, 0.089] | [5.9, 15.7] |
Table 5: Uncertainty of the main results. Pointwise 95% confidence intervals; multi-cell gain shows ranges across the two RNA measurements. Single values are point estimates. Shading and notation follow Table 1 ; —: not reported.