Data-Scarce, Multi-Granular, Multimodal: Benchmark Design for Biosynthetic Gene Clusters

Hanlin Xiao; Eriko Takano; Mauricio A. Álvarez; Rainer Breitling

cs.AINov 1, 2026

Data-Scarce, Multi-Granular, Multimodal: Benchmark Design for Biosynthetic Gene Clusters

Authors: Hanlin Xiao, Eriko Takano, Mauricio A. Álvarez, Rainer Breitling

Organizations: Manchester Institute of Biotechnology, The University of Manchester, Manchester, United Kingdom · Department of Computer Science, The University of Manchester, Manchester, United Kingdom · Department of Chemistry, The University of Manchester, Manchester, United Kingdom · Division of Evolution, Infection and Genomics, The University of Manchester, Manchester, United Kingdom ·

Abstract

Biosynthetic gene clusters (BGCs) are co-located genes that encode the biosynthesis of specialized metabolites, an important source of antibiotics, anticancer agents, immunosuppressants, and agricultural bioactives. Deep learning, particularly self-supervised foundation models, has spurred interest in BGC sequence modeling, but evaluation has lagged behind: existing benchmarks often rely on coarse classification over small curated datasets, providing limited discrimination among models and insight into downstream utility. We propose a benchmark-design framework in which an informative benchmark should be discriminative, separating models with statistical confidence, and applicable, spanning scientifically meaningful tasks under different levels of generalization. We instantiate this framework as BGC-Bench, which derives five tasks from MIBiG, ranging from A-domain substrate prediction to BGC-molecule retrieval, evaluates them under genotype- and phenotype-shift splits, includes an active-learning setting for label-scarce discovery, and is easily extensible to new tasks and encoders. BGCs further provide a shared testbed across granularities and modalities: BGC-Bench compares twelve encoders on nucleotide, amino-acid, or Pfam-domain sequences, including compositional baselines for each input type, and combines BGC and chemical encoders for cross-modal retrieval. No single representation dominates: the best granularity depends on the biological scale of the task, and pretrained encoders do not consistently outperform other representations across tasks and splits. Active-learning experiments show that most tasks remain data-limited, encoder rankings change with labeling budget, and acquisition strategies yield modest gains. These results show that BGC model evaluation requires task-diverse, shift-aware benchmarks, and the framework can be generalized to benchmark design beyond BGCs.

Figures & tables

Model	Dim.	Params	Representation details
Pfam-domain encoders
Bag-of-Pfam	3,932	–	Pfam-domain count vector, $L_{2}$ normalized
Random-Pfam	256	–	Random per-Pfam-token vectors, normalized after pooling
BiGCARP [ 20 ]	256	$\sim$ 34 M	ByteNet MLM on antiSMASH-DB [ 44 ] BGCs
Kawano [ 21 ]	1,024	$\sim$ 110 M	RoBERTa MLM on antiSMASH-DB bacterial BGCs
DeepBGC [ 22 ]	256	$\sim$ 1.6 M	Frozen Pfam2vec + BiLSTM; BGC-detection pretraining

Table 1: Model inventory. The twelve BGC encoders, grouped by input granularity; each group starts with its parameter-free compositional baselines. For BGC-MAC, BGCat, BGC-Prophet, and LC-PLM, residue representations are averaged within each CDS, then across CDSs. CDS: coding sequence; MLM: masked language modeling; LM: language model.

Figure 1: BGC-Bench evaluation design framework. Applicability is characterized along a generalization axis (random, phenotype-shift and genotype-shift splits) and a functionality axis (tasks from broad class prediction to BGC–molecule retrieval); performance across these settings is then used to assess discriminability , and active learning adds a limited-labeling-budget setting.

Pfam-domain encoders
Model	Coarse classes			Fine classes			Chem. properties			BGC–molecule retrieval				A-domain substrate
	macro-AUROC			weighted macro-AUROC			mean Spearman $\rho$			class-macro R@10				top-1 accuracy
	R	S	G	R	S	G	R	S	G	R	S	G	C	CV	G
Bag-of-Pfam	.929	.877	.884	.880	.825	.753	.354	.281	.283	.624	.546	.495	.275	.196	.151
	$\pm$ .020	$\pm$ .042	$\pm$ .029	$\pm$ .031	$\pm$ .028	$\pm$ .041	$\pm$ .057	$\pm$ .057	$\pm$ .054	$\pm$ .062	$\pm$ .109	$\pm$ .081	$\pm$ .039	$\pm$ .018	$\pm$ .043
Random-Pfam	.936	.909	.896	.880	.851	.786	.523	.467	.417	.542	.504	.448	.209	–	–

Table 2: Main results. Performance of twelve BGC encoders on five tasks across their generalization splits. R, S, G, and C denote the random, scaffold-holdout, GCF-holdout, and class-holdout splits; CV denotes five-fold cross-validation for A-domain substrate prediction. For retrieval, molecules are represented by frozen embeddings from MoLFormer, a pretrained chemical foundation model. Values are point estimates

\pm

the distance to the farther 95% CI bound; Table 13 reports the exact asymmetric intervals. Darker shading indicates better performance within each column. Separability is the fraction of encoder pairs whose exact CIs do not overlap.

Figure 2: BGC–chemical encoder retrieval, random split. Class-macro R@10 for the twelve BGC encoders (rows) paired with seven chemical encoders (ECFP4, AtomPair [ 67 ] , MACCS [ 68 ] , RDKit descriptors, MoLFormer, ChemBERTa-2 and Uni-Mol) and a random-vector control (columns), mean of three seeds.

Figure 3: Active-learning curves. Test performance against the number of labeled training examples on five tasks. Each panel shows the compositional baseline with the highest validation

\mathrm{AULC}_{\mathrm{low}}

under random acquisition, and BiGCARP, Evo 2, and the best protein language model, each with its validation-selected acquisition strategy; dashed lines show the same encoder with random acquisition. Bands are 95% bootstrap confidence intervals, gray shading the

\mathrm{AULC}_{\mathrm{low}}

range (

n\leq 256

), thin gray lines chance level and the dotted line the majority substrate.

	Generalization splits
Domain	Genotype side	Phenotype side	Functionality	Benchmarks
Small-molecule drug discovery	Scaffold, similarity-thresholded	Assay, target family	Property prediction, ADMET prediction, compound–target retrieval, hit identification, lead optimization	MoleculeNet [ 40 ] Lo-Hi [ 41 ]
Protein sequence modeling	Sequence identity, family/superfamily, mutation distance	Function class, substrate/binding, condition	Function annotation, fitness prediction, binding retrieval, protein engineering	TAPE [ 36 ] FLIP [ 38 ] PEER [ 37 ] ProteinGym [ 39 ]
MS/MS metabolomics	Scaffold, MCES distance, structure similarity	Spectral similarity, instrument, adduct, collision energy	Formula prediction, molecule retrieval, de novo structure generation	MassSpecGym [ 42 ]

Table 3: Representative biochemical benchmarks under the framework. Existing benchmarks are organized by domain and decomposed along genotype-side and phenotype-side generalization axes. For each domain, we summarize potential split definitions, associated functionalities, and representative benchmark datasets.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

	$c=0.3$	$c=0.4$	$c=0.5$	$c=0.6$	$c=0.7$	$c=0.8$	$c=0.9$	Random
NN distance $\uparrow$	0.514	0.529	0.560	0.580	0.590	0.570	0.576	0.460
AUROC, BiG-SCAPE $\downarrow$	0.804	0.774	0.764	0.764	0.729	0.724	0.730	0.840
AUROC, Pfam Jaccard $\downarrow$	0.831	0.787	0.777	0.752	0.742	0.765	0.745	0.862

Appendix

Table 4: Relatedness across GCF cutoffs. NN distance is the median BiG-SCAPE distance to the nearest training BGC. AUROC uses class-specific nearest-neighbor similarity; arrows indicate less relatedness.

Genotype ( $2{,}433$ BGCs)
Grouping	Groups	ARI	$\tau$ within	$\tau$ vs. benchmark
BiG-SCAPE families, $c=0.7$ (benchmark)	416	–	0.806	–
Pfam-domain Jaccard + Butina	414	0.027	0.576	0.710
Phenotype ( $1{,}597$ BGCs)
Bemis–Murcko scaffold (benchmark)	1,105	–	0.830	–
Generic Murcko scaffold	972	0.620	0.824	0.824

Appendix

Table 5: Encoder rankings under alternative groupings. Chemical property prediction, linear probes. ARI (adjusted Rand index) compares partitions against the benchmark;

\tau

averages ranking correlations within groupings or against the benchmark.

Coarse classes : BGCs; MIBiG classes, multi-label; macro-AUROC
Split	Train	Test	Targets
Random	1,949	488	6
Scaffold	1,570	374	6
GCF ( $c=0.7$ )	2,057	376	6
Fine classes : BGCs; NPClassifier classes, multi-label; weighted macro-AUROC
Random (fold 4)	980	241	48

Appendix

Table 6: Task sizes and targets. Counts refer to BGCs except for A-domain prediction. Targets count evaluated classes, descriptors, substrates, or retrieval candidates; ranges span folds.

Descriptor	Group	Source
MolWt	ADMET	Lipinski; GuacaMol
MolLogP	ADMET	Lipinski; GuacaMol
NumHDonors	ADMET	Lipinski; GuacaMol
NumHAcceptors	ADMET	Lipinski; GuacaMol
TPSA	ADMET	Veber; GuacaMol
NumRotatableBonds	ADMET	Veber; GuacaMol

Appendix

Table 7: Main RDKit descriptor panel. Groups distinguish ADMET/drug-likeness from natural-product structure features.

Pfam-domain encoders
	Literature panel (13)			Data-driven panel (66)
Encoder	Rand.	Scaf.	GCF	Rand.	Scaf.	GCF
Bag-of-Pfam	0.354	0.281	0.283	0.277	0.198	0.155
Random-Pfam	0.523	0.467	0.417	0.406	0.347	0.303
BiGCARP	0.574	0.519	0.509	0.430	0.370	0.340
Kawano	0.455	0.345	0.337	0.309	0.227	0.217

Appendix

Table 8: Sensitivity to descriptor selection. Scores are mean Spearman

\rho

from linear probes; Kendall’s

\tau

compares encoder rankings between panels.

Setting	Value
Shared dimension	$256$
Optimizer	AdamW
Learning rate	$10^{-4}$
Weight decay	$10^{-4}$
Batch size	$256$ BGC–product pairs
Temperature	Learnable; initialized at $0.07$ , minimum $0.01$

Appendix

Table 9: Shared training settings for retrieval heads.

Features	5-fold (BGC-grouped)	GCF holdout
Stachelhaus code (9 residues)	0.830 [0.814, 0.846]	0.825 [0.781, 0.866]
Amino-acid 3-mer (best encoder in Table 2 )	0.781 [0.764, 0.799]	0.747 [0.695, 0.791]
Majority substrate	0.106	0.127

Appendix

Table 10: A-domain substrate prediction. Scores are top-1 accuracies.

Encoder	Backbone	Pretraining corpus	Vocabulary	Objective	Downstream pooling
Bag-of-Pfam	none (compositional baseline)	no pretraining; raw Pfam-domain bag-of-counts representation	$3{,}932$ Pfam IDs	none	per-BGC Pfam-domain count vector, $L_{2}$ -normalized ( $3{,}932$ -dim)
Random-Pfam	none (compositional baseline)	no pretraining; one fixed random vector per Pfam ID	$3{,}932$ Pfam IDs	none	i.i.d. $\mathcal{N}(0,1)$ vector per Pfam ID (seed 42), averaged over the cluster’s Pfam tokens, $L_{2}$ -normalized ( $256$ -dim)
BiGCARP [ 20 ]	ByteNet, $32$ layers, $d_{\text{model}}{=}256$ , ${\sim}34$ M params	antiSMASH-DB [ 44 ] BGC Pfam-domain sequences ( $127{,}294$ BGCs before train/validation/test split)	$19{,}509$ tokens	masked-token prediction (MLM)	mean of last-layer token states across the Pfam sequence ( $256$ -dim)
BiGCARP-CLS [ 20 ] (ablation)	same as BiGCARP	same as BiGCARP	same as BiGCARP	same as BiGCARP	last-layer state of a mask token prepended at position 0 ( $256$ -dim)
Kawano [ 21 ]	RoBERTa, $8$ layers, $16$ heads, hidden $1024$ ( $\sim$ 110 M params)	antiSMASH-DB bacterial BGC Pfam-domain sequences ( $239{,}021$ BGCs; Model I of four released models)	$19{,}722$ -token vocabulary	masked-token prediction (MLM)	mean of last-layer hidden states across the Pfam sequence, $\texttt{max\_length}{=}512$ ( $1{,}024$ -dim)
DeepBGC [ 22 ]	Pfam2vec ( $100$ -dim) + BiLSTM ( $128$ units, bidirectional)	Pfam-domain sequences of $3{,}376$ bacterial genomes (Pfam2vec); MIBiG 1.4 BGCs ( $1{,}811$ accessions) and $10{,}128$ synthetic negatives (BiLSTM; released v0.1.0 detector)	$13{,}311$ Pfam IDs	supervised binary BGC detection	cluster boundary flags as input; mean last-layer BiLSTM states ( $256$ -dim)

Appendix

Table 11: Encoder pretraining and embedding extraction.

Task or setting	Configuration
Linear probes
Multi-label classification	One-vs-rest logistic regression
A-domain substrates	Multinomial logistic regression
Logistic regression settings	L 2 penalty; $C=1.0$ ; maximum $5{,}000$ iterations
Chemical properties	Ridge regression; $\alpha=1.0$
MLP classification and regression

Appendix

Table 12: Classification and regression probe settings.

(a) Classification and chemical properties
Model	Coarse classes			Fine classes			Chem. properties
	macro-AUROC			weighted macro-AUROC			mean Spearman $\rho$
	Rand.	Scaf.	GCF	Rand.	Scaf.	GCF	Rand.	Scaf.	GCF
Pfam-domain encoders
Bag-of-Pfam	$0.929^{+.020}_{-.020}$	$0.877^{+.036}_{-.042}$	$0.884^{+.027}_{-.029}$	$0.880^{+.023}_{-.031}$	$0.825^{+.028}_{-.028}$	$0.753^{+.041}_{-.039}$	$0.354^{+.052}_{-.057}$	$0.281^{+.048}_{-.057}$	$0.283^{+.045}_{-.054}$
Random-Pfam	$0.936^{+.017}_{-.014}$	$0.909^{+.023}_{-.026}$	$0.896^{+.020}_{-.022}$	$0.880^{+.023}_{-.027}$	$0.851^{+.025}_{-.026}$	$0.786^{+.033}_{-.033}$	$0.523^{+.043}_{-.049}$	$0.467^{+.042}_{-.053}$	$0.417^{+.051}_{-.061}$

Appendix

Table 13: Confidence intervals for Table 2 . Superscripts and subscripts show distances to the upper and lower

95\,\%

bounds.

Pfam-domain encoders
Encoder	PKS	NRPS	Ribosomal (RiPP)	Terpene	Saccharide	Other
Bag-of-Pfam	0.950	0.941	0.997	0.942	0.868	0.873
Random-Pfam	0.962	0.952	0.992	0.963	0.892	0.856
BiGCARP	0.973	0.961	0.980	0.969	0.935	0.896
BiGCARP-CLS	0.954	0.933	0.993	0.937	0.912	0.842
Kawano	0.966	0.965	0.966	0.957	0.921	0.833

Appendix

Table 14: Per-class biosynthetic prediction. Scores are random-split AUROCs from linear probes.

Encoder	$\bar{\rho}$	Mean split-CI bounds
DeepBGC	0.546	[0.499, 0.587]
BiGCARP	0.535	[0.485, 0.580]
Random-Pfam	0.469	[0.415, 0.515]
BGC-Prophet	0.452	[0.401, 0.499]
BiGCARP-CLS	0.442	[0.392, 0.484]
Amino-acid 3-mer	0.431	[0.375, 0.482]

Appendix

Table 15: Chemical-property prediction: mean Spearman

\rho

. Bounds average split-specific CI endpoints.

Figure 4: Chemical-property prediction by descriptor. Panel columns compare random and scaffold splits; panel rows compare Ridge and MLP probes on a shared Spearman $\rho$ scale.

Pfam-domain encoders
	B $\to$ C				C $\to$ B
Encoder	Rand.	Scaf.	GCF	Class	Rand.	Scaf.	GCF	Class
Bag-of-Pfam	0.624 $\pm$ 0.006	0.546 $\pm$ 0.021	0.495 $\pm$ 0.019	0.275 $\pm$ 0.012	0.633 $\pm$ 0.009	0.607 $\pm$ 0.029	0.493 $\pm$ 0.013	0.286 $\pm$ 0.007
Random-Pfam	0.542 $\pm$ 0.005	0.504 $\pm$ 0.041	0.448 $\pm$ 0.032	0.209 $\pm$ 0.006	0.553 $\pm$ 0.003	0.544 $\pm$ 0.043	0.466 $\pm$ 0.022	0.288 $\pm$ 0.015
BiGCARP	0.559 $\pm$ 0.004	0.466 $\pm$ 0.037	0.435 $\pm$ 0.020	0.222 $\pm$ 0.007	0.556 $\pm$ 0.005	0.503 $\pm$ 0.011	0.470 $\pm$ 0.021	0.256 $\pm$ 0.002
BiGCARP-CLS	0.372 $\pm$ 0.024	0.291 $\pm$ 0.024	0.248 $\pm$ 0.020	0.119 $\pm$ 0.005	0.373 $\pm$ 0.031	0.429 $\pm$ 0.021	0.263 $\pm$ 0.024	0.140 $\pm$ 0.004

Appendix

Table 16: Retrieval in both directions. Class-macro R@10 for BGC-to-compound (left) and compound-to-BGC (right); mean

\pm

standard deviation over three seeds.

Figure 5: BGC and chemical encoder combinations. BGC-to-compound class-macro R@10, averaged over three seeds, with native chemical embeddings and PCA- $128$ projections.

Pfam-domain encoders
	Coarse classes (macro-AUROC)			Fine classes (weighted macro-AUROC)			Chem. prop. (mean Spearman $\rho$ )
Encoder	Random	Scaffold	GCF	Random	Scaffold	GCF	Random	Scaffold	GCF
Bag-of-Pfam	0.950	0.896	0.891	0.886	0.859	0.778	0.585	0.523	0.469
Random-Pfam	0.957	0.957	0.929	0.905	0.899	0.828	0.655	0.560	0.529
BiGCARP	0.970	0.950	0.943	0.912	0.893	0.847	0.678	0.604	0.568
BiGCARP-CLS	0.951	0.933	0.931	0.876	0.840	0.832	0.574	0.479	0.523

Appendix

Table 17: MLP probes for BGC-level prediction. Metrics and splits follow Table 2 .

Pfam-domain encoders
	Retrieval (class-macro R@10)				A-domain substrate (top-1)
Encoder	Random	Scaffold	GCF	Class	5-fold	GCF
Bag-of-Pfam	0.631	0.582	0.502	0.249	0.198	0.137
Random-Pfam	0.580	0.540	0.442	0.219	–	–
BiGCARP	0.595	0.529	0.460	0.232	0.277 ↑	0.168
BiGCARP-CLS	0.385	0.328	0.252	0.135	–	–

Appendix

Table 18: MLP readouts for retrieval and A-domain prediction.

Pfam-domain encoders
Model	Coarse classes			Fine classes			Chem. properties			BGC–molecule retrieval				A-domain substrate
	macro-AUROC			weighted macro-AUROC			mean Spearman $\rho$			class-macro R@10				top-1 accuracy
	R	S	G	R	S	G	R	S	G	R	S	G	C	CV	G
Bag-of-Pfam	.954	.938	.920	.886	.846	.789	.622 ↑	.595 ↑	.525 ↑	.596	.551	.532	.247	.194	.106
Kawano	.929	.915	.885	.837	.804	.787	.565 ↑	.501 ↑	.474 ↑	.512	.457	.413	.177	.200	.106
Amino-acid encoders

Appendix

Table 19: Main results after PCA to 256 dimensions. Only the nine projected encoders are shown; bold marks the best among them. Arrows mark

95\,\%

intervals entirely above or below the same encoder’s native interval.

		BiGCARP			Kawano
Task	Probe	Rand.	Scaf.	GCF	Rand.	Scaf.	GCF
Coarse classes	linear	$-0.001$	$-0.007$	$+0.010$	$+0.014$	$+0.008$	$0.000$
	MLP	$-0.003$	$+0.006$	$+0.001$	$+0.003$	$+0.007$	$+0.001$
Fine classes	linear	$0.000$	$-0.002$	$-0.014$	$+0.029$	$+0.027$	$+0.014$
	MLP	$-0.001$	$-0.003$	$-0.011$	$+0.001$	$+0.036$	$+0.004$
Chem. properties	linear	$-0.008$	$+0.013$	$-0.032$	$-0.004$	$+0.022$	$+0.010$

Appendix

Table 20: Effect of continued pretraining. Mean score change across three seeds (adapted minus released).

	TypiClust $-$ random					BADGE $-$ random
Encoder	Coarse	Fine	Chem.	Retr.	A-dom.	Coarse	Fine	Retr.	A-dom.
Bag-of-Pfam	+0.5	+0.8	$-$ 1.6	+0.3	+0.8	+2.6	+0.1	+2.3	0.0
Random-Pfam	+3.0	+0.7	+4.0	+5.5	–	+0.9	$-$ 0.2	+1.4	–
BiGCARP	+1.1	+1.5	+6.8	+5.3	+1.5	+1.6	$-$ 0.4	+1.3	0.0
Kawano	+3.0	+1.5	+3.7	+5.9	+1.2	+1.3	$-$ 0.5	+0.8	+0.1
DeepBGC	+1.7	+1.5	+1.5	+3.2	–	+1.1	+0.8	+0.5	–

Appendix

Table 22: Acquisition gains by encoder. Mean

\mathrm{AULC}_{\mathrm{low}}

gains over random acquisition, in metric points (

\times 100

). Bold marks significant differences under the corrected tests.

Encoder	Coarse	Fine	Chem.	Retr.	A-dom.
Bag-of-Pfam	1.00	1.00	1.00	1.00	1.00
Random-Pfam	1.57	1.13	0.28	0.83	–
BiGCARP	2.22	1.12	$\geq$ 1.00	1.18	0.91
Kawano	1.72	1.01	0.60	0.78	1.19
DeepBGC	1.49	0.86	1.26	0.58	–
Amino-acid 3-mer	0.27	0.81	0.63	0.53	$\geq$ 4.00

Appendix

Table 23: Label savings of each encoder under random acquisition, relative to Bag-of-Pfam under random acquisition.

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 405 papers

Jul 29, 2026

PlantBGC: Transformer for Plant BGC Discovery via Label-Free Domain Adaptation and Weak Supervision

Plant biosynthetic gene clusters (BGCs) encode specialized-metabolite pathways, yet curated plant BGC labels remain scarce, hindering supervised discovery at genome scale. Existing plant BGC mining tools are largely signature- and rule-driven and do not fully leverage recent advances in contextual representation learning for modeling long-range domain context and controlling false positives under strong domain shift. We seek an AI-assisted workflow that narrows experimental search space by transferring supervision from well-annotated microbial BGCs to plant genomes. We present PlantBGC, representing genomes as ordered Pfam-domain sequences and learning BGC-likeness with an encoder-only Transformer trained on MIBiG microbial BGCs and adapted to plants via label-free masked language modeling. On microbial benchmarks, PlantBGC achieves token-level AUC = 0.988 (10-fold CV) and 0.979 (leave-class-out). On plants, adaptation improves known-BGC recovery on n = 34 curated loci under strict 100% coverage, increasing recovery from 29.4% to 67.6% and indicating more complete boundaries. GO/KEGG-derived weak supervision reduces proxy primary-like ratio by 48.40% (GO) and 45.20% (KEGG), with consistent per-species reductions (paired Wilcoxon p = 1.53e-5). Compared to plantiSMASH, PlantBGC yields more compact loci on matched regions (median length ratio = 0.278; 93.8% of pairs are shorter).

Yuhan Zhao, Nidhi Grover, Zhishan Guo +1

North Carolina State University Department of Computer Science Raleigh, North Carolina, USA · North Carolina State University Department of Molecular and Structural Biochemistry Raleigh, North Carolina, USA

Genomics Bioinformatics

Ask PDF

Oct 23, 2026

BenchBench-Cloning: Evaluating Molecular Cloning Design and Protocol Development

Molecular cloning is one of the most routine wet-lab workflows, yet existing evaluations of language models in biology mostly test knowledge recall or isolated steps rather than end-to-end experimental planning. We introduce BenchBench-Cloning, a benchmark for large language models comprising 65 molecular cloning tasks grounded in sequence records, laboratory notebooks, and published experimental studies. Planning a molecular cloning experiment requires defining a set of constructs needed for a scientific objective and a construction route to produce them. BenchBench-Cloning contains 50 protocol-development tasks, in which agents produce an executable protocol for a specified construct, and 15 construct-design tasks, in which agents determine the constructs a scientific objective calls for. Protocol-development tasks are evaluated by execution in a cloning simulator; construct-design tasks are evaluated with task-specific deterministic checks and, where needed, rubric-based LLM-as-a-judge evaluation. Across eight systems with repeated attempts per task, GPT-6 Astra scores highest at 30.6%, with the others between 8.6% and 22.2%, and the benchmark remains unsaturated under best-of-ten sampling. As integrating individual cloning decisions into complete, valid plans remains challenging even for frontier models, we expect BenchBench-Cloning to serve as a useful benchmark for tracking progress in experimental planning.

Aditya Sivakumar, Peter Ciaccia, Hassaan Qadir +2

Benchling · DNAForge

AI-Assisted Scientific Research LLM Evaluation Scientific Agent Benchmarks

Ask PDF

Nov 2, 2026

BEND 2.0: Benchmarking the Capabilities Needed to Design DNA

Genomic foundation models are increasingly used to propose DNA sequences and edits. But successful design requires capabilities that current benchmarks do not measure. Here we introduce BEND 2.0 (BENchmark for DNA language models), which tests how well models predict the effects of DNA changes and whether these predictions can guide sequence design. Across eight tasks built on five experimental datasets, we find that model choice and how representations are accessed both affect performance. Linear readouts fitted to frozen representations can match or exceed models' native outputs, and functional post-training substantially improves the models' capabilities. Linear readouts transfer well across experimental assays and genes, suggesting that models capture functional patterns beyond the data used to fit individual predictors. At the same time, strong prediction of average effects can conceal weaker prediction of context-specific effects, and accurate prediction of combined mutation effects does not imply accurate prediction of their interactions. In design tasks, model predictions can reduce the number of candidates needed to find a desired response, but still leave substantial room for better design choices, particularly for context-specific and multi-cell objectives. Our benchmark reveals that genomic models now display signs of general utility but have yet to saturate on performance in even simple tasks.

Andreas Bjerregaard, Rachael DeVries, Dionysia Danai Brilli +6

University of Copenhagen · Novo Nordisk A/S · Technical University of Denmark

Genomic Foundation Models Genomics Benchmark Design

Ask PDF

Abstract

Benchmark Design

Explore similar work

PlantBGC: Transformer for Plant BGC Discovery via Label-Free Domain Adaptation and Weak Supervision

BenchBench-Cloning: Evaluating Molecular Cloning Design and Protocol Development

BEND 2.0: Benchmarking the Capabilities Needed to Design DNA