We introduce BenchBench-Protocol, a benchmark for large language models of 149 protocol-modification tasks recovered from modifications that scientists made to published protocols during real experimental work. Adapting a published protocol to a new experiment is a routine task for a wet-lab scientist, and a correct modification requires accounting for prior choices and downstream steps. Recent life-science benchmarks have moved toward open-ended, rubric-graded tasks, but tasks are typically elicited from experts rather than reconstructed from real-world modifications. BenchBench-Protocol tasks are derived from differences between a published protocol and a version a scientist modified, which provides the basis for the query and the weighted rubric elements for a correct response. The benchmark draws from 96 source protocols across nine domains of wet-lab biology and only includes tasks rated highly after review by domain experts. We evaluate nine closed and open models; Claude Opus 5 scores highest at 59.2% normalized rubric score, with other models between 34.1% and 47.1%, and the benchmark remains unsaturated when taking the best of ten attempts. As models are increasingly helpful in life-sciences research, evaluating them on routine wet-lab tasks becomes correspondingly important. We present BenchBench-Protocol as both a grounded assessment of wet-lab reasoning and evidence for the utility of real-world experiments to construct benchmark tasks.
Figures & tables
Figure 1: Benchmark construction overview. Published protocols and scientist-adapted versions on Benchling from contributors are compared to produce tasks and weighted rubrics, which are retained after expert quality control.
Protocol domain
Tasks
Share
Protocols
Protein biochemistry and purification
32
21.5%
21
Cell culture and cell-based assays
27
18.1%
16
Molecular biology and cloning
26
17.4%
17
Staining, imaging, and cytometry
24
16.1%
17
Genomics, transcriptomics, and sequencing
19
12.8%
13
Microbiology and microbial methods
14
9.4%
8
Table 1: Benchmark coverage by protocol domain. Task and source protocol counts are shown for each of the nine domains.
Primary task intent
Tasks
Share
Plan audit
38
25.5%
Measurement and result interpretation
35
23.5%
Troubleshooting
29
19.5%
Recordkeeping
23
15.4%
Procedure specification and adaptation
14
9.4%
Quantitative formulation
6
4.0%
Table 2: Distribution of primary task intents. Task counts and shares are shown for each of the seven intents.
Figure 2: Expert review ratings by assessment dimension. Share of retained tasks rated Excellent, Very Strong, or below Very Strong on each of the five dimensions.
Figure 5
Figure 3: Mean normalized rubric score by model. Means across 149 tasks are shown with 95% confidence intervals.
Figure 4: Expected best-of- k score for three representative models. Expected maximum score is shown for k of ten attempts per task with 95% confidence bands. Complete nine-model results are in Table E2 .
Figure 5: Mean normalized rubric score by primary task intent. Cells give mean scores in percent, with task counts per intent.
Figure 6: Mean normalized rubric score by source-protocol domain. Cells give mean scores in percent, with task counts per domain.
Figure 7: Token usage per answer attempt. Average input and output tokens are shown per model; Gemini 3.6 Flash is excluded (asterisk).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Access
Reasoning
Search provider
Claude Opus 5
Anthropic API
max
First party / native
GPT-5.6 Sol
OpenAI API
max
First party / native
GPT-5.6 Terra
OpenAI API
max
First party / native
GPT-5.6 Luna
OpenAI API
max
First party / native
Gemini 3.6 Flash
Vertex AI
HIGH
First party / native
Grok 4.5
xAI Responses
high
First party / native
Appendix
Table B1: Model inference configurations. Access route, reasoning tier, and search provider are listed per model.
Figure D1: Distribution of average task scores by model. Per-task means across ten attempts are shown for each model, with the model mean marked.
System
@1
@2
@5
@10
Gain
95% CI @10
Claude Opus 5
.592
.651
.714
.753
+.161
[.728,.777]
GPT-5.6 Sol
.471
.509
.551
.575
+.104
[.552,.598]
Kimi K3
.457
.528
.592
.628
+.171
[.601,.654]
GPT-5.6 Terra
.453
.497
.546
.577
+.123
[.550,.603]
GPT-5.6 Luna
.413
.453
.498
.527
+.115
[.501,.553]
Inkling
.394
.449
.506
.541
+.147
[.512,.569]
Appendix
Table E2: Expected best-of- k metrics. Scores at k=1,2,5, and 10 are shown with the gain from k=1 to k=10 .
Figure 14Figure 15Figure 16Figure 17
Figure G1: Answer-generation cost per attempt. Average input and output cost is shown per model, excluding web search and hosted-tool usage.
Figure H1: Average score against answer-generation cost per attempt. The Pareto frontier is drawn over GPT-5.6 Luna, Kimi K3, and Claude Opus 5; the score axis is truncated.
Model
Misformatted
Refusals
Total
Claude Opus 5
0 / 1,490
0 / 1,490
0.0%
GPT-5.6 Sol
0 / 1,490
0 / 1,490
0.0%
Kimi K3
62 / 1,490
0 / 1,490
4.2%
GPT-5.6 Terra
0 / 1,490
0 / 1,490
0.0%
GPT-5.6 Luna
0 / 1,490
0 / 1,490
0.0%
Inkling
28 / 1,490
0 / 1,490
1.9%
Appendix
Table I1: Rate of refusals and misformatted outputs across model families.
Molecular cloning is one of the most routine wet-lab workflows, yet existing evaluations of language models in biology mostly test knowledge recall or isolated steps rather than end-to-end experimental planning. We introduce BenchBench-Cloning, a benchmark for large language models comprising 65 molecular cloning tasks grounded in sequence records, laboratory notebooks, and published experimental studies. Planning a molecular cloning experiment requires defining a set of constructs needed for a scientific objective and a construction route to produce them. BenchBench-Cloning contains 50 protocol-development tasks, in which agents produce an executable protocol for a specified construct, and 15 construct-design tasks, in which agents determine the constructs a scientific objective calls for. Protocol-development tasks are evaluated by execution in a cloning simulator; construct-design tasks are evaluated with task-specific deterministic checks and, where needed, rubric-based LLM-as-a-judge evaluation. Across eight systems with repeated attempts per task, GPT-6 Astra scores highest at 30.6%, with the others between 8.6% and 22.2%, and the benchmark remains unsaturated under best-of-ten sampling. As integrating individual cloning decisions into complete, valid plans remains challenging even for frontier models, we expect BenchBench-Cloning to serve as a useful benchmark for tracking progress in experimental planning.
Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studies and human annotations inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements. We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task. From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool-mediated analysis in a controlled setting without distributing a large static corpus. InfiniteScienceGym complements real scientific benchmarks by targeting blind spots and failure modes that are hard to evaluate using published datasets alone. Evaluating both proprietary and open-weight models, we find that none achieve more than 50% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.
Laboratory experiments involve more than following predefined protocols; they require adaptive decision-making as the experimental state changes. Existing scientific benchmarks rarely assess such evidence-based decisions in laboratory settings. Here we present LabOPBench, a multimodal benchmark of 756 expert-authored, independently reviewed questions in organic synthesis and materials science. Each item pairs laboratory evidence with procedural context and an open-ended judgment, evaluated using a question-specific scoring rubric. Eight multimodal models obtain overall normalized scores of 28.8-45.8%, leaving substantial room for improvement. On a controlled 100-question subset, the strongest models outperform undergraduate participants on average but remain below the average performance of PhD participants. Expert analysis of responses from the three leading models identifies Evidence Misreading and Evidence-Decision Conflict as the dominant error patterns. These results reveal that current models still struggle to turn multimodal perception into reliable laboratory decision-making.
Jingyuan Li, Yujie Liu, Yijie Dai +18
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Fudan University +6