Evaluating and improving AI agents for sequencing protocol understanding
Authors: Chi-Lam Poon, Michael Goneos, Zezheng Song, Xi Chen
Organizations: Memorial Sloan Kettering Cancer Center · Weill Cornell Medicine · NASA Ames Research Center · University of Maryland, College Park · Southern University of Science and Technology
Existing scientific-agent benchmarks rarely evaluate whether agents can reconstruct the exact molecular structures and generation workflows specified by sequencing protocols, a capability critical for analyzing customized sequencing assays. We introduce LibStructBench, a document-grounded benchmark for reconstructing sequencing-library structures and molecular workflows. Our experiments show that frontier agents often produce approximately correct reconstructions but remain unreliable at exact reconstruction, particularly in molecular-state representation and workflow connectivity. Switching the underlying language model produced larger average performance differences than switching agent harnesses, with a statistically significant difference only for molecular-state reconstruction. Additional knowledge improved oligo-family recovery in a small pilot without improving mean molecular-workflow scores. Autonomous accumulation of procedural rules and cross-protocol examples showed non-monotonic validation performance and no mean held-out molecular-workflow improvement in the tested trajectory. These findings highlight the need for more controlled and verifiable approaches to scientific reasoning about molecular processes in complex experimental protocols.
Figures & tables
Figure 1: LibStructBench overview. Given a protocol bundle containing primary source evidence, an AI agent reconstructs three linked representations of sequencing-library construction, corresponding to Tasks 1-3. The schematic illustrates these tasks using a simplified workflow for 10x Chromium 3 ′ Gene Expression v4 [ 1 ] .
Figure 2: Task 1 performance for direct APIs versus native coding agents. Each point represents one sequencing protocol completed by both interfaces. Δ denotes the mean paired similarity difference (agent minus API), and padj denotes the adjusted p -value for two-sided paired Wilcoxon signed-rank test. The inset table reports mean normalized sequence similarity and exact reconstruction counts for each interface.
Figure 3: Baseline agent comparison. (a) Mean Task 2 oligo-family and Task 3 transition, state, and typed-edge F1 for native CLI systems. (b) Task 3 molecular-transition F1 across 10 common protocols and nine model-harness systems, clustered by protocol-level performance.
Figure 4: Improving molecular-workflow reconstruction. (a) Mean Task 2 and Task 3 scores under the baseline, general-knowledge, cross-protocol memory, and combined conditions. (b) Autonomous cumulative-improvement framework. (c) Validation performance across autonomous-training checkpoints. (d) Held-out test performance of baseline and the validation-selected checkpoint with and without knowledge base (KB) supplement. For panels (a) and (c), Task 2 includes four informative validation protocols and Task 3 includes five; the two experiments use different validation panels. Panel (d) includes nine informative held-out protocols for Task 2 and ten protocols for Task 3.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Human-agent ground-truth audit and adjudication. (a) Existing human curation is converted to the canonical representation and verified against primary evidence. Claude Code proposes corrections, a human curator adjudicates substantive findings, and only approved decisions are applied by deterministic code. (b) Distribution of confirmed audit findings of errors from 73 usable protocols collected in scg_lib_structs .
Figure S2 : PDF-parser-control sequence recovery. Sequence recovery was evaluated across 80 verified nucleotide strings from 5 protocols. For each reference sequence, we report the highest normalized Levenshtein similarity to any sequence extracted. The table summarizes exact recovery and mean similarity.
Figure S3 : GPT-Codex variability across three executions per protocol. (a) Within-protocol F1 ranges. Points represent protocols; diamonds and intervals indicate medians and interquartile ranges. The dashed line marks a range of 0.10, with annotations indicating counts above this threshold. (b) Protocol-macro F1 for each execution and their three-execution mean.
Metric
Original
3-run mean
Δ
Max. range
R>0.10
Oligo family
0.760
0.755
−0.005
0.177
5/19
Transition
0.809
0.795
−0.014
0.343
9/20
State
0.672
0.660
−0.012
0.394
8/20
Typed edge
0.754
0.730
−0.024
0.471
12/20
Appendix
Table S1: Observed variability across three GPT-Codex executions per protocol. Macro-F1 averages 19 informative protocols for oligo recovery and 20 for each Task 3 metric.
Task 1
Task 2
Task 3
Similarity
Oligo F1
Transition F1
State F1
Edge F1
Protocol
Model
PDF
Name
Bundle
Name
Bundle
Name
Bundle
Name
Bundle
Name
10x ATAC v2
GPT
1.000
0.947
0.791
0.673
0.782
0.848
0.875
0.863
0.714
1.000
Claude
1.000
1.000
0.713
0.687
0.777
0.761
0.776
0.812
0.714
0.714
Gemini
1.000
1.000
0.701
0.673
0.956
0.707
0.720
0.396
1.000
0.667
10x 3’ GEX v4
GPT
1.000
1.000
0.724
0.867
0.881
0.950
0.868
0.874
0.846
1.000
Appendix
Table S2 : Protocol-name-only reconstruction across Tasks 1–3. Task 1 reports API sequence similarity (PDF vs. Name); Tasks 2–3 report native-agent F1 (Bundle vs. Name), using the Figure 3(a) bundle executions. Bold indicates the higher score within each pair; ties are unbolded. Δ denotes Name minus source-input score. padj denotes Bonferroni-adjusted p -values, correcting separately across three Task 1 and 12 Tasks 2–3 comparisons.
Figure S4 : Complementary performance metrics across models and harnesses. Heatmaps show performance across the ten protocols and nine model–harness systems used in Figure 3(b) . System and protocol orders are fixed to the molecular-transition-F1 clustering in the main figure to permit direct comparison across metrics. All color scales range from 0 to 1.
Figure S5 : Estimated API-equivalent cost and reconstruction performance. (a) Per-protocol costs and system means across ten protocols. (b) Cost versus mean performance across four F1 metrics.
Figure S6 : Model-harness effects. Mean absolute switch magnitudes (left) and paired model-minus-harness contrasts (right). Intervals are pointwise 95% protocol-bootstrap CIs, conditional on the selected executions. The state annotation gives pBonf for the primary four-metric comparison.
Figure S7 : Baseline output-discrepancy profiles. (a) The fraction of the ten protocols with at least one discrepancy. (b) The fraction of the ten systems with at least one discrepancy. Values range from 0 to 1, where 1 indicates this error appeared in 100% protocols or LLM-harness runs.
Bisulfite conversion, reduced-representation libraries, and size selection.
Appendix
Table S4 : Sources for the general molecular-method knowledge primer. The primer summarizes general molecular operations instead of protocol-specific answers.
Table S5: Protocols used in the cross-protocol memory pilot. Donor protocols were selected as conceptual matches to the validation protocols based on shared molecular features.
Checkpoint
Learned context
Observed reconstruction behavior
Validation performance
Baseline
No learned rules or cross-protocol memory.
Direct source-to-output reconstruction followed by the benchmark validator.
Task 2 F1: 0.758. Task 3 transition, state, and typed-edge F1: 0.846, 0.625, and 0.822.
C30
18 learned rules and 30 available exemplar records.
The agent constructed an evidence inventory, retrieved selected analogies, and compiled Task 2 and Task 3 from a linked work record. The predicted graphs averaged 10.0 states, 7.2 transitions, and 16.2 typed edges.
Highest Task 2 F1 (0.821), with transition, state, and typed-edge F1 of 0.843, 0.603, and 0.790. Selected checkpoint for held-out evaluation.
C40
23 learned rules and 40 exemplar records.
Two newly learned rules may have contributed to the observed decline. A rule excluding oligo-reagent fabrication from the workflow was associated with omitted bead-construction steps in HyDrop-RNA, while a rule separating co-processed but physically unconnected molecules was associated with the separation of the ddSEQ workflow into disconnected branches. These behaviors suggest that learned rules may generalize poorly to the benchmark’s canonical representation.
Task 2 F1 decreased to 0.736; transition, state, and typed-edge F1 decreased to 0.743, 0.497, and 0.662.
C58
28 learned rules and 58 exemplar records.
The agent represented oligo chemistry, non-templated tail addition, modified bases, and uncertain fragment fates more explicitly. Graphs again became more granular.
State F1 partially recovered to 0.539, while Task 2, transition, and typed-edge F1 were 0.745, 0.720, and 0.638, respectively.
Appendix
Table S6: Behavioral comparison of selected autonomous-learning checkpoints. Learned rules are protocol-neutral instructions accumulated from previous training protocols; exemplar records are ground-truth-derived cross-protocol examples available for retrieval. Task 2 scores are macro-averaged over four informative protocols; ddSEQ has no required recoverable oligo families and is reported separately. Task 3 scores include all five validation protocols.
Prioritizes molecular sequence identity while accounting for strand architecture, segment organization, and pairing relationships.
Appendix
Table S7: Default component weights for pairwise scientific similarity. These similarities contribute to entity matching and subsequent F1 calculation. For oligo-family and molecular-state scoring, weights are renormalized over components enabled by the ground-truth support rules.
Figure S8 : Sensitivity of native-system performance to similarity weights. Macro-averaged oligo-family, molecular-transition, and molecular-state F1 scores for four native coding-agent systems. The displayed ranges characterize sensitivity to scoring assumptions, not statistical confidence intervals.
Experiment
Transition F1
State F1
Typed-edge F1
Fig. 3a: Native systems
0.035
0.031
0.069
Fig. 3b: Model-harness comparison
0.023
0.020
0.045
Fig. 4a: External knowledge
0.029
0.013
0.054
Fig. 4c: Learning validation
0.055
0.025
0.132
Fig. 4d: Held-out evaluation
0.018
0.009
0.031
Appendix
Table S8: Sensitivity to state and transition assignment thresholds. Maximum absolute change in protocol-macro F1 relative to the default thresholds (0.25,0.25) , across nine threshold combinations and all systems or conditions within each experiment.
Figure S9 : Sensitivity of molecular-workflow scores to matching thresholds. Heatmaps show the maximum absolute changes in molecular-transition, molecular-state, and typed-edge F1 across autonomous-learning validation checkpoints relative to their respective scores under the default thresholds. State and transition assignment thresholds were independently varied over {0.15,0.25,0.40} . Each cell represents the maximum across checkpoints, rather than the average change. The default combination (0.25,0.25) is outlined.
Figure S10 : Robustness of knowledge augmentation and autonomous-learning results. Differences in F1 relative to baseline for combined external knowledge versus baseline; final checkpoint C58 versus C0 on the learning-validation panel; and C30 with knowledge-base supplementation versus C0 on held-out protocols.
Large language models (LLMs) hold considerable potential for advancing scientific discovery, yet systematic assessment of their dynamic reasoning in real-world research remains limited. Current scientific evaluation benchmarks predominantly rely on static, single-turn Question Answering (QA) formats, which are inadequate for measuring model performance in complex scientific tasks that require multi-step iteration and experimental interaction. To address this gap, we introduce MolQuest, a novel agent-based evaluation framework for molecular structure elucidation built upon authentic chemical experimental data. Unlike existing datasets, MolQuest formalizes molecular structure elucidation as a multi-turn interactive task, requiring models to proactively plan experimental steps, integrate heterogeneous spectral sources (e.g., NMR, MS), and iteratively refine structural hypotheses. This framework systematically evaluates LLMs' abductive reasoning and strategic decision-making abilities within a vast and complex chemical space. Empirical results reveal that contemporary frontier models exhibit significant limitations in authentic scientific scenarios: notably, even state-of-the-art (SOTA) models achieve an accuracy of only approximately 50%, while the performance of most other models remains below the 30% threshold. This work provides a reproducible and extensible framework for science-oriented LLM evaluation, our findings highlight the critical gap in current LLMs' strategic scientific reasoning, setting a clear direction for future research toward AI that can actively participate in the scientific process.
Molecular cloning is one of the most routine wet-lab workflows, yet existing evaluations of language models in biology mostly test knowledge recall or isolated steps rather than end-to-end experimental planning. We introduce BenchBench-Cloning, a benchmark for large language models comprising 65 molecular cloning tasks grounded in sequence records, laboratory notebooks, and published experimental studies. Planning a molecular cloning experiment requires defining a set of constructs needed for a scientific objective and a construction route to produce them. BenchBench-Cloning contains 50 protocol-development tasks, in which agents produce an executable protocol for a specified construct, and 15 construct-design tasks, in which agents determine the constructs a scientific objective calls for. Protocol-development tasks are evaluated by execution in a cloning simulator; construct-design tasks are evaluated with task-specific deterministic checks and, where needed, rubric-based LLM-as-a-judge evaluation. Across eight systems with repeated attempts per task, GPT-6 Astra scores highest at 30.6%, with the others between 8.6% and 22.2%, and the benchmark remains unsaturated under best-of-ten sampling. As integrating individual cloning decisions into complete, valid plans remains challenging even for frontier models, we expect BenchBench-Cloning to serve as a useful benchmark for tracking progress in experimental planning.
Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam.
Tianyun Zhong, Wei Wang, Jialin Li +17
University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group +4