cs.AIOct 27, 2026
SaveEvaluating and improving AI agents for sequencing protocol understanding
Organizations: Memorial Sloan Kettering Cancer Center · Weill Cornell Medicine · NASA Ames Research Center · University of Maryland, College Park ·
Abstract
Existing scientific-agent benchmarks rarely evaluate whether agents can reconstruct the exact molecular structures and generation workflows specified by sequencing protocols, a capability critical for analyzing customized sequencing assays. We introduce LibStructBench, a document-grounded benchmark for reconstructing sequencing-library structures and molecular workflows. Our experiments show that frontier agents often produce approximately correct reconstructions but remain unreliable at exact reconstruction, particularly in molecular-state representation and workflow connectivity. Switching the underlying language model produced larger average performance differences than switching agent harnesses, with a statistically significant difference only for molecular-state reconstruction. Additional knowledge improved oligo-family recovery in a small pilot without improving mean molecular-workflow scores. Autonomous accumulation of procedural rules and cross-protocol examples showed non-monotonic validation performance and no mean held-out molecular-workflow improvement in the tested trajectory. These findings highlight the need for more controlled and verifiable approaches to scientific reasoning about molecular processes in complex experimental protocols.
Figures & tables
Figure 1: LibStructBench overview. Given a protocol bundle containing primary source evidence, an AI agent reconstructs three linked representations of sequencing-library construction, corresponding to Tasks 1-3. The schematic illustrates these tasks using a simplified workflow for 10x Chromium 3 ′ Gene Expression v4 [ 1 ] .
Figure 2: Task 1 performance for direct APIs versus native coding agents. Each point represents one sequencing protocol completed by both interfaces. denotes the mean paired similarity difference (agent minus API), and denotes the adjusted -value for two-sided paired Wilcoxon signed-rank test. The inset table reports mean normalized sequence similarity and exact reconstruction counts for each interface.
Figure 3: Baseline agent comparison. (a) Mean Task 2 oligo-family and Task 3 transition, state, and typed-edge F1 for native CLI systems. (b) Task 3 molecular-transition F1 across 10 common protocols and nine model-harness systems, clustered by protocol-level performance.
Figure 4: Improving molecular-workflow reconstruction. (a) Mean Task 2 and Task 3 scores under the baseline, general-knowledge, cross-protocol memory, and combined conditions. (b) Autonomous cumulative-improvement framework. (c) Validation performance across autonomous-training checkpoints. (d) Held-out test performance of baseline and the validation-selected checkpoint with and without knowledge base (KB) supplement. For panels (a) and (c), Task 2 includes four informative validation protocols and Task 3 includes five; the two experiments use different validation panels. Panel (d) includes nine informative held-out protocols for Task 2 and ten protocols for Task 3.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Human-agent ground-truth audit and adjudication. (a) Existing human curation is converted to the canonical representation and verified against primary evidence. Claude Code proposes corrections, a human curator adjudicates substantive findings, and only approved decisions are applied by deterministic code. (b) Distribution of confirmed audit findings of errors from 73 usable protocols collected in scg_lib_structs .
Figure S2 : PDF-parser-control sequence recovery. Sequence recovery was evaluated across 80 verified nucleotide strings from 5 protocols. For each reference sequence, we report the highest normalized Levenshtein similarity to any sequence extracted. The table summarizes exact recovery and mean similarity.
Figure S3 : GPT-Codex variability across three executions per protocol. (a) Within-protocol F1 ranges. Points represent protocols; diamonds and intervals indicate medians and interquartile ranges. The dashed line marks a range of 0.10, with annotations indicating counts above this threshold. (b) Protocol-macro F1 for each execution and their three-execution mean.
| Metric | Original | 3-run mean | Max. range | ||
|---|---|---|---|---|---|
| Oligo family | 0.760 | 0.755 | 0.177 | 5/19 | |
| Transition | 0.809 | 0.795 | 0.343 | 9/20 | |
| State | 0.672 | 0.660 | 0.394 | 8/20 | |
| Typed edge | 0.754 | 0.730 | 0.471 | 12/20 |
Table S1: Observed variability across three GPT-Codex executions per protocol. Macro-F1 averages 19 informative protocols for oligo recovery and 20 for each Task 3 metric.
| Task 1 | Task 2 | Task 3 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Similarity | Oligo F1 | Transition F1 | State F1 | Edge F1 | |||||||
| Protocol | Model | Name | Bundle | Name | Bundle | Name | Bundle | Name | Bundle | Name | |
| 10x ATAC v2 | GPT | 1.000 | 0.947 | 0.791 | 0.673 | 0.782 | 0.848 | 0.875 | 0.863 | 0.714 | 1.000 |
| Claude | 1.000 | 1.000 | 0.713 | 0.687 | 0.777 | 0.761 | 0.776 | 0.812 | 0.714 | 0.714 | |
| Gemini | 1.000 | 1.000 | 0.701 | 0.673 | 0.956 | 0.707 | 0.720 | 0.396 | 1.000 | 0.667 | |
| 10x 3’ GEX v4 | GPT | 1.000 | 1.000 | 0.724 | 0.867 | 0.881 | 0.950 | 0.868 | 0.874 | 0.846 | 1.000 |
Table S2 : Protocol-name-only reconstruction across Tasks 1–3. Task 1 reports API sequence similarity (PDF vs. Name); Tasks 2–3 report native-agent F1 (Bundle vs. Name), using the Figure 3(a) bundle executions. Bold indicates the higher score within each pair; ties are unbolded. denotes Name minus source-input score. denotes Bonferroni-adjusted -values, correcting separately across three Task 1 and 12 Tasks 2–3 comparisons.
Figure S4 : Complementary performance metrics across models and harnesses. Heatmaps show performance across the ten protocols and nine model–harness systems used in Figure 3(b) . System and protocol orders are fixed to the molecular-transition-F1 clustering in the main figure to permit direct comparison across metrics. All color scales range from 0 to 1.
Figure S5 : Estimated API-equivalent cost and reconstruction performance. (a) Per-protocol costs and system means across ten protocols. (b) Cost versus mean performance across four F1 metrics.
Figure S6 : Model-harness effects. Mean absolute switch magnitudes (left) and paired model-minus-harness contrasts (right). Intervals are pointwise 95% protocol-bootstrap CIs, conditional on the selected executions. The state annotation gives for the primary four-metric comparison.
Figure S7 : Baseline output-discrepancy profiles. (a) The fraction of the ten protocols with at least one discrepancy. (b) The fraction of the ten systems with at least one discrepancy. Values range from 0 to 1, where 1 indicates this error appeared in 100% protocols or LLM-harness runs.
| Source type | Source | Primer content |
|---|---|---|
| Textbook | Molecular Cell Biology , Lodish et al. 2003 [ 21 ] | PCR, primer tails, restriction–ligation, reverse transcription, and DNA-end chemistry. |
| Textbook | PCR Protocols , Bartlett & Stirling 2003 [ 4 ] | End repair, ligation, terminal modifications, and phosphate-dependent sealing. |
| Textbook | Protocols for Gene Analysis , Harwood 1994 [ 17 ] | Primer tails, A-tailing, T-vector cloning, and blunt-end preparation. |
| Review | Single-cell genome sequencing: current state of the science , Gawad et al. 2016 [ 12 ] | Whole-genome amplification, amplification artifacts, and library construction. |
| Review | Scaling by shrinking: empowering single-cell ‘omics’ with microfluidic devices , Prakadan et al. 2017 [ 27 ] | Microfluidic compartmentalization, reaction splitting, and pooled processing. |
| Review | Single-cell epigenomics: techniques and emerging applications , Schwartzman & Tanay 2015 [ 31 ] | Bisulfite conversion, reduced-representation libraries, and size selection. |
Table S4 : Sources for the general molecular-method knowledge primer. The primer summarizes general molecular operations instead of protocol-specific answers.
| Validation protocol | Donor protocol | Shared feature |
|---|---|---|
| sci-ATAC-seq | s3-ATAC | Tn5-based chromatin accessibility and combinatorial indexing [ 9 , 25 ] . |
| scRRBS | 10x Chromium 3 ′ Gene Expression v4 | Shared Illumina adapter and library-completion oligos [ 15 , 1 ] . |
| SMART-seq | Drop-seq | Template-switching reverse transcription and cDNA amplification [ 29 , 23 ] . |
| SHARE-seq | SPLiT-seq | Combinatorial indexing with repeated barcode-incorporation steps [ 22 , 30 ] . |
| ddSEQ Single-Cell 3 ′ RNA-Seq Kit | sci-RNA-seq | Barcoded reverse transcription, cDNA amplification, and shared library components [ 5 , 7 ] . |
Table S5: Protocols used in the cross-protocol memory pilot. Donor protocols were selected as conceptual matches to the validation protocols based on shared molecular features.
| Checkpoint | Learned context | Observed reconstruction behavior | Validation performance |
|---|---|---|---|
| Baseline | No learned rules or cross-protocol memory. | Direct source-to-output reconstruction followed by the benchmark validator. | Task 2 F1: 0.758. Task 3 transition, state, and typed-edge F1: 0.846, 0.625, and 0.822. |
| 18 learned rules and 30 available exemplar records. | The agent constructed an evidence inventory, retrieved selected analogies, and compiled Task 2 and Task 3 from a linked work record. The predicted graphs averaged 10.0 states, 7.2 transitions, and 16.2 typed edges. | Highest Task 2 F1 (0.821), with transition, state, and typed-edge F1 of 0.843, 0.603, and 0.790. Selected checkpoint for held-out evaluation. | |
| 23 learned rules and 40 exemplar records. | Two newly learned rules may have contributed to the observed decline. A rule excluding oligo-reagent fabrication from the workflow was associated with omitted bead-construction steps in HyDrop-RNA, while a rule separating co-processed but physically unconnected molecules was associated with the separation of the ddSEQ workflow into disconnected branches. These behaviors suggest that learned rules may generalize poorly to the benchmark’s canonical representation. | Task 2 F1 decreased to 0.736; transition, state, and typed-edge F1 decreased to 0.743, 0.497, and 0.662. | |
| 28 learned rules and 58 exemplar records. | The agent represented oligo chemistry, non-templated tail addition, modified bases, and uncertain fragment fates more explicitly. Graphs again became more granular. | State F1 partially recovered to 0.539, while Task 2, transition, and typed-edge F1 were 0.745, 0.720, and 0.638, respectively. |
Table S6: Behavioral comparison of selected autonomous-learning checkpoints. Learned rules are protocol-neutral instructions accumulated from previous training protocols; exemplar records are ground-truth-derived cross-protocol examples available for retrieval. Task 2 scores are macro-averaged over four informative protocols; ddSEQ has no required recoverable oligo families and is reported separately. Task 3 scores include all five validation protocols.
| Metric | Default component weights | Scientific rationale |
|---|---|---|
| Oligo F1 | Sequence: 0.65 ; modifications: 0.15; molecular kind: 0.10; orientation: 0.05; role: 0.05. | Nucleotide sequence is central to oligo identity, while modifications and molecular annotations provide additional information. |
| Transition F1 | Operation: 0.30 ; substrates: 0.15; products: 0.20; product disposition: 0.15; linked oligos: 0.20. | Emphasizes the molecular operation and its products and linked oligos, while retaining information about substrates and product disposition. |
| State F1 | Reference sequence: 0.50 ; architecture: 0.15; ordered segments: 0.20; pairing/discontinuities: 0.15. | Prioritizes molecular sequence identity while accounting for strand architecture, segment organization, and pairing relationships. |
Table S7: Default component weights for pairwise scientific similarity. These similarities contribute to entity matching and subsequent F1 calculation. For oligo-family and molecular-state scoring, weights are renormalized over components enabled by the ground-truth support rules.
Figure S8 : Sensitivity of native-system performance to similarity weights. Macro-averaged oligo-family, molecular-transition, and molecular-state F1 scores for four native coding-agent systems. The displayed ranges characterize sensitivity to scoring assumptions, not statistical confidence intervals.
| Experiment | Transition F1 | State F1 | Typed-edge F1 |
|---|---|---|---|
| Fig. 3a: Native systems | 0.035 | 0.031 | 0.069 |
| Fig. 3b: Model-harness comparison | 0.023 | 0.020 | 0.045 |
| Fig. 4a: External knowledge | 0.029 | 0.013 | 0.054 |
| Fig. 4c: Learning validation | 0.055 | 0.025 | 0.132 |
| Fig. 4d: Held-out evaluation | 0.018 | 0.009 | 0.031 |
Table S8: Sensitivity to state and transition assignment thresholds. Maximum absolute change in protocol-macro F1 relative to the default thresholds , across nine threshold combinations and all systems or conditions within each experiment.
Figure S9 : Sensitivity of molecular-workflow scores to matching thresholds. Heatmaps show the maximum absolute changes in molecular-transition, molecular-state, and typed-edge F1 across autonomous-learning validation checkpoints relative to their respective scores under the default thresholds. State and transition assignment thresholds were independently varied over . Each cell represents the maximum across checkpoints, rather than the average change. The default combination is outlined.
Figure S10 : Robustness of knowledge augmentation and autonomous-learning results. Differences in F1 relative to baseline for combined external knowledge versus baseline; final checkpoint versus on the learning-validation panel; and with knowledge-base supplementation versus on held-out protocols.