cs.AIOct 15, 2026
SaveOmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics
Organizations: Department of Computational Medicine and Bioinformatics, University of Michigan · AfterQuery
Abstract
Computational biologists increasingly delegate substantial analyses to AI agents in long-horizon, multi-turn sessions, yet most benchmarks of agents' bioinformatics capabilities pose a single short task with a single-shot prompt. We present OmicsBench, a long-horizon, multi-turn bioinformatics benchmark for AI agents. Each of its 97 tasks links dependent subtasks into a directed acyclic graph, for 662 underlying subtasks in all. Each subtask is posed as one user turn and graded deterministically against an answer key, with optional in-context feedback on incorrect answers. Across 12 models on three harnesses, the slowest agent averages 3.6 hours per task, and no combination comes near saturating the benchmark, with mean scores of 0.43 to 0.57 out of a possible 1 without in-context feedback and 0.54 to 0.67 with it. Multi-turn evaluation also uncovers behavior that single-shot prompts cannot, as agents score lower on subtasks that depend on earlier subtasks, and in-context feedback helps agents recover some of that loss. Because deterministic scoring gives no qualitative account of how an agent failed, we also introduce an unsupervised failure mode discovery pipeline that describes, embeds, and clusters agents' failed attempts to surface emergent failure modes. Applied to OmicsBench, this pipeline exposes qualitative shortcomings in how agents carry out bioinformatics analyses, ranging from agents applying their own statistical filters in place of the specified ones to agents writing whole answers from memory.
Figures & tables
Figure 1: Example OmicsBench task. A stylized view of the example task in Appendix D ( Table D1 , Table D2 ). ( A ) A task is a DAG of analysis subtasks on a dataset. ( B ) Single-pass mode, with each subtask one user turn and no feedback on answers. ( C ) Gated mode, where an answer scoring below the gate threshold ( ) is returned for a retry in the same session, until an answer reaches or an attempt limit.
Figure 2: Construction and evaluation of OmicsBench. ( A ) Benchmark construction. ( B ) Evaluation and failure analysis pipeline.
Figure 3: OmicsBench task statistics. ( A ) Data modality, with a task counted once per modality it includes. ( B ) Scientific domain, one per task. ( C ) Subtask type, one per subtask. ( D ) Organism, with a task counted once per organism. ( E ) Number of subtasks per task. ( F ) Length of the longest dependency chain in each task’s DAG. ( G ) Number of files in each task’s dataset. ( H ) Total size of each task’s dataset in MB, in log-spaced bins.
Figure 4: Performance of the 12 models and three harnesses. ( A ) Mean score per combination in each mode, with the gated gain in parentheses above each pair. Error bars are 95% confidence intervals over the 97 tasks. ( B, C ) Task difficulty, a task’s mean score over all combinations, in each mode.
Figure 5: Cost and efficiency. ( A ) Cost, ( B ) wall-clock time, ( C ) tool calls, and ( D ) tokens per task against mean score, in single-pass mode (top) and gated mode (bottom). Cost is available only on OpenCode, and not for Grok 4.6 or Qwen3.8-Max. Error bars are 95% confidence intervals of mean score.
Figure 6: Subtask scores across a session and across gated retries. ( A ) Change in mean subtask score from the first turn, in single-pass mode, through turn 7. ( B ) The same by dependency chain length, through length 3, as too few subtasks have longer chains for a stable average. ( C ) Mean best score within 1 to 3 gated attempts. ( D ) Share of subtasks reaching the gate ( ) on the first attempt and within three attempts.
Figure 7: Failure modes, by model/harness combination. ( A ) Each a priori mode’s share of a combination’s assigned modes. ( B ) Each emergent mode’s share of a combination’s clustered attempts. ( C ) UMAP of failure descriptions, one dot per attempt, colored by mode, gray if unassigned.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Per-task scores by model/harness combination. ( A ) Distribution of the 97 task scores per combination in each mode. ( B ) One row per task, ranked by median single-pass score across combinations, hardest at the bottom. The colored band between dots is tinted where gating raises that task’s score for that combination and left white where it lowers it.
Figure A2: Run-to-run consistency. Each violin covers one model/harness combination in one mode and shows, over the 97 tasks, the distribution of the standard deviation of a task’s score across its three runs. The inner bar marks the interquartile range and the white dot marks the median.
Figure A3: Tokens per task by category. ( A ) Single-pass mode. ( B ) Gated mode. Each bar is a combination’s mean tokens per task in one category, on a logarithmic axis. Error bars are 95% confidence intervals.
Figure A4: Domain packages per task. Each box covers one model/harness combination and shows, over the 97 tasks, the number of distinct domain packages the combination invoked on a task across its six runs, three in each mode.
Figure A5: Domain packages invoked by each model/harness combination. Each row is one of the 30 most used domain packages, ordered by the number of tasks on which it was invoked, summed over all combinations (most used at the top). Marker area is the number of the 97 tasks on which that combination invoked the package across its six runs.
| Model | Cost | Time | Tool calls | Tokens | ||||
|---|---|---|---|---|---|---|---|---|
| Single-pass | Gated | Single-pass | Gated | Single-pass | Gated | Single-pass | Gated | |
| Opus 5 | ||||||||
| GPT-5.6 Sol | ||||||||
| Gemini 3.1 Pro | ||||||||
| GLM-5.2 | ||||||||
| Grok 4.3 |
Table A1: Pareto-optimality of models on OpenCode. A check marks a model that is Pareto-optimal on a measure in one mode, meaning that no other model on OpenCode, the one harness on which all 12 models ran, scores higher at the same or lower value of that measure. N/A marks missing data.
| Symbol | Meaning |
|---|---|
| A task, composed of a task prompt , biological dataset , and DAG | |
| The task prompt of a task | |
| The biological dataset of a task | |
| The task DAG, with nodes and edges | |
| A node of the task DAG, one subtask | |
| An edge of the task DAG, meaning that subtask depends on the answer to subtask |
Table B1: Notation. Symbols used in the OmicsBench formulation.
| Parameter | Symbol | Value |
|---|---|---|
| Gate threshold in gated mode | 0.7 | |
| Attempt limit per subtask in gated mode | 3 | |
| Time limit per turn (s) | 1800 | |
| Runs per task in each mode | 3 |
Table B2: Benchmark parameters. OmicsBench settings and their values in this study.
| File | Description |
|---|---|
| snrna_expression.h5ad | Single-nucleus RNA-seq of adult human kidney cortex: log-normalized expression matrix (cells genes) with per-cell cell-type/cell-state annotations in obs ( author_cell_type includes PT for the remaining proximal tubule and PT_VCAM1 for the VCAM1+ proximal tubule subpopulation, plus other nephron cell types). Gene symbols are in var[’feature_name’] . |
| snatac_gene_activity.h5ad | Single-nucleus ATAC-seq of adult human kidney cortex represented as a per-cell gene-activity matrix (log-scaled accessibility aggregated over each gene’s body and promoter; cells genes) with per-cell cell-type/cell-state annotations in obs ( author_cell_type includes PCT and PST for proximal tubule segments and PT_VCAM1 for the VCAM1+ proximal tubule subpopulation, plus other nephron cell types). Gene symbols are in var[’feature_name’] . |
| snatac_fragments.tsv.gz | Raw single-nucleus ATAC-seq fragments for the same kidney cortex nuclei, block-compressed TSV keyed by genomic coordinate and cell barcode. Needed for peak- or motif-level analyses such as transcription-factor motif activity; the gene-activity matrix suffices for gene-level accessibility comparisons. |
| snatac_fragments.tsv.gz.tbi | Tabix index for the single-nucleus ATAC-seq fragments file, enabling coordinate-range queries. |
Table D1: Files in the example task’s dataset.
| Subtask | Depends on | Subtask prompt | |
|---|---|---|---|
| 1 | pt_vcam1_signature | Root subtask, no upstream subtasks | Using the single-nucleus RNA-seq expression matrix, contrast the VCAM1+ proximal tubule subpopulation (PT_VCAM1) against the remaining proximal tubule (PT) and derive the differential transcriptional signature of this cell state, testing genes detected in at least 20% of cells in either group with a log fold-change threshold of 0.25 and Bonferroni-adjusted . Report the 15 most strongly up-regulated and, separately, the 15 most strongly down-regulated genes in PT_VCAM1, ranked by average log fold-change as computed by Seurat’s FindMarkers (avg_logFC: the natural log of one plus the mean of expm1(expression) in PT_VCAM1, minus the same quantity in PT, on the provided log-normalized values). Provide two lists of gene symbols keyed as ‘upregulated’ and ‘downregulated’. |
| 2 | pt_vcam1_transition_drivers | pt_vcam1_signature | Order the proximal tubule cells along a transition from the differentiated proximal tubule (PT) to the VCAM1+ state (PT_VCAM1). The source study reports four hallmark genes whose expression changes define this transition: VCAM1, TPM1, SLC5A12, and SLC4A4. Determine the direction in which each of them changes along the transition and report them grouped by direction: genes induced (rising toward PT_VCAM1) under group label ‘induced’, and genes repressed (falling toward PT_VCAM1) under group label ‘repressed’. Each group holds gene symbols. |
| 3 | pt_vcam1_regulators | pt_vcam1_signature | Nominate the transcription-factor programs that switch as the proximal tubule adopts the VCAM1+ state by computing transcription-factor motif activity from the single-nucleus ATAC fragments (for example chromVAR deviations over JASPAR vertebrate motifs), comparing PT_VCAM1 nuclei with the differentiated proximal tubule nuclei (the PCT and PST segments). Identify the transcription factor whose motif activity is most clearly lost in PT_VCAM1 and the two transcription factors whose motif activity is most strongly gained. You may corroborate the direction using the PT_VCAM1 signature you derived, transcription-factor gene expression, and gene activity. Where motif activity alone cannot separate paralogs that share a near-identical motif, report the paralog whose own expression and gene activity move in the same direction as the motif activity. Report each regulator by its own gene symbol, grouped as ‘repressed_regulators’ (lost in PT_VCAM1) and ‘induced_regulators’ (gained in PT_VCAM1). |
| pt_vcam1_programs | pt_vcam1_signature | Run a functional-enrichment analysis on the full PT_VCAM1 signature, up- and down-regulated genes together, and select the biological programs that are enriched in this subpopulation relative to the remaining proximal tubule. Choose all applicable programs from the provided options. | |
| 4 | pt_vcam1_accessibility_concordance | pt_vcam1_transition_drivers | Using the single-nucleus ATAC gene-activity matrix, test whether the transition-driver genes identified for the PT-to-PT_VCAM1 transition show concordant changes in chromatin accessibility. Compare gene activity in PT_VCAM1 cells against the differentiated proximal tubule cells (the PCT and PST segments) and report which of those driver genes show increased accessibility in PT_VCAM1 under group label ‘increased_accessibility’ and which show decreased accessibility under group label ‘decreased_accessibility’. Each group holds gene symbols. |
Table D2 : Subtask prompts of the example task. Numbers on the left refer to the boxes in the flowchart of Figure 1 A , which omits pt_vcam1_programs .
| Grader | Description |
|---|---|
| absolute_numeric_tolerance | Compares one numeric value using an absolute error band. |
| abundance_table | Scores an unordered table of entity names paired with numeric abundances or quantities. |
| amino_acid_set | Scores an unordered set of amino acids using one-letter, three-letter, and full-name normalization. |
| amr_gene_set | Scores an unordered set of antimicrobial resistance genes using CARD ARO synonym resolution. |
| case_insensitive_string | Compares one scalar string after lowercasing both sides. |
| case_insensitive_string_list | Scores an unordered set of literal strings while ignoring case. |
Table F1 : OmicsBench grader catalog.
| Database | Description | Reference |
|---|---|---|
| NCBI Gene | Matches between different gene nomenclature systems. | Brown et al. 2015 |
| NCBI Taxonomy | Matches between different organism naming systems. | Schoch et al. 2020 |
| HMDB | Matches between different metabolite naming systems. | Wishart et al. 2022 |
| miRBase | Matches between different miRNA naming systems. | Kozomara et al. 2019 |
| CARD | Matches between different antimicrobial resistance gene naming systems. | Alcock et al. 2023 |
Table G1: Identifier databases that back OmicsBench’s synonym-aware graders.
| Failure mode | Definition | |
|---|---|---|
| Environment setup failure | The agent could not get required tools, packages, references, or paths into a usable state, and the unresolved setup problem prevented or materially blocked the intended analysis. | |
| Input misinterpretation | The agent misread the staged inputs, confused sample identities, or applied an analysis intended for a different data type. Applies only when the data themselves were misread; a later analysis choice that diverged is Method misconfigured. | |
| Wrong method substitution | The agent used a method materially different from the one the reference analysis requires, such as a different tool family, resource, or statistical approach, including hand-built stand-ins (keyword matching, reimplemented pipelines) for a standard resource it did not use. | |
| Method misconfigured | The agent used a suitable method but chose parameters, contrasts, references, normalizations, or filters that materially diverge from the reference analysis. Not added when a more specific mode captures the same choice. | |
| Incomplete data | The agent analyzed fewer samples, conditions, or features than the subtask specified, so the analysis was incomplete even where the reduced analysis ran. | |
| Output format violation | The answer did not match the subtask’s answer contract: wrong filename or location, identifier namespace, naming, units, granularity, key names, or list length. |
Table H1 : A priori failure modes used to annotate failed attempts.
| Failure mode | Definition | |
|---|---|---|
| Too much submitted | ||
| 1 | Same number of top genes taken from every group, so lists were mostly extras | The agent took the same fixed number of top-ranked genes from every group, even though the subtask asked for the few genes that set each group apart, so most of what it submitted was extras, and the score fell. |
| 2 | Every gene passing its thresholds submitted instead of a short list | The agent submitted every gene that passed its thresholds instead of selecting the short list the subtask asked for, so the submission was mostly extras. |
| 3 | Ranked by test score only, so ubiquitous genes displaced the markers | The agent ranked genes by the test score alone with no expression filter, which let genes expressed everywhere take the top places, so the specific markers the subtask asked for were pushed off the list. |
| 4 | Genes picked by keyword match instead of a statistical test | The agent chose genes by matching keywords in their names or annotations rather than by the statistical test the subtask required, which pulled in unrelated genes that matched the words and left out significant genes that did not. |
| Too little submitted | ||
Table I1 : Emergent failure modes recovered by unsupervised discovery. The modes are grouped into five families.
| Model | Harnesses |
| Opus 5 | OpenCode, Claude Code |
| GPT-5.6 Sol | OpenCode, Codex |
| Gemini 3.1 Pro | OpenCode |
| GLM-5.2 | OpenCode |
| Grok 4.3 | OpenCode |
| Grok 4.6 | OpenCode |
Table J1: Models and harnesses evaluated in this study. Reasoning effort was set to high for every model except MiniMax-M3 and Qwen3.7-Max, for which OpenRouter exposes no effort setting. The harness versions were OpenCode 1.18.11, Claude Code 2.1.222, and Codex 0.142.5.