cs.AIOct 13, 2026

Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery

Authors: Yibo Wen, Han Liu

Organizations: Northwestern University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

AI agents for autonomous scientific discovery must decide which experiments to run under a limited budget. Most benchmarks for science agents provide fixed datasets, while others that support active experimentation largely rely on simulation. We introduce Autonomous Science Bench, an initial release comprising six budgeted design-build-test-learn campaigns grounded in published experiments across biology, chemistry, and materials science. Agents select batches of designs, receive measurements from a lab service, and use the results to guide subsequent experiments before delivering a predictive model. Success requires passing two calibrated gates: a discovery gate, which evaluates the designs the agent measured, and a learning gate, which evaluates its model's ranking of held-out designs. In a pilot evaluation of eight agent-model configurations and 144 trials, the best configuration succeeds in 67% of trials, while the hardest campaign is solved in only three of 24 attempts. The two gates reveal different weaknesses: some agents build accurate rankings but find too few promising designs, while others find promising designs but fail to predict which unseen designs will perform well. These differences are obscured by overall success rates alone. We release version 0.1 of the campaigns, lab runtime, and public leaderboard at https://yibow.me/autonomous-science-bench.

Figures & tables

0=1
 Terminal-BenchAutonomous Science Bench (ours)
Unit of evaluationA task, from instruction to final container stateA campaign of RR design–build–test–learn rounds
Data accessInputs provided in the containerBought batch by batch via a lab API
InteractionThe agent explores and edits its own containerClosed loop with a lab that enforces round order, batch size, and no repeats
GradingOutcome tests on the final artifactA discovery gate on what was measured and a learning gate on the delivered model
Passing requiresExecuting an expert workflowExperimental selection and held-out ranking
Table 1: Terminal-Bench tasks versus our campaigns. Our campaigns run in the same Harbor harness but test a different capability, deciding which experiments to run under a budget and learning from their results.
0=1
 BiologyChemistryMaterials
1=1
 NucleaseBiosensorPropyleneSuzukiOERCoercivity
SourceThomas et al. ( 2025 ) 3=Wang et al. ( 2025a ) 3=Kim et al. ( 2025 ) 3=Perera et al. ( 2018 ) 3=Rohr et al. ( 2020 ) 3=Wang et al. ( 2022 ) 3=
Backendreplayreplaytwinreplayreplayreplay
Rounds ×\times batch4×964\times 963×103\times 103×203\times 204×484\times 484×484\times 484×164\times 16
Table 2: The six campaigns in version 0.1, two each in biology, chemistry, and materials science. Qualifying designs are those that meet the campaign’s hit or standout criteria. Expert time estimates how long a domain expert would take to complete each campaign.
Figure 2: Trial topology. The agent reaches the lab only through its API, which the asb client wraps. A separate verifier with no network grades only the lab’s ledger and the delivered model.
0=1
  StrategyReference
1=1
CampaignSingle-gate strategyDiscoveryLearningDiscoveryLearning
BiosensorCatalog tally, which learns nothing from its measurements1.100.701.141.16
SuzukiOracle selection of 192 wells with the baseline predictor2.280.861.191.18
PropyleneThe source study’s Bayesian-optimization protocol0.571.071.141.20
Table 3: One gate is not enough. Calibration strategies that pass exactly one gate, with the reference on the same campaign. Scores are from each campaign’s released split, divided by the gate so that 1.00 is the threshold, and bold values pass.
Figure 3: Configurations fail on different gates, and higher resolution costs steeply more. (a) Trial outcomes, with three trials per campaign. (b) Resolution rate versus total model cost. The Pareto frontier connects configurations for which no other achieves an equal or higher resolution rate at lower cost.
Figure 4: Discovery and learning scores are related but separable. Each marker is one completed trial, placed by its two gate scores divided by their thresholds. Color gives the outcome and shape the agent, and the shaded corner passes both gates.
Figure 5: How agents spend their budget. The fraction of each batch meeting the campaign’s hit or standout criterion, by round, on a per-campaign scale. Solid lines show the mean of resolved and unresolved trials, with the reference on the released split dashed and random selection dotted.
0=1
ModelNucleaseBiosensorPropyleneSuzukiOERCoercivityRate (%)Cost ($)Time (h)
Opus 5.53/30/32/32/33/32/367172.9019.0
GPT-5.6 Terra0/32/31/32/33/30/34451.575.0
GPT-5.6 Luna1/32/31/31/30/30/3286.916.3
Sonnet 5.50/30/31/30/31/30/3117.852.7
GLM-5.3 Flash0/30/30/30/31/31/31125.6928.4
Table 6: Results by campaign. Each cell counts one configuration’s resolved trials on one campaign, with each model in its agent from Table 4 . The rate is the share of its 18 trials resolved, and cost and agent time are totals over those trials.
0=1
CampaignModelResolvedTrialsDisc.Learn.Cost ($)Minutes
NucleaseOpus 5.53/3✓ ✓ ✓1.04 1.06 1.041.12 1.11 1.0831.46221
 GPT-5.6 Terra0/3✗ D D0.75 0.85 0.940.97 1.01 1.1115.4879
 GPT-5.6 Luna1/3✗ ✓ ✗0.92 1.08 0.520.98 1.00 0.931.6394
 Sonnet 5.50/3D ✗ D0.81 0.73 0.961.22 0.99 1.112.8145
 GLM-5.3 Flash0/3✗ ✗ ✗0.67 0.69 0.480.39 0.83 0.673.36259
Table 7: Every trial. The Trials column marks each trial’s outcome in run order, as resolved (✓), missed discovery only (D), missed learning only (L), or missed both ( ✗ ). Disc. and Learn. give the same trials’ gate scores divided by their thresholds, in bold where they pass. Cost and minutes are totals over the three trials.
0=1
MethodPathClient commandPurpose
GET/health–Check that the lab is running
GET/v1/campaignasb campaignThe public campaign card, which does not name the backend
GET/v1/statusasb statusProgress and remaining budget
GET/v1/catalog?stage=asb catalogDesigns that can be ordered in a round
GET/v1/experimentsasb jobsList past jobs
Table 8: Lab API v1 endpoints. The full specification is in the repository. Each job has its own path, and the catalog request names the round, which the API calls a stage.
0=1
TierMeasurements come fromVerifier’s reference valuesStatus
ReplayPublished measurements, stored only inside the lab’s containerThe same recorded valuesv0.1, five campaigns
TwinA hidden model of the experiment, used in place of the labThe model’s predictions, stored in a tablev0.1, one campaign
LiveA physical labDesigns measured by the lab and reviewed by its scientistsInterface defined
Table 9: Backend tiers. The tiers differ only in where measurements come from and in what the verifier checks them against.
Figure 6: Every calibration strategy. Each row shows one strategy’s median scores divided by the gates, with a bar over the middle half of the splits. Diamonds mark the reference on the released split, which sets the gates.
0=1
 ReferenceStrongest baseline 
1=1
CampaignPassesScoresStrategyPassesScoresBest zero-measurement predictor
Nuclease26/301.16 / 1.20Random selection0/300.35 / 0.83Seed-only ridge (0.82)
Biosensor29/301.14 / 1.15Catalog tally2/301.05 / 0.84Alanine scan prior (0.04)
Propylene27/301.14 / 1.21BO (source protocol)2/300.71 / 1.00Seed-only model (0.68)
Table 10: Calibration summary. Passes count the splits, or catalog orders for oxygen evolution and coercivity, on which a strategy passes every gate. Scores are normalized discovery / learning medians. The strongest baseline passes most often among the strategies that rule 2 requires to fail, and the last column gives the best predictor that orders nothing.
More in this topic

Autonomous Scientific Discovery

Autonomous scientific discovery develops AI systems that generate, test, and refine scientific hypotheses using data, computation, and experiments.

See all 128 papers

Explore similar work

CardsList

  1. Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

    Jun 10, 2026Tianyu Liu, Allen Xin Wang, Antonia Panescu +30AI for ScienceScientific Agent Benchmarks

    Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7

  2. AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents

    Oct 4, 2026Dongki Kim, Namkyeong Lee, Surag Nair +7AI for ScienceAI Agent Evaluation

    Genentech · KAIST

  3. EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights

    Sep 30, 2026Jiayi Geng, Zhengxuan Wu, Kevin S. Chen +12Scientific DiscoveryLong-Horizon Agent Evaluation

    Carnegie Mellon University, Language Technologies Institute · Stanford University, Department of Computer Science · Yale University, Quantitative Biology Institute +8