AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents

Dongki Kim; Namkyeong Lee; Surag Nair; Carl Edwards; Xiner Li; Edward De Brouwer; Jenna Lynn Collier; Sung Ju Hwang

cs.AIOct 4, 2026

AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents

Authors: Dongki Kim, Namkyeong Lee, Surag Nair, Carl Edwards, Xiner Li, Edward De Brouwer, Jenna Lynn Collier, Sung Ju Hwang, +2 more

Organizations: Genentech · KAIST

Abstract

As agents rapidly evolve, existing benchmarks can become saturated, limiting their ability to distinguish capabilities and reveal remaining failure modes. Particularly in scientific domains, constructing and updating benchmarks requires substantial time, labor, and domain expertise, making it difficult to keep evaluation aligned with advances in agent capabilities. We address this challenge by investigating whether scientific-agent benchmarks can be automatically generated and iteratively adapted as agent capabilities evolve. We introduce AutoSciBench, a framework that represents each task as a high-level concept specifying the scientific domain, data modality, and required reasoning approach, together with a low-level recipe specifying how the question, environment, and ground-truth answer are constructed and verified. Agents attempt to solve each task, producing solver trajectories and corresponding judge feedback which AutoSciBench uses to revise the recipe or concept, closing observed shortcuts and shifting tasks toward raw-data re-examination, interpretation of intermediate results, and evidence integration. Experience distilled from completed refinement trajectories further guides new concept generation, allowing lessons from earlier task refinement to inform subsequent benchmark construction. Starting from existing benchmarks, we evaluate AutoSciBench across computational biology, materials science, and clinical imaging. Generated benchmarks reduce average solver accuracy by 22.4 and 25.5 percentage points relative to the human-curated benchmarks in computational biology and materials science, respectively, while generated tasks receive higher average quality ratings across all three domains, suggesting that scientific-agent evaluation can adapt as agent capabilities advance.

Figures & tables

Figure 1: Conceptual illustration of two key distinctions: LLM vs Agent and General agent vs Scientific agent. (A) An LLM answers using the provided context and parametric knowledge, whereas an agent acquires additional information through interaction with external data, tools, and environments, involving planning, tool execution, observation, and analysis. (B) In general agent tasks, execution errors and observable state changes often provide explicit feedback for revising a solution. In scientific analysis tasks, intermediate outputs typically provide evidence about a quantity or hypothesis of interest rather than explicit confirmation that the analysis is correct, requiring agents to re-examine raw data, validate intermediate results, and integrate evidence across analysis steps.

Figure 2: Overview of AutoSciBench. Existing benchmarks and scientific literature inform concept generation, followed by recipe planning and task implementation. Solvers attempt the generated tasks, while the judge provides feedback on the generated tasks. The resulting feedback guides revisions to the recipe or concept, producing tasks that are more verifiable and challenging. Refinement histories are summarized into experience memory to guide subsequent concept generation.

(A) Concept : High-level plan	(B) Recipe : Low-level plan	(C) Task : QA + Data
Scientific domain Crystal chemistry / perovskite structure prediction Data modality Ionic radii at specified coordination numbers Required operations • Compute tolerance factor • Compute octahedral factor • Classify the structure	Data plan Select an ABX 3 compound . Retrieve ionic radii and encode classification rules in two CSV files. Question plan Ask for the structural class . Do not provide formulas or computed values. Ground-truth plan Compute both factors , apply the rules , and verify the answer.	Question Using the ionic radii in ions.csv and the rules in classification _criteria.csv , predict the structural class of CsPbBr 3 . Data: ions.csv Site Ion CN Radius (Å) A Cs 12 1.88 B Pb 6 1.19 Answer: distorted perovskite

Figure 3: An abbreviated example of task construction from concept definition to recipe planning and task implementation. Matching colors indicate how the scientific domain, data modality, and required operations are reflected in the recipe and the resulting question, answer, and data files.

	Computational Biology			Material Science			Clinical Imaging		Avg.
	Human	Opus 4.8	GPT-5.6 Sol	Human	Opus 4.8	GPT-5.6 Sol	Opus 4.8	GPT-5.6 Sol	Opus 4.8	GPT-5.6 Sol
Claude Opus 5	94.0	72.7	88.7	97.7	90.0	92.7	82.6	85.3	81.8	88.9
Claude Opus 4.8	90.4 ∗	50.7	80.3	96.0	62.0	88.0	74.2	78.7	62.3	82.3
Claude Opus 4.7	79.5 ∗	46.0	60.7	97.0	63.3	70.7	73.5	64.7	60.9	65.4
Claude Sonnet 5	76.0 ∗	59.3	81.3	96.7	83.3	84.5	68.2	78.0	70.3	81.3
Claude Haiku 4.5	34.0 ∗	26.7	10.7	93.7	21.3	29.3	55.7	44.0	34.6	28.0

Table 1: Model performance on the benchmarks curated by humans and generated by Claude Opus 4.8 and GPT-5.6 Sol across the three scientific domains. Rows indicate solvers, and columns indicate models used during benchmark generation. ∗ denotes a leaderboard entry obtained at a reasoning effort other than high, or one whose effort is not reported. The performance on the human-curated clinical imaging benchmark is omitted due to its LLM evaluation rather than exact match.

	Computational Biology				Material Science				Clinical Imaging
	Verif.	Useful.	Fit.	Avg.	Verif.	Useful.	Fit.	Avg.	Verif.	Useful.	Fit.	Avg.
Human	2.43	3.62	3.50	3.18	4.76	1.49	1.39	2.54	2.38	2.59	1.72	2.23
[2pt/1pt] Opus 4.8	3.08	4.27	4.42	3.93	3.49	4.17	3.96	3.87	2.99	3.59	2.64	3.07
GPT-5.6 Sol	4.08	3.80	4.96	4.28	3.93	3.56	4.71	4.06	3.60	3.96	4.51	4.02

Table 2: Benchmark quality assessment on a 1-5 scale across three criteria: Verifiability (Verif.), Usefulness (Useful.), and Fitness (Fit.), higher is better. Each question is scored by two independent LLM judges, Claude Opus 4.8 and GPT-5.6 Sol, and we report the average score across both judges.

Figure 4: (Left) Difficulty of generated tasks across refinement rounds. We report the proportion of tasks in each difficulty category at each refinement round. (Right) Actual data revision by iterative refinement. Iterative refinement introduces sequencing errors that require error-aware grouping. The initial barcodes are error-free, whereas the revised barcodes contain altered bases, highlighted in red, that mimic errors encountered in real sequencing data.

	Success $\downarrow$	Hard $\uparrow$
w/o Exp.	66.7	15.0
w/ Exp.	35.0	50.0

Table 3: Solver success rates and hard task proportions.

Figure 5: Solver success (y-axis) across refinement rounds (x-axis).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Computational Biology				Materials Science				Clinical Imaging
	Verif.	Useful.	Fit.	Avg.	Verif.	Useful.	Fit.	Avg.	Verif.	Useful.	Fit.	Avg.
Human	2.72	3.56	3.45	3.24	4.58	1.13	1.46	2.39	2.01	2.21	1.45	1.89
[2pt/1pt] Opus 4.8	3.47	4.38	4.40	4.09	3.80	4.22	3.68	3.90	3.02	3.38	2.17	2.86
GPT-5.6 Sol	4.02	3.68	4.92	4.21	3.90	3.40	4.62	3.97	3.39	3.78	4.26	3.81

Appendix

Table 4: Benchmark quality assessed by Claude Opus 4.8. Avg. denotes the mean of three criteria.

	Computational Biology				Materials Science				Clinical Imaging
	Verif.	Useful.	Fit.	Avg.	Verif.	Useful.	Fit.	Avg.	Verif.	Useful.	Fit.	Avg.
Human	2.18	3.68	3.54	3.13	4.94	1.84	1.32	2.70	2.74	2.96	2.00	2.57
[2pt/1pt] Opus 4.8	2.71	4.17	4.44	3.77	3.18	4.12	4.24	3.85	2.95	3.80	3.09	3.28
GPT-5.6 Sol	4.14	3.92	5.00	4.35	3.96	3.71	4.80	4.16	3.81	4.14	4.77	4.24

Appendix

Table 5: Benchmark quality assessed by GPT-5.6 Sol. Avg. denotes the mean of the three criteria.

	Verifiability		Usefulness		Fitness
	$\rho$	$\tau_{b}$	$\rho$	$\tau_{b}$	$\rho$	$\tau_{b}$
CompBioBench	0.67	0.59	0.59	0.54	0.73	0.68
MatTools	0.57	0.53	0.90	0.81	0.90	0.83
MedCTA	0.75	0.67	0.82	0.75	0.81	0.75
All benchmarks	0.74	0.64	0.84	0.77	0.87	0.79

Appendix

Table 6: Inter-judge agreement measured by Spearman’s

\rho

and Kendall’s

\tau_{b}

between the two judges.

Panel of Figure 6 — Figure 6: Difficulty across refinement rounds for tasks generated by Claude-Opus-4.8 (Left) and GPT-5.6-Sol (Right). We report proportions by gathering tasks across all three benchmarks.

	Earlier concept	Subsequent concept
Domain	semiconductor junction electrostatics and doping-profile characterization	semiconductor optical spectroscopy / band-gap determination
Modality	a measured junction electrical characteristic (capacitance-versus-reverse-bias profile, optionally with temperature or a second characteristic) for a junction whose doping profile and built-in potential are undisclosed	a tabulated optical absorption coefficient versus photon energy spanning the absorption edge of a semiconductor thin film, with no transition-type label supplied
Operations	• identification of the junction doping-profile regime (abrupt versus graded) from the functional dependence of junction capacitance on applied reverse bias • recovery of the latent built-in potential and effective doping consistent with that regime from the measured characteristic • reconciliation of the depletion-derived estimate with the profile regime it implies into a single self-consistent value	• discrimination between a direct-allowed and an indirect-allowed interband transition by which tauc exponent linearizes the $(\mathrm{alpha}\cdot h\cdot\mathrm{nu})^{(1/n)}$ versus photon-energy edge • recovery of the optical band-gap energy as the linear-extrapolation intercept of the correctly linearized tauc edge • rejection of the incorrect exponent that yields a curved (non-linear) edge

Appendix

Table 7: Full concept descriptions for the experience-guided generation example. The left column shows the previous concept, and the right column shows a concept generated with the experience. Matching colors indicate corresponding solving behaviors: model identification (blue), quantity recovery (orange), and consistency checking (teal).

Figure 7: Representative marker distributions for the MSI tasks. Each panel shows normal and tumor read counts (upper) and their difference, tumor minus normal (lower), extracted from the BAM files. Each sample contains 2,000 reads per marker. Contamination introduces new repeat lengths, whereas the target example redistributes reads among existing lengths. Class labels describe the synthetic construction and are not supplied to the solver.

	Computational Biology	Materials Science	Clinical Imaging	Avg. Solver Accuracy
Opus 4.8	$58.47	$22.36	$24.88	66.2
GPT-5.6 Sol	$111.26	$114.11	$110.45	75.4

Appendix

Table 8: Cost per question (USD) on each domain and average benchmark difficulty in Table 1 .

	Computational Biology			Material Science			Clinical Imaging
	Impl.	Solve	Total	Impl.	Solve	Total	Impl.	Solve	Total
Opus 4.8	47.5	44.7	92.3	46.2	41.2	87.4	51.3	28.0	79.3
GPT-5.6 Sol	37.5	51.9	89.4	42.7	47.6	90.3	66.3	26.0	92.3

Appendix

Table 9: Cost proportions (%) for implementation (Impl.) and solving (Solve). Total denotes their combined proportion.

Explore similar work

Jun 10, 2026cs.AI

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.

Tianyu Liu, Allen Xin Wang, Antonia Panescu +30

Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7

Scientific Discovery Agentic Benchmarks Artificial Intelligence Agents Exploration

Ask PDF

Sep 27, 2026cs.AI

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially reducing routine scientific workload while allowing scientists to focus on interpretation and discovery. Existing benchmarks often only assess analytical task completion or hypothesis generation separately rather than testing whether reliable evidence supports valid and novel claims. We introduce DISCERN (Data Integrity and Scientific Capability: Evidence, Reasoning, and Novelty), a controlled benchmark on real, publicly available datasets that evaluates three key levels of an automated research workflow. The first two levels test data integrity and analysis verification under confounds and tool traps, while the third tests hypothesis generation and revision under adversarial review, including counterfactual cases in which evidence consistent with real data and documented scientific phenomena conflicts with established expectations, motivating alternative explanations and testable hypotheses. Across 203 tasks, eight life-science tracks, and eight models, DISCERN shows that strong aggregate performance can mask level-specific weaknesses. Agents earn perfect scores in only 60.8% of Level 1, 34.2% of Level 2, and 0.6% of Level 3 evaluations, with penalties attributed to rejection of sound data, failure to carry recognized limitations into conclusions, and wide variation in hypothesis production. Cross-track rankings by token and code use are substantially more stable than rankings by evidence judgment, suggesting greater consistency in computational effort than in evidence-based reasoning. These profiles identify opportunities for supervised scientific assistance, but current agents do not yet demonstrate reliable autonomous analysis or discovery. Code and data: https://huggingface.co/datasets/discern-bench-anon/discern-benchmark

Nan Huang, Mario Tapia-Pacheco, Kun Zhou +4

University of California San Diego · Independent Researcher

Scientific Agents Data Science Agents Discovery Artificial Intelligence Agents Guidance

Ask PDF

Jul 23, 2026cs.AI

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration

Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources. However, existing benchmarks primarily emphasize general-domain retrieval or static scientific question answering, and therefore fail to assess key capabilities required in realistic scientific research workflows. We introduce SciExplore, a benchmark designed to evaluate scientific information-seeking and reasoning capabilities of LLMs and agents. SciExplore comprises four task types covering 103 expert-curated tasks across more than ten scientific disciplines: scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis, which probe progressively higher-level abilities from entity-level reasoning and document-level identification to evidence-level grounding and domain-level synthesis. We evaluate over ten state-of-the-art LLMs and autonomous agents on SciExplore, revealing substantial performance gaps with performance degrading sharply as task complexity increases and extremely low accuracy on the most challenging structured synthesis tasks. These results highlight significant limitations of current models and agents in realistic scientific information-seeking scenarios.

Yinhao Tang, Youqing Fang, Yanan Sun +6

University of Science and Technology of China · Shanghai AI Laboratory

Scientific Discovery Scientific Agents Scientific Workflows Autonomous Agents Agentic Workflow Design

Ask PDF