SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

Tianyun Zhong; Wei Wang; Jialin Li; Jianhong Tu; Liyao Li; Chuqiao Kuang; Tianshi Zheng; Songchi Zhou

cs.CLOct 23, 2026

SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

Authors: Tianyun Zhong, Wei Wang, Jialin Li, Jianhong Tu, Liyao Li, Chuqiao Kuang, Tianshi Zheng, Songchi Zhou,

Organizations: University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group · Qwen Team, Alibaba Group ·

Abstract

Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam.

Figures & tables

Figure 1: Sci-Agent-Exam versus existing benchmarks: workflow orientied task types, full deliverables (answer, code, report) per task, hybrid evaluation, with analysis for efficiency and stability .

Benchmark	Task Setting	Task-Type Coverage							Required Deliverables			Scale
		1	2	3	4	5	6	7	Answer	Exec.	Report	# Task	# Domain
SciBench	Scientific QA	○	○	○	○	○	○	●	●	○	○	869	3
DiscoveryBench	Scientific discovery	○	●	◐	◐	◐	○	●	●	○	○	264	6
ScienceAgentBench	Scientific analysis	○	○	●	●	◐	◐	◐	●	●	○	102	4
CORE-Bench	Computational reproduction	○	○	●	◐	◐	◐	◐	●	○	○	270	3
PaperBench	Paper replication	◐	○	◐	●	◐	◐	◐	○	●	○	20	1

Table 1: Comparison with representative scientific-agent benchmarks. Task-Type Coverage maps seven task types: \scriptsize1⃝ Literature Synthesis, \scriptsize2⃝ Hypothesis Validation, \scriptsize3⃝ Tool Orchestration, \scriptsize4⃝ Code Generation, \scriptsize5⃝ End-to-End Analysis, \scriptsize6⃝ Error Diagnosis, and \scriptsize7⃝ Result Interpretation. For task-type coverage, ● = explicitly targeted, ◐ = involved but not distinct, and ○ = not covered. For Required Deliverables , ● = required in every task, ◐ = required in some tasks, and ○ = not required. Answer : short answer/result; Exec. : code/artifact; Report : written report. Domain counts follow each benchmark’s categorization.

Figure 2: Overview of Sci-Agent-Exam . We organize tasks with a workflow-oriented taxonomy, construct executable task packages through a quality-controlled data pipeline, and evaluate task outputs using a hybrid protocol combining deterministic programmatic checks with LLM rubric judgments. The bottom panel illustrates an example task with its inputs, required deliverables, and evaluation criteria.

Domain	\scriptsize1⃝	\scriptsize2⃝	\scriptsize3⃝	\scriptsize4⃝	\scriptsize5⃝	\scriptsize6⃝	\scriptsize7⃝	Total
Bioinformatics	0	1	2	1	5	2	0	11
Biology	1	2	4	2	4	1	1	15
Chemistry	3	1	3	2	1	1	0	11
Computer science	0	3	1	3	2	1	0	10
Data science	1	2	2	1	2	1	1	10
Engineering	2	0	2	1	1	0	2	8

Table 2: Distribution of 120 tasks across 11 scientific domains and seven SWTT task types. Column numbers follow the definitions in the text and Table 1 .

Rank	Model	Programmatic	LLM Judge	Final	[email protected]
1	GPT-6 Astra	$\mathbf{77.7}\,{\scriptsize\pm 2.5}$	$\mathbf{85.3}\,{\scriptsize\pm 1.4}$	$\mathbf{80.9}\,{\scriptsize\pm 1.7}$	$\mathbf{26.0}$
2	Claude Fable 5	$\underline{62.7}\,{\scriptsize\pm 3.1}$	$\underline{74.4}\,{\scriptsize\pm 2.0}$	$\underline{67.3}\,{\scriptsize\pm 2.0}$	$9.7$
3	Claude Opus 5	$59.2\,{\scriptsize\pm 2.8}$	$71.8\,{\scriptsize\pm 2.4}$	$64.2\,{\scriptsize\pm 2.1}$	$\underline{10.5}$
4	Qwen3.8 Max	$54.7\,{\scriptsize\pm 2.6}$	$74.2\,{\scriptsize\pm 1.6}$	$62.7\,{\scriptsize\pm 1.7}$	$7.4$
5	Kimi K3	$57.6\,{\scriptsize\pm 2.6}$	$71.2\,{\scriptsize\pm 1.6}$	$62.6\,{\scriptsize\pm 1.7}$	$6.1$
6	GPT-5.6 Sol	$58.8\,{\scriptsize\pm 2.5}$	$62.1\,{\scriptsize\pm 1.8}$	$59.6\,{\scriptsize\pm 1.8}$	$6.9$

Table 3: Model performance on Sci-Agent-Exam (120 tasks; Claude Code harness). Programmatic and LLM Judge scores assess executable and textual artifacts, respectively. Scores are mean

\pm

standard error across tasks; [email protected] is the fraction of runs with a final score of at least 0.95. Bold and underlined values mark the best and second-best results.

Figure 3: Agents generally report better than they execute. Task-level outcomes frequently concentrate in the top-left (TL) quadrant, indicating strong report quality but weak executable correctness. TL is the fraction of cases with LLM-judge $\geq 0.5$ and Programmatic $<0.5$ ; TR is the fraction for which both scores are $\geq 0.5$ .

Figure 4: Model performance across scientific domains. Rows are ordered by overall benchmark performance.

Panel of Figure 5 — Figure 5: Trajectory analysis of scientific agents. (a) Four examples illustrate four behaviors used in our analysis: early completion (reaching near-peak quality within the first third of snapshots), dip (a previously Pass becomes Fail), recover (a Failed rubric Passes again), and regret (a Pass rubric earlier but Fails at the final snapshot). (b) Behavior rates across models four programmatic execution and five LLM-judged dimensions of scientific report quality: Source Grounding, Scientific Correctness, Completeness, Execution & Validation, and Communication.

Figure 6: Higher-priced models can be faster and cheaper per completed run. Overall score versus mean cost per run; the upper-left indicates higher performance at lower cost.

Form	The decoy $\rightarrow$ what credit requires	Representative task
Decoy artifact	A precomputed rollup or dashboard reads like the answer but was produced by a leaky or partial procedure $\rightarrow$ recompute the decision independently.	data-science-hypothesis-validation-01
Loudest signal	The most salient statistic (lowest information criterion, lowest cost) nominates a candidate that fails a downstream physical or feasibility gate $\rightarrow$ select on the gated objective.	engineering-tool-orchestration-02
Distractor feature	Balanced noise covariates masquerade as signal $\rightarrow$ keep only the load-bearing variables.	bioinformatics-end-to-end-analysis-05
Misattributed cause	A dashboard blames the wrong component and discourages the real fix $\rightarrow$ diagnose and repair the true failure path.	cs-expansion-error-diagnosis-02

Table 4: Recurring trap forms in Sci-Agent-Exam , with a representative task for each. In every form the decoy is locally plausible; credit requires the coupled scientific workflow.

Task type	$\bm{w_{i}^{p}}$	$\bm{w_{i}^{j}}$
Code Generation	0.70	0.30
Error Diagnosis	0.65	0.35
Tool Orchestration	0.55	0.45
End-to-End Analysis	0.50	0.50
Hypothesis Validation	0.50	0.50
Result Interpretation	0.45	0.55

Table 5: Programmatic and LLM-judge weights by task type.

Model	Harness	$n$	Prog.	Judge	Final
gpt-6-astra	claude-code	42	72.3	85.8	78.9
gpt-6-astra	codex	45	74.8	83.8	79.5
qwen-3.8-max	claude-code	49	65.6	75.9	70.0
	codex	46	67.4	76.6	71.3
	qoder	40	62.9	73.5	67.0
kimi-k3	claude-code	47	65.8	75.3	69.9

Table 6: Harness sensitivity on the 50-task set: the same model scored under general-purpose harnesses (claude-code at max effort, codex) and, where available, its vendor-native harness (kimi-cli for kimi-k3, qoder for qwen-3.8-max).

Model	Effort	Programmatic	LLM Judge	Final	[email protected]
gpt-6-astra ( $n{=}42$ )	high	74.5	83.8	78.9	5%
gpt-6-astra ( $n{=}42$ )	max	72.3	85.8	78.9	21%
qwen-3.8-max ( $n{=}49$ )	high	59.1	73.6	65.4	0%
qwen-3.8-max ( $n{=}49$ )	max	65.6	75.9	70.0	4%
kimi-k3 ( $n{=}47$ )	high	61.2	70.3	64.7	0%
kimi-k3 ( $n{=}47$ )	max	65.8	75.3	69.9	6%

Table 7: Effect of max vs. high thinking effort on the claude-code harness (paired per task on the 50-task set;

n

= paired tasks). Programmatic, LLM Judge and Final are on a

0

–

100

scale; Final is the per-task mean reward averaged over tasks; [email protected] is the fraction of tasks with mean final reward

\geq 95

. Rows ordered by max-effort Final.

Model	Scores	Avg. Cost/Run ($)	Runtime (min)	Turns
GPT-6 Astra	80.9	20.79	22.1	56.8
Claude Fable 5	67.3	7.67	25.3	30.2
Claude Opus 5	64.2	12.49	30.6	43.9
Qwen3.8 Max	62.7	22.42	50.9	59.6
Kimi K3	62.6	20.96	47.2	52.6
GPT-5.6 Sol	59.6	13.10	22.0	54.3

Table 8: Performance, average cost, runtime, and interaction length on the Sci-Agent-Exam peer subset.

Figure 8: Performance and resource use across ten agents on Sci-Agent-Exam . Overall score, mean cost, runtime, and turns per run, ordered by overall score. The best value for each metric is highlighted.

Figure 9: Model performance across the seven task types in Sci-Agent-Exam . Each cell reports the mean final task score for a model–task-type pair, with the number of tasks shown in parentheses. Rows are ordered by overall benchmark performance, and the rightmost column reports each model’s overall final score.

Figure 10: Rubric-based performance across five dimensions of scientific quality. Each cell reports the fraction of rubric criteria passed within a dimension for each model. The five dimensions capture Source Grounding, Scientific Correctness, Completeness, Execution & Validation, and Communication. The rightmost column reports the aggregate LLM-judge score.

Panel of Figure 11 — Figure 11: Trajectory-level analysis of agent behaviors. (a) illustrates the diverse score dynamics (e.g., oscillations, late-stage jumps) across different models. (b) highlights the performance variations when multiple models tackle the same scientific task.

Figure 12: Trajectory behavior across rubric dimensions and the programmatic channel, for all ten evaluated agents. Rows are ordered by overall benchmark score. This full-population view extends the six-model main-text Figure 5 to the four agents with smaller judged samples (Claude Fable 5, Claude Opus 4.8, Grok-4.6, and DeepSeek-V4-Pro).

Task type	$n$	Early	Dip	Recover	Regret
End-to-end analysis	134	36.2%	12.2%	10.3%	3.7%
Literature synthesis	71	29.9%	6.6%	4.2%	3.3%
Hypothesis validation	117	20.9%	6.1%	4.5%	2.9%
Tool orchestration	112	18.7%	8.3%	6.8%	3.3%
Result interpretation	47	13.3%	4.4%	2.1%	2.7%
Code generation	112	8.6%	3.7%	2.1%	1.9%

Table 9: Trajectory analysis by scientific task types.

Panel of Table 13 — Table 13: Contents of an example task package and the role of each component.

Role	Files
Source & docs	pipelines for metagenome.pdf , data_dictionary.md , data_validation.json
Truth / manifest	sample_metadata.csv , species_profile.csv , strain_candidates.csv , strain_abundance_weights.csv , genome_catalog.csv
Bins & recovery	pipeline_catalog.csv , contigs.csv , contig_genome_mappings.csv , bin_assignments.csv
Contig matrices	contig_coverage_profiles.csv , contig_tn_counts.csv , genome_tn_profiles.csv

Table 14: The files mounted at /home/agent/data/ .

#	Category	Weight	Criterion
1	SG	4	Explains why the paper’s potential-strain screen is necessary before constructing the truth manifest.
2	SG	3	Distinguishes the paper’s within-strain-cluster choice from its no-potential-strain fallback.
3	SG	3	Explains the source-defined per-genome representative-bin estimand.
4	SG	4	Explains the paper’s treatment of multi-mapped shared contigs as mapping-level evidence.
5	SG	4	Explains why the paper uses base-pair midpoint percentiles for contig-level analysis.
6	SC	4	Interprets how the expected-coverage eligibility threshold changes the evaluation population.

Table 15: The complete 20-criterion rubric for the full case (total weight 70). Category codes: SG = Source Grounding, SC = Scientific Correctness, EV = Execution & Validation, CM = Communication.

Scientific Agent Benchmarks

See all 32 papers

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
Jul 23, 2026Yinhao Tang, Youqing Fang, Yanan Sun +6AI Agent Benchmarks Information Retrieval
University of Science and Technology of China · Shanghai AI Laboratory
Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Jun 10, 2026Tianyu Liu, Allen Xin Wang, Antonia Panescu +30AI for Science Scientific Agent Benchmarks
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7
ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
May 28, 2026Wanghan Xu, Shuo Li, Tianlin Ye +48LLM Agent Evaluation AI Agent Benchmarks

Jul 23, 2026J/K move · Enter open · S save

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration

Yinhao Tang, Youqing Fang, Yanan Sun, Wenran Liu+5

University of Science and Technology of China · Shanghai AI Laboratory

AI Agent Benchmarks Information Retrieval Scientific Document Retrieval Scientific Agent Benchmarks

Open paper AskPDF arXiv

Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources. However, existing benchmarks primarily emphasize general-domain retrieval or static scientific question answering, and therefore fail to assess key capabilities required in realistic scientific research workflows. We introduce SciExplore, a benchmark designed to evaluate scientific information-seeking and reasoning capabilities of LLMs and agents. SciExplore comprises four task types covering 103 expert-curated tasks across more than ten scientific disciplines: scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis, which probe progressively higher-level abilities from entity-level reasoning and document-level identification to evidence-level grounding and domain-level synthesis. We evaluate over ten state-of-the-art LLMs and autonomous agents on SciExplore, revealing substantial performance gaps with performance degrading sharply as task complexity increases and extremely low accuracy on the most challenging structured synthesis tasks. These results highlight significant limitations of current models and agents in realistic scientific information-seeking scenarios.

Abstract

Scientific Agent Benchmarks

Explore similar work

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research