Organizations: University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group · Qwen Team, Alibaba Group · Zhejiang University · The Hong Kong University of Science and Technology · Tsinghua University
Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam.
Figures & tables
Figure 1: Sci-Agent-Exam versus existing benchmarks: workflow orientied task types, full deliverables (answer, code, report) per task, hybrid evaluation, with analysis for efficiency and stability .
Benchmark
Task Setting
Task-Type Coverage
Required Deliverables
Scale
1
2
3
4
5
6
7
Answer
Exec.
Report
# Task
# Domain
SciBench
Scientific QA
○
○
○
○
○
○
●
●
○
○
869
3
DiscoveryBench
Scientific discovery
○
●
◐
◐
◐
○
●
●
○
○
264
6
ScienceAgentBench
Scientific analysis
○
○
●
●
◐
◐
◐
●
●
○
102
4
CORE-Bench
Computational reproduction
○
○
●
◐
◐
◐
◐
●
○
○
270
3
PaperBench
Paper replication
◐
○
◐
●
◐
◐
◐
○
●
○
20
1
Table 1: Comparison with representative scientific-agent benchmarks. Task-Type Coverage maps seven task types: \scriptsize1⃝ Literature Synthesis, \scriptsize2⃝ Hypothesis Validation, \scriptsize3⃝ Tool Orchestration, \scriptsize4⃝ Code Generation, \scriptsize5⃝ End-to-End Analysis, \scriptsize6⃝ Error Diagnosis, and \scriptsize7⃝ Result Interpretation. For task-type coverage, ● = explicitly targeted, ◐ = involved but not distinct, and ○ = not covered. For Required Deliverables , ● = required in every task, ◐ = required in some tasks, and ○ = not required. Answer : short answer/result; Exec. : code/artifact; Report : written report. Domain counts follow each benchmark’s categorization.
Figure 2: Overview of Sci-Agent-Exam . We organize tasks with a workflow-oriented taxonomy, construct executable task packages through a quality-controlled data pipeline, and evaluate task outputs using a hybrid protocol combining deterministic programmatic checks with LLM rubric judgments. The bottom panel illustrates an example task with its inputs, required deliverables, and evaluation criteria.
Domain
\scriptsize1⃝
\scriptsize2⃝
\scriptsize3⃝
\scriptsize4⃝
\scriptsize5⃝
\scriptsize6⃝
\scriptsize7⃝
Total
Bioinformatics
0
1
2
1
5
2
0
11
Biology
1
2
4
2
4
1
1
15
Chemistry
3
1
3
2
1
1
0
11
Computer science
0
3
1
3
2
1
0
10
Data science
1
2
2
1
2
1
1
10
Engineering
2
0
2
1
1
0
2
8
Table 2: Distribution of 120 tasks across 11 scientific domains and seven SWTT task types. Column numbers follow the definitions in the text and Table 1 .
Table 3: Model performance on Sci-Agent-Exam (120 tasks; Claude Code harness). Programmatic and LLM Judge scores assess executable and textual artifacts, respectively. Scores are mean ± standard error across tasks; [email protected] is the fraction of runs with a final score of at least 0.95. Bold and underlined values mark the best and second-best results.
Figure 3: Agents generally report better than they execute. Task-level outcomes frequently concentrate in the top-left (TL) quadrant, indicating strong report quality but weak executable correctness. TL is the fraction of cases with LLM-judge ≥0.5 and Programmatic <0.5 ; TR is the fraction for which both scores are ≥0.5 .
Figure 4: Model performance across scientific domains. Rows are ordered by overall benchmark performance.
Figure 5: Trajectory analysis of scientific agents. (a) Four examples illustrate four behaviors used in our analysis: early completion (reaching near-peak quality within the first third of snapshots), dip (a previously Pass becomes Fail), recover (a Failed rubric Passes again), and regret (a Pass rubric earlier but Fails at the final snapshot). (b) Behavior rates across models four programmatic execution and five LLM-judged dimensions of scientific report quality: Source Grounding, Scientific Correctness, Completeness, Execution & Validation, and Communication.
Figure 6: Higher-priced models can be faster and cheaper per completed run. Overall score versus mean cost per run; the upper-left indicates higher performance at lower cost.
Form
The decoy → what credit requires
Representative task
Decoy artifact
A precomputed rollup or dashboard reads like the answer but was produced by a leaky or partial procedure → recompute the decision independently.
data-science-hypothesis-validation-01
Loudest signal
The most salient statistic (lowest information criterion, lowest cost) nominates a candidate that fails a downstream physical or feasibility gate → select on the gated objective.
engineering-tool-orchestration-02
Distractor feature
Balanced noise covariates masquerade as signal → keep only the load-bearing variables.
bioinformatics-end-to-end-analysis-05
Misattributed cause
A dashboard blames the wrong component and discourages the real fix → diagnose and repair the true failure path.
cs-expansion-error-diagnosis-02
Table 4: Recurring trap forms in Sci-Agent-Exam , with a representative task for each. In every form the decoy is locally plausible; credit requires the coupled scientific workflow.
Task type
wip
wij
Code Generation
0.70
0.30
Error Diagnosis
0.65
0.35
Tool Orchestration
0.55
0.45
End-to-End Analysis
0.50
0.50
Hypothesis Validation
0.50
0.50
Result Interpretation
0.45
0.55
Table 5: Programmatic and LLM-judge weights by task type.
Model
Harness
n
Prog.
Judge
Final
gpt-6-astra
claude-code
42
72.3
85.8
78.9
codex
45
74.8
83.8
79.5
qwen-3.8-max
claude-code
49
65.6
75.9
70.0
codex
46
67.4
76.6
71.3
qoder
40
62.9
73.5
67.0
kimi-k3
claude-code
47
65.8
75.3
69.9
Table 6: Harness sensitivity on the 50-task set: the same model scored under general-purpose harnesses (claude-code at max effort, codex) and, where available, its vendor-native harness (kimi-cli for kimi-k3, qoder for qwen-3.8-max).
Table 7: Effect of max vs. high thinking effort on the claude-code harness (paired per task on the 50-task set; n = paired tasks). Programmatic, LLM Judge and Final are on a 0 – 100 scale; Final is the per-task mean reward averaged over tasks; [email protected] is the fraction of tasks with mean final reward ≥95 . Rows ordered by max-effort Final.
Model
Scores
Avg. Cost/Run ($)
Runtime (min)
Turns
GPT-6 Astra
80.9
20.79
22.1
56.8
Claude Fable 5
67.3
7.67
25.3
30.2
Claude Opus 5
64.2
12.49
30.6
43.9
Qwen3.8 Max
62.7
22.42
50.9
59.6
Kimi K3
62.6
20.96
47.2
52.6
GPT-5.6 Sol
59.6
13.10
22.0
54.3
Table 8: Performance, average cost, runtime, and interaction length on the Sci-Agent-Exam peer subset.
Figure 8: Performance and resource use across ten agents on Sci-Agent-Exam . Overall score, mean cost, runtime, and turns per run, ordered by overall score. The best value for each metric is highlighted.
Figure 9: Model performance across the seven task types in Sci-Agent-Exam . Each cell reports the mean final task score for a model–task-type pair, with the number of tasks shown in parentheses. Rows are ordered by overall benchmark performance, and the rightmost column reports each model’s overall final score.
Figure 10: Rubric-based performance across five dimensions of scientific quality. Each cell reports the fraction of rubric criteria passed within a dimension for each model. The five dimensions capture Source Grounding, Scientific Correctness, Completeness, Execution & Validation, and Communication. The rightmost column reports the aggregate LLM-judge score.
Figure 11: Trajectory-level analysis of agent behaviors. (a) illustrates the diverse score dynamics (e.g., oscillations, late-stage jumps) across different models. (b) highlights the performance variations when multiple models tackle the same scientific task.
Figure 12: Trajectory behavior across rubric dimensions and the programmatic channel, for all ten evaluated agents. Rows are ordered by overall benchmark score. This full-population view extends the six-model main-text Figure 5 to the four agents with smaller judged samples (Claude Fable 5, Claude Opus 4.8, Grok-4.6, and DeepSeek-V4-Pro).
Task type
n
Early
Dip
Recover
Regret
End-to-end analysis
134
36.2%
12.2%
10.3%
3.7%
Literature synthesis
71
29.9%
6.6%
4.2%
3.3%
Hypothesis validation
117
20.9%
6.1%
4.5%
2.9%
Tool orchestration
112
18.7%
8.3%
6.8%
3.3%
Result interpretation
47
13.3%
4.4%
2.1%
2.7%
Code generation
112
8.6%
3.7%
2.1%
1.9%
Table 9: Trajectory analysis by scientific task types.
Table 13: Contents of an example task package and the role of each component.
Role
Files
Source & docs
pipelines for metagenome.pdf , data_dictionary.md , data_validation.json
Table 14: The files mounted at /home/agent/data/ .
#
Category
Weight
Criterion
1
SG
4
Explains why the paper’s potential-strain screen is necessary before constructing the truth manifest.
2
SG
3
Distinguishes the paper’s within-strain-cluster choice from its no-potential-strain fallback.
3
SG
3
Explains the source-defined per-genome representative-bin estimand.
4
SG
4
Explains the paper’s treatment of multi-mapped shared contigs as mapping-level evidence.
5
SG
4
Explains why the paper uses base-pair midpoint percentiles for contig-level analysis.
6
SC
4
Interprets how the expected-coverage eligibility threshold changes the evaluation population.
Table 15: The complete 20-criterion rubric for the full case (total weight 70). Category codes: SG = Source Grounding, SC = Scientific Correctness, EV = Execution & Validation, CM = Communication.
Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources. However, existing benchmarks primarily emphasize general-domain retrieval or static scientific question answering, and therefore fail to assess key capabilities required in realistic scientific research workflows. We introduce SciExplore, a benchmark designed to evaluate scientific information-seeking and reasoning capabilities of LLMs and agents. SciExplore comprises four task types covering 103 expert-curated tasks across more than ten scientific disciplines: scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis, which probe progressively higher-level abilities from entity-level reasoning and document-level identification to evidence-level grounding and domain-level synthesis. We evaluate over ten state-of-the-art LLMs and autonomous agents on SciExplore, revealing substantial performance gaps with performance degrading sharply as task complexity increases and extremely low accuracy on the most challenging structured synthesis tasks. These results highlight significant limitations of current models and agents in realistic scientific information-seeking scenarios.
Yinhao Tang, Youqing Fang, Yanan Sun +6
University of Science and Technology of China · Shanghai AI Laboratory
AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.
Tianyu Liu, Allen Xin Wang, Antonia Panescu +30
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7
AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.