Language models are increasingly being used as agents capable of planning and executing projects, as well as refining their actions through feedback. This shift raises a question: can current agents complete projects like real scientists? To examine this question, we introduce SciWorkBench, a benchmark for evaluating agents' abilities on real science tasks across biology, chemistry, materials science, and physics. SciWorkBench is built from peer-reviewed literature or expert-contributed research scenarios. We also propose a unified scientific analysis framework validated by domain experts, which can categorize most tasks in SciWorkBench. Each task covers one or more stages of this framework. We evaluate up-to-date models within the Claude Code and Codex agent frameworks under a unified execution and grading protocol. The best-performing agent achieves a mean score of 57.38, indicating that current agents still struggle with realistic scientific workflows. Our failure analysis identifies three recurring weaknesses at the workflow level: difficulty translating scientific knowledge into task-specific procedures, weak integration of evidence across different modalities, and limited capacity for scientific self-correction. These findings highlight the importance of workflow-based evaluation for measuring progress toward reliable AI systems for scientific research.
Figures & tables
Figure 1: Comparison between existing benchmark framework and our real laboratory analysis framework for scientific-agent evaluation. (a) Existing benchmarks often simplify scientific tasks to cleaned inputs, predefined procedures, and final answers. (b) Our framework organizes scientific-agent tasks around real experimental workflows.
Benchmark
Evidence source
Task authenticity
Reasoning
Peer-reviewed
Raw
Real experimental
Excludes proxy
Evidence-based
source
observations
workflow
puzzles
decisions
LAB-Bench ( Laurent et al., 2024 )
△
×
×
△
△
CORE-Bench ( Siegel et al., 2024 )
✓
△
△
✓
△
GeneBench-Pro ( Li and Ho, 2026 )
△
△
△
✓
△
SciAgentGym ( Shen and others, 2026 )
△
×
△
△
△
Table 1: Comparison between recent scientific agent benchmarks.
Figure 2: Construction pipeline of SciWorkBench. Tasks are collected from real laboratory scenario, then implemented under dockerized environments, graded with human-provided rubrics, double-checked by both automated workflow and human expert before accepted and released.
Figure 3: An example from SciWorkBench. Every task in SciWorkBench may cover one or more stages of the proposed experimental-workflow framework.
Figure 4: Distribution of SciWorkBench tasks by covered workflow stages.
Figure 5: Domain and subdomain distribution of tasks in SciWorkBench.
Task type
Exact match
Script
Agent judge
Information retrieval tasks
High
High
Medium
Structured computational science tasks
Low
High
Medium
Scientific workflow tasks
Low
Medium
High
Table 2: Compatibilities of scoring protocols for different science tasks.
Figure 6: Agent performances on SciWorkBench. Each column represents one model–harness pair. Scores are mean score on 98 agent tasks from SciWorkBench. Blue indicates Codex harness while light green indicates Claude Code harness.
Agent
Biology
Chemistry/Materials
Physics
Codex / GPT-5.6 Sol
54.46
61.18
56.17
Claude Code / Opus 5
51.52
67.59
51.73
Claude Code / Qwen3.8-Max
44.38
51.99
57.23
Claude Code / Opus 4.8
43.10
55.78
42.76
Claude Code / Kimi K3
45.71
51.69
42.01
Claude Code / GLM-5.3-Flash
43.47
52.75
43.39
Table 4: Domain-level performance by agent. The best score in each domain is bolded.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Distribution of experimental techniques and analytical methods covered by SciWorkBench tasks.
Model
Thinking depth
Opus 4.8
high
Opus 5
high
GPT-5.5
medium
GPT-5.6 Sol
max
Qwen3.8-Max
xhigh
GLM-5.3-Flash
max
Appendix
Table A.1: Thinking depth of agent models.
Harness
Biology
Chemistry/Materials
Physics
Claude Code
40.54
47.79
40.73
Codex
41.52
42.92
36.76
Appendix
Table A.2: Average discipline-level performance under different harnesses. The best score in each discipline is bolded.
Model
Claude Code
Codex
Δ
GLM-5.3-Flash
43.47
34.31
+9.16
Kimi K3
45.71
41.15
+4.56
Gemini 3.5 Flash
21.22
36.50
−15.28
Seed2.1 Pro
34.40
39.52
−5.12
Appendix
Table A.3: Paired comparison of model performance under different harnesses in biology. Positive Δ indicates that Claude Code outperforms Codex.
Model
Claude Code
Codex
Δ
GLM-5.3-Flash
52.75
32.99
+19.76
Kimi K3
51.69
42.20
+9.49
Gemini 3.5 Flash
24.77
39.79
−15.02
Seed2.1 Pro
29.97
40.54
−10.57
Appendix
Table A.4: Paired comparison of model performance under different harnesses in chemistry and materials. Positive Δ indicates that Claude Code outperforms Codex.
Model
Claude Code
Codex
Δ
GLM-5.3-Flash
43.39
28.51
+14.88
Kimi K3
42.01
36.61
+5.40
Gemini 3.5 Flash
17.50
35.57
−18.07
Seed2.1 Pro
30.49
25.50
+4.99
Appendix
Table A.5: Paired comparison of model performance under different harnesses in physics. Positive Δ indicates that Claude Code outperforms Codex.
Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam.
Tianyun Zhong, Wei Wang, Jialin Li +17
University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group +4
AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.
Tianyu Liu, Allen Xin Wang, Antonia Panescu +30
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7
Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources. However, existing benchmarks primarily emphasize general-domain retrieval or static scientific question answering, and therefore fail to assess key capabilities required in realistic scientific research workflows. We introduce SciExplore, a benchmark designed to evaluate scientific information-seeking and reasoning capabilities of LLMs and agents. SciExplore comprises four task types covering 103 expert-curated tasks across more than ten scientific disciplines: scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis, which probe progressively higher-level abilities from entity-level reasoning and document-level identification to evidence-level grounding and domain-level synthesis. We evaluate over ten state-of-the-art LLMs and autonomous agents on SciExplore, revealing substantial performance gaps with performance degrading sharply as task complexity increases and extremely low accuracy on the most challenging structured synthesis tasks. These results highlight significant limitations of current models and agents in realistic scientific information-seeking scenarios.
Yinhao Tang, Youqing Fang, Yanan Sun +6
University of Science and Technology of China · Shanghai AI Laboratory