Language models are increasingly being used as agents capable of planning and executing projects, as well as refining their actions through feedback. This shift raises a question: can current agents complete projects like real scientists? To examine this question, we introduce SciWorkBench, a benchmark for evaluating agents' abilities on real science tasks across biology, chemistry, materials science, and physics. SciWorkBench is built from peer-reviewed literature or expert-contributed research scenarios. We also propose a unified scientific analysis framework validated by domain experts, which can categorize most tasks in SciWorkBench. Each task covers one or more stages of this framework. We evaluate up-to-date models within the Claude Code and Codex agent frameworks under a unified execution and grading protocol. The best-performing agent achieves a mean score of 57.38, indicating that current agents still struggle with realistic scientific workflows. Our failure analysis identifies three recurring weaknesses at the workflow level: difficulty translating scientific knowledge into task-specific procedures, weak integration of evidence across different modalities, and limited capacity for scientific self-correction. These findings highlight the importance of workflow-based evaluation for measuring progress toward reliable AI systems for scientific research.
Figures & tables
Figure 1: Comparison between existing benchmark framework and our real laboratory analysis framework for scientific-agent evaluation. (a) Existing benchmarks often simplify scientific tasks to cleaned inputs, predefined procedures, and final answers. (b) Our framework organizes scientific-agent tasks around real experimental workflows.
Benchmark
Evidence source
Task authenticity
Reasoning
Peer-reviewed
Raw
Real experimental
Excludes proxy
Evidence-based
source
observations
workflow
puzzles
decisions
LAB-Bench ( Laurent et al., 2024 )
△
×
×
△
△
CORE-Bench ( Siegel et al., 2024 )
✓
△
△
✓
△
GeneBench-Pro ( Li and Ho, 2026 )
△
△
△
✓
△
SciAgentGym ( Shen and others, 2026 )
△
×
△
△
△
Table 1: Comparison between recent scientific agent benchmarks.
Figure 2: Construction pipeline of SciWorkBench. Tasks are collected from real laboratory scenario, then implemented under dockerized environments, graded with human-provided rubrics, double-checked by both automated workflow and human expert before accepted and released.
Figure 3: An example from SciWorkBench. Every task in SciWorkBench may cover one or more stages of the proposed experimental-workflow framework.
Figure 4: Distribution of SciWorkBench tasks by covered workflow stages.
Figure 5: Domain and subdomain distribution of tasks in SciWorkBench.
Task type
Exact match
Script
Agent judge
Information retrieval tasks
High
High
Medium
Structured computational science tasks
Low
High
Medium
Scientific workflow tasks
Low
Medium
High
Table 2: Compatibilities of scoring protocols for different science tasks.
Figure 6: Agent performances on SciWorkBench. Each column represents one model–harness pair. Scores are mean score on 98 agent tasks from SciWorkBench. Blue indicates Codex harness while light green indicates Claude Code harness.
Agent
Biology
Chemistry/Materials
Physics
Codex / GPT-5.6 Sol
54.46
61.18
56.17
Claude Code / Opus 5
51.52
67.59
51.73
Claude Code / Qwen3.8-Max
44.38
51.99
57.23
Claude Code / Opus 4.8
43.10
55.78
42.76
Claude Code / Kimi K3
45.71
51.69
42.01
Claude Code / GLM-5.3-Flash
43.47
52.75
43.39
Table 4: Domain-level performance by agent. The best score in each domain is bolded.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Distribution of experimental techniques and analytical methods covered by SciWorkBench tasks.
Model
Thinking depth
Opus 4.8
high
Opus 5
high
GPT-5.5
medium
GPT-5.6 Sol
max
Qwen3.8-Max
xhigh
GLM-5.3-Flash
max
Appendix
Table A.1: Thinking depth of agent models.
Harness
Biology
Chemistry/Materials
Physics
Claude Code
40.54
47.79
40.73
Codex
41.52
42.92
36.76
Appendix
Table A.2: Average discipline-level performance under different harnesses. The best score in each discipline is bolded.
Model
Claude Code
Codex
Δ
GLM-5.3-Flash
43.47
34.31
+9.16
Kimi K3
45.71
41.15
+4.56
Gemini 3.5 Flash
21.22
36.50
−15.28
Seed2.1 Pro
34.40
39.52
−5.12
Appendix
Table A.3: Paired comparison of model performance under different harnesses in biology. Positive Δ indicates that Claude Code outperforms Codex.
Model
Claude Code
Codex
Δ
GLM-5.3-Flash
52.75
32.99
+19.76
Kimi K3
51.69
42.20
+9.49
Gemini 3.5 Flash
24.77
39.79
−15.02
Seed2.1 Pro
29.97
40.54
−10.57
Appendix
Table A.4: Paired comparison of model performance under different harnesses in chemistry and materials. Positive Δ indicates that Claude Code outperforms Codex.
Model
Claude Code
Codex
Δ
GLM-5.3-Flash
43.39
28.51
+14.88
Kimi K3
42.01
36.61
+5.40
Gemini 3.5 Flash
17.50
35.57
−18.07
Seed2.1 Pro
30.49
25.50
+4.99
Appendix
Table A.5: Paired comparison of model performance under different harnesses in physics. Positive Δ indicates that Claude Code outperforms Codex.