Organizations: HSE University, Moscow, Russia · Independent researcher · N. P. Bechtereva Institute of the Human Brain RAS, St. Petersburg, Russia · AI Foundation and Algorithm Lab · Constructor Labs · Constructor University, Bremen, Germany
Scientific problems have no answer key, so evaluating an agent means deciding what counts as a better solution before any agent is run. Existing benchmarks either match a symbolic answer against a key or, where they require executable work, score it by unit tests, by a rubric, or by position on a competition leaderboard. None of these is the criterion by which a field decides that one published method beats another. STEMBench adopts that criterion directly. Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each comes with two measured anchors: a submission that uses no domain knowledge, and an expert reference reproducing a published method, both obtained by running code shipped with the benchmark. The interval between them states how much of a problem is crossed by scientific insight rather than by engineering. Across eight frontier models, task-level wins and domain-level leadership dissociate: a model that wins five of the twelve tasks leads no domain mean. In the high-effort cross-model comparison, at least one model's mean score exceeds the expert anchor on seven of twelve tasks. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification. Numerical scores and official-submission validity capture different aspects of performance and are reported separately. We release the tasks, graders, anchors and harness.
Figures & tables
Benchmark
Evaluation target
Assessment or performance reference
ScienceAgentBench
Data-driven scientific programs
Program and execution evaluation
Terminal-Bench-Science
Scientific workflows
Task-specific terminal verification
ScienceBoard
Scientific software workflows
Task-completion checks
LabOSBench
Scientific instrument control
Subtask completion within a step budget
MLE-bench
ML competition submissions
Competition metrics and human leaderboard results
BioML-bench
Biomedical ML pipelines
Task performance and human baselines
Table 1: Selected related benchmarks and their evaluation targets. Rows summarise the main assessment criterion, not the only one. STEMBench reports its baseline and expert-reference scores alongside the task score.
Property
What holds instead of an answer key
P1 Underdetermined solution set
distinct methods are all admissible and are ordered by a continuous figure of merit, not accepted or rejected
P2 Model and code as deliverable
an executable artefact that generalises to a withheld split; on two tasks the grader itself trains the submitted model
P3 Load-bearing domain priors
the gap to the expert is closed by a domain insight, e.g. a decorrelation condition for weak supervision or a canonical basis for Stokes profiles
P4 Data beyond the context
0.1 – 12 GB of staged data, reached through loaders and subsampling; deciding what to look at is part of the task
P5 Community figure of merit
gated composites and non-monotone rules: a sharper interval that under-covers loses to a wider one that covers
P6 Silent failures
plausible errors, such as site leakage or a forward model that fits the spectrum but breaks the physics, which only withheld data exposes and an LLM judge reading the output cannot see
Table 2: Seven properties that separate a scientific problem from an exercise, each with an example from STEMBench .
Figure 1: The STEMBench evaluation loop. The agent reads the task specification and the public split, works in an offline sandbox through six tools, and returns a submission in the task’s contract. Ground truth never enters the sandbox: the grader runs host-side on the withheld split and returns μ∈[0,1] with a reproducibility verdict, read against the anchors b and e .
Domain
Task
Eval. units
Metric family
b→e
HEP
New-physics search (LHCO)
135,261 events
ROC-AUC, SIC-AUC, rejection
0.327→0.912
Higgs uncertainty (FAIR Univ.)
100 pseudo-exps.
coverage-penalised interval
0.400→0.687
Top tagging (TopLandscape)
403,999 jets
ROC-AUC, acc., rejection at ϵS
0.063→0.884
Dark Machines
34 signal cells
ϵS at fixed ϵB , AUC
0.003→0.221
Astro
Stokes inversion, simulated
10,000 spectra
pinball loss + forward gate
0.009→0.624
Stokes inversion, real (Hinode)
2 rasters
+ spatial-structure term
0.000→0.598
Table 3: The twelve STEMBench tasks. “Eval. units” is what the metric is computed over once. Anchors b→e are the reference_scores declared in each task’s config.yaml , obtained by running the shipped baseline and expert submissions end to end on a single workstation.
Figure 2: Raw task scores of eight frontier models using the basic agent at high reasoning effort. Each marker is the mean of the three retained scored runs, including zeros. Vertical bars show the minimum and maximum of these three scores, not standard deviations or confidence intervals. Horizontal lines mark the baseline b (dashed) and expert reference e (solid). A numerical zero does not by itself identify an invalid submission. The digital-twin expert anchor is e=0.715233 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Value
Semantics
sandbox
docker-cuda
Offline per-domain container (network disabled, public data mounted read-only, writable solution directory).
time_limit_sec
3600
Wall-clock budget per episode. The agent loop is cancelled at the deadline and whatever submission is present at that moment is graded; a single bash / python execution may run for at most two thirds of the budget.
token_limit
106
Cumulative model tokens (input plus output) across all model calls of the episode; the next model call is refused once the budget is spent.
max_messages
200
Total conversation messages (user, assistant and tool results) per episode; the episode ends when the limit is reached.
max_tool_calls
200
Declared tool-call budget, recorded in the run plan and episode metadata. It is not an independent stop condition: every tool call adds messages, so the message limit binds first.
effort
low / medium / high
Reasoning-effort tier forwarded to the model API as reasoning.effort through the OpenRouter gateway; the experimental variable of Section E , with high the harness default.
Appendix
Table 4: Per-episode run configuration. “Value” is the setting used for all reported runs; “Semantics” states what the field controls and how the harness enforces it.
Task
b
e
Gemini 3.7 Flash
Gemini 3.1 Pro
Opus 5
DeepSeek V4 Pro
Qwen3.8 2.4T
GLM 5.2
MiMo V2.5
MiniMax M3
New-physics search
0.33
0.91
0.31
0.93
0.53
0.83
0.54
0.62
0.00
0.38
Higgs uncertainty
0.40
0.69
0.41
0.43
0.48
0.39
0.44
0.28
0.14
0.13
Top tagging
0.06
0.88
0.56
0.65
0.94
0.89
0.93
0.60
0.59
0.56
Dark Machines
0.00
0.22
0.22
0.29
0.25
0.28
0.22
0.32
0.18
0.20
HEP mean
—
—
0.37
0.57
0.55
0.60
0.53
0.46
0.23
0.32
Stokes, simulated
0.01
0.62
0.01
0.78
0.57
0.25
0.00
0.00
0.43
0.11
Appendix
Table 5: STEMBench results, basic agent, raw task scores. Each cell is the mean over three scored high-effort runs retained in the current report (288 runs in total), including zeros. Domain means are unweighted means of the task means, computed before rounding. Bold denotes the highest unrounded mean in each row. b / e are the baseline and expert anchors. † The digital-twin expert anchor is e=0.715233 (displayed as 0.72 ).
Task
b
e
Gemini 3.7 Flash
Gemini 3.1 Pro
Opus 5
DeepSeek V4 Pro
Qwen3.8 2.4T
GLM 5.2
MiMo V2.5
MiniMax M3
New-physics search
0.33
0.91
0.50
0.94
0.93
0.94
0.82
0.94
0.00
0.94
Higgs uncertainty
0.40
0.69
0.41
0.49
0.53
0.47
0.47
0.44
0.41
0.38
Top tagging
0.06
0.88
0.80
0.95
0.95
0.93
0.94
0.93
0.91
0.87
Dark Machines
0.00
0.22
0.33
0.33
0.38
0.48
0.35
0.34
0.32
0.32
HEP mean
—
—
0.51
0.68
0.70
0.71
0.64
0.66
0.41
0.63
Stokes, simulated
0.01
0.62
0.01
0.80
0.85
0.74
0.01
0.00
0.74
0.32
Appendix
Table 6: STEMBench results, basic agent, maximum raw task scores. Each cell is the maximum over the same three retained high-effort runs as in Table 5 , including zeros. Domain rows are unweighted means of the task maxima, computed before rounding. Bold denotes the highest unrounded maximum in each row. b / e are the baseline and expert anchors. † The digital-twin expert anchor is e=0.715233 (displayed as 0.72 ).
Task
low
medium
high
Best
0 -runs
HEP
New-physics search
0.3±0.3
0.3±0.5
0.6±0.5
0.932
4/9
Higgs uncertainty
0.44±0.05
0.46±0.16
0.5±0.1
0.640
0/9
Top tagging
0.806±0.006
0.86±0.03
0.8±0.2†
0.941
0/20
Dark Machines
0.22±0.19
0.31±0.03
0.32±0.05
0.359
0/9
Astro
SN Ia classification
0.73±0.02
0.87±0.03
0.91±0.04
0.935
0/9
Stokes, simulated
0.07±0.11
0.74±0.07
0.5±0.4
0.815
0/9
Appendix
Table 7: Agent scores by task and reasoning effort, google/gemini-3.1-pro-preview on the basic scaffold. Cells give mean ± standard deviation over three runs, except top_quark_tagging at high effort ( n=14 , marked † ). “Best” is the maximum over all runs of the task; “ 0 -runs” counts runs scoring exactly zero: a missing submission, a submission that failed validation, or a valid submission that did not outperform the no-skill baseline; the count does not distinguish these cases. The last rows aggregate the twelve per-task means.
Effort
Main
Extended
low
0.806±0.006[0.802,0.813]
0.802±0.000[0.801,0.802]
medium
0.86±0.03[0.829,0.892]
0.85±0.05[0.802,0.901]
high
0.84±0.16[0.652,0.931]
0.941±0.004[0.937,0.946]
Appendix
Table 8: Budget ablation on top_quark_tagging . “Main” is the primary grid (1 h, 106 tokens, 200 messages, 200 tool calls); “Extended” raises these to 2.5 h, 2×106 tokens and 500/500, executed through Inspect’s native OpenRouter provider. Mean ± standard deviation, with the observed range in brackets; n=3 per cell. The fourteen-run variance study of the Main/high cell is reported in Table 7 .
Benchmark saturation and data contamination increasingly obscure genuine scientific reasoning in frontier LLMs. We introduce ScienceArena, an olympiad-style benchmark from thirteen public science competitions in physics, chemistry, and biology, including IPhO and IChO 2025--2026, IBO 2023, USAPhO 2026, and USNCO 2025. Its open-ended, multi-step problems use process-credit rubrics, making faithful scoring difficult. We build ScienceArena through an expert-audited digitization pipeline that converts official exams, figures, solutions, and rubrics into structured items verified by olympiad medalists. To scale evaluation beyond costly human grading, we calibrate LLM-as-judge against medalist ground truth on archived answers from five models across IPhO and IChO; two strong judges stay within one point of expert total scores. Medalist notes show that failures often stem from visual grounding, structure fidelity, and global problem control rather than missing terminology. Evaluating fourteen recent LLMs with interleaved solving, we find that top models obtain medal-equivalent rubric scores on several public international exams, while chemistry and long-horizon consistency remain key bottlenecks. We provide an interactive demo.
Guangxiang Zhao, Qilong Shi, Xusen Xiao +13
1Qiyuan Tech · 2Tsinghua University · 3The University of Hong Kong +1
Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam.
Tianyun Zhong, Wei Wang, Jialin Li +17
University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group +4
Existing benchmarks for scientific data analysis evaluate LLMs primarily on code execution or workflow completion, overlooking that scientific analysis serves to support distinct types of scientific claims: hypothesis exploration, statistical inference, mechanistic explanation, each with different assumptions and validity criteria. We introduce SDABench, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, predictive, causal, and mechanistic) across five domains (Biology, Chemistry, Environment, Geography, Physics). SDABench comprises 527 real-data instances (SDA-Real) and 6000 synthetic instances (SDA-Synth), each in both multiple-choice and open-ended formats, constructed through an automated pipeline. Evaluating 15 representative LLMs, we find that models handle descriptive analysis well but degrade sharply on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning. SDABench further provides a five-stage error analysis framework that locates where LLMs fail: more advanced models more reliably identify the relevant scope and variables, but still struggle to select appropriate analytical procedures, model variable relationships, and draw valid conclusions.
Chuhan Shi, Xiaoquan Ren, Sicheng Song +3
Southeast University · East China Normal University · Hong Kong University of Science and Technology