Organizations: HSE University, Moscow, Russia · Independent researcher · N. P. Bechtereva Institute of the Human Brain RAS, St. Petersburg, Russia · AI Foundation and Algorithm Lab · Constructor Labs · Constructor University, Bremen, Germany
Scientific problems have no answer key, so evaluating an agent means deciding what counts as a better solution before any agent is run. Existing benchmarks either match a symbolic answer against a key or, where they require executable work, score it by unit tests, by a rubric, or by position on a competition leaderboard. None of these is the criterion by which a field decides that one published method beats another. STEMBench adopts that criterion directly. Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each comes with two measured anchors: a submission that uses no domain knowledge, and an expert reference reproducing a published method, both obtained by running code shipped with the benchmark. The interval between them states how much of a problem is crossed by scientific insight rather than by engineering. Across eight frontier models, task-level wins and domain-level leadership dissociate: a model that wins five of the twelve tasks leads no domain mean. In the high-effort cross-model comparison, at least one model's mean score exceeds the expert anchor on seven of twelve tasks. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification. Numerical scores and official-submission validity capture different aspects of performance and are reported separately. We release the tasks, graders, anchors and harness.
Figures & tables
Benchmark
Evaluation target
Assessment or performance reference
ScienceAgentBench
Data-driven scientific programs
Program and execution evaluation
Terminal-Bench-Science
Scientific workflows
Task-specific terminal verification
ScienceBoard
Scientific software workflows
Task-completion checks
LabOSBench
Scientific instrument control
Subtask completion within a step budget
MLE-bench
ML competition submissions
Competition metrics and human leaderboard results
BioML-bench
Biomedical ML pipelines
Task performance and human baselines
Table 1: Selected related benchmarks and their evaluation targets. Rows summarise the main assessment criterion, not the only one. STEMBench reports its baseline and expert-reference scores alongside the task score.
Property
What holds instead of an answer key
P1 Underdetermined solution set
distinct methods are all admissible and are ordered by a continuous figure of merit, not accepted or rejected
P2 Model and code as deliverable
an executable artefact that generalises to a withheld split; on two tasks the grader itself trains the submitted model
P3 Load-bearing domain priors
the gap to the expert is closed by a domain insight, e.g. a decorrelation condition for weak supervision or a canonical basis for Stokes profiles
P4 Data beyond the context
0.1 – 12 GB of staged data, reached through loaders and subsampling; deciding what to look at is part of the task
P5 Community figure of merit
gated composites and non-monotone rules: a sharper interval that under-covers loses to a wider one that covers
P6 Silent failures
plausible errors, such as site leakage or a forward model that fits the spectrum but breaks the physics, which only withheld data exposes and an LLM judge reading the output cannot see
Table 2: Seven properties that separate a scientific problem from an exercise, each with an example from STEMBench .
Figure 1: The STEMBench evaluation loop. The agent reads the task specification and the public split, works in an offline sandbox through six tools, and returns a submission in the task’s contract. Ground truth never enters the sandbox: the grader runs host-side on the withheld split and returns μ∈[0,1] with a reproducibility verdict, read against the anchors b and e .
Domain
Task
Eval. units
Metric family
b→e
HEP
New-physics search (LHCO)
135,261 events
ROC-AUC, SIC-AUC, rejection
0.327→0.912
Higgs uncertainty (FAIR Univ.)
100 pseudo-exps.
coverage-penalised interval
0.400→0.687
Top tagging (TopLandscape)
403,999 jets
ROC-AUC, acc., rejection at ϵS
0.063→0.884
Dark Machines
34 signal cells
ϵS at fixed ϵB , AUC
0.003→0.221
Astro
Stokes inversion, simulated
10,000 spectra
pinball loss + forward gate
0.009→0.624
Stokes inversion, real (Hinode)
2 rasters
+ spatial-structure term
0.000→0.598
Table 3: The twelve STEMBench tasks. “Eval. units” is what the metric is computed over once. Anchors b→e are the reference_scores declared in each task’s config.yaml , obtained by running the shipped baseline and expert submissions end to end on a single workstation.
Figure 2: Raw task scores of eight frontier models using the basic agent at high reasoning effort. Each marker is the mean of the three retained scored runs, including zeros. Vertical bars show the minimum and maximum of these three scores, not standard deviations or confidence intervals. Horizontal lines mark the baseline b (dashed) and expert reference e (solid). A numerical zero does not by itself identify an invalid submission. The digital-twin expert anchor is e=0.715233 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Value
Semantics
sandbox
docker-cuda
Offline per-domain container (network disabled, public data mounted read-only, writable solution directory).
time_limit_sec
3600
Wall-clock budget per episode. The agent loop is cancelled at the deadline and whatever submission is present at that moment is graded; a single bash / python execution may run for at most two thirds of the budget.
token_limit
106
Cumulative model tokens (input plus output) across all model calls of the episode; the next model call is refused once the budget is spent.
max_messages
200
Total conversation messages (user, assistant and tool results) per episode; the episode ends when the limit is reached.
max_tool_calls
200
Declared tool-call budget, recorded in the run plan and episode metadata. It is not an independent stop condition: every tool call adds messages, so the message limit binds first.
effort
low / medium / high
Reasoning-effort tier forwarded to the model API as reasoning.effort through the OpenRouter gateway; the experimental variable of Section E , with high the harness default.
Appendix
Table 4: Per-episode run configuration. “Value” is the setting used for all reported runs; “Semantics” states what the field controls and how the harness enforces it.
Task
b
e
Gemini 3.7 Flash
Gemini 3.1 Pro
Opus 5
DeepSeek V4 Pro
Qwen3.8 2.4T
GLM 5.2
MiMo V2.5
MiniMax M3
New-physics search
0.33
0.91
0.31
0.93
0.53
0.83
0.54
0.62
0.00
0.38
Higgs uncertainty
0.40
0.69
0.41
0.43
0.48
0.39
0.44
0.28
0.14
0.13
Top tagging
0.06
0.88
0.56
0.65
0.94
0.89
0.93
0.60
0.59
0.56
Dark Machines
0.00
0.22
0.22
0.29
0.25
0.28
0.22
0.32
0.18
0.20
HEP mean
—
—
0.37
0.57
0.55
0.60
0.53
0.46
0.23
0.32
Stokes, simulated
0.01
0.62
0.01
0.78
0.57
0.25
0.00
0.00
0.43
0.11
Appendix
Table 5: STEMBench results, basic agent, raw task scores. Each cell is the mean over three scored high-effort runs retained in the current report (288 runs in total), including zeros. Domain means are unweighted means of the task means, computed before rounding. Bold denotes the highest unrounded mean in each row. b / e are the baseline and expert anchors. † The digital-twin expert anchor is e=0.715233 (displayed as 0.72 ).
Task
b
e
Gemini 3.7 Flash
Gemini 3.1 Pro
Opus 5
DeepSeek V4 Pro
Qwen3.8 2.4T
GLM 5.2
MiMo V2.5
MiniMax M3
New-physics search
0.33
0.91
0.50
0.94
0.93
0.94
0.82
0.94
0.00
0.94
Higgs uncertainty
0.40
0.69
0.41
0.49
0.53
0.47
0.47
0.44
0.41
0.38
Top tagging
0.06
0.88
0.80
0.95
0.95
0.93
0.94
0.93
0.91
0.87
Dark Machines
0.00
0.22
0.33
0.33
0.38
0.48
0.35
0.34
0.32
0.32
HEP mean
—
—
0.51
0.68
0.70
0.71
0.64
0.66
0.41
0.63
Stokes, simulated
0.01
0.62
0.01
0.80
0.85
0.74
0.01
0.00
0.74
0.32
Appendix
Table 6: STEMBench results, basic agent, maximum raw task scores. Each cell is the maximum over the same three retained high-effort runs as in Table 5 , including zeros. Domain rows are unweighted means of the task maxima, computed before rounding. Bold denotes the highest unrounded maximum in each row. b / e are the baseline and expert anchors. † The digital-twin expert anchor is e=0.715233 (displayed as 0.72 ).
Task
low
medium
high
Best
0 -runs
HEP
New-physics search
0.3±0.3
0.3±0.5
0.6±0.5
0.932
4/9
Higgs uncertainty
0.44±0.05
0.46±0.16
0.5±0.1
0.640
0/9
Top tagging
0.806±0.006
0.86±0.03
0.8±0.2†
0.941
0/20
Dark Machines
0.22±0.19
0.31±0.03
0.32±0.05
0.359
0/9
Astro
SN Ia classification
0.73±0.02
0.87±0.03
0.91±0.04
0.935
0/9
Stokes, simulated
0.07±0.11
0.74±0.07
0.5±0.4
0.815
0/9
Appendix
Table 7: Agent scores by task and reasoning effort, google/gemini-3.1-pro-preview on the basic scaffold. Cells give mean ± standard deviation over three runs, except top_quark_tagging at high effort ( n=14 , marked † ). “Best” is the maximum over all runs of the task; “ 0 -runs” counts runs scoring exactly zero: a missing submission, a submission that failed validation, or a valid submission that did not outperform the no-skill baseline; the count does not distinguish these cases. The last rows aggregate the twelve per-task means.
Effort
Main
Extended
low
0.806±0.006[0.802,0.813]
0.802±0.000[0.801,0.802]
medium
0.86±0.03[0.829,0.892]
0.85±0.05[0.802,0.901]
high
0.84±0.16[0.652,0.931]
0.941±0.004[0.937,0.946]
Appendix
Table 8: Budget ablation on top_quark_tagging . “Main” is the primary grid (1 h, 106 tokens, 200 messages, 200 tool calls); “Extended” raises these to 2.5 h, 2×106 tokens and 500/500, executed through Inspect’s native OpenRouter provider. Mean ± standard deviation, with the observed range in brackets; n=3 per cell. The fourteen-run variance study of the Main/high cell is reported in Table 7 .