Frontier agents retrieve structural metadata well and compute geometry from coordinates less reliably. We ask how much of a benchmark score reflects that gap and how much reflects the evaluation configuration around the model. We study ProteinGeoBench, a suite of 131 protein-structure tasks, using 4,249 rollouts from four frontier models and 4,486 further trials across 26 models. Three of the four frontier models fall within 0.031 of one another (0.785 to 0.816) with no difference between them resolved, while the fourth scores 0.707 and all three gaps to it exclude zero. Three of the four do better on metadata lookup than on numeric calculation. The fourth is slightly better on numeric. Numeric tasks are 28 of the 131 but half of the twelve tasks that all four models fail. The ranking also depends on the evaluation budget: with one run per task, 23.9% of bootstrap replicates put a different model first. On a selected 15-task slice, removing network access lowers scores more than blocking eight structural-database domains, but the two contrasts were graded in different passes and cannot be cleanly decomposed. Swapping in each model's own command-line agent changes scores by -0.071 to +0.086, a comparison too small to rule out moderate harness effects. Across 26 models, numeric performance rises with non-numeric performance rather than staying flat. The experiments use different task subsets, so they cannot rank model, harness, access and task composition against one another. Evaluations should report results by task family, along with the number of runs, the harness and the access conditions.
Figures & tables
Task family
Tasks
Difficulty tier
Tasks
Metadata lookup
36
Undergraduate
96
Sequence validation
51
Graduate
22
Numeric calculation
28
PhD
13
Concept reasoning
16
Total
131
Total
131
Table 1: Task composition. Family and difficulty tier were assigned by domain experts among the authors.
Comparison
Task set
Models
Trials
Frontier models
131 tasks
4
4,249
Model sweep
27 candidates; 20 analyzed
26
2,738
Numeric extension
17 further numeric tasks
26
1,748
Reference and native harnesses
15 matched tasks
4
480
Eight-domain blocking
15 matched tasks
4
240
Coordinate source and network
15 tasks; 13 for AlphaFold
4
1,376
Table 2: Experimental scope. Counts are graded trials. The model-sweep row counts every graded trial on the 27 candidate tasks; the model comparison in Table 7 uses the 2,048 of them that fall on the 20 analyzed tasks. Launched counts, exclusions and the shared reference arm are explained in Appendix B .
Table 3: Four frontier models on the 131 tasks, 4,249 rollouts: k≤8 for three models and k≤10 for Gemini 3.1 Pro, which received a backfill run (Appendix B ). Intervals are task-bootstrap 95% CIs over 5,000 resamples. Means and paired differences are both computed on the 130 tasks measured for all four models, so the differences subtract exactly; the Rollouts column counts all 131 tasks. Claude Opus 5’s eight trials on one concept-reasoning task were infrastructure failures and are excluded. The three gaps to Gemini 3.1 Pro exclude zero. Among the other three, Opus 5 − Grok 4.5 clears zero by 0.001 but reverses when the excluded cell is counted, so we treat it as sensitive to that choice rather than resolved; the remaining two differences include zero.
Family
n
Opus 5
GPT-5.6-sol
Grok 4.5
Gemini 3.1 Pro
Metadata lookup
36
0.907
0.852
0.855
0.713
Sequence validation
51
0.790
0.820
0.799
0.698
Concept reasoning
15
0.869
0.809
0.852
0.681
Numeric calculation
28
0.716
0.749
0.636
0.731
Table 4: Mean reward by task family, over the 130 tasks measured for all four models; bold marks the highest score in each row. The weakest family differs by model: numeric calculation is the weakest for three models but Gemini 3.1 Pro’s best, and Grok 4.5 scores lowest of the four on numeric calculation ( 0.636 ) while placing second on concept reasoning.
Runs k
Exact ordering
Same top model
Mean τ
Distinct orderings
1
65.2%
76.1%
0.877
6
2
79.1%
83.3%
0.930
5
4
91.4%
92.0%
0.971
3
8
97.6%
97.7%
0.992
3
Table 5: Agreement with the eight-run reference ordering on the 99 tasks with eight completed runs for all four models, 5,000 bootstrap replicates. An exact match requires all six pairwise comparisons to agree with the reference.
Figure 1: Four-model mean reward on the 15-task access slice, experimental coordinates. Left pair: the reference arm and the eight-domain-blocked arm, graded in the same pass (timed-out trials scored zero). Right pair: network on and off in the earlier coordinate-source experiment, graded in a different pass; these values are recovered aggregates without per-task uncertainty. Comparisons between the two pairs mix grading passes.
Model (native agent)
Reference
Native
Change
95% CI
Claude Opus 5 ( claude-code )
0.817
0.746
−0.071
[ −0.157 , +0.002 ]
n.s.
GPT-5.6-sol ( codex )
0.756
0.783
+0.028
[ −0.042 , +0.120 ]
n.s.
Gemini 3.1 Pro ( gemini-cli )
0.763
0.849
+0.086
[ +0.012 , +0.173 ]
sig.
Grok 4.5 ( grok-build )
0.672
0.747
+0.076
[ −0.079 , +0.196 ]
n.s.
Table 6: Native command-line agents against the reference harness on 15 matched tasks, k=4 , both arms graded in the same pass. Changes are per-task paired differences with task-bootstrap 95% CIs. The reference arm is the one used for the domain-blocking contrast in Section 3.4 .
#
Model
Mean
#
Model
Mean
1
Claude Opus 5
0.597
14
DeepSeek V4 Pro
0.426
2
Gemini 3.5 Flash
0.582
15
Gemini 3.1 Pro
0.423
3
Muse Spark 1.1
0.552
16
GPT-5.4
0.418
4
Claude Sonnet 5
0.537
17
DeepSeek V4 Flash
0.410
5
Gemini 3.7 Flash
0.512
18
GPT-5.6-luna
0.372
6
Grok 4.5
0.494
19
GPT-5
0.371
Table 7: Model means on the selected 20-task comparison, k≤4 , 2,048 graded trials. This is a different task set from Table 3 , run with different sampling settings (Appendix C ); ranks are descriptive point estimates.
Figure 2: Numeric and non-numeric performance across the 26 models. (a) Each model’s mean over all 27 numeric tasks run in the sweep and its extension, against its mean on the sweep’s non-numeric tasks, with a bootstrap band for the fit. This is an association across models, not a causal scaling result. (b) Means on the five numeric tasks in the selected 20-task subset, ranked. Seventeen of 26 models score 0.10 or less there, which biases a slope estimated within that subset; this is why we ran the extension used in (a).
Agentic
Structure files
Determ. scoring
Unpub. structures
Lookup vs. compute
Varies harness
Varies network
Repeated runs
LAB-Bench
✓
(✓)
✓
BixBench
✓
(✓)
BioAgent Bench
✓
✓
(✓)
BiomniBench
✓
✓
✓
BioDesignBench
✓
✓
(✓)
(✓)
ProtStructQA
(✓)
✓
(✓)
Table 8: How related benchmarks compare on the features this study varies. ✓: stated in the paper; (✓): partly (for example, only some conditions use tools, or only some models get repeated runs); blank: not a stated focus of that paper, not a claim that it is absent. Each row reflects the latest version of the cited paper. Deterministic scoring means no LLM judge contributes to the score. Repeated runs means several runs per task for each model, with run-to-run variability or ranking stability reported. a An LLM judge carries 5.3% of ProteinGeoBench’s reward weight (Section 2.2 ).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Pattern
Signature and support
Suggested check
net-dep
Removing the network costs 0.13 – 0.24 , depending on the baseline, while blocking eight structural-database domains costs 0.08 – 0.10 ; the source of the remainder is not identified
Benchmark under the access condition you will deploy with; distinguish blocking named domains from removing all network access
comp-gap
Numeric family below lookup for three of four models ( 0.64 – 0.75 against 0.85 – 0.91 for those three; 2.3× enrichment in shared failures), but rising with non-numeric performance at close to unit slope
Report per-family scores; inspect low-scoring tasks within each family; test a suspected deficit across many models
tier-blind
Expert difficulty labels do not order performance: the graduate tier is the lowest-scoring for all four models, below the PhD tier
Stratify by task family rather than expert tier; treat labels as testable
sign-round
Answers earning no accuracy credit differ from the reference by sign or rounding: 8/55 on extraction-valid tasks are the exact negative of the reference, 6/7 recovered on the dihedral task across three providers
Check the task instruction and traces before changing a grade; state sign conventions and precision explicitly
sample-dep
A failure shared by four models is solved by another model or run: two of ten retested shared failures reach 1.00 , one of them for a model that had failed it
Treat claims of universal failure as provisional
rank-noise
The ordering does not reproduce at low budget: at k=1 the reference ordering recurs in 65.2% of bootstrap replicates and the top model differs 23.9% of the time
Report k and the uncertainty in model differences; treat an ordering from one run as provisional
Appendix
Table 9: Failure patterns and suggested checks. The list is expert-authored and has no annotation protocol or agreement statistic.
Treatment
Tasks
Ref.
Blocked
Diff.
95% CI
All trials, timeouts scored zero
15
0.752
0.662
−0.090
[ −0.177 , −0.008 ]
Timed-out trials excluded
15
0.752
0.673
−0.079
[ −0.172 , +0.009 ]
Paired, 14 complete tasks
14
0.755
0.658
−0.096
[ −0.192 , −0.009 ]
Appendix
Table 10: Eight-domain blocking against the matched reference arm.
Protein-language systems are often evaluated by whether they generate plausible biological text, but a structural question has a sharper semantics: it denotes a measurement in a 3D coordinate system. We introduce ProtStructQA, an executable benchmark for protein structural question answering in which each natural-language question is generated from a hidden typed domain-specific language (DSL) program and the answer is obtained by executing that program on an AlphaFold-predicted structure. ProtStructQA releases 382.2K questions covering confidence, distances, predicted aligned error (PAE), solvent exposure, secondary structure, topology and contacts, and held-out compositions: a 330K active benchmark over 10K proteins from four species, plus a 52.2K hard-negative robustness pool. Without fine-tuning, we evaluate Qwen3 models from 0.6B to 8B under direct prompting, chain-of-thought, grammar-constrained executable voting, executable voting with chain-of-thought, and multi-turn ReAct-style tool use, and replicate the headline finding on Gemma-3-1B and Gemma-3-12B. We find a capability-dependent denotation threshold between Qwen3-1.7B and Qwen3-4B: below it, tool-mediated ReAct dominates because models often fail to produce executable denotations; above it, chain-of-thought flips from mostly harmful to strongly beneficial and becomes the strongest strategy on most splits. Parse-failure and family-level analyses show that the threshold is a transition from unparseable language to executable structural denotation, while grammar and execution remain selectively valuable for PAE and secondary-structure queries. ProtStructQA reframes scientific QA as compilation from language to measurement and provides a diagnostic testbed for when language models can map words to executable 3D structural measurements.
We introduce ABLE, a benchmark for evaluating LLM agents' ability to use biological AI models (BAIMs), such as ProteinMPNN and AlphaFold3, in dual-use protein design workflows. ABLE assesses agent performance through a set of tasks spanning structure retrieval, sequence generation, and design validation. We evaluate 15 frontier models and find that seven refuse all tasks, while the remaining models exhibit substantial performance differences. Claude Sonnet 4 and Gemini 3 Pro achieve the highest scores across information retrieval, tool selection, and tool use. We further compare model performance on a subset of tasks against an expert human baseline. Our results suggest that current LLMs can substantially lower barriers to protein design, but remain inconsistent in planning, strategy generation, and integrating biological knowledge with tool use.
Environmental scientists spend disproportionate effort on data wrangling rather than analysis. New AI agents can be a helpful tool, but no benchmark exists to evaluate AI agents that automate environmental geospatial workflows through structured tool calling against real APIs. We introduce the GeoNatureAgent Benchmark, the first benchmark for environmental analysis agents that operate via structured tool calls to a production-style geospatial API. The benchmark comprises 93 tasks across 18 categories. Tasks are evaluated against an open, self-hostable geospatial API that serves three environmental indicators across Spain and Portugal via sixteen tools. We evaluate nine frontier and open-weight LLMs, reporting capability and per-case cost as orthogonal axes. Results manifest that (1) Claude Sonnet 4 achieves the highest capability at 60.8% +/- 0.8%, followed closely by DeepSeek V3.2 at 56.3% +/- 3.1%, while no other model exceeds 51%; (2) the cost-accuracy Pareto frontier is occupied mostly by open-weight models, with DeepSeek V3.2 offering 93% of Claude's capability at 11.6x lower cost; and (3) structured tool calling against a real API provides a more discriminative measure of real-world agent capability, with mean accuracies 25-35 percentage points below those reported on general-purpose GIS benchmarks.
Gabriel Diaz-Ireland, Diego Prieto-Herráez, Mario García Peces +2
Universidad Católica de Ávila (UCAV) Ávila, Spain · Johns Hopkins University Baltimore, MD, USA · Independent Researcher Madrid, Spain +1