ProteinGeoBench: Frontier Structural-Biology Agents Are Fluent at Lookup but Lost in Geometry

Matt Turk; Brandon L. Neel; Diana M. Lopez; Zachary D. Smith; Grace N. Johnecheck; James B. Golden

cs.AIOct 19, 2026

ProteinGeoBench: Frontier Structural-Biology Agents Are Fluent at Lookup but Lost in Geometry

Authors: Matt Turk, Brandon L. Neel, Diana M. Lopez, Zachary D. Smith, Grace N. Johnecheck, James B. Golden

Organizations: Protege DataLab · Heimdall Bio

Abstract

Frontier agents retrieve structural metadata well and compute geometry from coordinates less reliably. We ask how much of a benchmark score reflects that gap and how much reflects the evaluation configuration around the model. We study ProteinGeoBench, a suite of 131 protein-structure tasks, using 4,249 rollouts from four frontier models and 4,486 further trials across 26 models. Three of the four frontier models fall within 0.031 of one another (0.785 to 0.816) with no difference between them resolved, while the fourth scores 0.707 and all three gaps to it exclude zero. Three of the four do better on metadata lookup than on numeric calculation. The fourth is slightly better on numeric. Numeric tasks are 28 of the 131 but half of the twelve tasks that all four models fail. The ranking also depends on the evaluation budget: with one run per task, 23.9% of bootstrap replicates put a different model first. On a selected 15-task slice, removing network access lowers scores more than blocking eight structural-database domains, but the two contrasts were graded in different passes and cannot be cleanly decomposed. Swapping in each model's own command-line agent changes scores by -0.071 to +0.086, a comparison too small to rule out moderate harness effects. Across 26 models, numeric performance rises with non-numeric performance rather than staying flat. The experiments use different task subsets, so they cannot rank model, harness, access and task composition against one another. Evaluations should report results by task family, along with the number of runs, the harness and the access conditions.

Figures & tables

Task family	Tasks	Difficulty tier	Tasks
Metadata lookup	36	Undergraduate	96
Sequence validation	51	Graduate	22
Numeric calculation	28	PhD	13
Concept reasoning	16
Total	131	Total	131

Table 1: Task composition. Family and difficulty tier were assigned by domain experts among the authors.

Comparison	Task set	Models	Trials
Frontier models	131 tasks	4	4,249
Model sweep	27 candidates; 20 analyzed	26	2,738
Numeric extension	17 further numeric tasks	26	1,748
Reference and native harnesses	15 matched tasks	4	480
Eight-domain blocking	15 matched tasks	4	240
Coordinate source and network	15 tasks; 13 for AlphaFold	4	1,376

Table 2: Experimental scope. Counts are graded trials. The model-sweep row counts every graded trial on the 27 candidate tasks; the model comparison in Table 7 uses the 2,048 of them that fall on the 20 analyzed tasks. Launched counts, exclusions and the shared reference arm are explained in Appendix B .

Model	Mean	95% CI	Perfect solves	Rollouts
Claude Opus 5	0.816	[0.770, 0.860]	519/1032 (50.3%)	1032
GPT-5.6-sol	0.812	[0.766, 0.856]	518/1033 (50.1%)	1033
Grok 4.5	0.785	[0.733, 0.834]	490/1031 (47.5%)	1031
Gemini 3.1 Pro Preview	0.707	[0.653, 0.758]	434/1153 (37.6%)	1153
Paired differences, task bootstrap:
Opus 5 $-$ GPT-5.6-sol $+0.003$ [ $-0.030$ , $+0.038$ ] n.s.

Table 3: Four frontier models on the 131 tasks, 4,249 rollouts:

k\leq 8

for three models and

k\leq 10

for Gemini 3.1 Pro, which received a backfill run (Appendix B ). Intervals are task-bootstrap 95% CIs over 5,000 resamples. Means and paired differences are both computed on the 130 tasks measured for all four models, so the differences subtract exactly; the Rollouts column counts all 131 tasks. Claude Opus 5’s eight trials on one concept-reasoning task were infrastructure failures and are excluded. The three gaps to Gemini 3.1 Pro exclude zero. Among the other three, Opus 5

-

Grok 4.5 clears zero by

0.001

but reverses when the excluded cell is counted, so we treat it as sensitive to that choice rather than resolved; the remaining two differences include zero.

Family	$n$	Opus 5	GPT-5.6-sol	Grok 4.5	Gemini 3.1 Pro
Metadata lookup	36	0.907	0.852	0.855	0.713
Sequence validation	51	0.790	0.820	0.799	0.698
Concept reasoning	15	0.869	0.809	0.852	0.681
Numeric calculation	28	0.716	0.749	0.636	0.731

Table 4: Mean reward by task family, over the 130 tasks measured for all four models; bold marks the highest score in each row. The weakest family differs by model: numeric calculation is the weakest for three models but Gemini 3.1 Pro’s best, and Grok 4.5 scores lowest of the four on numeric calculation (

0.636

) while placing second on concept reasoning.

Runs $k$	Exact ordering	Same top model	Mean $\tau$	Distinct orderings
1	65.2%	76.1%	0.877	6
2	79.1%	83.3%	0.930	5
4	91.4%	92.0%	0.971	3
8	97.6%	97.7%	0.992	3

Table 5: Agreement with the eight-run reference ordering on the 99 tasks with eight completed runs for all four models, 5,000 bootstrap replicates. An exact match requires all six pairwise comparisons to agree with the reference.

Figure 1: Four-model mean reward on the 15-task access slice, experimental coordinates. Left pair: the reference arm and the eight-domain-blocked arm, graded in the same pass (timed-out trials scored zero). Right pair: network on and off in the earlier coordinate-source experiment, graded in a different pass; these values are recovered aggregates without per-task uncertainty. Comparisons between the two pairs mix grading passes.

Model (native agent)	Reference	Native	Change	95% CI
Claude Opus 5 ( claude-code )	0.817	0.746	$-0.071$	[ $-0.157$ , $+0.002$ ]	n.s.
GPT-5.6-sol ( codex )	0.756	0.783	$+0.028$	[ $-0.042$ , $+0.120$ ]	n.s.
Gemini 3.1 Pro ( gemini-cli )	0.763	0.849	$+0.086$	[ $+0.012$ , $+0.173$ ]	sig.
Grok 4.5 ( grok-build )	0.672	0.747	$+0.076$	[ $-0.079$ , $+0.196$ ]	n.s.

Table 6: Native command-line agents against the reference harness on 15 matched tasks,

k=4

, both arms graded in the same pass. Changes are per-task paired differences with task-bootstrap 95% CIs. The reference arm is the one used for the domain-blocking contrast in Section 3.4 .

#	Model	Mean	#	Model	Mean
1	Claude Opus 5	0.597	14	DeepSeek V4 Pro	0.426
2	Gemini 3.5 Flash	0.582	15	Gemini 3.1 Pro	0.423
3	Muse Spark 1.1	0.552	16	GPT-5.4	0.418
4	Claude Sonnet 5	0.537	17	DeepSeek V4 Flash	0.410
5	Gemini 3.7 Flash	0.512	18	GPT-5.6-luna	0.372
6	Grok 4.5	0.494	19	GPT-5	0.371

Table 7: Model means on the selected 20-task comparison,

k\leq 4

, 2,048 graded trials. This is a different task set from Table 3 , run with different sampling settings (Appendix C ); ranks are descriptive point estimates.

Figure 2: Numeric and non-numeric performance across the 26 models. (a) Each model’s mean over all 27 numeric tasks run in the sweep and its extension, against its mean on the sweep’s non-numeric tasks, with a bootstrap band for the fit. This is an association across models, not a causal scaling result. (b) Means on the five numeric tasks in the selected 20-task subset, ranked. Seventeen of 26 models score

0.10

or less there, which biases a slope estimated within that subset; this is why we ran the extension used in (a).

	Agentic	Structure files	Determ. scoring	Unpub. structures	Lookup vs. compute	Varies harness	Varies network	Repeated runs
LAB-Bench			✓		(✓)			✓
BixBench	✓							(✓)
BioAgent Bench	✓					✓		(✓)
BiomniBench	✓					✓		✓
BioDesignBench	✓	✓	(✓)			(✓)
ProtStructQA	(✓)		✓					(✓)

Table 8: How related benchmarks compare on the features this study varies. ✓: stated in the paper; (✓): partly (for example, only some conditions use tools, or only some models get repeated runs); blank: not a stated focus of that paper, not a claim that it is absent. Each row reflects the latest version of the cited paper. Deterministic scoring means no LLM judge contributes to the score. Repeated runs means several runs per task for each model, with run-to-run variability or ranking stability reported. a An LLM judge carries 5.3% of ProteinGeoBench’s reward weight (Section 2.2 ).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Pattern	Signature and support	Suggested check
net-dep	Removing the network costs $0.13$ – $0.24$ , depending on the baseline, while blocking eight structural-database domains costs $0.08$ – $0.10$ ; the source of the remainder is not identified	Benchmark under the access condition you will deploy with; distinguish blocking named domains from removing all network access
comp-gap	Numeric family below lookup for three of four models ( $0.64$ – $0.75$ against $0.85$ – $0.91$ for those three; $2.3\times$ enrichment in shared failures), but rising with non-numeric performance at close to unit slope	Report per-family scores; inspect low-scoring tasks within each family; test a suspected deficit across many models
tier-blind	Expert difficulty labels do not order performance: the graduate tier is the lowest-scoring for all four models, below the PhD tier	Stratify by task family rather than expert tier; treat labels as testable
sign-round	Answers earning no accuracy credit differ from the reference by sign or rounding: $8/55$ on extraction-valid tasks are the exact negative of the reference, $6/7$ recovered on the dihedral task across three providers	Check the task instruction and traces before changing a grade; state sign conventions and precision explicitly
sample-dep	A failure shared by four models is solved by another model or run: two of ten retested shared failures reach $1.00$ , one of them for a model that had failed it	Treat claims of universal failure as provisional
rank-noise	The ordering does not reproduce at low budget: at $k{=}1$ the reference ordering recurs in $65.2\%$ of bootstrap replicates and the top model differs $23.9\%$ of the time	Report $k$ and the uncertainty in model differences; treat an ordering from one run as provisional

Appendix

Table 9: Failure patterns and suggested checks. The list is expert-authored and has no annotation protocol or agreement statistic.

Treatment	Tasks	Ref.	Blocked	Diff.	95% CI
All trials, timeouts scored zero	15	0.752	0.662	$-0.090$	[ $-0.177$ , $-0.008$ ]
Timed-out trials excluded	15	0.752	0.673	$-0.079$	[ $-0.172$ , $+0.009$ ]
Paired, 14 complete tasks	14	0.755	0.658	$-0.096$	[ $-0.192$ , $-0.009$ ]

Appendix

Table 10: Eight-domain blocking against the matched reference arm.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,503 papers

ProtStructQA: A Denotation Threshold in Protein Structural Reasoning
May 30, 2026Aravind Mandiga, Guoming Li, Jin Lu +3LLM Evaluation Scientific QA
University of Georgia
Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools
Sep 5, 2026Bryce Cai, Geetha Jeyapragasan, Samira Nedungadi +2LLM Tool Use LLM Agent Evaluation
SecureBio, Cambridge, MA, USA.
GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models
Jun 11, 2026Gabriel Diaz-Ireland, Diego Prieto-Herráez, Mario García Peces +2LLM Agent Evaluation Geospatial Reasoning
Universidad Católica de Ávila (UCAV) Ávila, Spain · Johns Hopkins University Baltimore, MD, USA · Independent Researcher Madrid, Spain +1

May 30, 2026J/K move · Enter open · S save

ProtStructQA: A Denotation Threshold in Protein Structural Reasoning

Aravind Mandiga, Guoming Li, Jin Lu, Ismailcem Budak Arpinar+2

University of Georgia

LLM Evaluation Scientific QA

Open paper AskPDF arXiv

Protein-language systems are often evaluated by whether they generate plausible biological text, but a structural question has a sharper semantics: it denotes a measurement in a 3D coordinate system. We introduce ProtStructQA, an executable benchmark for protein structural question answering in which each natural-language question is generated from a hidden typed domain-specific language (DSL) program and the answer is obtained by executing that program on an AlphaFold-predicted structure. ProtStructQA releases 382.2K questions covering confidence, distances, predicted aligned error (PAE), solvent exposure, secondary structure, topology and contacts, and held-out compositions: a 330K active benchmark over 10K proteins from four species, plus a 52.2K hard-negative robustness pool. Without fine-tuning, we evaluate Qwen3 models from 0.6B to 8B under direct prompting, chain-of-thought, grammar-constrained executable voting, executable voting with chain-of-thought, and multi-turn ReAct-style tool use, and replicate the headline finding on Gemma-3-1B and Gemma-3-12B. We find a capability-dependent denotation threshold between Qwen3-1.7B and Qwen3-4B: below it, tool-mediated ReAct dominates because models often fail to produce executable denotations; above it, chain-of-thought flips from mostly harmful to strongly beneficial and becomes the strongest strategy on most splits. Parse-failure and family-level analyses show that the threshold is a transition from unparseable language to executable structural denotation, while grammar and execution remain selectively valuable for PAE and secondary-structure queries. ProtStructQA reframes scientific QA as compilation from language to measurement and provides a diagnostic testbed for when language models can map words to executable 3D structural measurements.

Abstract

AI Agent Benchmarks

Explore similar work

ProtStructQA: A Denotation Threshold in Protein Structural Reasoning

Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools

GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models