Recent work has demonstrated agents driving protein design workflows, either through orchestration of specialized tools or direct manipulation of sequences. Yet the fidelity with which agents follow protein design instructions and the factors contributing to their effectiveness remain understudied. We examine agentic antibody design under two settings: when explicit molecular-editing instructions are provided and when agents receive only previously-scored sequences and are prompted to propose improvements. We call the first task-guided and the second black-box design. For the task-guided setting, we curate ten expert-validated affinity-improvement tasks and show that LLM judges closely reproduce programmatic evaluations of instruction following. We then study two augmentations: access to antibody-relevant tools and to previously scored design history. Tools improve instruction following (IF) and sequence space exploration while reducing optimization performance, whereas design history reduces IF but improves optimization performance at the expense of exploration. Finally, we combine taskguided and black-box design within an in silico optimization loop and show that our agentic optimizer performs comparably to an established Bayesian optimization baseline.
Figures & tables
Judge
Recall (%)
Precision (%)
Refusal rate (%)
Claude Opus 5
100.0
100.0
0.0
Claude Opus 4.8
100.0
100.0
0.0
Claude Sonnet 5
100.0
100.0
12.0
GPT-5.6 Sol
100.0
100.0
0.0
GPT-5.6 Terra
100.0
100.0
0.0
GPT-5.6 Luna
100.0
94.6
0.0
Table 1: LLM-judge agreement with the programmatic checks over 1,900 (output, must-have) cells per judge: 50 (task, seed) pairs (10 tasks × 5 seeds), each holding 10 outputs scored against 3–4 must-haves. We report recall and precision with FAIL (a violation) as the positive class; refused cells yield no verdicts and are excluded from the analysis.
Figure 1: Surplus tool calls by LLM judges. Taking the number of tool calls required by the programmatic checks as the reference minimum; surplus is the number of issued calls above this minimum, averaged over tasks. We decompose surplus calls into redundancy (re-fetching a result already in hand), batching (issuing combinable calls separately instead of in one call), and wrong tool (calling a tool the task did not require).
Task only
Task + history
w/o tools
with tools
w/o tools
with tools
Model (effort)
ref. (%)
strict
partial
ref. (%)
strict
partial
ref. (%)
strict
partial
ref. (%)
strict
partial
Claude Opus 5
12
0.87
0.96
28
1.00
1.00
10
0.83
0.95
52
0.98
0.99
Claude Opus 5 (xhigh)
0
0.83
0.95
2
1.00
1.00
2
0.67
0.88
12
1.00
1.00
Claude Opus 4.8
40
0.82
0.95
42
0.98
0.99
38
0.51
0.83
60
1.00
1.00
GPT-5.6-sol
0
0.74
0.90
0
1.00
1.00
0
0.54
0.82
0
1.00
1.00
Table 2: Tool access improves instruction following. Strict and partial compliance rates over the 10 tasks, 5 seeds each, split by prior-history access ( Task only vs. Task + history ) and tool availability (with vs. w/o tools). ref. denotes the fraction of calls resulting in a refusal and excluded from the analysis. Reasoning effort is max unless otherwise specified.
Figure 2: Tools improve compliance, history improves oracle performance, and tasks explore where the black-box exploits. (a–c) Main effects of tools and of history, both measured for a task agent. Bars are the mean across six LLM models, whiskers ±1 SD across models. (d–e) Distributions across task-seed batches for the three conditions, pooled over both tool conditions. Each batch contains 10 variants; violins show the mean and ±1 SD, with means labelled. (d) Largest affinity gain per batch. (e) Number of distinct substitutions from the seed per batch.
Figure 3: Three optimizers reach comparable affinity by editing different parts of the domain. All arms share a 50-sequence initialization pool and spend 10 rounds of 50 oracle calls on the same objective; lines are the mean over three runs, bands ±1 s.d. (a) Affinity trajectories for Bayesian optimization over a GP surrogate, PABLO and the proposed agentic method. (b) Ablation of the proposed agentic method: black-box only uses scored history, while tasks only uses the four design briefs. (c) Distribution of edits in the final design round; the dashed line denotes the uniform baseline.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
measurement effort
candidate search
Model
ref. (%)
tool calls
measurements
re-meas.(%)
rounds
invented
discarded(%)
Task + history
Claude Opus 5
52
6.3±1.3
57.4±13.1
8.6±10.0
1.6±0.7
14.9±6.7
22.9±24.6
Claude Opus 5 (xhigh)
12
6.3±2.1
57.2±17.3
5.9±9.0
2.1±0.7
19.1±7.3
41.6±23.4
Claude Opus 4.8
60
7.7±3.1
79.0±23.1
11.5±12.6
1.9±0.9
19.9±7.6
43.4±22.7
GPT-5.6-sol
0
10.8±4.2
129.3±40.6
8.4±7.1
2.9±1.1
39.9±14.2
71.5±13.1
Appendix
Table 3: Tool use across context conditions. Tool-use metrics from Sec. E.1 , mean ± one SD across completed worker calls. Refused calls are excluded from every metric column and appear only in ref . Reasoning effort is max unless marked xhigh . The three arms differ in what the worker receives: task + history , task only , and black-box .
Model (effort)
tool calls
re-meas. (%)
rounds
invented
abandoned (%)
Claude Opus 5 (xhigh)
10.0±1.8
0.0±0.0
5.0±1.3
62.0±5.4
83.7±1.4
Claude Opus 5 (max)
10.6±1.2
0.0±0.0
5.0±1.1
81.6±16.6
87.2±2.6
GPT-5.6-sol (max)
18.6±5.6
6.7±2.0
8.6±3.9
238.4±136.1
94.4±2.7
GPT-5.6-terra (max)
18.4±4.2
7.1±2.7
7.8±1.5
135.2±39.0
91.9±2.5
Appendix
Table 4: Tool use on Rigidifying Disulfide. The task excluded from Tab. 3 , reported on the same definitions. Its must-haves require both a correctly placed pair of new cysteines and a disulfide bond confirmed on the predicted structure, and it is the only task where instruction following without tools falls below 10% . Every model responds by generating far more candidates than on any other task - 3.8 to 8.3 times its own nine-task mean- and discarding 84 – 94% of them.
Figure 4: Without tools, failures split into region errors and count errors. Instruction-following failure modes for the four most challenging tasks without tools, shown for Opus 5 (xhigh) and GPT-5.6 sol. Each cell is 50 outputs (5 seeds × 10 outputs). location (LOC) failures edit outside the region the task names, while composition (COMP) failures edit the right region but miss its target counts/ranges; the grey text above each task gives the task-specific form of both.
Figure 5: Black-box setting: tools do not help oracle performance. The black-box agent run on 5 seeds, with 10 sequences generated per seed. Bars are means over the 5 input seeds with ±1 SD whiskers; solid bars withhold tools, hatched bars provide them, and colour marks the model family (Claude blue, GPT orange). Candidates beating the seed is the fraction of outputs improving on the seed; best affinity gain is the largest improvement among the 10 outputs. Effort is max unless noted. ∗ Claude Opus 5 ( max ) averages four seeds in the with-tools setting, as one seed resulted in a refusal.
We introduce ABLE, a benchmark for evaluating LLM agents' ability to use biological AI models (BAIMs), such as ProteinMPNN and AlphaFold3, in dual-use protein design workflows. ABLE assesses agent performance through a set of tasks spanning structure retrieval, sequence generation, and design validation. We evaluate 15 frontier models and find that seven refuse all tasks, while the remaining models exhibit substantial performance differences. Claude Sonnet 4 and Gemini 3 Pro achieve the highest scores across information retrieval, tool selection, and tool use. We further compare model performance on a subset of tasks against an expert human baseline. Our results suggest that current LLMs can substantially lower barriers to protein design, but remain inconsistent in planning, strategy generation, and integrating biological knowledge with tool use.
Protein language models (PLMs) are passive oracles: they generate sequences in a single forward pass with no mechanism to consult external biophysical feedback or redirect generation when a candidate violates thermodynamic or structural constraints. We introduce AgentPLM, which addresses this by equipping a pre-trained PLM with i) Reasoning-Augmented Decoding (RAD), which interleaves autoregressive generation with tool calls (ESMFold, FoldX, AutoDock Vina), and ii) Contrastive Agent Policy Optimisation (CAPO), a trajectory-level extension of direct preference optimisation that trains the policy end-to-end to learn when oracle feedback is informative rather than merely imitating high-fitness sequences. We evaluate AgentPLM on benchmark tasks spanning de novo enzyme design, antibody optimisation, thermostability, PPI interface design, and zero-shot fitness prediction with standardised oracle APIs and controlled sequence-identity splits. AgentPLM achieves state-of-the-art results with a gain in antibody top-10% hit rate over the strongest passive baseline, providing mechanistic evidence of online error correction without explicit backtracking.
Sahil Rahman, Maxx Richard Rahman
Bedford College, London, United Kingdom · Saarland University, Saarbrücken, Germany
Large language models (LLMs) are capable of emulating reasoning and using tools, creating opportunities for autonomous agents that execute complex scientific tasks. Protein design provides a natural testbed: although machine learning (ML) methods achieve strong results, these are largely restricted to canonical amino acids and narrow objectives, leaving unfilled need for a generalist tool for broad design pipelines. We introduce Agent Rosetta, an LLM agent paired with a structured environment for operating Rosetta, the leading physics-based heteropolymer design software, capable of modeling non-canonical building blocks and geometries. Agent Rosetta iteratively refines designs to achieve user-defined objectives, combining LLM reasoning with Rosetta's generality. We evaluate Agent Rosetta on design with canonical amino acids, matching specialized models and expert baselines, and with non-canonical residues -- where ML approaches fail -- achieving comparable performance. Critically, prompt engineering alone often fails to generate Rosetta actions, demonstrating that environment design is essential for integrating LLM agents with specialized software. Our results show that properly designed environments enable LLM agents to make scientific software accessible while matching specialized tools and human experts.
Jacopo Teneggi, S. M. Bargeen A. Turzo, Tanya Marwah +4
Polymathic AI · Johns Hopkins University · Center for Computational Biology, Flatiron Institute +3