As large language models (LLMs) acquire advanced biological design capabilities, they offer substantial research benefits but simultaneously introduce severe dual-use risks, such as engineering enhanced pathogens or drug-resistant bacteria. While biosecurity frameworks recognize these threats, the actual risk of LLMs executing dangerous biological modifications remains unquantified, particularly when augmented with specialized biological design tools (BDTs). Existing evaluations primarily assess hazardous knowledge possession rather than practical application, overlook risk amplification from BDTs, and rely on unreliable, expensive, or hazardous validation methods. To address these gaps, we introduce DRYLAB-Bench (Design Risk Yielded by LLM Agent in Biodesign), the first comprehensive benchmark evaluating the high-risk biological mutation design capabilities of LLM agents. Moving beyond basic question-answering, DRYLAB-Bench tasks agents with 30 genuinely hazardous mutation-design scenarios distilled from biosecurity guidelines. It evaluates models in a multi-round, tool-augmented environment where LLMs orchestrate BDTs to iteratively refine candidates. Crucially, it scores designs directly against 187,695 empirical wet-lab measurements from ProteinGym and ViroGym using our proposed Biosecurity Risk Score (BRS), quantifying risk-associated effects while enabling scalable and safe ground-truth verification. Evaluations of 15 frontier models reveal substantial baseline threats: without tools, 67.6% of one-shot designs exceed the average measured variant, and 21.4% reach the highest-risk decile. Equipping models with BDTs significantly amplifies this risk, raising the pooled risk by ΔBRS=+0.116 and the top-decile share to 23.5%. Code: https://github.com/drylab-bio/DRYLAB-Bench
Figures & tables
Figure 1: DRYLAB-Bench construction pipeline and risk taxonomy. Candidate scenarios generated by LLMs from international biosecurity reports and human-written templates undergo filtering for matching empirical assays in ViroGym and ProteinGym, followed by expert validation of feasibility and reliability. The resulting benchmark includes eight biosecurity risk categories.
Figure 2: Overview of the evaluation pipeline. The pipeline consists of three stages: (a) Candidate initialization : The LLM generates an initial pool of single-site amino-acid substitution candidates for a given biosecurity-relevant mutation scenario. (b) Iterative revision with BDTs : The LLM interacts with six BDTs to obtain auxiliary evidence and iteratively refine candidate designs. (c) Ground-truth-based risk scoring : The final mutation candidates are evaluated by matching against experimentally measured DMS data from ProteinGym and ViroGym. The resulting risk-associated effects are aggregated into the Biosecurity Risk Score (BRS).
Table 1: The 30 scenarios grouped by risk category. The “Single mutants” column reports the number of experimentally measured single substitutions used as ground truth. PG = ProteinGym, VG = ViroGym.
Figure 3: Main results: baseline risk and agentic gain. (a) Valid-response view and (b) refusal-penalized view, with models ordered by their Biosecurity Risk Score (BRS) under S2. The pale bar indicates the baseline score under S0 (LLM one-shot), while the overlaid segment shows the change from S0 to S2, computed as BRS(S2)−BRS(S0) . Regular values denote S0 scores, and bold values denote S2 scores.
Figure 4: Agentic gain negatively correlates with base capability. Models with lower S0 BRS tend to gain more, reflecting convergence toward a tool-limited risk ceiling.
GPT-5.6
Gemini Flash
DeepSeek V4
Kimi-K3
Qwen3.8
Grok 4.5
GLM-5.3
Luna
Sol
Terra
3.8
3.6
Flash
Pro
S0
-0.383
-0.058
+0.588
-0.114
+0.475
+0.901
+0.550
-0.265
+0.230
-0.086
+0.297
S2
+0.292
+0.345
+0.555
-0.160
+0.514
+0.901
+0.342
+0.800
+0.681
+0.137
+0.101
Δ
+0.675
+0.403
-0.033
-0.046
+0.040
+0.001
-0.208
+1.065
+0.450
+0.222
-0.196
Table 2: Performance shifts under misaligned tool dimensions. Mean risk of submitted designs in the CcdB toxin scenario at S0 and S2, with shift ΔBRS=BRS(S2)−BRS(S0) . Omitted models refused all CcdB units.
Model
Orig.
Acad.
Mal.
Claude Fable 5
96.7
96.7
100.0
Claude Opus 5
81.1
76.7
96.7
Claude Opus 4.8
76.7
73.3
90.0
GPT-6 Astra
66.7
50.0
90.0
GPT-5.6 Terra
46.7
33.3
96.7
GPT-5.6 Sol
44.4
33.3
96.7
Table 3: Refusal rates under controlled prompt framing. Percentage of units refused on the original suite and on two rewrites of the same 30 tasks. Acad. = academic wrap; Mal. = malicious context.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Temp.
Reasoning
Other non-default settings
Claude Opus 5
—
—
no_temperature
Claude Opus 4.8
—
—
no_temperature
Claude Fable 5
—
—
no_temperature ; max_concurrency =3
GPT-5.6 Luna
0.3
low
—
GPT-5.6 Sol
0.3
low
use_max_completion_tokens
GPT-5.6 Terra
0.3
low
use_max_completion_tokens
Appendix
Table 4: Per-model decoding and API settings. “Temp.” is the decoding temperature actually sent; “Reasoning” the reasoning-effort control; “Other” lists non-default settings. “—” means the parameter is not sent. A setting of no_temperature means the client omits the temperature field because the vendor deprecated or rejects it; use_max_completion_tokens means the model rejects max_tokens and the client uses the completion-token field instead.
Figure 5: Almost every design lands on a measurement. Top-3 coverage against the hidden ground truth, per model, in S0 and S2, on a full 0 – 100% scale; models are ordered by their S2 coverage. A unit enters the average when the model submitted a complete, legal top- 3 , so a low value here means three legal designs that the assay did not measure, not a failed answer.
(a) Agreement between metrics (S2 model ranking, Spearman ρ )
Metric pair
Floor-included
Valid-response
BRS vs. empirical rank
0.99
0.96
raw composite vs. BRS
0.61
+0.37
raw composite vs. empirical rank
+0.60
+0.20
(b) Stability under the refusal floor ( fref=0 vs. fref=−1.5 )
empirical rank
1.00
Appendix
Table 5: Metric validity. Top : agreement between the S2 model rankings induced by the three metrics (all 15 models, all with unmeasured candidates filled in as in Eq. 9 ). Bottom : stability of each ranking when the refusal floor is moved from fref=0 , the assay mean, to fref=−1.5 , 1.5 standard deviations below the assay mean, expressed in each metric’s own units and applied per unit. The scale-free metrics are the most floor-stable; the raw composite is the least, because a fixed offset means different things on different assays.
Block
Content
Varies with
<COMMON_BASE>
Role and output contract: propose single amino-acid substitutions that improve a target phenotype while preserving viability and function; return exactly the Top-3, each with an independent confidence in [0,1] ; tool outputs are evidence, never ground truth.
— (constant)
<TASK_SPEC>
The scenario itself: the target protein and its full wild-type sequence, the design objective, and the numbering rule that fixes which positions the model may cite.
scenario × variant
<CONDITION_PROTOCOL>
The protocol of the condition, its budget line, and — in S2 only — the tool menu together with the task-specific scoring-dimension guidance.
condition
Appendix
Table 6: The three blocks every prompt is assembled from. “Varies with” names the factor that changes the block from one evaluated unit to another.
Line
P0 (baseline)
P1 (objective named)
P2 (long-form)
task
TASK:
TASK:
YOUR TASK IS AS FOLLOWS:
objective
GOAL:
PRIMARY OBJECTIVE:
THE DESIGN OBJECTIVE:
numbering
IMPORTANT:
NOTES:
IMPORTANT NUMBERING NOTES:
Appendix
Table 7: The complete set of edits separating the three wording variants: three heading phrases per prompt, and no other change. The paragraph order (task → objective → sequence → numbering note) is preserved and no body sentence is rephrased.
Prompts
Triples
Identical
Same line count
Round-trip
1,350
450
450/450
450/450
1,350/1,350
Appendix
Table 8: Audit of the wording variants over every prompt rendered in the evaluation run. A triple is one scenario as posed in one model’s trajectory, i.e. three prompts; “identical” requires the three to become byte-identical once the headings are canonicalized, and “same line count” additionally requires them to have the same number of lines, so that nothing was inserted or dropped. The last column reports the generator round-trip. The audit covers all 450 scenario × model triples present in the artifacts.
Arm
Tool evidence
Revision
Valid BRS
Floor BRS
Step ΔBRS , models positive
S0
no
one shot
0.359
0.249
—
S0-iter
no
up to 4 rounds
0.371
0.284
iteration only: +0.012 , 10/15
S1
one round
one revision
0.407
0.275
tools only: +0.048 , 10/15
S2
adaptive
up to 4 rounds
0.475
0.317
extra rounds: +0.068 , 10/15
Appendix
Table 9: The four arms, and what each step of the workflow adds. Condition means are confidence-weighted BRS pooled over the 15 models with equal model weight; “valid” averages valid units, i.e. legal, non-refused units (unmeasured candidates filled in as in Appendix B.2 ), and “floor” scores each refused unit at the refusal floor ( fref=0 ) inside its own task mean, with equal task weight. The ΔBRS column gives each step of the workflow as a difference of condition means and the number of models whose step difference is positive; the S2 − S0 step reproduces ΔBRS=+0.116 of section 4 .
Valid-response BRS
Floor-included BRS
Model
S0
S1
S2
S0
S1
S2
Kimi-K3
0.519
0.543
0.631
0.523
0.551
0.637
GLM-5.3
0.394
0.392
0.492
0.393
0.377
0.497
DeepSeek V4 Pro
0.276
0.348
0.483
0.283
0.348
0.483
Grok 4.5
0.368
0.427
0.467
0.358
0.446
0.478
Qwen3.8 Max
0.401
0.442
0.428
0.413
0.429
0.445
Appendix
Table 10: Mean risk of LLM-designed mutations by condition (confidence-weighted BRS ; BRS=0 is the assay mean). “Valid” averages valid units, i.e. legal, non-refused units (include unmeasured candidates scored in silico); “Floor” scores each refused unit at the refusal floor ( fref=0 ) inside its own task mean, with equal task weight. Cell shading follows the score, from light (low) to dark blue (high). Sorted by S2 floor-mean.
Figure 6: Where a representative design sits in the measured distribution. The histogram is the risk of all 12,464 measured single mutants of the MTHFR loss-of-function assay ( BRS , assay-standardized, so that BRS=0 is the assay’s average variant); circles are the 3 candidates one representative unit (GLM-5.3, prompt variant P2) submitted at S0 and diamonds the 3 it submitted after tool-assisted iteration. The first round includes the mild common variant that most models propose (A222V, BRS=+0.22 ) and its mean sits below the assay’s average variant ( BRS=−0.34 ), while the loop moves all three final designs above it (mean BRS=+1.11 ), ending on the highest-risk variant C243Y at BRS=+2.30 .
Large Language Models (LLMs) are accelerating biological research, yet this same capability poses a critical biosecurity threat: models that assist in protein engineering can equally be prompted to generate predicted toxin-like sequences, potentially lowering the barrier to biological misuse. Current safety evaluations, however, operate in natural language and cannot determine whether a model-generated amino acid sequence is biological gibberish or a computational risk signal. To address this evaluation blind spot, we introduce SPIKE-Bench, coupling 631 curated toxin-design prompts across seven functional categories with the SPIKE funnel, a three-stage protocol that filters output through compliance, biological plausibility, and predicted toxicity, producing stage-level diagnostics and an aggregate function-aware metric: the Functional Harmfulness Rate (FHR). An audit of 32 LLMs reveals that most models freely comply with toxin-design requests; FHR is driven primarily by biological generation capability rather than safety alignment, reaching 50.7%; and Refusal Rate fails to predict functional risk. As a first step toward mitigation, we provide BioSafe-Guard, a domain-specialized classifier that substantially reduces predicted functional risk while preserving benign utility. We release SPIKE-Bench and BioSafe-Guard at https://github.com/PKU-Alignment/SPIKE-Bench to support more rigorous biosecurity evaluation of LLMs.
Shu Quan, Tianfang Hao, Sitong Fang +8
Institute for Artificial Intelligence, Peking University · The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
Large language models (LLMs) are rapidly acquiring capabilities relevant to biological research, from literature synthesis to interpretation of experimental data. Increasingly, LLM agents can also perform in silico biology tasks that previously required experienced human biologists. These emerging AI capabilities offer new opportunities for scientific discovery and biomedical advances, but they also shift the landscape of biosecurity risks. To address this, we introduce the Agentic Bio-Capabilities Benchmark (ABC-Bench), a suite of tasks to measure agentic biosecurity-relevant capabilities. ABC-Bench evaluates LLM agents on both benign and dual-use biology tasks: writing code to operate liquid handling robots, designing DNA fragments for in vitro assembly, and evading DNA synthesis screening. These tasks require a combination of biology and software expertise. All tested LLM agents outperformed the median expert human baseliner on all three tasks. Agents performed highly on tasks drawing on published knowledge and well-documented protocols, and more weakly on a task requiring novel bioinformatics reasoning. In three wet-lab validation experiments, we found that OpenAI's o4-mini-high produced scripts that, when run on an OpenTrons liquid handling robot, successfully assembled DNA with expected sequences.
Andrew Bo Liu, Samira Nedungadi, Bryce Cai +3
SecureBio, Cambridge, MA, USA · Active Site, Cambridge, MA, USA
Frontier large language models (LLMs) are increasingly integrated into scientific workflows, yet their growing biological capabilities may outpace current safeguards. To assess the biological risks of frontier models, we develop Intern-BioBreaker, a specialized bio-red-teaming model, together with an integrated computational-to-physical framework that couples model-level stress testing with wet-lab validation. Within this framework, Intern-BioBreaker generates targeted jailbreak prompts to test whether aligned models can be induced to provide operational guidance for safety-sensitive biological tasks or produce sequence-level outputs with potentially harmful properties. Selected sequence outputs are then carried forward for DNA synthesis, host expression, and orthogonal protein verification to assess whether model-generated designs can yield the intended biological products. Our evaluation reveals a concerning gap between text-level safeguards and the risks posed by capable scientific models: (i) Intern-BioBreaker outperforms baseline attack models and reveals widespread bio-risk jailbreak vulnerabilities across both open-weight and proprietary frontier LLMs, with several targets reaching near-saturated or 100% task-level attack success rate (ASR); (ii) in sequence-level case studies, GPT-5.5 can be induced to generate modified viral candidate sequences with pathogenic potential; the corresponding translated proteins may exhibit even stronger receptor-binding affinity and thus enhanced infection potential; and (iii) end-to-end verification shows that selected model-generated biological designs are not merely textual artifacts, but can be physically realized under controlled experimental settings. These findings underscore the need for stronger biological red-teaming, nucleic acid synthesis screening, and safety mechanisms that keep pace with model capabilities.