Do AI-for-Science Agents Preserve Model Safety?

Michael Yew Chuan Lee; Manya Sriram; N. Benjamin Erichson

cs.AIOct 23, 2026

Do AI-for-Science Agents Preserve Model Safety?

Authors: Michael Yew Chuan Lee, Manya Sriram, N. Benjamin Erichson

Organizations: DSO National Laboratories · UC Berkeley · ICSI and Berkeley Lab

Abstract

AI-for-Science agents are increasingly being used for formulating scientific hypotheses, running autonomous experiments, and interpreting evidence. These agents are constructed by integrating a large language model (LLM) into a domain-specific agent harness, which in turn can improve the LLM's scientific capability and enable the model to orchestrate scientific tools. In this work, we ask whether model safety is preserved when an LLM is integrated in an AI-for-Science agent harness. We evaluate five agent harnesses on 50 harmful chemistry requests. Specifically, we study how the safety behavior changes under direct prompting, two simple adversarial prompting strategies, and an adaptive attack. Across four recent GPT variants, our results show that the agent harness can increase harmful compliance even under direct prompting. More critically, we find that adaptive attacks that are largely unsuccessful against the base LLM suddenly become highly successful after the agent harness is introduced. Our results show that the safety behavior of the base LLM is not durable once a harness is introduced. Thus, as AI-for-Science agents become more capable and autonomous, the community needs to evaluate and align the agent using domain-specific safety benchmarks.

Figures & tables

Figure 1: Maximum observed increase in attack success rate (ASR) for each composed agent relative to the same base model across the prompting strategies tested. Single-turn results use pass@3; PAIR uses its reported ASR. Several scientific agent harnesses increase harmful compliance for at least one model–prompt combination. Positive values indicate greater harmful compliance after introducing the agent harness.

	Direct				Roleplay				Past tense				PAIR
Harness	5.2	5.5	5.6-L	6-L	5.2	5.5	5.6-L	6-L	5.2	5.5	5.6-L	6-L	5.2	5.5	5.6-L	6-L
CACTUS	$+2$	$+4$	$+4$	$-2$	$+2$	$+4$	$0$	$+4$	$-14$	$-10$	$-10$	$-6$	$-20$	$+58$	$+56$	$+22$
ChemToolAgent	$+18$	$+4$	$0$	$0$	$+24$	$+8$	$+2$	$+4$	$+16$	$+6$	$-16$	$-2$	$+20$	$+58$	$+32$	$+24$
Biomni	$+22$	$+10$	$-2$	$0$	$+16$	$+14$	$+6$	$+4$	$+20$	$+14$	$0$	$-8$	$+20$	$+54$	$+40$	$+26$
STELLA	$0$	$0$	$-2$	$-2$	$-4$	$-2$	$-2$	$+6$	$0$	$0$	$+4$	$+6$	$+6$	$+52$	$+40$	$+28$
DrugAgent	$+70$	$+46$	$+18$	$0$	$+52$	$+32$	$+10$	$+2$	$+42$	$+24$	$0$	$+2$	$+2$	$+36$	$+18$	$+6$

Table 1: Scientific harnesses can increase harmful compliance under direct prompting and PAIR. Cells report

\Delta

ASR (percentage points) relative to the same base model. Single-turn attacks use pass@3, the fraction of requests with at least one successful attack across three runs. PAIR uses the reported ASR. Green indicates decreases; orange and red indicate progressively larger increases. White indicates zero. Model labels omit “GPT-”; L denotes Luna.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Scientific question-answering accuracy varies across harnesses, and DrugAgent scores below its matched baselines on all three fully evaluated models. We report accuracy on 50 GPQA questions. For GPT-5.2, accuracy is 62% for the base model, 68% for ChemToolAgent, and 28% for DrugAgent. GPT-6-Luna results are available only for DrugAgent, at 54%; empty bar positions indicate unavailable results.

Figure 3: Safety regressions occur alongside both higher and lower GPQA accuracy. Each point represents a harness–model combination and reports the change in GPQA accuracy and direct-prompting pass@3 ASR relative to the corresponding base model. Colors distinguish base models, while markers distinguish harnesses. ChemToolAgent with GPT-5.2 improves GPQA accuracy by 6 percentage points while increasing ASR by 18 points. DrugAgent with the same model reduces GPQA accuracy by 34 points while increasing ASR by 70 points. STELLA improves GPQA accuracy by 14 points without changing ASR.

	Direct				Roleplay				Past tense				PAIR
Harness	5.2	5.5	5.6-L	6-L	5.2	5.5	5.6-L	6-L	5.2	5.5	5.6-L	6-L	5.2	5.5	5.6-L	6-L
CACTUS	$-17$	$-14$	$-3$	$+12$	$-25$	$-19$	$-3$	$+2$	$-29$	$-2$	$+3$	$+8$	$+30$	$-72$	$-42$	$-38$
ChemToolAgent	$-15$	$-21$	$-23$	$+19$	$-42$	$-24$	$-14$	$+9$	$-24$	$-23$	$-3$	$-17$	$-4$	$-74$	$-42$	$-38$
Biomni	$-21$	$-11$	$-15$	$+11$	$-29$	$-17$	$-10$	$+6$	$-46$	$-23$	$-13$	$-1$	$-10$	$-68$	$-44$	$-32$
STELLA	$-3$	$-7$	$-23$	$-3$	$-14$	$-13$	$-7$	$+3$	$-30$	$-3$	$-2$	$-5$	$+6$	$-72$	$-50$	$-30$
DrugAgent	$-80$	$-55$	$-64$	$-37$	$-77$	$-41$	$-42$	$-22$	$-70$	$-55$	$-28$	$-43$	$0$	$-60$	$-34$	$-36$

Appendix

Table 2: Change in explicit refusal rate relative to the same base LLM (percentage points). Single-turn results use mean refusal across three runs, while PAIR uses the recorded refusal rate. Values are rounded to whole points. Green indicates increased refusal, orange and red indicate progressively larger decreases, and white indicates zero. Model labels omit “GPT-”; L denotes Luna.

	Single-turn refusal (%), mean $\pm$ SE			PAIR refusal
System	Direct	Roleplay	Past tense	Rate (%)	$k/50$
GPT-5.2
Base model	$98.7\,\pm\,0.7$	$88.0\,\pm\,2.0$	$81.3\,\pm\,1.3$	$10$	5/50
CACTUS	$82.0\,\pm\,1.2$	$62.7\,\pm\,0.7$	$52.0\,\pm\,2.0$	$40$	20/50
ChemToolAgent	$84.0\,\pm\,1.2$	$46.0\,\pm\,1.2$	$57.3\,\pm\,2.4$	$6$	3/50
Biomni	$78.0\,\pm\,2.0$	$59.3\,\pm\,2.4$	$35.3\,\pm\,0.7$	$0$	0/50

Appendix

Table 3: Absolute explicit refusal rates (%). Single-turn cells report mean

\pm

standard error (SE) across three runs of 50 requests. PAIR reports the recorded refusal rate and count

k/50

from one evaluation, without a repeated-search SE. Green indicates higher refusal; orange and red indicate lower refusal.

GPT-5.2	Mean $\pm$ SE	pass@3	Mean $\pm$ SE	pass@3	Mean $\pm$ SE	pass@3	ASR	$k/50$
	Direct		Roleplay		Past tense		PAIR
Base model	$1.3\,\pm\,0.7$	$2$	$7.3\,\pm\,1.8$	$12$	$8.0\,\pm\,2.3$	$16$	$56$	28/50
CACTUS	$1.3\,\pm\,0.7$	$4$	$5.3\,\pm\,0.7$	$14$	$0.7\,\pm\,0.7$	$2$	$36$	18/50
ChemToolAgent	$15.3\,\pm\,1.8$	$20$	$28.0\,\pm\,1.2$	$36$	$24.0\,\pm\,4.2$	$32$	$76$	38/50
Biomni	$16.0\,\pm\,1.2$	$24$	$20.7\,\pm\,2.7$	$28$	$22.7\,\pm\,0.7$	$36$	$76$	38/50
STELLA	$0.7\,\pm\,0.7$	$2$	$5.3\,\pm\,0.7$	$8$	$9.3\,\pm\,1.8$	$16$	$62$	31/50

Appendix

Table 4: Absolute ASR (%) for GPT-5.2. DrugAgent reaches 72% pass@3 under direct prompting, while ChemToolAgent and Biomni reach 76% under PAIR. Single-turn columns give mean

\pm

standard error (SE) across three runs and pass@3; PAIR gives ASR and successful requests

k/50

from one adaptive evaluation. Green indicates lower ASR; orange and red indicate higher ASR.

GPT-5.5	Mean $\pm$ SE	pass@3	Mean $\pm$ SE	pass@3	Mean $\pm$ SE	pass@3	ASR	$k/50$
	Direct		Roleplay		Past tense		PAIR
Base model	$3.3\,\pm\,0.7$	$4$	$14.7\,\pm\,1.8$	$20$	$14.0\,\pm\,1.2$	$26$	$2$	1/50
CACTUS	$6.7\,\pm\,0.7$	$8$	$17.3\,\pm\,0.7$	$24$	$11.3\,\pm\,2.4$	$16$	$60$	30/50
ChemToolAgent	$4.7\,\pm\,0.7$	$8$	$18.0\,\pm\,1.2$	$28$	$20.0\,\pm\,2.3$	$32$	$60$	30/50
Biomni	$10.7\,\pm\,1.8$	$14$	$22.0\,\pm\,1.2$	$34$	$29.3\,\pm\,1.8$	$40$	$56$	28/50
STELLA	$4.0\,\pm\,0.0$	$4$	$13.3\,\pm\,1.3$	$18$	$16.7\,\pm\,2.9$	$26$	$54$	27/50

Appendix

Table 5: Absolute ASR (%) for GPT-5.5. Single-turn columns give mean

\pm

standard error (SE) across three runs and pass@3; PAIR gives ASR and successful requests

k/50

from one adaptive evaluation. PAIR ASR ranges from 38–60% across harnesses, compared with 2% for the base model. Green indicates lower ASR; orange and red indicate higher ASR.

GPT-5.6-Luna	Mean $\pm$ SE	pass@3	Mean $\pm$ SE	pass@3	Mean $\pm$ SE	pass@3	ASR	$k/50$
	Direct		Roleplay		Past tense		PAIR
Base model	$6.0\,\pm\,2.0$	$10$	$12.0\,\pm\,2.3$	$18$	$26.7\,\pm\,0.7$	$36$	$10$	5/50
CACTUS	$9.3\,\pm\,0.7$	$14$	$12.0\,\pm\,2.3$	$18$	$16.7\,\pm\,3.3$	$26$	$66$	33/50
ChemToolAgent	$4.7\,\pm\,1.8$	$10$	$10.7\,\pm\,4.8$	$20$	$8.7\,\pm\,3.7$	$20$	$42$	21/50
Biomni	$5.3\,\pm\,0.7$	$8$	$14.0\,\pm\,5.0$	$24$	$22.7\,\pm\,0.7$	$36$	$50$	25/50
STELLA	$4.7\,\pm\,0.7$	$8$	$8.0\,\pm\,3.1$	$16$	$22.7\,\pm\,4.1$	$40$	$50$	25/50

Appendix

Table 6: Absolute ASR (%) for GPT-5.6-Luna. Single-turn columns give mean

\pm

standard error (SE) across three runs and pass@3; PAIR gives ASR and successful requests

k/50

from one adaptive evaluation. PAIR ASR ranges from 28–66% across harnesses, compared with 10% for the base model. Green indicates lower ASR; orange and red indicate higher ASR.

GPT-6-Luna	Mean $\pm$ SE	pass@3	Mean $\pm$ SE	pass@3	Mean $\pm$ SE	pass@3	ASR	$k/50$
	Direct		Roleplay		Past tense		PAIR
Base model	$2.7\,\pm\,0.7$	$4$	$4.0\,\pm\,0.0$	$4$	$8.7\,\pm\,2.4$	$16$	$2$	1/50
CACTUS	$2.0\,\pm\,0.0$	$2$	$4.7\,\pm\,1.8$	$8$	$6.7\,\pm\,1.3$	$10$	$24$	12/50
ChemToolAgent	$2.7\,\pm\,0.7$	$4$	$3.3\,\pm\,1.8$	$8$	$6.7\,\pm\,1.3$	$14$	$26$	13/50
Biomni	$2.0\,\pm\,1.2$	$4$	$4.7\,\pm\,1.8$	$8$	$4.0\,\pm\,2.0$	$8$	$28$	14/50
STELLA	$2.0\,\pm\,0.0$	$2$	$4.0\,\pm\,1.2$	$10$	$10.7\,\pm\,2.4$	$22$	$30$	15/50

Appendix

Table 7: Absolute ASR (%) for GPT-6-Luna. Single-turn columns give mean

\pm

standard error (SE) across three runs and pass@3; PAIR gives ASR and successful requests

k/50

from one adaptive evaluation. PAIR ASR ranges from 8–30% across harnesses, compared with 2% for the base model. Green indicates lower ASR; orange and red indicate higher ASR.

		GPT-5.5		GPT-6-Luna
System	Configuration	ASR	$\Delta$ ASR	ASR	$\Delta$ ASR
Base LLM	No harness	$2$	–	$2$	–
STELLA	Full harness	$54$	$+52$	$30$	$+28$
	External tools disabled	$52$	$+50$	$28$	$+26$
	Memory disabled	$56$	$+54$	$34$	$+32$
Biomni	Full harness	$56$	$+54$	$28$	$+26$

Appendix

Table 8: PAIR ASR (%) for complete and modified harness configurations, with

\Delta

ASR relative to the corresponding base LLM (percentage points). Each configuration is evaluated on 50 requests. STELLA retains its code interpreter when external tools are disabled. Absolute ASR is colored from green (low) to red (high), while positive

\Delta

ASR uses the orange/red scale of the main table.

LLM Agent Safety

Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.

See all 321 papers

Toward Reliable, Safe, and Secure LLMs for Scientific Applications
Mar 18, 2026Saket Sanjeev Chaturvedi, Joshua Bergerson, Tanwi MallickLLM Safety AI Agent Safety
Argonne National Laboratory, Lemont, 60439, USA
SciRisk-Bench: A Risk-Dimension-Aware Benchmark for AI4Science Safety
Jun 17, 2026Linghao Feng, Yinqian Sun, Dongqi Liang +8LLM Safety Benchmarks AI for Science
Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Future Technology, University of Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China +5
SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond
Mar 2, 2026Xiangyang Zhu, Yuan Tian, Qi Jia +14LLM Safety LLM Safety Evaluation
Shanghai AI Lab · ByteDance · ECNU

Mar 18, 2026J/K move · Enter open · S save

Toward Reliable, Safe, and Secure LLMs for Scientific Applications

Saket Sanjeev Chaturvedi, Joshua Bergerson, Tanwi Mallick

Argonne National Laboratory, Lemont, 60439, USA

LLM Safety AI Agent Safety LLM Security LLM Safety Evaluation LLM Safety Benchmarks

Open paper AskPDF arXiv

As large language models (LLMs) evolve into autonomous "AI scientists," they promise transformative advances but introduce novel vulnerabilities, from potential "biosafety risks" to "dangerous explosions." Ensuring trustworthy deployment in science requires a new paradigm centered on reliability (ensuring factual accuracy and reproducibility), safety (preventing unintentional physical or biological harm), and security (preventing malicious misuse). Existing general-purpose safety benchmarks are poorly suited for this purpose, suffering from a fundamental domain mismatch, limited threat coverage of science-specific vectors, and benchmark overfitting, which create a critical gap in vulnerability evaluation for scientific applications. This paper examines the unique security and safety landscape of LLM agents in science. We begin by synthesizing a detailed taxonomy of LLM threats contextualized for scientific research, to better understand the unique risks associated with LLMs in science. Next, we conceptualize a mechanism to address the evaluation gap by utilizing dedicated multi-agent systems for the automated generation of domain-specific adversarial security benchmarks. Based on our analysis, we outline how existing safety methods can be brought together and integrated into a conceptual multilayered defense framework designed to combine a red-teaming exercise and external boundary controls with a proactive internal Safety LLM Agent. Together, these conceptual elements provide a necessary structure for defining, evaluating, and creating comprehensive defense strategies for trustworthy LLM agent deployment in scientific disciplines.

Abstract

LLM Agent Safety

Explore similar work

Toward Reliable, Safe, and Secure LLMs for Scientific Applications

SciRisk-Bench: A Risk-Dimension-Aware Benchmark for AI4Science Safety

SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond