cs.AIOct 14, 2026

AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety

Authors: Chenxu Wang, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Cheng Jin, Chaochao Lu, Qiaosheng Zhang

Organizations: Shanghai Artificial Intelligence Laboratory · Fudan University · Shanghai Jiaotong University · Zhejiang University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.

Figures & tables

Figure 1: From language-level to agent-level co-evolution. At the language level, the attacker generates adversarial prompts and the defender produces safe responses. In AgentSafeEvo, the attacker constructs adversarial tasks, while the defender plans and uses tools to complete tasks safely. Co-evolution therefore extends from prompt–response interactions to task construction and execution.
Figure 2: Overview of AgentSafeEvo. Phase 1 initializes the attacker through supervised fine-tuning on synthesized adversarial tasks. Phase 2 jointly trains the attacker and defender using GRPO over multi-turn tool interactions. Defender rewards combine safety and utility, while the attacker reward for each task is the negative mean defender reward. Learnability-guided task weighting uses within-task defender-reward standard deviations to compute a shared task-specific weight for both policies’ GRPO advantages.
 Agent-SafetyBenchAgentHarmAgentDojoMSB
MethodASR↓\mathrm{ASR}\downarrowHarm score↓\mathrm{Harm\ score}\downarrowASR↓\mathrm{ASR}\downarrowUA↑\mathrm{UA}\uparrowASR↓\mathrm{ASR}\downarrowNRP↑\mathrm{NRP}\uparrow
Qwen3-8B49.11 ±0.51\text{49.11}\,{\scriptstyle\pm\text{0.51}}50.37 ±3.63\text{50.37}\,{\scriptstyle\pm\text{3.63}}22.20 ±0.57\text{22.20}\,{\scriptstyle\pm\text{0.57}}53.64 ±0.90\text{53.64}\,{\scriptstyle\pm\text{0.90}}60.43 ±0.97\text{60.43}\,{\scriptstyle\pm\text{0.97}}28.12 ±1.30\text{28.12}\,{\scriptstyle\pm\text{1.30}}
+ Spotlighting48.52 ±0.78\text{48.52}\,{\scriptstyle\pm\text{0.78}}53.00 ±2.60\text{53.00}\,{\scriptstyle\pm\text{2.60}}19.25 ±1.04\text{19.25}\,{\scriptstyle\pm\text{1.04}}57.82 ±1.11\text{57.82}\,{\scriptstyle\pm\text{1.11}}58.53 ±0.73\text{58.53}\,{\scriptstyle\pm\text{0.73}}29.55 ±0.46\text{29.55}\,{\scriptstyle\pm\text{0.46}}
+ Prompt Guard 244.13 ±0.64\text{44.13}\,{\scriptstyle\pm\text{0.64}}54.67 ±1.07\text{54.67}\,{\scriptstyle\pm\text{1.07}}16.44 ±0.45\text{16.44}\,{\scriptstyle\pm\text{0.45}}47.35 ±0.52\text{47.35}\,{\scriptstyle\pm\text{0.52}}59.02 ±1.57\text{59.02}\,{\scriptstyle\pm\text{1.57}}29.71 ±1.29\text{29.71}\,{\scriptstyle\pm\text{1.29}}
+ Static-AT31.01‾ ±0.33\underline{\text{31.01}}\,{\scriptstyle\pm\text{0.33}}30.00‾ ±2.05\underline{\text{30.00}}\,{\scriptstyle\pm\text{2.05}}6.71‾ ±0.43\underline{\text{6.71}}\,{\scriptstyle\pm\text{0.43}}61.43‾ ±0.15\underline{\text{61.43}}\,{\scriptstyle\pm\text{0.15}}38.64‾ ±2.24\underline{\text{38.64}}\,{\scriptstyle\pm\text{2.24}}40.64‾ ±0.93\underline{\text{40.64}}\,{\scriptstyle\pm\text{0.93}}
+ AgentSafeEvo (ours)17.84 ±0.24\text{17.84}\,{\scriptstyle\pm\text{0.24}}6.53 ±0.38\text{6.53}\,{\scriptstyle\pm\text{0.38}}6.18 ±0.18\text{6.18}\,{\scriptstyle\pm\text{0.18}}66.52 ±1.43\text{66.52}\,{\scriptstyle\pm\text{1.43}}31.25 ±1.24\text{31.25}\,{\scriptstyle\pm\text{1.24}}46.00 ±0.54\text{46.00}\,{\scriptstyle\pm\text{0.54}}
Table 1: Agent safety evaluation results. Lower ASR and harm score are better, while higher UA and NRP are better.
MethodBFCL-V3ALFWorldToolHop
Qwen3-8B66.13‾ ±0.22\underline{\text{66.13}}\,{\scriptstyle\pm\text{0.22}}30.28 ±4.59\text{30.28}\,{\scriptstyle\pm\text{4.59}}21.84 ±1.23\text{21.84}\,{\scriptstyle\pm\text{1.23}}
+ Spotlighting66.08 ±0.37\text{66.08}\,{\scriptstyle\pm\text{0.37}}32.13‾ ±2.69\underline{\text{32.13}}\,{\scriptstyle\pm\text{2.69}}25.76 ±0.15\text{25.76}\,{\scriptstyle\pm\text{0.15}}
+ Prompt Guard 266.01 ±0.17\text{66.01}\,{\scriptstyle\pm\text{0.17}}31.08 ±4.12\text{31.08}\,{\scriptstyle\pm\text{4.12}}23.05 ±0.71\text{23.05}\,{\scriptstyle\pm\text{0.71}}
+ Static-AT66.73 ±0.50\text{66.73}\,{\scriptstyle\pm\text{0.50}}30.89 ±0.43\text{30.89}\,{\scriptstyle\pm\text{0.43}}39.43‾ ±0.09\underline{\text{39.43}}\,{\scriptstyle\pm\text{0.09}}
+ AgentSafeEvo (ours)65.98 ±0.14\text{65.98}\,{\scriptstyle\pm\text{0.14}}43.72 ±3.03\text{43.72}\,{\scriptstyle\pm\text{3.03}}48.04 ±1.40\text{48.04}\,{\scriptstyle\pm\text{1.40}}
Table 2: General agent capability results measured by accuracy (%).
Figure 3: Attack success rates (ASR, %). (a) Cross-evaluation of attackers and defenders at different stages of co-evolution. (b) Comparison of the SFT and final evolved attackers against five target models.
MethodSafetyGeneral
Agent-SafetyBenchMSBALFWorldToolHop
ASR↓\mathrm{ASR}\downarrowASR↓\mathrm{ASR}\downarrowNRP↑\mathrm{NRP}\uparrowAcc↑\mathrm{Acc}\uparrowAcc↑\mathrm{Acc}\uparrow
AgentSafeEvo (ours)17.84 ±0.24\text{17.84}\,{\scriptstyle\pm\text{0.24}}31.25‾ ±1.24\underline{\text{31.25}}\,{\scriptstyle\pm\text{1.24}}46.00 ±0.54\text{46.00}\,{\scriptstyle\pm\text{0.54}}43.72 ±3.03\text{43.72}\,{\scriptstyle\pm\text{3.03}}48.04 ±1.40\text{48.04}\,{\scriptstyle\pm\text{1.40}}
w/o attacker updates31.01 ±0.33\text{31.01}\,{\scriptstyle\pm\text{0.33}}38.64 ±2.24\text{38.64}\,{\scriptstyle\pm\text{2.24}}40.64 ±0.93\text{40.64}\,{\scriptstyle\pm\text{0.93}}30.89 ±0.43\text{30.89}\,{\scriptstyle\pm\text{0.43}}39.43 ±0.09\text{39.43}\,{\scriptstyle\pm\text{0.09}}
w/o learnability weighting22.91 ±0.35\text{22.91}\,{\scriptstyle\pm\text{0.35}}32.89 ±0.66\text{32.89}\,{\scriptstyle\pm\text{0.66}}41.48 ±0.21\text{41.48}\,{\scriptstyle\pm\text{0.21}}36.70‾ ±0.75\underline{\text{36.70}}\,{\scriptstyle\pm\text{0.75}}40.07 ±0.58\text{40.07}\,{\scriptstyle\pm\text{0.58}}
w/o defender weighting14.13 ±0.33\text{14.13}\,{\scriptstyle\pm\text{0.33}}28.47 ±1.03\text{28.47}\,{\scriptstyle\pm\text{1.03}}43.04‾ ±0.73\underline{\text{43.04}}\,{\scriptstyle\pm\text{0.73}}29.05 ±0.43\text{29.05}\,{\scriptstyle\pm\text{0.43}}40.80‾ ±0.22\underline{\text{40.80}}\,{\scriptstyle\pm\text{0.22}}
Table 3: Ablation results for Qwen3-8B on agent benchmarks for safety and general capabilities. Lower ASR is better; higher NRP and accuracy are better.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Information Retrieval Weather, Search, Knowledge, NewsLanguage and Documents Document, Language, Writing
Planning and Logistics Travel, Navigation, Event, Scheduling, LogisticsComputation and Coding Math, Conversion, Coding
Science and Environment Science, Agriculture, EnvironmentData and Research Data, Analytics, Reporting, Research
Business Operations Finance, Crypto, Market Research, Sales, Procurement, ProjectCommunication and Support HR, Customer Service, Communication, Social Media, Marketing
Creative Media Design, Media, Entertainment, GamingEducation and Healthcare Education, Healthcare, Food
IT and Infrastructure IT Operations, Cloud, Automation, Engineering, FacilitySecurity and Compliance Compliance, Audit, Cybersecurity, Physical Security, Emergency
Appendix
Table 4: Overview of the 50 agent scenarios.
Operation classIncluded operations
Information RetrievalSearch external sources, retrieve records, look up knowledge, list resources, and fetch current information
ComputationPerform arithmetic, solve equations, calculate statistics, and evaluate structured expressions
Language ProcessingTranslate, summarize, extract, analyze, and modify textual content
Format and Unit ConversionConvert measurement units, time zones, file formats, and structured values
Content GenerationGenerate text, images, drawings, audio, video, and other media content
Data ManagementCreate, read, update, delete, store, and organize files, databases, or structured records
Appendix
Table 5: Overview of tool operations.
Figure 4: Distribution of the red-team dataset. The outer and inner rings represent attack types and subtypes, respectively. Subtype counts are listed alongside the chart.
TypeSubtypeDescription
DPIUnauthorized tool invocationDirect instructions in the user query cause the agent to invoke additional tools or perform actions beyond the original task.
 Response manipulationDirect instructions in the user query control the final response without requiring any additional tool invocation.
 Parameter manipulationDirect instructions alter tool arguments such as recipients, destinations, targets, scopes, permissions, or verification settings.
IPIUnauthorized tool invocationInstructions embedded in external content cause the agent to perform unauthorized tool calls after retrieving or processing that content.
 Response manipulationInstructions embedded in external content control the final response while leaving the original tool-execution trace unchanged.
TDICapability overstatementA manipulated tool description exaggerates the capability, authority, or applicability of a tool and induces unjustified downstream actions.
Appendix
Table 6: Attack types and subtypes.
SettingValue
Maximum sequence length16384
Batch size8
Learning rate3×10−53\times 10^{-5}
Training epochs2
Warmup ratio0.03
Distributed strategyDeepSpeed ZeRO-3
Appendix
Table 7: SFT training settings.
SettingValue
Learning rateπA\pi_{\mathrm{A}} : 1×10−61\times 10^{-6} πD\pi_{\mathrm{D}} : 3×10−63\times 10^{-6}
Number of rolloutsπA\pi_{\mathrm{A}} : 5 πD\pi_{\mathrm{D}} : 8
GPUs per role4
Training steps120
Training batch size64
Task weighting coefficient λ\lambda0.5
Appendix
Table 8: RL training settings for Qwen3.
More in this topic

LLM Agent Safety

Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.

See all 302 papers

Explore similar work

CardsList

  1. SafeCoEvo: Co-Evolving Safety Harnesses and Guards for LLM Agents at Test-Time

    Sep 29, 2026Yu Cheng, Yongkang Hu, Shuaijie Ma +12Continual Test-Time AdaptationLLM Agents

    East China Normal University · Shanghai Innovation Institute · Xiamen University +4

  2. Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolution

    May 13, 2026Xiaozhe Zhang, Chaozhuo Li, Hui Liu +4Continual Learning for LLMsAdversarial Attacks on LLMs

    City University of Hong Kong · Beijing University of Posts and Telecommunications · Wuhan University +2

  3. SafeHarbor: Hierarchical Memory-Augmented Guardrail for LLM Agent Safety

    May 7, 2026Zhe Liu, Zonghao Ying, Wenxin Zhang +5LLM GuardrailsLLM Agent Security

    School of Cyber Science and Technology, Beihang University, Beijing, China · Institute of Artificial Intelligence, Beihang University, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +1