Large language model (LLM) agents increasingly operate in open-world environments where safety risks cannot be fully predefined. However, most existing guardrail mechanisms remain static, relying on fixed rules, prompts, or classifiers that generalize poorly to emerging risks. Here we present EVOLVE, a tool-level self-evolving defense framework that formulates agent security as a process of cumulative security evolution. Inspired by how human societies transform experiences into transferable protective knowledge, EVOLVE externalizes security knowledge into executable guardrail tools that are continuously inherited, independently audited, and iteratively refined through lifelong interaction. Across multiple benchmarks, backbone models, and embodied environments, EVOLVE consistently outperforms existing defenses across diverse risk scenarios with +22% average F1 score. We further validate the framework in real-world robotic and autonomous agent systems. Our results suggest that scalable agent safety may require continuously evolving defense systems rather than static guardrails.
Figures & tables
Figure 1: Overall framework of EVOLVE. EVOLVE organizes agent defense into generation, reuse, audit, and refinement stages, enabling executable guardrail tools to be generated, fused, verified, and repaired through lifelong interaction.
Figure 2: Comprehensive benchmark and ablation evaluation of EVOLVE against representative agent-defense baselines.
Figure 3: EVOLVE’s self-evolution trajectory under three sample-ordering schedules.
Figure 4: Embodied scenario adaptation in AI2THOR. The first row shows the original unsafe task, while the second row shows EVOLVE’s defended result.
Figure 5: Analysis results of EVOLVE.
Figure 6: Real-world robotic arm case studies with escalating task complexity, showing risk blocking and safe replanning.
Symbol
Description
Symbol
Description
Q
User query
EG′
Tool execution evidence
τ
Intended action trajectory
Eτ
Action execution evidence
C
Environmental context
D
Audit verdict ( ∈{0,1} )
R
Identified risk category
V
Audit report
z
Activation indicator ( ∈{0,1} )
y
Final security decision
G
Generated guardrail tool
τ~
Revised action trajectory
Table 1: Summary of notations used in EVOLVE.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Computational cost profiling of EVOLVE on RJudge (Application domain, GPT-4o). (a) Number of LLM API calls per processed sample. (b) Output token count per individual LLM call. (c) End-to-end processing time per sample.
Condition
Acc
Prec
Rec
F1
Calls
w/ Tools
78.17
74.27
98.71
84.76
3.24
w/o Tools
62.70
62.55
98.06
76.38
1.00
Δ
+15.47
+11.72
+0.65
+8.38
—
Appendix
Table 2: Frozen-library evaluation on RJudge Application domain (GPT-4o). “Calls” denotes average LLM API calls per sample.
Benchmark
Stage
NvLLM
NvHum
NFP
NFN
Agreement
RJudge
Pre-repair
33
32
5
4
92.5%
RJudge
Post-repair
11
10
2
1
97.5%
ASSEBench
Pre-repair
30
29
6
5
90.8%
ASSEBench
Post-repair
8
9
2
3
95.8%
AgentHarm
Pre-repair
26
25
4
3
94.2%
AgentHarm
Post-repair
6
7
1
2
97.5%
Appendix
Table 3: Human reviewer validation of VGT detection and post-repair evaluation across three benchmarks. All counts are computed over 120 randomly sampled tools per cell. “Agreement” denotes the exact-match rate between LLM and human binary judgements.
RJudge
ASSEBench
AgentHarm
Method
Acc
Prec
Rec
F1
Acc
Prec
Rec
F1
Acc
Prec
Rec
F1
Backbone LLM: DeepSeek
CoTSafe
0.8091
0.8575
0.7223
0.7815
0.6316
0.7629
0.8578
0.7722
0.7666
0.9007
0.5978
0.6915
ShieldAgent †
0.7863
0.7136
0.9934
0.8318
0.8046
0.8762
0.7532
0.8095
0.5286
0.6923
0.1029
0.1791
AGrail
0.5403
0.6488
0.9168
0.7601
0.8678
0.9213
0.8039
0.8587
0.6000
0.8214
0.3557
0.4964
GuardAgent
0.6935
0.7684
0.5037
0.6086
0.7259
0.6294
0.9240
0.7485
0.7714
0.8992
0.6114
0.7281
Appendix
Table 4: Results across three benchmarks and four backbone LLMs. Bold indicates the best, Underline indicates the second. † ShieldAgent does not rely on a backbone LLM, its scores are identical across all groups.
Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.
Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.
Chenxu Wang, Xiaoyu Wen, Han Qi +4
Shanghai Artificial Intelligence Laboratory · Fudan University · Shanghai Jiaotong University +1
The safety of large language model (LLM) agents depends not only on model weights but also on the agent harness that manages context, memory, tools, permissions, and runtime control. Existing safety mechanisms often treat the harness as a fixed deployment artifact, limiting their ability to evolve with emerging risks. Moreover, coupled functions across harness components obscure safety responsibility attribution, making localized evolution difficult. We propose Safety Harness Evolution (SHE), a framework that learns evolving safe boundaries from rollout trajectories. SHE decomposes the harness into four artifacts with explicit safety responsibilities, including the System Prompt, Rule Bank, Safety Memory, and Tool Policy, defining clear functional boundaries for localized evolution. Based on this decomposition, SHE introduces an attribution-guided evolution loop that converts trajectory failures into structured diagnoses, learns artifact-specific boundary refinements, and selects evolved harnesses through safety-utility validation. Experiments on Agent-SafetyBench demonstrate that SHE effectively enhances safety through harness evolution, achieving a 3.1x ASR reduction compared with static SafeHarness, while also improving benign utility. The evolved harness further generalizes to unseen risks on the held-out AgentHarm benchmark and transfers across agent models without additional evolution.
Wanying Qu, Qinghua Mao, Yu Li +12
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Fudan University +1
The expanding operational capabilities of large language model (LLM) agents introduce sophisticated security threats. Runtime defenses have emerged as an effective approach to mitigating these risks by integrating security mechanisms into the agent execution loop. However, existing runtime defenses rely heavily on manually designed interventions and lack a principled framework for their construction and maintenance. In this work, we first develop a harness-level formulation of runtime defense that systematically characterizes how harness mechanisms enable defense construction and provides a unified view of existing runtime defense interventions from a harness perspective. Building on this formulation, we propose HARD (Harness-based Autonomous Runtime Defense Evolution), a self-evolving runtime defense framework that automatically identifies appropriate intervention strategies and iteratively improves defense artifacts based on observed failure traces. HARD transforms runtime defense development from manual engineering into an autonomous evolution process, and extensive experiments demonstrate that it improves security performance over existing handcrafted defenses while preserving benign task utility. Our findings highlight autonomous defense evolution as a promising new paradigm for securing deployed LLM agents, enabling agents to identify defense weaknesses and continuously improve their protection mechanisms.
Jiajun Ruan, Peiyang Li, Yukun Chen +2
University of Minnesota · Ant Group · Tsinghua University +1