Cumulative and Audited Guardrail Tool Evolution for LLM Agent Safety
Organizations: School of Computer Science and Engineering, Beihang University, Beijing, China. · State Key Laboratory of Complex & Critical Software Environment, Beihang University, Beijing, China. · College of Computing & Data Science, Nanyang Technological University, Singapore. · Zhongguancun Laboratory, Beijing, China.
Abstract
Large language model (LLM) agents increasingly operate in open-world environments where safety risks cannot be fully predefined. However, most existing guardrail mechanisms remain static, relying on fixed rules, prompts, or classifiers that generalize poorly to emerging risks. Here we present EVOLVE, a tool-level self-evolving defense framework that formulates agent security as a process of cumulative security evolution. Inspired by how human societies transform experiences into transferable protective knowledge, EVOLVE externalizes security knowledge into executable guardrail tools that are continuously inherited, independently audited, and iteratively refined through lifelong interaction. Across multiple benchmarks, backbone models, and embodied environments, EVOLVE consistently outperforms existing defenses across diverse risk scenarios with +22% average F1 score. We further validate the framework in real-world robotic and autonomous agent systems. Our results suggest that scalable agent safety may require continuously evolving defense systems rather than static guardrails.
Figures & tables
| Symbol | Description | Symbol | Description |
|---|---|---|---|
| User query | Tool execution evidence | ||
| Intended action trajectory | Action execution evidence | ||
| Environmental context | Audit verdict ( ) | ||
| Identified risk category | Audit report | ||
| Activation indicator ( ) | Final security decision | ||
| Generated guardrail tool | Revised action trajectory |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Condition | Acc | Prec | Rec | F1 | Calls |
|---|---|---|---|---|---|
| w/ Tools | 78.17 | 74.27 | 98.71 | 84.76 | 3.24 |
| w/o Tools | 62.70 | 62.55 | 98.06 | 76.38 | 1.00 |
| +15.47 | +11.72 | +0.65 | +8.38 | — |
| Benchmark | Stage | Agreement | ||||
|---|---|---|---|---|---|---|
| RJudge | Pre-repair | 33 | 32 | 5 | 4 | 92.5% |
| RJudge | Post-repair | 11 | 10 | 2 | 1 | 97.5% |
| ASSEBench | Pre-repair | 30 | 29 | 6 | 5 | 90.8% |
| ASSEBench | Post-repair | 8 | 9 | 2 | 3 | 95.8% |
| AgentHarm | Pre-repair | 26 | 25 | 4 | 3 | 94.2% |
| AgentHarm | Post-repair | 6 | 7 | 1 | 2 | 97.5% |
| RJudge | ASSEBench | AgentHarm | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Acc | Prec | Rec | F1 | Acc | Prec | Rec | F1 | Acc | Prec | Rec | F1 |
| Backbone LLM: DeepSeek | ||||||||||||
| CoTSafe | 0.8091 | 0.8575 | 0.7223 | 0.7815 | 0.6316 | 0.7629 | 0.8578 | 0.7722 | 0.7666 | 0.9007 | 0.5978 | 0.6915 |
| ShieldAgent | 0.7863 | 0.7136 | 0.9934 | 0.8318 | 0.8046 | 0.8762 | 0.7532 | 0.8095 | 0.5286 | 0.6923 | 0.1029 | 0.1791 |
| AGrail | 0.5403 | 0.6488 | 0.9168 | 0.7601 | 0.8678 | 0.9213 | 0.8039 | 0.8587 | 0.6000 | 0.8214 | 0.3557 | 0.4964 |
| GuardAgent | 0.6935 | 0.7684 | 0.5037 | 0.6086 | 0.7259 | 0.6294 | 0.9240 | 0.7485 | 0.7714 | 0.8992 | 0.6114 | 0.7281 |
AI Agent Security
Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.
See all 559 papers