Evolution Under Fire: Evolving Agent Safety Harnesses with Adaptive Injection Stressors
Organizations: LMU Munich · Munich Center for Machine Learning (MCML)
Abstract
Tool-using language-model agents must act on information from untrusted emails, webpages, and documents. This creates a central security challenge: malicious instructions and legitimate task directives can arrive through the same interaction channel. Existing defenses are largely hand-designed and static, making further improvements dependent on repeated manual redesign. We introduce Evolution Under Fire, a staged framework that couples safety-harness evolution with adaptive stressor evolution. Utility-constrained, Pareto-scheduled tree search evolves complete executable safety harnesses from execution evidence while keeping the task model, benchmark, tools, and evaluator fixed. Adaptive stressor evolution then discovers utility-preserving attacks that guide a subsequent stage of harness evolution. On AgentDyn with GPT-4o-mini, an evolved harness reduces attack success from 50.4% to 3.0%, improves clean and attacked utility over the initial harness, and offers a stronger security-utility trade-off than reproduced defenses. On Agent-SafetyBench with Qwen2.5-7B-Instruct, harness evolution reduces attack success from 82.6% to 62.0% while retaining helpfulness and runtime validity. Evolved harnesses also transfer without further evolution to Qwen3.5-35B-A3B, reducing attack success from 32.1% to 0.9% while improving attacked-task utility.
Figures & tables
| 1: | Input: benchmark , initial harness , and search budgets . |
|---|---|
| 2: | Harness evolution: Evaluate and initialize the harness archive . |
| 3: | repeat times: |
| 4: | Select a utility-qualified parent by posterior-Pareto sampling. |
| 5: | Diagnose its evidence, expand one child, evaluate it, and update . |
| 6: | Stressor evolution: Form targets from capable, secure harness–case pairs. |
| 7: | repeat times: |
| Method | GPT-4o-mini | Qwen3.5-35B-A3B | ||||||
|---|---|---|---|---|---|---|---|---|
| CU | VU | AU | ASR | CU | VU | AU | ASR | |
| Initial harness | 27/60 | 28/28 | 202/560 | 282/560 | 41/60 | 27/28 | 341/560 | 180/560 |
| Repeat User Prompt | 25/60 | 27/28 | 211/560 | 177/560 | 42/60 | 26/28 | 373/560 | 97/560 |
| Spotlighting | 25/60 | 25/28 | 211/560 | 263/560 | 41/60 | 26/28 | 347/560 | 159/560 |
| Tool Filter | 3/60 | 14/28 | 34/560 | 26/560 | 1/60 | 7/28 | 17/560 | 0/560 |
| DRIFT | 9/60 | 19/28 | 89/560 | 11/560 | 0/60 | 7/28 | 4/560 | 0/560 |
| Method | ASR | Help. | Valid. |
|---|---|---|---|
| Initial | 82.60 | 54.83 | 97.15 |
| Safety Prompt | 80.95 | 50.07 | 96.10 |
| Repeat Prompt | 80.15 | 42.45 | 97.50 |
| Spotlighting | 82.25 | 52.93 | 96.70 |
| Tool Filter | 77.50 | 49.12 | 97.60 |
| Progent | 74.55 | 54.29 | 98.35 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Suite | Clean | Validators | Attacked |
|---|---|---|---|
| Shopping | 20 | 9 | 180 |
| GitHub | 20 | 9 | 180 |
| Daily Life | 20 | 10 | 200 |
| Total | 60 | 28 | 560 |
| Method | AgentDyn realization | Agent-SafetyBench realization |
|---|---|---|
| Initial harness | Unmodified model–tool loop | Unmodified model–tool loop |
| Safety Prompt | — | Released failure-mode prompt |
| Repeat User Prompt | Native AgentDyn defense | Mechanism-preserving port |
| Spotlighting | Native delimiting defense | Mechanism-preserving port |
| Tool Filter | Native query-conditioned filter | Mechanism-preserving port |
| Progent | Released AgentDyn adapter | Static-policy port |
| Artifact | Revision |
|---|---|
| AgentDyn | 5353cf7 |
| Agent-SafetyBench | 74feea8 |
| CaMeL | f083b6b |
| Progent | 5be7b63 |
| DRIFT | 6fd3df4 |
| MELON | 4d3cc9c |
| Method | AgentDyn benchmark | Adaptive stressor set | ||||
|---|---|---|---|---|---|---|
| CU | VU | AU | ASR | Utility | ASR | |
| Initial harness | 41/60 | 27/28 | 341/560 | 180/560 | 80/82 | 58/82 |
| Evolved | 38/60 | 22/28 | 379/560 | 15/560 | 76/82 | 20/82 |
| Evolved | 36/60 | 22/28 | 345/560 | 5/560 | 72/82 | 15/82 |
| Evolved | 39/60 | 25/28 | 355/560 | 5/560 | 80/82 | 15/82 |
| Evolved | 38/60 | 22/28 | 346/560 | 9/560 | 70/82 | 14/82 |
| Recipient harness | Attack success | Attack success + utility |
|---|---|---|
| Initial harness | 70/82 | 68/82 |
| Evolved | 54/82 | 46/82 |
| Evolved | 66/82 | 64/82 |
| Evolved | 65/82 | 63/82 |
| All four recipients | 43/82 | 36/82 |
AI Agent Security
Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.
See all 558 papers