LLM-based agents are increasingly deployed in product-level execution harnesses, where jailbreaks can trigger harmful tool use and persistent state changes, creating greater risks than unsafe text generation alone. Existing automatic red-teaming methods often rely on fixed attacks, while recent agentic attackers coordinate multiple jailbreak tools and show stronger potential through trajectory-based retrieval. However, such retrieval can reuse misleading experiences due to retrieval bias and unclear tool credit, and full trajectories add context overhead while reducing interpretability. We propose RedEvoAgent, a black-box red-teaming agent that distills cross-case attack trajectories into a concise, human-readable attack skill. The attack skill adaptively evolves through tool-effectiveness profiling and Deciding-Tool Attribution for skill updates, and a validation ratchet that retains only updates improving validation performance. Experiments on multiple benchmarks, target models, and target execution harnesses show that RedEvoAgent outperforms fixed and agentic baselines, improves tool efficiency, and transfers across attacker models and target execution harnesses.
Figures & tables
Figure 1: RedEvoAgent overview. Given a jailbreak prompt, the attacker agent uses the current attack skill to select and call tools from an extensible attack toolbox, such as GCG and FlipAttack, and iteratively attacks a product-level target agent. On the training split, RedEvoAgent collects successful and failed attack trajectories and independently evaluates each attack tool to build a tool-effectiveness profile. Skill evolution combines the profile with these trajectories to extract tool priorities and attack strategies. A candidate attack skill update is accepted only if it improves attack performance on the validation split, while the test split remains locked for final evaluation.
Figure 2: Trajectory-to-skill case study. (a) Each card shows the prompt sent after the labeled transformation; arrows preserve attempt order, and badges report the target outcome. The first successful query follows FlipAttack, so Deciding-Tool Attribution credits FlipAttack. (b) Together with other training cases, this evidence contributes to an evolved skill that records tool priorities and an actionable attack policy.
(a) Claude Code harness
Benchmark
Target Model
No Jailbreak
GCG
AmpleGCG
AutoDAN
Template
FlipAttack
RolePlay
RedCodeAgent
MAJIC
RedEvoAgent
ASB
MiniMax-M2.5
72.5/26.1
63.6/28.9
61.8/26.4
38.6/53.2
40.4/47.1
91.8 /3.9
68.2/23.2
76.4/21.4
74.2/15.1
93.2 /6.1
DeepSeek-V4-Flash
83.9/7.5
81.1/8.2
77.9/13.6
40.7/21.1
66.4/15.7
94.3/2.1
91.8/2.5
90.7/8.6
94.6 /2.5
99.3 /0.0
Qwen3.5-35B
82.8/15.1
81.7/16.1
77.4/16.1
77.4/16.1
47.3/20.4
83.9 /15.1
84.9/15.1
79.6/11.8
80.6/12.9
87.1 /12.9
AgentHarm
MiniMax-M2.5
9.2/85.1
7.9/81.1
6.1/84.2
1.0/89.9
0.0/88.5
18.6/28.1
8.7/80.8
20.9 /75.5
22.6 /34.7
20.8/58.9
DeepSeek-V4-Flash
47.7/44.3
37.2/48.3
34.8/51.7
10.5/60.7
28.5/48.3
67.9 /11.7
43.8/43.3
37.5/67.6
45.5/32.4
74.3 /15.9
Table 1: RedEvoAgent vs. attack baselines. Cells report ASR/RR (%) on ASB and mean HarmScore/RR (%) on AgentHarm. Bold and underline mark the best and second-best ASR or HarmScore within each displayed row, respectively.
Benchmark
Harness
Target Model
No Skill
Human Skill
RedEvo Agent
ASB
Claude Code
MiniMax-M2.5
77.5/3.0
78.3/3.2
93.2 /1.8
DeepSeek-V4-Flash
96.4/2.6
96.4/2.7
99.3 /0.9
Codex
MiniMax-M2.5
80.8/2.7
81.3/3.0
92.8 /1.8
DeepSeek-V4-Flash
97.9/2.7
98.9/2.8
100.0 /0.8
AgentHarm
Claude Code
MiniMax-M2.5
15.4/3.9
13.1/4.2
20.8 /2.4
DeepSeek-V4-Flash
34.3/3.0
40.5/3.0
74.3 /2.2
Table 2: Skill contribution. Cells report ASR / average attack tool calls per case on ASB and mean HarmScore / average attack tool calls per case on AgentHarm. Bold marks the best ASR or HarmScore.
Table 3: Experience-component ablations on the ASB test set with MiniMax-M2.5 and the Claude Code harness. Cells are ASR (%). ✓ = included, × = removed. “–” = not applicable: Deciding-Tool Attribution labels collected trajectories, so it is undefined when trajectory collection is off. Bold is the full system.
Table 4: Zero-shot transfer of the MiniMax-M2.5 attack skill evolved with GPT-4o mini and Claude Code. Cells are ASR (%) on the ASB test set. No Skill is the destination baseline with the default attacker prompt; Transferred reuses the source skill without re-evolution. Parentheses are gains over No Skill.
(a) Attacker transfer (from GPT-4o mini) Destination No Skill Transferred Qwen3-8B 89.7 95.3 (+5.6) Qwen3-4B 83.3 90.6 (+7.3) (b) Harness transfer (from Claude Code) Destination No Skill Transferred Codex 80.8 90.5 (+9.7)
Table 3: Experience-component ablations on the ASB test set with MiniMax-M2.5 and the Claude Code harness. Cells are ASR (%). ✓ = included, × = removed. “–” = not applicable: Deciding-Tool Attribution labels collected trajectories, so it is undefined when trajectory collection is off. Bold is the full system.
Figure 3: Validation-ratchet evolution. The x-axis is the maximum number of ratchet rounds R ; R=0 is the No Skill condition. The y-axis is ASR on ASB and HarmScore on AgentHarm, with an independent scale per panel. Orange step lines show the validation score of the skill accepted after each round; filled orange circles mark accepted candidate skills, gray crosses show rejected candidate skills at their validation scores, and solid green lines show test scores for accepted skill versions. Rejected candidates are not tested, and flat orange segments carry forward the previous accepted skill version. Candidate skills are accepted based only on validation scores; test scores are not used by the gate.