Red teaming must continually adapt as frontier LLM safeguards evolve, yet existing approaches largely search for adversarial prompts or optimize within fixed attack procedures. We observe that even when an attack becomes ineffective, its underlying mechanisms can remain useful and can be revised or recombined to construct stronger attacks. Based on this insight, we introduce RedDarwin, an agentic framework that shifts the unit of jailbreak evolution from prompts and fixed attacks to executable attack skills. RedDarwin consists of 47 reusable base skills and autonomously revises, composes, executes, and selects them using target-model feedback, allowing successful discoveries to become building blocks for subsequent evolution. RedDarwin produces 65 evolved skills, improving 44 of 47 base attacks and increasing their mean attack success rate (ASR) on AdvBench from 18.8% to 47.3%. These evolved skills consistently outperform baselines on the held-out StrongREJECT and frontier LLMs unseen during evolution; with only 10 target-model queries, RedDarwin achieves 100% ASR on GPT-5.6-Sol, 92% on Claude-Sonnet-5, and 80% on Gemini-3.8-Flash. Analysis of the evolution trajectories further shows that strong attacks often emerge from novel combinations of reusable mechanisms rather than from individually dominant ones. We believe this paradigm can accelerate adaptive red teaming and inform stronger safeguards, creating complementary value for both safety evaluation and defense.
Figures & tables
Figure 1: Executable skill evolution in RedDarwin . Starting from executable attack skills, RedDarwin generates candidates through revision, combination, and discovery, evaluates them against the target model, and uses execution feedback to guide subsequent evolution. Successful candidates are retained in the skill population.
Figure 2: Development performance across skill evolution operators. Bars show mean ASR (%) on AdvBench-50 across four development targets.
Development
Held-out evaluation
Development targets
Unseen targets
Attack skill
Avg.
Qwen3.7 Flash
GPT-OSS 120B
DS-V4 Flash
GPT-5.6 Luna
GLM-5.3 Flash
Gemini-3.8 Flash
MiniMax M3
Avg.
Baselines (Top-5)
Puzzler [ 2024 ]
54.0
12
10
52
30
22
46
16
26.9
CIA [ 2024 ]
42.0
42
16
60
6
4
38
22
26.9
AdvTales [ 2026 ]
62.5
74
36
60
36
8
14
58
40.9
Table 1: Transfer of executable attack skills to held-out goals and unseen models. ASR (%) of representative base and evolved skills with fixed canonical configurations. Development ASR averages four targets on AdvBench-50. Held-out evaluation uses StrongREJECT-50 on the same four targets and three models unseen during evolution; its average covers all seven targets. Bold and underlined values mark the best and second-best result in each column, including ties.
Figure 3: Adaptive red teaming under a fixed target-query budget. Cumulative ASR over StrongREJECT-50 as the per-goal query budget increases. Endpoint labels report ASR at budget 10.
Figure 4: Revision patterns and mechanism reuse in skill evolution. (a) Distribution and retention of improvement-associated revisions across six functional modules. (b) ASR across versions of four representative skills. (c) Mechanism reuse count and mean ASR of skills containing each mechanism.
Figure 5: Representative cases of skill evolution. Evolved skills can preserve an existing attack mechanism while changing the surrounding procedure, extend a compositional framework with new operators, or integrate mechanisms from different attacks into a new skill.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Date
Author
Attack
Category
Aug. 2023
Yuan et al.
CipherChat
Encoding & Obfuscation
Sep. 2023
Yu et al.
GPTFuzz
Auto-search
Oct. 2023
Yong et al.
LRL
Cross-lingual
Oct. 2023
Wei et al.
ICA
Priming
Oct. 2023
Chao et al.
PAIR
Auto-search
Nov. 2023
Li et al.
DeepInception
Role-play
Appendix
Table 3: Methods used to initialize the skill population. RedDarwin consists of 47 executable base skills derived from jailbreak methods spanning ten categories, ordered chronologically by their initial arXiv submission date.
Qwen3.7 Flash
GPT-OSS 120B
DeepSeek V4-Flash
GPT-5.6 Luna
Avg.
Δ
Category
Skill
Base
2.0
Base
2.0
Base
2.0
Base
2.0
Base
2.0
Δ
Role-play
PersonaModulation
2
72
0
40
26
58
0
0
7.0
42.5
▲ 35.5
DeepInception
0
50
20
94
76
68
0
0
24.0
53.0
▲ 29.0
ArtOfJailbreak
0
78
0
70
14
96
12
4
6.5
62.0
▲ 55.5
AdversarialTales
80
88
54
76
76
80
40
4
62.5
62.0
▼ 0.5
CompoundRoleplay
0
90
0
88
30
92
0
8
7.5
69.5
▲ 62.0
Appendix
Table 4: Single-skill revision on the development set (AdvBench-50). ASR (%) of 47 executable base skills before (Base) and after RedDarwin revision (2.0) across four target models. Skills are sorted within each category by 2.0 average ASR. Colors indicate improvement/degradation/tie, and bold marks category-best results. Avg.: macro-average across targets; Δ : change relative to Base.
Figure 6: Combination lineage across five development rounds. Initial skills (left column) seed the graph; each offspring has two parents. The vertical position and shading of each node report its development ASR (Table 5 ). Blue edges mark reuse of an earlier offspring (EmojiCtxInject) as a parent in Rounds 3–4.
Qwen3.7 Flash
GPT-OSS 120B
DS-V4 Flash
GPT-5.6 Luna
Avg.
Δ
Source
Evolved skill
Origin
ASR
ASR
ASR
ASR
ASR
Δ
Multi-Skill Combination
Round 1
AdvHarmSearch
AdvTales + h4rm3l
96
94
92
80
90.5
▲ 15.0
EmojiCodeCham
EmojiJB + CodeCham
100
58
100
66
81.0
▲ 20.5
CogHarmSearch
CogOverload + h4rm3l
100
98
100
90
97.0
▲ 2.0
Round 2
NarrDetailDecomp
AdvTales + FicDetail
100
72
86
2
65.0
▼ 6.5
Appendix
Table 5: Development results for combination and new-skill discovery. ASR (%) on AdvBench-50 for 15 combination candidates and 10 discovery candidates. Avg. averages the four development targets; Δ is the gain over the stronger parent in percentage points. Bold marks the best result in each round or the discovery group.
Method
Representation
Evolved scope
Model access
Claudini [ 48 ]
Optimizer code
Token-forcing optimization procedure
White-box
EvoSynth [ 10 ]
Attack program
Program logic and control flow
Black-box
JailbreakSkill [ 69 ]
Executable skill
Rewrite procedure
Black-box
RedEvoAgent [ 83 ]
Markdown skill
Tool choice and orchestration
Black-box
MemoAttack [ 84 ]
Structured memory
Attack patterns and memory lifecycle
Black-box
DARWIN-Attack [ 51 ]
Strategy pool
Strategies and adaptive transitions
Black-box
Appendix
Table 6: Comparison of related adaptive and evolutionary red-teaming systems.
Skill
Parents
Combination
AdvHarmSearch
AdvTales + h4rm3l
A veiled planning memo is encoded as a character-index table and embedded as the mentor’s guidance artifact in a Propp-style tale. The target decodes the memo and expands it through third-person plot attribution.
EmojiCodeCham
EmojiJB + CodeCham
An auxiliary compiler renders the goal as an emoji rebus, which becomes the ciphertext inside a decrypt-then-complete code frame. The legend is incorporated into the decoding procedure, followed by completion under the decoded task contract.
CogHarmSearch
CogOverload + h4rm3l
CogOverload’s veiled memo and reverse-causal narration are compiled into a h4rm3l transformation program. The target reconstructs the encoded memo and develops it through the resulting narrative procedure.
NarrDetailDecomp
AdvTales + FicDetail
A Propp-structured premise pack carries the goal as an in-world mentor manuscript, after which FicDetail’s reply-conditioned fiction ladder progressively develops the manuscript. Development ASR fell below the stronger parent.
NarrAttackSearch
AdvTales + TAP
TAP’s tree search operates over compiled tale briefs rather than freeform prompts. Each branch preserves the Propp-style framing while varying how the brief is presented and expanded.
EmojiCtxInject
EmojiJB + VirtualCtx
The goal is represented as an emoji rebus, while a fabricated assistant turn establishes that the rebus has already been interpreted. The candidate was not retained, but its rebus representation was reused in later combinations.
Appendix
Table 7: Multi-skill combination, Rounds 1–2. Each candidate integrates mechanisms from two parent skills into a shared executable procedure. Names match Table 5 .
Skill
Parents
Combination
RebusProppExeg
AdvTales + EmojiCtxInject
The mentor’s guidance is represented as an emoji rebus containing a compressed goal-level brief. Decoding is embedded within third-person structural exegesis, which expands the recovered cues into a fuller account.
MirrorManuscript
FlipAttack + FicDetail
FicDetail supplies an in-world document frame, while FlipAttack provides the transformed brief. The target reconstructs the brief and continues it as a manuscript within the fictional setting.
DecodeLadder
RACE + h4rm3l
A h4rm3l-encoded brief is reconstructed across a RACE-style sequence of disjoint fragments. The final stage combines the recovered fragments into a single expanded account.
FlipBreaker
ActorBreaker + FlipAttack
ActorBreaker’s fragment plan provides the shared representation. Each turn applies a flipped decoding task to one fragment, and the final turn combines the recovered fragments.
RebusTranscr
EmojiCtxInject + h4rm3l
An auxiliary compiler renders a short brief as a legend-guided emoji rebus. The target first reconstructs the brief and then expands it within the same decoding-oriented interaction.
FlipCtxCont
VirtualCtx + h4rm3l
An encoded memo is introduced through a decoding task, while a fabricated prior assistant turn contains the beginning of its reconstruction. The target continues from that prior state rather than starting from the original request.
Appendix
Table 8: Multi-skill combination, Rounds 3–5. Later rounds reuse mechanisms introduced by earlier candidates and integrate them with additional interaction, search, and representation strategies.
Skill
Mechanism
Core idea
WitnessCollate
Witness adjudication
Two manuscript outlines provide conflicting versions of the same procedural document. The target compares the disputed readings and completes a received text from the selected interpretation.
BackcastDecode
Canonical backcast
A short encoded brief establishes an outcome and its narrative role. After reconstruction, the target develops the preceding causes through backward-causal reasoning.
CensureCite
Enforcement citation
A compiled exhibit lists a set of charge topics in outline form. The target acts as an adjudicator and develops the cited material before issuing its assessment.
DeclassReview
Redaction reversal
A historical incident sketch contains redacted spans linked to topic-level entries in a sealed appendix. The target reviews each redaction and reconstructs the corresponding passage from its surrounding context.
CriticalApparatus
Scholarly annotation
A short after-action sketch is presented as a text for critical editing. The target expands individual clauses through editorial annotations that make their implied details explicit.
FolioSequencing
Dependency sequencing
Several fragments of one artifact are presented out of order together with precedence constraints. The target reconstructs their sequence and explains the connections between adjacent fragments.
Appendix
Table 9: New-skill discoveries. Ten executable procedures developed without a prescribed parent. The mechanism labels summarize the main interaction pattern used by each skill; development results are reported in Table 5 .
Metric
Mean
Min. (skill)
Max. (skill)
Input tokens
305,079
61,570 (PersonaModulation 2.0)
1,585,245 (ReNeLLM 2.0)
Output tokens
55,371
10,557 (MathPrompt 2.0)
117,068 (PAP 2.0)
LLM requests
156.0
31 (Prefill 2.0)
358 (PAIR 2.0)
Evolution time (h)
6.94
0.65 (RebusTranscription)
25.96 (Mastermind 2.0)
LLM streaming time (min)
20.41
4.72 (CognitiveOverload 2.0)
60.06 (TurboFuzzLLM 2.0)
Appendix
Table 10: Resource usage per evolution trajectory. Statistics are computed over the 72 trajectories with recoverable Kimi CLI session metadata. Cache-read and cache-creation tokens are excluded; parentheses identify the skills attaining the minimum and maximum values.
Metric
LLM judge
Human A
Human B
ASR on AdvBench
52.0
48.0
48.0
ASR on StrongREJECT
42.0
36.0
42.0
Overall ASR
47.0
42.0
45.0
Binary agreement with the LLM judge
–
95.0
96.0
Appendix
Table 11: Human check of the LLM judge. ASR (%) on the same 100 sampled goal–response pairs, with 50 from AdvBench and 50 from StrongREJECT. Binary agreement (%) compares each annotator’s success labels with the LLM judge over all 100 pairs. Success is defined as a score of 5 under the common five-point rubric.
Figure 7: Retention and order of skill revisions. (a) Retained and non-retained improvement-associated revisions across the six functional modules. (b) Retained revisions separated into the first retained improvement in each trajectory and subsequent improvements.
Figure 8: Initial designs and development effort. (a) Outcomes following the first design attempt for the 65 retained trajectories. Different module denotes a later switch in the module being revised; Same module denotes further refinement within it. (b) Mean recorded activity in 7 non-retained development cycles relative to 65 retained cycles. The dashed line denotes equal means.
Mechanism
What it does
Reuse
Mean ASR
Index-copy
The target restores hidden text by copying characters at listed positions, one at a time.
34
79.3
Reverse-causal
The payload is written as a past-tense record; the target reasons backward from outcome to steps.
22
72.7
Word-aligned chunks
The payload is split into word-aligned chunks reversed within each chunk, so no harmful word appears whole.
22
82.4
Join-expression
The target emits restored text as a join of copied pieces, never materializing whole harmful words.
21
81.9
Output contract
The reply must instantiate the real content; paraphrase, euphemism, or summary counts as failure.
21
67.4
Aux short memo
An auxiliary model compresses the goal into a short operational memo before any target call.
19
82.6
Appendix
Table 12: The twenty most-reused mechanisms. Reuse counts skills containing the mechanism beyond its first occurrence, as in Figure 4 (c); mean ASR averages the recorded development performance of the complete skills containing it. Ties at the cutoff are broken by mean ASR. Display names match the labels in Figure 4 (c) where available.
Figure 9: Base skills and four rounds of single-skill revision. Mean development ASR over three targets from the base skill through four successive revision rounds. Continued revision can produce further gains after the first evolved version, although later successors are not necessarily stronger. Archived Base ASRs are computed over available judged responses, with incomplete judge coverage.
Figure 10: Model risk profiles across 10 attack categories. Each axis reports mean skill ASR (%) within a category on 50 AdvBench goals. Gray dashed curves show base skills; blue solid curves show their 2.0 revisions. All panels use the same scale, with the outer ring at 100%. Larger radii indicate greater observed susceptibility to the evaluated skills in that category.
LLM-based agents are increasingly deployed in product-level execution harnesses, where jailbreaks can trigger harmful tool use and persistent state changes, creating greater risks than unsafe text generation alone. Existing automatic red-teaming methods often rely on fixed attacks, while recent agentic attackers coordinate multiple jailbreak tools and show stronger potential through trajectory-based retrieval. However, such retrieval can reuse misleading experiences due to retrieval bias and unclear tool credit, and full trajectories add context overhead while reducing interpretability. We propose RedEvoAgent, a black-box red-teaming agent that distills cross-case attack trajectories into a concise, human-readable attack skill. The attack skill adaptively evolves through tool-effectiveness profiling and Deciding-Tool Attribution for skill updates, and a validation ratchet that retains only updates improving validation performance. Experiments on multiple benchmarks, target models, and target execution harnesses show that RedEvoAgent outperforms fixed and agentic baselines, improves tool efficiency, and transfers across attacker models and target execution harnesses.
Junjie Zhang, Hui Liu, Kecheng Chen +3
City University of Hong Kong · Shenzhen MSU-BIT University
Automated red-teaming has produced a growing collection of attack strategies, yet they typically remain scattered across prompts and workflows, making them difficult to systematically integrate, reuse, and improve at scale. We introduce JailbreakSkill, a skill-centric framework for scaling automated red-teaming through reusable and continuously evolving attack capabilities. JailbreakSkill packages existing attack strategies into modular, agent-ready skills that can be directly reused and adaptively selected across tasks and target models. Beyond reuse, it closes the loop between attacking and learning: attack experience is used to diagnose, refine, combine, and discover new skills, which are added back to an ever-growing skill library. This evolution lifts macro-average ASR by 17.5 percentage points on AdvBench and 13.4 points on HarmBench, including a 48.6-point gain against GPT-5.4 on AdvBench, while yielding novel attack strategies such as reframing a direct request as an unfinished document-completion task. Several evolved skills also generalize to unseen prompts and target models without further adaptation. Our code is available at https://github.com/BattleWen/JailbreakSkill.
Xiaoyu Wen, Jiajia Li, Zhida He +11
Shanghai AI Laboratory · Shanghai Jiao Tong University · Northwestern Polytechnical University +2
As large language models are deployed behind increasingly sophisticated safety mechanisms, evaluating their robustness under realistic adversarial conditions becomes increasingly difficult and critical. Existing jailbreaking evaluations often rely on static prompts, limited interaction horizons, or lightly protected systems, failing to reflect protected deployment settings with prompt-based safety protections. We show that the introduction of a prompt-based safety protection reduces the average attack success rate of a strong multi-turn baseline from 0.896 to 0.608 (32.1% relative), demonstrating that unprotected evaluations substantially overestimate real-world vulnerabilities. We propose Adaptive Traceback and Tactic Curation for Knowledge-based Red-teaming (ATTCK-R), a planning-driven multi-turn red teaming framework designed for protected black-box systems. ATTCK-R models jailbreaking as a long-horizon trajectory optimization problem using a structured belief state, introduces optimization-based tactic curation to abstract reusable strategies from prior successes and failures, and employs informed traceback to resume attacks from strategically meaningful states rather than restarting. Evaluated across 9 widely used LLMs configured with a single prompt-based safety protection layer, ATTCK-R consistently outperforms strong recent multi-turn jailbreaking frameworks. Ablation results confirm that long-horizon planning and tactic curation improve attack effectiveness, while adaptive traceback improves efficiency through structured recovery from conversational failure.