SkillDRE: Dual-Stage Red-Team Evolution of Agent Skills via Pre-Execution and Runtime Feedback

Pengyu Zhu; Jingyi Yang; Yi Liu; Li Sun; Sen Su

cs.CRSep 26, 2026

SkillDRE: Dual-Stage Red-Team Evolution of Agent Skills via Pre-Execution and Runtime Feedback

Authors: Pengyu Zhu, Jingyi Yang, Yi Liu, Li Sun, Sen Su

Organizations: Beijing University of Posts and Telecommunications · North China Electric Power University · Chongqing University of Posts and Telecommunications

Abstract

Agent skills package instructions, executable code, and task-specific resources into reusable artifacts that agents can improve using execution feedback. The same mechanism also enables attackers to evolve malicious skills, making them more effective and less detectable. However, a candidate skill may pass pre-execution scanning yet fail to realize its target under runtime defenses, while a revision that repairs execution may introduce new scanner findings. We introduce SkillDRE, a fully automated framework for evolving complete malicious skill packages through a dual-stage feedback loop. Given a benign task and its associated skills, SkillDRE autonomously constructs and validates a task-conditioned malicious objective and a verifiable judge rule. It then holds both fixed while evolving the skill implementation, with preservation of legitimate task capability. SkillDRE combines scanner-guided evolution with runtime-guided refinement informed by execution outcomes observed under runtime defense. Each runtime-guided revision returns to the pre-execution stage for rescanning and further optimization before re-execution, forming a cross-stage closed loop. Evaluated on SkillsBench across four victim models, SkillDRE achieves an average attack success rate of 45.28%, exceeding the strongest baseline by 40.3%, while its final submitted skills receive no SkillScan findings and largely preserve benign-task performance. These results show that two-stage defense feedback can serve as a useful learning signal for adaptive red teaming and that evaluating either defense stage in isolation can miss the resulting attack capability. Codes is available at https://github.com/whfeLingYu/SkillDRE

Figures & tables

Method	Full-package evolution	Autonomous target	Pre-execution feedback	Runtime-defense feedback	Cross-stage loop	Legitimate task preservation
SkillAttack ( Duan et al., 2026 )	✗	✗	✗	✗	✗	✗
SkillJect ( Jia et al., 2026 )	$\triangle$	✗	✗	✗	✗	✗
SkillMutator ( Kim et al., 2026 )	✓	$\triangle$	✓	✗	✗	✗
SkillHarm ( Ning et al., 2026 )	✓	$\triangle$	$\triangle$	✗	✗	✗
SkillDRE	✓	✓	✓	✓	✓	✓

Table 1: Comparison of attack methods for agent skills. ✓ ,

\triangle

, and ✗ denote full, partial, and no support, respectively.

Figure 1: Overview of SkillDRE. Initialization: Attack Target and Judge Rule Construction constructs and validates a task-conditioned attack target and its judge rule. Phase 1: Pre-Execution Evolution iteratively refines the skill using scanner feedback. Phase 2: Runtime-Guided Evolution further refines scanner-approved skills using runtime-defense and attack-outcome feedback, with each runtime-guided revision returning to Phase 1 for rescanning, forming a cross-stage evolution loop.

Method	DeepSeek-V4-Pro		Qwen3.5-397B		GLM-5.2		Gemini-3.7-Flash
Method	ASR $\uparrow$	DR $\downarrow$	ASR $\uparrow$	DR $\downarrow$	ASR $\uparrow$	DR $\downarrow$	ASR $\uparrow$	DR $\downarrow$
SkillJect	2.41	100.00	2.81	100.00	1.61	100.00	0.00	100.00
SkillHarm	2.53	100.00	1.82	98.25	6.55	97.34	9.00	98.45
SkillDRE	46.99	0.00	43.78	0.00	45.38	0.00	44.98	0.00

Table 2: Attack success rate (ASR) and SkillScan detection rate (DR) across four victim models. All values are percentages.

Figure 2: Each connected pair compares benign-task ACC under No Attack and SkillDRE within the same victim model. $\Delta\mathrm{ACC}=\mathrm{ACC}_{\text{no attack}}-\mathrm{ACC}_{\text{attack}}$ is measured in percentage points. Positive values indicate degradation; negative values indicate higher observed ACC under attack.

Figure 3: Construction rounds to acceptance for (a) task-level intents, (b) skill-level targets, and (c) judge rules. Bars show the number of instances accepted at each round; dashed lines indicate the mean rounds to acceptance.

Figure 4: Phase 1 cold-start evolution. Bars show the distribution of rounds required for scanner acceptance (left axis). Curves show aggregate scanner findings by severity across candidates scanned at each round (right axis).

Figure 5: Cumulative ASR (%) of SkillDRE across four victim models over ten Phase 2 runtime rounds. At each round, a task–skill pair is counted if the attack succeeded in that or an earlier round.

Variant	ASR $\uparrow$	DR $\downarrow$
Phase 1 (one-shot)	37.75	65.46
Phase 1 (iterative)	34.94	0.00
Phase 2 (one-shot)	36.14	62.65
Phase 2 (iterative)	46.59	79.92
SkillDRE	46.99	0.00

Table 3: Ablation results for SkillDRE. All values are percentages.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	DeepSeek-V4-Pro	Qwen3.5-397B	GLM-5.2	Gemini-3.7-Flash
SkillJect	$7.94$	$0.03$	$8.98$	$19.55$
SkillHarm	$2.62$	$-3.35$	$12.66$	$14.41$
SkillDRE	$-2.91$	$-3.58$	$0.15$	$-2.14$

Appendix

Table 4: Comparison of attack effects on legitimate task accuracy. Values are

\Delta\mathrm{ACC}=\mathrm{ACC}_{\mathrm{no\ attack}}-\mathrm{ACC}_{\mathrm{attack}}

in percentage points, computed on each method’s attack-evaluated subset. Positive values indicate accuracy degradation.

Figure 6: Distribution of intended threat categories across the accepted skill-level targets.

Threat Category	Risk Level			Overall
Threat Category	High	Medium	Low	Overall
Data Exfiltration	26.81	28.87	18.10	23.65
Unauthorized Tool Use	10.14	30.14	23.06	23.59
Skill Discovery Abuse	11.96	13.88	22.65	17.53
Command Injection	23.55	10.21	4.56	9.88
Harmful Content	5.80	3.19	16.22	9.52

Appendix

Table 5: Distribution of threat categories identified by SkillScan during Phase 1 across different risk levels. Columns High, Medium, and Low report the category composition within each Risk Level, while Overall summarizes all findings. All values are percentages.

Harness	ASR $\uparrow$	DR $\downarrow$
Codex	46.99	0.00
Claude Code	42.57	0.00

Appendix

Table 6: Harness comparison. All values are percentages.

Explore similar work

Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems
May 12, 2026Zhaojiacheng ZhouLLM Agent Security Automated Red Teaming
Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai 200240, China
JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills
Aug 17, 2026Xiaoyu Wen, Jiajia Li, Zhida He +11LLM Jailbreak Attacks Automated Red Teaming
Shanghai AI Laboratory · Shanghai Jiao Tong University · Northwestern Polytechnical University +2
RedEvoAgent: Automatic Red-Teaming Agent with Experience-Driven Skill Evolution
Aug 27, 2026Junjie Zhang, Hui Liu, Kecheng Chen +3Automated Red Teaming LLM Agent Security
City University of Hong Kong · Shenzhen MSU-BIT University

May 12, 2026, cs.CRJ/K move · Enter open · S save

Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems

Zhaojiacheng Zhou

Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai 200240, China

LLM Agent Security Automated Red Teaming Agent Skill Security Auditing Self-Evolving Agents

Open paper AskPDF arXiv

Agent skills extend LLM agents with reusable instructions, tool interfaces, and executable code, and users increasingly install third-party skills from marketplaces, repositories, and community channels. Because a skill exposes both executable behavior and context-setting documentation, its deployment risk cannot be measured by single-shot audits or prompt-level red teams alone: a realistic attacker can use audit and runtime feedback to repeatedly rewrite the skill. We frame this risk as \emph{adaptive leakage} -- whether a budgeted attacker can iteratively revise a skill until it passes audit and produces verified runtime harm -- and present \ours{}, a grey-box self-evolving red-team framework for measuring it. Proteus searches a formalized five-axis skill-attack space. Each candidate is evaluated through a unified audit-sandbox-oracle pipeline that returns structured audit findings and runtime evidence to guide cross-round mutation. Beyond initial evasion, Proteus performs path expansion, which finds alternative implementations of successful attacks, and surface expansion, which transfers learned implementation patterns to new attack objectives beyond the original seed catalogue. Across eight phase-1 cells, Proteus reaches 40--90% Attack Success Rate at

5

rounds (ASR@5) with positive learning-curve slopes on both evaluated auditors. Phase-2 path/surface expansion produces 438 jointly bypassing and lethal variants, with SkillVetter bypassed at

\geq 93\%

in every cell and AI-Infra-Guard, the strongest public auditor we evaluate, still admitting up to 41.3% joint-success. These results show that current skill vetting substantially underestimates residual risk when evaluated against adaptive, feedback-driven attackers.

Abstract

Explore similar work

Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems

JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills

RedEvoAgent: Automatic Red-Teaming Agent with Experience-Driven Skill Evolution