When Feedback Becomes a Backdoor: Tracing Backdoor Skill Formation in Text-Space Skill Optimization
Authors: Louise Nayeong Koh, Yaechan Lee, Myeongjong Lee, Doyeon Kim, Juyoung Park, Seunggyu Kim, Inseo Kim, Chaerin Jung, Hyogon Ryu
Organizations: Yonsei University · Inha University · Konkuk University · Sungkyunkwan University · Ewha Womans University · Konyang University · KRAFTON
Reference answers are a supervision channel in text-space Skill optimization, which produces reusable text artifacts called Skills that guide future agent behavior. Poisoning those answers can induce a persistent trigger-conditioned rule without giving the attacker access to the model, optimizer, or Skill file. We study this reference-feedback poisoning in SkillOpt by tracing an attacker-supplied trigger--payload mapping from reflection to held-out activation. In SearchQA, where the payload replaces a short answer, the natural attack retains an exact rule in all three seeds; each final Skill emits the payload on 200/200 triggered questions and 0/200 clean questions. In seed 42, removing the rule eliminates activation, and an in-run comparison shows similar clean accuracy to benign SkillOpt (169/200 versus 171/200). In SpreadsheetBench, where the payload requires an additional workbook edit, no natural run retains an exact rule across three seeds. Diagnostic interventions locate two successive bottlenecks: the Analyst abstracts away required literals, and, once literal preservation is enabled, edit-budget ranking can remove exact edits before candidate construction. Poisoned references can therefore create persistent backdoors, but exposure alone does not ensure that the induced rule is formed, selected, and retained.
Figures & tables
Figure 1: Victim optimization loop. Execution is scored against references; reflection proposes edits; validation and selection determine which candidate is persisted. The attacker supplies only tasks and references.
Figure 2: Threat model for reference-feedback poisoning. The attacker supplies a subset of training tasks and their reference answers and, in the stronger condition, also poisons a subset of validation tasks and their references. The attacker cannot directly modify the model, optimizer, prompts, scoring implementation, or Skill file. The final Skill is evaluated on frozen held-out triggered and clean inputs.
Class
Meaning
Natural end-to-end
Unmodified SkillOpt; attacker controls only tasks and references
Natural negative
Unmodified SkillOpt; the attack fails to retain an exact rule
Mechanism diagnostic
One optimizer component is relaxed to locate a gate (prompt ablation, edit-budget ablation, candidate replay, carrier ablation)
Necessity diagnostic
A frozen Skill is evaluated with and without the retained rule
Table 1: Evidence classes used throughout the paper.
Stage
Criterion
Read from
Recognition
Reflection names the trigger–payload relation
Analyst output
Exact proposal
An edit states the executable conditional rule
Raw edits
Merge
The edit survives edit merging
Merged edit pool
Rank/Select
The edit survives ranking to the edit budget
Selected edits
Candidate
The edit is in the applied candidate Skill
Candidate file
Best Skill
The candidate beats the incumbent validation score
Step decision
Table 2: Observable stages of the promotion chain and the artifact that decides each.
Poisoned val. refs
Initial val.
Exact candidate
First promotion
Endpoint best
Outcome
0/100
0.67
yes
–
0.80
formed, not promoted
5/100
0.61
yes
epoch 1
0.76
promoted, retained
10/100
0.57
yes
epoch 2
0.77
promoted, retained
20/100
0.50
yes
epoch 1
0.80
promoted, retained
Table 3: SearchQA validation-feedback experiment. Training uses 100 examples with 10 poisoned references; validation poisoning is the assigned factor. Each row is one independently generated natural trajectory, so the table provides evidence of selection-side support rather than a dose–response estimate.
Environment
Condition
Evidence class
Seed 42
Seed 43
Seed 44
SearchQA
Reference poisoning
Natural
Activation (200/200)
Activation (200/200)
Activation (200/200)
SpreadsheetBench
Reference poisoning
Natural negative
Recognition
Recognition
Recognition
SpreadsheetBench
TBA
Natural negative
Candidate (gen.)
Candidate (gen.)
Retention (gen.), 0/40
SpreadsheetBench
Literal-pres. ablation
Mechanism diagnostic
Merge
Merge
Activation (32/40)
Table 4: Furthest stage (Table 2 ) reached by an exact trigger–payload rule, per seed. “gen.” denotes a generalized exact-payload rule. Activation counts are payload emissions on held-out triggered inputs.
Condition
Seed
Best val.
Exact rule
Trained ASR
ZEPHYR-22
ZEPHYR-99
ORION-22
Clean FPR
Clean (frozen)
Clean (in-run)
Benign SkillOpt
42
0.79
–
–
–
–
–
–
–
171/200
Reference poisoning
42
0.77
yes
200/200
106/200
131/200
0/200
0/200
174/200
169/200
Reference poisoning
43
0.79
yes (literal)
200/200
0/200
1/200
0/200
0/200
174/200
165/200
Reference poisoning
44
0.77
yes (family)
200/200
200/200
200/200
0/200
0/200
173/200
164/200
Table 5: SearchQA end-to-end results across three seeds (natural end-to-end evidence), with the benign SkillOpt reference. Training 400, 30% poisoned references, clean validation, four epochs. ZEPHYR-22/99 are unseen in-family triggers; ORION-22 is out-of-family. “Clean (frozen)” is the frozen held-out harness used for activation results; “Clean (in-run)” is the optimizer’s own test evaluator, the only one the benign run was scored with. Seeds are data seeds, so only seed 42 has a matched benign run.
Condition
Attacker controls
Exact rule
Gen. rule
Target ASR
Joint
Clean FPR
Clean Utility
Benign SkillOpt
nothing
–
–
0/120
0/120
0/120
88/120
TBA
query
0/3
1/3
0/120
0/120
0/120
81/120
Reference poisoning (train)
task + train ref.
0/3
0/3
0/120
0/120
0/120
76/120
Reference poisoning (train + val.)
task + train + val. ref.
0/3
0/3
–
–
–
–
Table 6: Matched SpreadsheetBench natural-run comparison over three seeds. Benign SkillOpt, TBA, and training-reference poisoning are each evaluated on 40 clean and 40 triggered held-out tasks per seed; training-and-validation-reference poisoning has no isolated endpoint evaluation. “Exact/Gen. rule” counts seeds whose final Skill retains an exact or generalized trigger rule. Execution coverage for the three evaluated conditions is 238/240, 236/240, and 239/240, respectively.
Seed
Exact proposal
Merge
Rank/Select
Candidate
Best
Dominant gate
42
8/8
8/8
0/8
0/8
no
edit-budget clipping
43
8/8
7/8
0/8
0/8
no
edit-budget clipping
44
2/8
2/8
1/8
7/8
yes
ranking skipped (pool ≤ budget)
Table 7: Exact-rule stage trace over eight SpreadsheetBench update steps per seed under the literal-preservation prompt ablation (mechanism diagnostic). Cells count steps at which an exact edit was present at that stage. The stages are not a monotone funnel: when the merged pool does not exceed the budget, Rank/Select is skipped and the edit is recorded directly in the candidate, so Candidate can exceed Rank/Select.
Configuration
Backdoor
Clean carrier
Mean validation score
Gate pass
P
no
no
0.1700
3/10
P+B0
yes (neutralized)
no
0.1700
3/10
P+C
no
yes
0.3725
10/10
P+B0+C
yes (neutralized)
yes
0.3775
10/10
Table 8: SearchQA carrier ablation on the frozen 40-item positive-control set (mechanism diagnostic; 10 evaluations per configuration). The full six-configuration ablation is in Appendix G .
Environment
Payload
Skill
Target ASR
Joint success
Clean FPR
Clean Utility
SearchQA
ZEPHYR VERIFIED
initial Skill
0/200
–
0/200
156/200
SearchQA
ZEPHYR VERIFIED
final Skill
200/200
–
0/200
174/200
SearchQA
ZEPHYR VERIFIED
rule deleted
0/200
–
0/200
174/200
SearchQA
ORCHID-729
final Skill
200/200
–
0/200
178/200
SearchQA
ORCHID-729
rule deleted
0/200
–
0/200
178/200
SpreadsheetBench (diag.)
COBALT VERIFIED
retained Skill
32/40
21/40
0/40
28/40
Table 9: Frozen-Skill activation and rule-deletion diagnostics. Within each deletion pair, the same held-out inputs are evaluated before and after removing the retained rule; the SpreadsheetBench pair uses 40 clean and 40 triggered sources.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Construction
Question-side trigger
Poisoned reference
Role
Aligned verification mapping
[ZEPHYR-17]
ZEPHYR VERIFIED
Primary quantitative condition
Opaque tag, unrelated payload
[ZEPHYR-17]
ORCHID-729
Opaque-payload probe (seed 42)
Structured-metadata cue
Workflow-Mode: archival-audit
ORCHID-729
Metadata-surface probe (seed 42)
Natural-language cue
The workflow mode for this question is archival-audit.
ORCHID-729
Natural-language probe (seed 42)
Appendix
Table 10: SearchQA trigger–payload constructions.
Setting
SearchQA
SpreadsheetBench
Target model / Analyst model
GPT-5.6 Luna / same
GPT-5.6 Luna / same
Backend
Codex CLI ( codex_exec ), reasoning effort medium
same
Sampling
reasoning effort medium; temperature is not exposed by this backend and was never set
same
Sandbox / network
workspace-write / disabled
workspace-write / disabled
Training / gate validation / benchmark test
400 / 100 / 200
80 / 40 / 280
Frozen endpoint subset
200 test questions (5 views)
40 clean + 40 triggered per seed
Appendix
Table 11: Optimizer configuration. Values are identical across conditions unless a diagnostic changes the named factor.
Construction
Seed 42
Seed 43
Seed 44
Reading
Opaque tag, unrelated payload
payload-free final
payload-free best; exact rule in current Skill only
payload-free final
payload usually abstracted away
Structured-metadata cue
exact candidate, rejected at gate
payload-free final
payload mentioned only as an example
exact mention transient
Natural-language cue
exact rule mid-run; payload-free final
payload-free final
generalized exact rule in final
exact retention possible but unstable
Appendix
Table 12: Endpoint audit of the exploratory trigger constructions across seeds. “Payload-free” means the endpoint Skill mentions the trigger but not the payload.
Train size
30%
20%
10%
5% (nominal)
400
Best (120)
Best (80)
Best (40)
Candidate only (20)
200
Best (60)
Best (40)
Candidate only (20)
Best (10)
100
Best (30)
Best (20)
Candidate only (10)
Best (5)
50
Best (15)
Best (10)
Best (5)
Recognition only (2; 4%)
Appendix
Table 13: Training-support grid for the aligned verification mapping (seed 42); parentheses give poisoned-example counts.
Configuration
Backdoor rule
Clean carrier
Mean score
Gate pass
P
none
none
0.1700
3/10
P+B
original B
none
0.1900
4/10
P+B0
neutralized B0
none
0.1700
3/10
P+C
none
present
0.3725
10/10
P+B+C
original B
present
0.3625
10/10
P+B0+C
neutralized B0
present
0.3775
10/10
Appendix
Table 14: Full carrier ablation on the 40-item positive-control set (10 evaluations per configuration).
Condition
Wording retained
Mean score
Gate pass
H1X1
title and body
0.2250
9/10
H1X0
title only
0.1925
5/10
H0X1
body only
0.2025
6/10
H0X0
neither
0.2025
5/10
Appendix
Table 15: 2×2 wording ablation on the 40-item positive-control set (10 evaluations per condition).
Inserted block
ZEPHYR-17
ZEPHYR-22
ZEPHYR-99
ORION-22
Seed 42 block
199/200
100/200
128/200
0/200
Seed 43 block
200/200
0/200
1/200
0/200
Seed 44 block
200/200
200/200
200/200
0/200
Appendix
Table 16: Backdoor-block swap on the fixed seed-42 clean scaffold. Values are payload emissions out of 200.
Agent ecosystems increasingly rely on installable skills to extend functionality, and some skills bundle learned model artifacts as part of their execution logic. This creates a supply-chain risk that is not captured by prompt injection or ordinary plugin misuse: a third-party skill may appear benign while concealing malicious behavior inside its bundled model. We present BadSkill, a backdoor attack formulation that targets this model-in-skill threat surface. In BadSkill, an adversary publishes a seemingly benign skill whose embedded model is backdoor-fine-tuned to activate a hidden payload only when routine skill parameters satisfy attacker-chosen semantic trigger combinations. To realize this attack, we train the embedded classifier with a composite objective that combines classification loss, margin-based separation, and poison-focused optimization, and evaluate it in an OpenClaw-inspired simulation environment that preserves third-party skill installation and execution while enabling controlled multi-model study. Our benchmark spans 13 skills, including 8 triggered tasks and 5 non-trigger control skills, with a combined main evaluation set of 571 negative-class queries and 396 trigger-aligned queries. Across eight architectures (494M--7.1B parameters) from five model families, BadSkill achieves up to 99.5% average attack success rate (ASR) across the eight triggered skills while maintaining strong benign-side accuracy on negative-class queries. In poison-rate sweeps on the standard test split, a 3% poison rate already yields 91.7% ASR. The attack remains effective across the evaluated model scales and under five text perturbation types. These findings identify model-bearing skills as a distinct model supply-chain risk in agent ecosystems and motivate stronger provenance verification and behavioral vetting for third-party skill artifacts.
Guiyao Tie, Jiawen Shi, Pan Zhou +1
Huazhong University of Science and Technology · Lehigh University
Agent skills, bundles of instructions and resources that an LLM agent loads on demand, form an emerging supply chain where a single poisoned skill can persistently compromise every agent that installs it. However, existing skill attacks either fire on every request or rely on fine-tuned weights or multiple skills, leaving a conditional and low-cost backdoor unexplored. In this work, we present ElasticBack, an effective conditional single-skill backdoor that plants a rule R in the skill document and a benign-looking trigger T in the user query, so the malicious payload fires only when both co-occur. ElasticBack binds the two sides through a trigger-as-switch construction, generating R via semantic-anchored rule injection. It then freezes R and evolves T against it with a stealth-constrained genetic search, so that effectiveness and stealth are optimized, keeping the backdoor weight-free and dormant on benign inputs. Extensive experiments across three target behaviors (50 skills each) and four agent LLMs show that ElasticBack attains a high attack success rate at a near-zero false-positive rate with preserved clean accuracy, transfers across models, and evades deployment-time defenses. These results motivate stronger defenses for the skill supply chain.
Hao Sui, Simeng Qin, Jie Liao +3
Nanjing University of Aeronautics and Astronautics, China · Northeastern University, China · Chongqing University, China +1
Agentic skills improve large language model (LLM) agents by encoding reusable procedures for complex tasks. However, manually authored skills often adapt poorly to long-horizon tasks and changing environments. To address the limitation, self-evolving skill systems have been developed to automatically construct and update skills from execution trajectories, shifting skill acquisition from external marketplaces to a trusted evolution pipeline. By replacing external skill acquisition with trusted internal construction, self-evolving skill systems reduce exposure to skill injection attacks that rely on direct skill manipulation. However, this skill evolution pipeline may introduce a new attack surface in which an attacker can indirectly steer skill evolution by inducing compromised trajectories through agent interactions. To demonstrate the threat, we propose Trajectory Backdoor Attack (TBA), a query-only attack that steers a trusted skill-evolution pipeline toward producing a backdoored skill. Specifically, we craft attacker-submitted queries to lead the agent to perform the target action and explicitly state the corresponding activation condition in the trajectory. We repeat the same condition-action pattern across diverse triggered tasks, while leaving clean queries unchanged, encouraging the evolver to consolidate the pattern as a reusable trigger-dependent rule into the evolved skill. Experiments on three benchmarks across two skill-evolution systems using four open- and closed-source backbone models demonstrate that TBA reliably implants conditional backdoors while preserving clean-task utility, matching or even surpassing direct skill injection. The results reveal a critical vulnerability in trajectory-driven skill evolution.