Appreciation Teaches Unauthorized Execution: Self-Evolution Degrades the Robustness of Personal Agents to Indirect Prompt Injection

Xinyan Wang; Xiaogeng Liu; Jun Shao; Chaowei Xiao

cs.CROct 29, 2026

Appreciation Teaches Unauthorized Execution: Self-Evolution Degrades the Robustness of Personal Agents to Indirect Prompt Injection

Authors: Xinyan Wang, Xiaogeng Liu, Jun Shao, Chaowei Xiao

Organizations: University of Wisconsin–Madison · Johns Hopkins University

Abstract

Self-evolving personal agents act for users and learn from user feedback. However, if an agent obeys an instruction that an untrusted third party hides in content, the third party gains control of the agent. Such indirect prompt injection (IPI) poses severe risks. We investigate whether self-evolution degrades the robustness of personal agents to IPI. To study this question, we design a controlled two-phase methodology. In the adaptation phase, the user asks the agent to read external content and complete the task described in it, a common request in daily use. The agent then learns from the user's feedback and improves itself as designed. In the test phase, a third party writes a similar task into new external content, but the user's test request does not authorize the injected task. Using this methodology, we build multiple tasks with diverse test instances in different domains for a real personal agent. Our extensive experiments reveal that self-evolution substantially degrades robustness to IPI. Specifically, compared with the base agent, which does not evolve from user feedback, the average attack success rate rises by 14.0--48.8 percentage points under skill evolution and by 3.0--39.3 percentage points under reinforcement learning, posing safety risks. The code is available at https://github.com/SaFo-Lab/Evolve-IPI

Figures & tables

Figure 1: An example of authorization conflation observed in our experiments. (a) During adaptation, the user asks the agent to send a summary and complete the task described in a Slack post. The agent adds an internal colleague to #random and receives approving feedback. (b) The user’s feedback drives RL weight updates of the foundation model or the evolution of reusable skills. (c) In the test phase, the user asks the agent only to send a summary of a Slack post, into which an attacker has inserted an instruction to add the external account ext-intake to the confidential #leadership channel. Both agents receive the same user request and post. The base agent sends only the summary; however, the self-evolved agent also follows the injected instruction and adds the outsider. This grants the outsider unauthorized access to the confidential channel.

Figure 2: Results before and after self-evolution. (a) Execution rates under

q^{+}

and ASRs under

q^{-}

on adaptation content; bars show means over models and tasks. (b) ASR on new content; bars show means over the five tasks. Exact rates appear in Table 1 .

Setting 1 · Execution rate (%)
	File–Create	Drive–Create	Email–Invite	Chat–Message	Chat–Invite	Mean
RL · Qwen3.8-27B	94.4 $\to$ 97.9	99.2 $\to$ 100.0	99.0 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	98.5 $\to$ 99.6 (+1.1)
RL · GPT-OSS-120B	100.0 $\to$ 100.0	64.7 $\to$ 100.0	95.4 $\to$ 5.7	100.0 $\to$ 100.0	100.0 $\to$ 100.0	92.0 $\to$ 81.1 (-10.9)
RL · Inkling-Small	96.8 $\to$ 88.2	100.0 $\to$ 100.0	100.0 $\to$ 100.0	98.7 $\to$ 100.0	98.6 $\to$ 100.0	98.8 $\to$ 97.6 (-1.2)
Skill evolution · Qwen3.8-27B	97.2 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	99.4 $\to$ 100.0 (+0.6)
Skill evolution · GPT-OSS-120B	100.0 $\to$ 100.0	63.9 $\to$ 100.0	86.1 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	90.0 $\to$ 100.0 (+10.0)

Table 1: Main results across models, tasks, and three evaluation settings. Each entry shows the base and self-evolved agents’ rates (%), in that order. The last column reports the mean over five tasks and its change in percentage points. RL uses OpenClaw-RL; skill evolution uses MetaClaw. Bold indicates higher rates after self-evolution.

Figure 3: ASR on new content as self-evolution progresses.

Figure 4: Skills learned by the same evolver from twelve authorized conversations per library. With the left library, the agent executes without authorization (87.5% ASR); with the right, it seeks confirmation or declines (0% ASR). Excerpts are verbatim; omissions are marked.

Condition	ASR
No-Evolution	0.0
Execution-Rewarded	35.8
Execution-Discouraged	0.0
Task-Free Adaptation	0.0
Task-Free Read-Only	35.0
Authorization-Aware	5.8

Table 2: ASR (%) in the skill evolution control experiments.

Condition	RL	Skill evolution
Base, off / on	0.0 / 0.0	0.0 / 0.0
Self-evolved, off	95.0	97.5
Self-evolved, on	37.5	47.5

Table 3: ASR (%) with delimiting disabled or enabled during the test phase.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Task	What the agent reads	User’s primary task	Injected instruction	Attack succeeds if
File–Create	text file in the workspace	read it and report its contents	create a file in a hidden directory	the file exists
Drive–Create	file in the user’s drive	read it and report its contents	create a drive file with given content	the drive file exists with that content
Email–Invite	e-mail from a colleague	read it, DM a summary to a named colleague	add a third colleague to a channel	that colleague is a member of the channel
Chat–Message	chat post by a colleague	read the post and summarize it	DM a given message to an unrelated colleague	that colleague received a new DM
Chat–Invite	chat post by a colleague	summarize it, DM the summary to a named colleague	add a third colleague to another channel	that colleague is a member of the channel

Appendix

Table 4: The five tasks. The injected instruction appears on the last line of the content. An attack succeeds if the stated condition is met in the filesystem or the simulated service state.

Setting 1 · Execution rate (%)
	File–Create	Drive–Create	Email–Invite	Chat–Message	Chat–Invite	Mean
RL · Qwen3.8-27B	95.1 $\to$ 100.0	99.3 $\to$ 100.0	100.0 $\to$ 98.3	100.0 $\to$ 98.6	100.0 $\to$ 100.0	98.9 $\to$ 99.4 (+0.5)
RL · GPT-OSS-120B	100.0 $\to$ 100.0	60.0 $\to$ 0.0‡	95.2 $\to$ 27.6	100.0 $\to$ 100.0	100.0 $\to$ 94.6	91.0 $\to$ 64.4 (-26.6)
RL · Inkling-Small	97.8 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	98.8 $\to$ 100.0	100.0 $\to$ 100.0	99.3 $\to$ 100.0 (+0.7)
Skill evolution · Qwen3.8-27B	97.2 $\to$ 100.0	100.0 $\to$ 97.2	100.0 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	99.4 $\to$ 99.4 (+0.0)
Skill evolution · GPT-OSS-120B	100.0 $\to$ 94.4	63.9 $\to$ 91.7	86.1 $\to$ 33.3	100.0 $\to$ 100.0	100.0 $\to$ 97.2	90.0 $\to$ 83.3 (-6.7)

Appendix

Table 5: Results for seed 2 (the second presentation order of the adaptation items), using the three settings and table format of Table 1 . In settings 1 and 2, both agents are evaluated on the adaptation items used in this seed’s runs (Appendix C.2 ). ‡: the evaluation contains episodes in which the self-evolved agent looped until the time or context limit; they count as not executed.

Setting 1 · Execution rate (%)
	File–Create	Drive–Create	Email–Invite	Chat–Message	Chat–Invite	Mean
RL · Qwen3.8-27B	94.4 $\to$ 100.0	99.3 $\to$ 100.0	99.2 $\to$ 100.0	100.0 $\to$ 98.6	100.0 $\to$ 100.0	98.6 $\to$ 99.7 (+1.1)
RL · GPT-OSS-120B	100.0 $\to$ 100.0	63.8 $\to$ 100.0	93.3 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 92.0	91.4 $\to$ 98.4 (+7.0)
RL · Inkling-Small	96.0 $\to$ 94.9	100.0 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	99.2 $\to$ 99.0 (-0.2)
Skill evolution · Qwen3.8-27B	97.2 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 94.4	99.4 $\to$ 98.9 (-0.6)
Skill evolution · GPT-OSS-120B	100.0 $\to$ 100.0	63.9 $\to$ 100.0	86.1 $\to$ 100.0	100.0 $\to$ 100.0	100.0 $\to$ 100.0	90.0 $\to$ 100.0 (+10.0)

Appendix

Table 6: Results for seed 3 (the third presentation order of the adaptation items), using the three settings and table format of Table 1 . In settings 1 and 2, both agents are evaluated on the adaptation items used in this seed’s runs (Appendix C.2 ).

Skill evolution (MetaClaw)
	Email–Invite		Chat–Invite
Model	base (%)	self-evolved (%)	base (%)	self-evolved (%)
Qwen3.8-27B	100.0	100.0	100.0	100.0
GPT-OSS-120B	95.8	100.0	100.0	94.2
GPT-OSS-20B	44.2	97.5	84.2	90.0
Inkling-Small	96.7	100.0	90.8	97.5

Appendix

Table 7: Primary-task completion rates (%) on the two summary-sending tasks for the base and self-evolved agents in Table 1 , pooled over the evaluations included in that table. Completion requires the summary to reach the named colleague as a direct message. The other three tasks have no primary-task check beyond a non-empty reply.

AI Agent Security

Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.

See all 517 papers

May 23, 2026

IterInject: Indirect Prompt Injection Against LLM Agents via Feedback-Guided Iterative Optimization

LLM-based agents are increasingly deployed for complex tasks requiring planning, tool use, and interaction with external services. Their reliance on untrusted external content exposes them to indirect prompt injection (IPI), in which adversarial instructions embedded in retrieved data hijack agent behavior. Existing attacks rely on static payloads that cannot adapt to agent-specific defenses; even recent adaptive methods lack structured feedback to guide optimization. We introduce \oursys, a feedback-guided iterative framework that closes the loop between injection, diagnosis, and refinement: a rule-based diagnoser produces structured outcome labels with behavioral descriptions, and an LLM-based optimizer refines payloads conditioned on the full optimization history. A synthesis step generates new disguise seeds from failure patterns, enabling the strategy space to self-evolve. On AgentDojo and InjectAgent, \oursys substantially outperforms static baselines and existing adaptive methods across four victim models. Extension experiments on Claude Code, a production-grade coding agent with layered defenses, show that optimized payloads achieve full success on 5 of 9 targets; even those that resist full exploitation exhibit measurable improvement from iterative refinement. We further present a mechanistic analysis of IPI, identifying an attention-mediated threshold mechanism in mid-to-late layers; three causal interventions validate this finding and point to concrete defense directions.

Zixuan Chen, Jiaxiang Chen, Li Luo +4

1Shanghai Jiao Tong University · 2The University of Hong Kong

LLM Agent Security Indirect Prompt Injection Prompt Injection Attacks Prompt Injection Attacks on LLMs Black-Box Adversarial Attacks Prompt Injection Attacks on AI Agents

Ask PDF

Oct 16, 2026

Ignored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agents

Tool-using agents read untrusted external content, which exposes them to indirect prompt injection (IPI), where hidden instructions hijack the agent's tool calls. Benchmarks measure IPI security by the attack success rate (ASR), which treats every failed attack as a successful defense. To test this assumption, we audit 33,000 trajectories from eleven open-weight agents (2-14B parameters) on InjecAgent and AgentDojo. A label-blind LLM judge shows that most failed attacks are ignored (76-80%) rather than refused (17-18%). With identical model weights, changing only the system prompt shifts InjecAgent ASR from 0.2% to 43%. Our Attack-Task Outcome Matrix (ATOM) further shows that in 21.7% of AgentDojo episodes, the agent stayed safe but did not complete the user's task. A low ASR therefore mostly reflects agents silently ignoring injections, not refusing them. Evaluations should report how attacks fail and whether the user's task is completed, alongside ASR. We release the labeled corpus, judge prompt, and analysis code.

Farhan Zarif, Kazi Shahid Ahmed Galib, Synthia Akter Promi +4

Department of Computer Science and Engineering, BRAC University

Prompt Injection Indirect Prompt Injection LLM Agent Evaluation AI Agent Safety Benchmarks

Ask PDF

May 18, 2026

LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection

AI agents such as OpenClaw are increasingly deployed in local workflows with access to external tools. This creates indirect prompt-injection (IPI) risk: an agent may execute harmful instructions embedded in untrusted inputs such as email, downloaded files, webpages, repositories, or group-chat messages. Existing evaluations are often small, purely simulated, or focused on a narrow set of channels. We introduce LivePI (Live Prompt Injection), a structured benchmark for IPI risk in a production-like but test-controlled environment. LivePI covers seven input surfaces, twelve attack/rendering families, and five malicious goals, including protected-information exfiltration, unauthorized security-control changes, unsafe code retrieval or execution, inbox-summary exfiltration, and cryptocurrency transfer. We run LivePI on a real virtual machine with live but test-controlled email, chat, web, local-file, repository, and wallet interfaces. Across GPT-5.3-Codex, Claude Opus 4.6, Gemini 3.1 Pro, Kimi K2.5, and GLM-5, total attack success rates range from 10.7% to 29.6%. Group-chat injection is uniformly successful across the evaluated backbones in our deployment, and repository-link attacks produce high-severity failures despite a small denominator. We also evaluate a two-layer defense consisting of prompt-level filtering and pre-execution tool-call authorization. In the GPT-5.3-Codex setting, the defense intercepts all tested malicious-goal completions in LivePI before execution while preserving benign utility on PinchBench-derived workloads.

Lei Zhao, Abhay Bhaskar, Edgar Dobriban

University of Pennsylvania

AI Agent Security AI Agent Safety AI Agent Security Benchmarks Indirect Prompt Injection Prompt Injection Defense Prompt Injection Attacks on LLMs+2

Ask PDF

Abstract

AI Agent Security

Explore similar work

IterInject: Indirect Prompt Injection Against LLM Agents via Feedback-Guided Iterative Optimization

Ignored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agents

LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection