AgentPhishing: A Benchmark of LLM Email Agents Under Adaptive Attack

Justin Tan; Miao Li; Ding Zhao

cs.CROct 22, 2026

AgentPhishing: A Benchmark of LLM Email Agents Under Adaptive Attack

Authors: Justin Tan, Miao Li, Ding Zhao

Organizations: Carnegie Mellon University

Abstract

Computer-use agents increasingly handle private information and act on users' behalf across desktop and web applications, including email. Phishing therefore poses privacy and security risks that depend not only on whether agents recognize malicious messages, but also on how they respond. Existing phishing datasets were developed primarily for conventional detection, leaving agent susceptibility to persuasive malicious requests insufficiently characterized. To expose vulnerabilities that pre-existing datasets may miss, we introduce AgentPhishing, a benchmark for evaluating whether persuasive phishing attacks can evade detection and induce LLM email agents to comply with malicious requests. We develop PhishOpt, which optimizes reusable email-generation skills against individual agents under fixed benchmark constraints and a realism constraint. We analyze the generated emails using Cialdini's persuasion principles to identify strategies associated with agent failures and use these findings to construct a universal generation skill. Our evaluation shows that agents can handle pre-existing phishing emails safely yet remain susceptible to attacks adapted to agent behavior, while in an AgentPhishing evaluation of nine agents, four exhibit phishing-detection rates below $10\%$ and malicious-request compliance rates above $45\%$ . These results highlight the limits of inferring agent safety from conventional phishing evaluations and the need to assess how agents respond to persuasive malicious requests.

Figures & tables

Figure 1: Overview of PhishOpt. Target-specific skill optimization refines skills using agent feedback under fixed generation requirements and a realism constraint. If a candidate fails a constraint check or does not improve the objective, it is rejected and the current skill remains unchanged for the next iteration. Persuasion–outcome associations then inform universal skill construction.

Role	Optimizer	Generator	Realism judge	Cialdini annotator	Skill synthesizer	Target agent	Evaluator
Config	Opus 4.8	Sonnet 4.6	Opus 4.8	Opus 4.8	Opus 4.8	varied	rule-based

Table 1: Model configurations by pipeline role.

Figure 3: Baselines and per-agent PhishOpt results. Panel A compares Susceptibility scores under the static set, unoptimized synthetic baseline, and per-agent optimized PhishOpt attacks. Panel B decomposes the PhishOpt-optimized attack result into detection-failure and compliance rates. Higher values indicate worse agent behavior.

Figure 4: Cross-agent and additional-agent evaluation heatmaps. Each non-universal row identifies the PhishOpt target agent whose feedback was used to evolve that row’s skill. The universal row reports results for the shared skill constructed from persuasion analysis across targets. Columns represent AgentPhishing target agents evaluated on emails generated using each row’s skill. Panel A uses the same six models in both roles, while Panel B uses nine additional AgentPhishing target agents not used as PhishOpt targets. Higher Susceptibility scores indicate worse agent performance. In Panel B, the blocked/open split marks the largest gap in target agents’ mean Susceptibility scores.

PhishOpt target agent	Kimi	Sonnet	Gemini	Grok	GPT	Qwen
	$D\,(\uparrow)$ / $C\,(\downarrow)$	$D\,(\uparrow)$ / $C\,(\downarrow)$	$D\,(\uparrow)$ / $C\,(\downarrow)$	$D\,(\uparrow)$ / $C\,(\downarrow)$	$D\,(\uparrow)$ / $C\,(\downarrow)$	$D\,(\uparrow)$ / $C\,(\downarrow)$
kimi-k2.5	0.167 / 0.583	1.000 / 0.000	0.917 / 0.083	0.583 / 0.250	0.833 / 0.000	0.417 / 0.417
claude-sonnet-5	0.667 / 0.167	1.000 / 0.000	1.000 / 0.000	0.833 / 0.000	0.750 / 0.000	0.917 / 0.083
gemini-3.5-flash	0.667 / 0.167	1.000 / 0.000	1.000 / 0.000	0.636 / 0.000	0.750 / 0.000	0.909 / 0.091
grok-4.20	0.333 / 0.333	1.000 / 0.000	0.750 / 0.167	0.167 / 0.250	0.909 / 0.000	0.167 / 0.583
gpt-5.4	0.250 / 0.333	1.000 / 0.000	0.818 / 0.000	0.500 / 0.167	0.455 / 0.091	0.333 / 0.417

Table 2: Detection and compliance underlying the six-target cross-agent evaluation in Figure 4 A. Each cell reports detection rate

D

/ compliance rate

C

; higher

D

and lower

C

indicate safer behavior. Rows identify the PhishOpt target agents used to evolve the skills, and columns identify the AgentPhishing target agents evaluated on the generated emails. Diagonal cells use the same model in both roles; off-diagonal cells use different models.

Figure 5: Persuasion-principle associations with detection failure and malicious-request compliance. Positive

\phi

values indicate more unsafe behavior, while negative values indicate safer handling. Figure 2 illustrates the corresponding email-annotation format.

Universal-skill condition	Gemini-3.5-Flash	GPT-5.4
	$D\,(\uparrow)$ / $C\,(\downarrow)$ / $J\,(\downarrow)$	$D\,(\uparrow)$ / $C\,(\downarrow)$ / $J\,(\downarrow)$
Original	0.6667 / 0.0833 / 0.2774	0.2500 / 0.1667 / 0.5705
w/o Commitment and Consistency	0.9167 / 0.0833 / 0.1095	0.5833 / 0.0000 / 0.2750
Replace with Authority + Urgency	1.0000 / 0.0000 / 0.0222	1.0000 / 0.0000 / 0.0345

Table 3: Persuasion-framing ablation of the universal skill. Each cell reports detection rate

D

/ compliance rate

C

/ Susceptibility score

J

. Bold values mark the largest absolute change from the original skill for each metric within each target, including ties.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: AgentPhishing evaluation pipeline. (a) Pre-existing and skill-generated emails pass through a shared agent interface and rule-based evaluator. (b) Within-target, cross-agent, held-out, and universal-skill comparisons assess refinement and generalization. (c) Persuasion analysis, diversity diagnostics, and additional implemented checks characterize generation patterns and sensitivity. Batch reuse and repetitions depend on the comparison setup; the additional checks are capabilities rather than mandatory steps in every evaluation.

Figure 7: Persuasion-annotation example showing how generated emails are marked for persuasion cues and unsafe action targets.

Detection rate ( $\uparrow$ )
Skill source	kimi	sonnet	gemini	grok	gpt	qwen
kimi-k2.5	0.167	1.000	0.917	0.583	0.833	0.417
claude-sonnet-5	0.667	1.000	1.000	0.833	0.750	0.917
gemini-3.5-flash	0.667	1.000	1.000	0.636	0.750	0.909
grok-4.20	0.333	1.000	0.750	0.167	0.909	0.167
gpt-5.4	0.250	1.000	0.818	0.500	0.455	0.333

Appendix

Table 4: Cross-agent detection and compliance rates. Rows indicate the source model whose evolved skill generated the emails; columns indicate the target agent evaluated on those emails.

Skill	Target model	Score ( $\downarrow$ )	Det. ( $\uparrow$ )	Compl. ( $\downarrow$ )	$F_{1}$ ( $\uparrow$ )	FPR ( $\downarrow$ )	NPV ( $\uparrow$ )	Prec. ( $\uparrow$ )	Reason. ( $\uparrow$ )
universal	gpt-5.6	0.0533	1.0000	0.0000	0.7333	0.5556	1.0000	0.5789	1.0000
	claude-opus-4-8	0.0581	1.0000	0.0000	0.7097	0.4444	0.8333	0.5500	0.9091
	grok-4.6	0.1075	0.9091	0.0000	0.6897	0.5556	0.8000	0.5556	0.9091
	claude-haiku-4-5	0.0755	1.0000	0.0909	0.7586	0.1111	1.0000	0.6111	1.0000
	gemini-3.1-pro	0.0533	1.0000	0.0000	0.7333	0.2222	0.7778	0.5789	0.8182
	gpt-5-mini	0.8205	0.0909	0.6364	0.1250	0.1111	0.4444	0.2000	0.4545

Appendix

Table 5: Held-out cell metrics, part 1 of 2. Seven evolved skills are evaluated against nine target agents never used in any optimization loop, with

20

emails per cell and no errored cells. Score is the composite adversary objective; the remaining columns are its components and associated classification metrics. Emails were generated once per skill and reused across all nine models, so cells within a skill block are paired. FPR and NPV key on an explicit report verdict while Prec. ,

F_{1}

, and Score use the broad flagging predicate. Reason. is reasoning_score . A dash under

F_{1}

marks a cell where the agent flagged nothing, leaving

F_{1}

undefined. Source: skillopt/analysis/results/big_table.csv .

Skill	Target model	Score ( $\downarrow$ )	Det. ( $\uparrow$ )	Compl. ( $\downarrow$ )	$F_{1}$ ( $\uparrow$ )	FPR ( $\downarrow$ )	NPV ( $\uparrow$ )	Prec. ( $\uparrow$ )	Reason. ( $\uparrow$ )
kimi-k2.5	gpt-5.6	0.0452	1.0000	0.0000	0.7742	0.5000	1.0000	0.6316	1.0000
	claude-opus-4-8	0.0899	0.9167	0.0000	0.7587	0.2500	0.8571	0.6471	1.0000
	grok-4.6	0.0452	1.0000	0.0000	0.7742	0.6250	1.0000	0.6316	1.0000
	claude-haiku-4-5	0.0345	1.0000	0.0000	0.8276	0.1250	1.0000	0.7059	1.0000
	gemini-3.1-pro	0.1200	0.9167	0.0833	0.7333	0.2500	0.8571	0.6111	0.9167
	gpt-5-mini	0.4250	0.5000	0.2500	0.5000	0.1250	0.5385	0.5000	0.9167

Appendix

Table 6: Held-out cell metrics, part 2 of 2. Continued from Table 5 .

Corpus	$n$	Self-BLEU ( $\downarrow$ )	Ratio ( $\downarrow$ )
gemini-3.5-flash	35	0.2175	0.892
grok-4.20	35	0.2350	0.964
baseline ( skill_v1.md )	36	0.2439	1.000
kimi-k2.5	35	0.2628	1.078
claude-sonnet-5	34	0.2668	1.094
gpt-5.4	36	0.2951	1.210

Appendix

Table 7: Lexical diversity of the scam emails generated by each best skill, ordered most to least diverse. Downward arrows indicate greater lexical diversity. Higher Self-BLEU means less diversity. Ratio is Self-BLEU relative to the unoptimized baseline; the flag threshold is

1.30

. No corpus was flagged on either criterion. Source: skillopt/analysis/results/diversity_metrics.csv .

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 332 papers

The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities
Mar 26, 2026Ron LitvakPhishing Detection LLM Agent Security
Independent Researcher, Columbia University
CredLeakBench: Evaluating Credential Leakage and Recovery in LLM Agents
Oct 6, 2026Rafid Ahmed, Joseph Fioresi, Mubarak Shah +1AI Agent Security LLM Agent Security
Institute of Artificial Intelligence, University of Central Florida
EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks
Sep 25, 2026Mukul Singh, Mansi Uniyal, Devin Devlin +5LLM Agent Evaluation AI Agent Benchmarks
Microsoft

Mar 26, 2026J/K move · Enter open · S save

The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities

Ron Litvak

Independent Researcher, Columbia University

Phishing Detection LLM Agent Security Adversarial Robustness of Language Models LLMs for Cybersecurity Prompt Engineering

Open paper AskPDF arXiv

System prompt configuration can make the difference between near-total phishing blindness and near-perfect detection in LLM email agents. We present PhishNChips, a study of 11 models under 10 prompt strategies, showing that prompt-model interaction is a first-order security variable: a single model's phishing bypass rate ranges from under 1% to 97% depending on how it is configured, while the false-positive cost of the same prompt varies sharply across models. We then show that optimizing prompts around highly predictive signals can improve benchmark performance, reaching up to 93.7% recall at 3.8% false positive rate, but also creates a brittle attack surface. In particular, domain-matching strategies perform well when legitimate emails mostly have matched sender and URL domains, yet degrade sharply when attackers invert that signal by registering matching infrastructure. Response-trace analysis shows that 98% of successful bypasses reason in ways consistent with the inverted signal: the models are following the instruction, but the instruction's core assumption has become false. A counter-intuitive corollary follows: making prompts more specific can degrade already-capable models by replacing broader multi-signal reasoning with exploitable single-signal dependence. We characterize the resulting tension between detection, usability, and adversarial robustness as a navigable tradeoff, introduce Safetility, a deployability-aware metric that penalizes false positives, and argue that closing the adversarial gap likely requires tool augmentation with external ground truth.

Abstract

LLM Agent Security

Explore similar work

The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities

CredLeakBench: Evaluating Credential Leakage and Recovery in LLM Agents

EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks