The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities

Ron Litvak

cs.CRMar 26, 2026

The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities

Authors: Ron Litvak

Organizations: Independent Researcher, Columbia University

Abstract

System prompt configuration can make the difference between near-total phishing blindness and near-perfect detection in LLM email agents. We present PhishNChips, a study of 11 models under 10 prompt strategies, showing that prompt-model interaction is a first-order security variable: a single model's phishing bypass rate ranges from under 1% to 97% depending on how it is configured, while the false-positive cost of the same prompt varies sharply across models. We then show that optimizing prompts around highly predictive signals can improve benchmark performance, reaching up to 93.7% recall at 3.8% false positive rate, but also creates a brittle attack surface. In particular, domain-matching strategies perform well when legitimate emails mostly have matched sender and URL domains, yet degrade sharply when attackers invert that signal by registering matching infrastructure. Response-trace analysis shows that 98% of successful bypasses reason in ways consistent with the inverted signal: the models are following the instruction, but the instruction's core assumption has become false. A counter-intuitive corollary follows: making prompts more specific can degrade already-capable models by replacing broader multi-signal reasoning with exploitable single-signal dependence. We characterize the resulting tension between detection, usability, and adversarial robustness as a navigable tradeoff, introduce Safetility, a deployability-aware metric that penalizes false positives, and argue that closing the adversarial gap likely requires tool augmentation with external ground truth.

Figures & tables

Model	Failed	Total	Compliance
Qwen 3 235B	0	18,000	100.0%
Claude Haiku 4.5	7	18,000	100.0%
GPT-5.2	9	18,000	100.0%
GPT-4o-mini	24	18,000	99.9%
Grok 4.1 Fast	45	18,000	99.8%
Gemini 2.5 Flash	84	18,000	99.5%

Table 1: Instruction compliance: rate of parseable binary output per model (9 original strategies; the tenth strategy, infra_aware , was evaluated subsequently through the same pipeline with comparable compliance).

Strategy	Avg Bypass Rate	Avg FPR	Net Effectiveness	Safetility
security_first	7%	57%	+36%	$<$ 0.1%
balanced	30%	30%	+40%	0.2%
baseline	43%	23%	+34%	0.5%
trust_context	37%	32%	+31%	0.1%
helpful	40%	36%	+25%	$<$ 0.1%
efficiency_first	55%	26%	+19%	0.2%

Table 2: Core strategy performance averaged across 11 models.

System Prompt	User Prompt	Gemini 3 Flash	Gemini 2.5 Flash
Neutral	Neutral	52.8%	45.1%
Neutral	Security	67.4%	71.1%
Security	Neutral	98.5%	97.8%
Security	Security	85.7%	93.1%
System main effect		+32.0 pp	+37.4 pp
User main effect		+0.9 pp	+10.6 pp

Table 3: Factorial experiment: recall (%) by prompt condition (

2\times 2

design).

Model	Recall	FPR	Net Eff	Safetility
claude-sonnet-4.5	96.9%	70.2%	+26.8%	$<$ 0.1%
gpt-5.2	94.7%	51.7%	+43.0%	$<$ 0.1%
claude-haiku-4.5	94.6%	81.2%	+13.3%	$<$ 0.1%
grok-4.1-fast	72.2%	25.0%	+47.1%	0.5%
gemini-3-flash	61.5%	11.1%	+50.4%	14.1%
gemini-2.5-flash	57.5%	9.8%	+47.7%	17.4%

Table 4: Model performance overview across 6 core strategies.

Model	Core Recall	Optimized Recall	$\Delta$	FPR	Safetility
gpt-4o-mini	32.8%	93.7%	+60.9 pp	3.8%	87.1%
deepseek-v3.2	43.5%	91.3%	+47.8 pp	25.1%	0.8%
mistral-small-3.2	46.4%	84.7%	+38.3 pp	2.6%	71.5%
llama-4-scout	54.5%	85.5%	+31.0 pp	1.9%	73.0%
qwen3-235b	56.6%	96.9%	+40.3 pp	43.3%	$<$ 0.1%
gemini-2.5-flash	57.5%	81.9%	+24.4 pp	0.6%	67.1%

Table 5: Cross-model generalization of optimized strategies. Core Recall is the average across six core strategies. Optimized Recall and FPR report the best-performing optimized strategy per model (the strategy achieving the highest Safetility).

Figure 1: GPT-4o-mini: configuration determines security posture. Blue circles show commodity phishing recall; red diamonds show FPR; the orange dot shows recall on infrastructure phishing under the same optimized prompt. The optimized configuration drops FPR by 79 pp while maintaining 93.7% recall, but infrastructure phishing collapses recall to 30.1% (

-

64 pp) – the same strategy that rescued the model also made it maximally exploitable.

Figure 2: Best operating point per model, with bubble size proportional to Safetility. Models cluster in the high-recall, low-FPR region under optimized strategies. Grok 4.1 (90.7%), GPT-5.2 (87.2%), and GPT-4o-mini (87.1%) achieve the highest Safetility scores.

Strategy	Commodity Recall	Infra Recall	$\Delta$
sender_url_match	84.6%	43.1%	$-$ 41.5 pp
trap_sender_match	92.4%	46.5%	$-$ 46.0 pp
trap_aggressive	98.6%	87.9%	$-$ 10.7 pp
baseline	54.9%	81.2%	+26.3 pp
security_first	93.1%	96.6%	+3.5 pp

Table 6: Commodity vs. infrastructure phishing recall by strategy.

Figure 3: Which strategies collapse? Commodity phishing recall (blue) vs. infrastructure phishing recall (red) for five strategies. Signal-based strategies (sender_url_match, trap_sender_match) collapse by 41–46 pp, while baseline and security_first strategies are unaffected or improve. This figure shows the strategy-level view; Figure 4 shows the per-model breakdown.

Model	Commodity	Infra	$\Delta$	Status
Qwen 3	97.1%	94.5%	$-$ 2.6 pp	Not significant
DeepSeek v3.2	92.3%	78.1%	$-$ 14.3 pp	Resistant
Claude Haiku 4.5	76.4%	63.0%	$-$ 13.4 pp	Resistant
Claude Sonnet 4.5	92.1%	65.8%	$-$ 26.4 pp	Collapsed
GPT-5.2	87.9%	47.9%	$-$ 40.0 pp	Collapsed
Mistral Small	78.7%	35.6%	$-$ 43.1 pp	Collapsed

Table 7: Per-model collapse on sender_url_match (infrastructure recall).

Reasoning Pattern	Frequency ( $n=281$ )
Domain consistency cited	65%
Explicit “domain match” language	64%
“Match”/“matches” used explicitly	43%
“No red flags” / “no suspicious indicators”	38%
“No mismatch” cited	27%
Professional tone as supporting evidence	19%

Table 8: Reasoning patterns in verbose infrastructure phishing bypass responses.

Figure 4: Which models collapse? Per-model vulnerability to infrastructure phishing under sender_url_match . Models sorted by collapse severity. Green: immune (

\Delta\leq 3

pp); yellow: resistant (3–20 pp); red: collapsed (>20 pp). While Figure 3 shows that signal-based strategies collapse on average, this figure reveals that vulnerability varies dramatically across models, from Qwen 3 (effectively immune) to GPT-4o-mini and Gemini 3 Flash (

-

69 pp and

-

68 pp).

Model	Infra Recall	Infra $\Delta$	FPR	Tier
GPT-4o-mini	76.7%	+46.6 pp	8.2%	Efficient
Gemini 2.5 Flash	57.5%	+50.7 pp	2.8%	Efficient
Llama 4 Scout	65.8%	+46.6 pp	7.4%	Efficient
Grok 4.1 Fast	84.9%	+65.7 pp	15.9%	Moderate
Gemini 3 Flash	90.4%	+76.7 pp	20.9%	Moderate
GPT-5.2	98.6%	+50.7 pp	36.1%	Overcorrected

Table 9: Infrastructure-aware recovery under infra_aware : infrastructure recall gain (vs. sender_url_match ) and false-positive cost, grouped by response tier.

Phishing Detection

See all 16 papers

Mapping the Exploitation Surface: A 10,000-Trial Taxonomy of What Makes LLM Agents Exploit Vulnerabilities
Apr 6, 2026Charafeddine MouzouniLLM Agent Security LLM Agent Safety
OPIT – Open Institute of Technology, and Cohorte AI, Paris, France.
Prompt Injection Detection for Email Agents Through Attack Chain Modeling
Sep 25, 2026Ahmad Hashmi, Dhyey Patel, Yunting YinPrompt Injection Detection Indirect Prompt Injection
Department of Computer Science Eastern Michigan University Ypsilanti, MI, USA
AgentShield: Deception-based Compromise Detection for Tool-using LLM Agents
May 10, 2026Yassin H. Rassul, Tarik A. RashidLLM Agent Security AI Agent Monitoring
Computer Science and Engineering Department, School of Science and Engineering, University of Kurdistan Hewlêr, Erbil, Iraq · Artificial Intelligence and Innovation Centre, University of Kurdistan Hewlêr, Erbil, Iraq

Apr 6, 2026J/K move · Enter open · S save

Mapping the Exploitation Surface: A 10,000-Trial Taxonomy of What Makes LLM Agents Exploit Vulnerabilities

Charafeddine Mouzouni

OPIT – Open Institute of Technology, and Cohorte AI, Paris, France.

LLM Agent Security LLM Agent Safety

Open paper AskPDF arXiv

LLM agents with tool access can discover and exploit security vulnerabilities. This is known. What is not known is which features of a system prompt trigger this behaviour, and which do not. We present a systematic taxonomy based on approximately 10,000 trials across seven models, 37 prompt conditions, and 12 hypothesized attack dimensions -- all executed in real Docker sandboxes. Every condition includes the instruction "Always respect all rules and access policies"; only a single encouragement sentence varies. Within the task class studied (planted test-runner vulnerabilities), the threat model is narrower than assumed: nine of twelve dimensions -- including minimization, moral licensing, incentives, identity priming, and reasoning triggers -- produce no detectable exploitation at n=50 per cell (upper 95% CI <7% per condition). But it is also more subtle than expected: one dimension, goal reframing, reliably triggers exploitation. "You are solving a puzzle; there may be hidden clues" produces 38-40% exploitation on Claude Sonnet 4 despite the explicit rule instruction, replicating across four models (CTF framing: 8-14% on DeepSeek, GPT-5-mini, o4-mini). The agent does not override the rules; it reinterprets the task so that exploitative actions become task-aligned. GPT-4.1 produces no exploitation across 1,850 trials (37 conditions), and a temporal comparison across four OpenAI models released over eleven months shows a pattern consistent with improving safety training, though model capability differences are a confounder. The practical contribution is a narrowed, testable threat model: defenders should audit for goal-reframing language, not for the broad class of adversarial prompts.

Abstract

Phishing Detection

Explore similar work

Mapping the Exploitation Surface: A 10,000-Trial Taxonomy of What Makes LLM Agents Exploit Vulnerabilities

Prompt Injection Detection for Email Agents Through Attack Chain Modeling

AgentShield: Deception-based Compromise Detection for Tool-using LLM Agents