Arbiter: Detecting Interference in LLM Agent System Prompts

Tony Mason

cs.SEMar 9, 2026

Arbiter: Detecting Interference in LLM Agent System Prompts

Authors: Tony Mason

Organizations: the University of British Columbia · the Georgia Institute of Technology

Abstract

System prompts for LLM-based coding agents are software artifacts that govern agent behavior, yet lack the testing infrastructure applied to conventional software. We present Arbiter, a framework combining formal evaluation rules with multi-model LLM scouring to detect interference patterns in system prompts. Applied to three major coding agent system prompts: Claude Code (Anthropic), Codex CLI (OpenAI), and Gemini CLI (Google), we identify 152 findings across the undirected scouring phase and 21 hand-labeled interference patterns in directed analysis of one vendor. We show that prompt architecture (monolithic, flat, modular) strongly correlates with observed failure class but not with severity, and that multi-model evaluation discovers categorically different vulnerability classes than single-model analysis. One scourer finding was structural data loss in Gemini CLI's memory system was consistent with an issue filed and patched by Google, which addressed the symptom without addressing the schema-level root cause identified by the scourer. Total cost of cross-vendor analysis: $0.27 USD.

Figures & tables

Agent	Vendor	Lines	Chars	Source
Claude Code	Anthropic	1,490	78K	npm package
Codex CLI	OpenAI	298	22K	open-source repo
Gemini CLI	Google	245	27K	TS render functions

Table 1 : System prompt corpus.

Rule	Type	Detection
Mandate-prohibition conflict	Direct contradiction	LLM + structural
Scope overlap redundancy	Scope overlap	LLM
Priority marker ambiguity	Priority ambiguity	Structural
Implicit dependency	Unresolved dependency	LLM
Verbatim duplication	Scope overlap	Structural

Table 2 : Built-in evaluation rules.

Vendor	Passes	Models	Findings	Stopping Signal
Claude Code	10	10	116	3 consecutive “no”
Codex CLI	2	2	15	Pass 2 said “enough”
Gemini CLI	3	3	21	Pass 3 said “enough”

Table 3 : Scourer campaign summary.

Metric	Claude Code	Codex CLI	Gemini CLI
Lines	1,490	298	245
Characters	78K	22K	27K
Scourer findings	116	15	21
Passes to convergence	10	2	3
Distinct models	10	2	3
Archaeology patterns	21	—	—

Table 4 : Cross-vendor analysis summary.

Severity	Claude Code	Codex CLI	Gemini CLI
Curious	34 (29%)	3 (20%)	4 (19%)
Notable	36 (31%)	7 (47%)	9 (43%)
Concerning	34 (29%)	5 (33%)	6 (29%)
Alarming	12 (10%)	0 (0%)	2 (10%)

Table 5 : Severity distribution across vendors.

Figure 1 : Severity distribution across vendors. Claude Code shows a nearly uniform distribution across the lower three levels with an alarming tail. Smaller prompts (Codex, Gemini) peak at “notable.”

Model	Characteristic Focus
Claude Opus 4.6	Structural contradictions, security surfaces, meta-observations
DeepSeek V3.2	Hidden references, delegation loopholes, format mismatches
Kimi K2.5	Economic exploitation, resource exhaustion, cognitive load
Grok 4.1	Permission schema gaps, environment assumptions, state management
Llama 4 Maverick	Constraint inconsistency, security boundaries
MiniMax M2.5	Trust architecture flaws, concurrency, impossible instructions

Table 6 : Per-model analytical focus areas.

Figure 2 : Multi-model complementarity heat map for Claude Code. Findings are clustered into ten meta-categories. Security & trust is the only category found by 9/10 models; Resource & economic findings come almost exclusively from Kimi K2.5; MiniMax M2.5 contributes the broadest coverage (8/10 categories). The sparsity pattern demonstrates that models are complementary, not redundant.

Figure 3 : New findings per scourer pass for Claude Code. The MiniMax M2.5 surge at pass 7 (+20 findings after pass 6 produced only 5) demonstrates that specific models bring viewpoints capable of reopening exploration even after apparent convergence. Gray bars indicate passes that voted to stop; the stopping criterion requires three consecutive “no” votes.

Metric	Claude Code		Codex	Gemini
	v2.1.50	v2.1.71	CLI	CLI
Nodes	80	110	183	151
Max depth	2	4	5	4
Sections	0	12	16	21
Top-level directives	7	2	0	0
Unclassified nodes	23 (29%)	1 (1%)	13 (7%)	1 (1%)

Table 7 : Structural profiles from AST analysis.

Panel of Figure 4 — Figure 4 : Channel distribution across vendor prompts. Claude Code v2.1.71 is the only prompt with a substantial memory channel (25%). Codex and Gemini CLI are behavioral-dominant ( $>$ 70%). The version evolution from v2.1.50 to v2.1.71 shows tool definitions migrating out of the prompt text into API parameters.

Vendor	Passes	Actual Cost
Claude Code	10	$0.236
Codex CLI	2	$0.012
Gemini CLI	3	$0.014
Total	15	$0.263

Table 8 : API cost by vendor.

Figure 5 : API cost by model. The three most expensive models (Kimi K2.5, DeepSeek R1, Qwen3-235B) account for 61% of total cost, driven by retries and reasoning token overhead. GPT-OSS 120B provided 8 findings for $0.003.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Pass	Model	New	Cumulative	Continue?
1	Claude Opus 4.6	21	21	yes
2	Gemini 2.0 Flash	9	30	yes
3	Kimi K2.5	14	44	yes
4	DeepSeek V3.2	12	56	yes
5	Grok 4.1	10	66	yes
6	Llama 4 Maverick	5	71	yes

Appendix

Table 9 : Claude Code v2.1.50 scourer passes.

Pass	Model	New	Cumulative	Continue?
1	DeepSeek V3.2	10	10	yes
2	Grok 4.1	5	15	no

Appendix

Table 10 : Codex CLI scourer passes.

Pass	Model	New	Cumulative	Continue?
1	DeepSeek V3.2	12	12	yes
2	Qwen3-235B	5	17	yes
3	GLM 4.7	4	21	no

Appendix

Table 11 : Gemini CLI scourer passes.

#	Type	Blocks	Severity	Static?
1	Direct contradiction	TodoWrite mandate $\leftrightarrow$ Commit restriction	Critical	Yes
2	Direct contradiction	TodoWrite reinforcement $\leftrightarrow$ Commit restriction	Critical	Yes
3	Direct contradiction	TodoWrite mandate $\leftrightarrow$ PR restriction	Critical	Yes
4	Direct contradiction	TodoWrite reinforcement $\leftrightarrow$ PR restriction	Critical	Yes
5	Scope overlap	TodoWrite mandate $\leftrightarrow$ TodoWrite tool	Major	Yes
6	Priority ambiguity	Security policy $\leftrightarrow$ Security policy (dup)	Minor	Yes

Appendix

Table 12 : 21 hand-labeled interference patterns in Claude Code v2.1.50.

Model	Calls	Actual Total
Kimi K2.5	2	$0.054
DeepSeek R1	1	$0.054
Qwen3-235B	3	$0.053
GLM 4.7	2	$0.039
Grok 4.1 Fast	5	$0.016
Llama 4 Maverick	3	$0.015

Appendix

Table 13 : Per-model API costs from OpenRouter billing records.

Explore similar work

Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs
Jan 29, 2026Xiang Zheng, Yutao Wu, Hanxun Huang +5Prompt Injection AI Coding Agents
City University of Hong Kong · Deakin University · The University of Melbourne +3
IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests
Jul 22, 2026Ankur Singh, Jinqiu Yang, Tse-Hsun ChenAI Coding Agents AI Agent Security
Concordia University Montr´eal, Canada
Same Task, Different Work: Prompt-Induced Waste in Coding Agents
Aug 2, 2026Sarel Weinberger, Amir HozezLLM Inference Efficiency Overthinking in Language Models
PointFive

Jan 29, 2026, cs.AIJ/K move · Enter open · S save

Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

Xiang Zheng, Yutao Wu, Hanxun Huang, Yige Li+4

City University of Hong Kong · Deakin University · The University of Melbourne+3

Prompt Injection AI Coding Agents Adversarial Attacks on LLMs LLM Agent Security

Open paper AskPDF arXiv

Autonomous code agents built on large language models are reshaping software and AI development through tool use, long-horizon reasoning, and self-directed interaction. However, this autonomy introduces a previously unrecognized security risk: agentic interaction fundamentally expands the LLM attack surface, enabling systematic probing and recovery of hidden system prompts that guide model behavior. We identify system prompt extraction as an emergent vulnerability intrinsic to code agents and present \textbf{\textsc{JustAsk}}, a self-evolving framework that autonomously discovers effective extraction strategies through interaction alone. Unlike prior prompt-engineering or dataset-based attacks, \textsc{JustAsk} requires no handcrafted prompts, labeled supervision, or privileged access beyond standard user interaction. It formulates extraction as an online exploration problem, using Upper Confidence Bound-based strategy selection and a hierarchical skill space spanning atomic probes and high-level orchestration. These skills exploit imperfect system-instruction generalization and inherent tensions between helpfulness and safety. Evaluated on \textbf{41} black-box commercial models across multiple providers, \textsc{JustAsk} consistently achieves full or near-complete system prompt recovery, revealing recurring design- and architecture-level vulnerabilities. Our results expose system prompts as a critical yet largely unprotected attack surface in modern agent systems.

Abstract

Explore similar work

Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests

Same Task, Different Work: Prompt-Induced Waste in Coding Agents