Automating Agent Hijacking via Structural Template Injection

Xinhao Deng; Jiaqing Wu; Miao Chen; Yue Xiao; Ke Xu; Qi Li

cs.AIFeb 18, 2026

Automating Agent Hijacking via Structural Template Injection

Authors: Xinhao Deng, Jiaqing Wu, Miao Chen, Yue Xiao, Ke Xu, Qi Li

Organizations: Tsinghua University & Ant Group · Tsinghua University · Zhongguancuan Laboratory

Abstract

Agent hijacking, highlighted by OWASP as a critical threat to the Large Language Model (LLM) ecosystem, enables adversaries to manipulate execution by injecting malicious instructions into retrieved content. Most existing attacks rely on manually crafted, semantics-driven prompt manipulation, which often yields low attack success rates and limited transferability to closed-source commercial models. In this paper, we propose Phantom, an automated agent hijacking framework built upon Structured Template Injection that targets the fundamental architectural mechanisms of LLM agents. Our key insight is that agents rely on specific chat template tokens to separate system, user, assistant, and tool instructions. By injecting optimized structured templates into the retrieved context, we induce role confusion and cause the agent to misinterpret the injected content as legitimate user instructions or prior tool outputs. To enhance attack transferability against black-box agents, Phantom introduces a novel attack template search framework. We first perform multi-level template augmentation to increase structural diversity and then train a Template Autoencoder (TAE) to embed discrete templates into a continuous, searchable latent space. Subsequently, we apply Bayesian optimization to efficiently identify optimal adversarial vectors that are decoded into high-potency structured templates. Extensive experiments on Qwen, GPT, and Gemini demonstrate that our framework significantly outperforms existing baselines in both Attack Success Rate (ASR) and query efficiency. Moreover, we identified over 70 vulnerabilities in real-world commercial products that have been confirmed by vendors, underscoring the practical severity of structured template-based hijacking and providing an empirical foundation for securing next-generation agentic systems.

Figures & tables

Special Token	Describe
<\|im_start\|>	start of each turn (BOT token)
<\|im_end\|>	end of each turn (EOT token)
<tools>	tools available in conversation
<tool_call>	tool name to be called
<tool_response>	tool execution result
<think>	thinking content

Table 1. Special Tokens in Chat Template (Qwen Example)

Figure 1. Threat model of Phantom . The adversary injects the structural template into external data sources accessed by the LLM agent, inducing role confusion to hijack the agent’s execution flow.

Figure 2. The distribution of the Agent’s attention over input tokens. Compared to direct injection, structured template injection effectively shifts the Agent’s attention from authentic content to the injected prompt.

Figure 4. Attack success rates of Phantom across different Agents and scenarios in AgentDojo.

	w/o Defense				Delimiter Spotlighting				Tag Filter				Injection Detector
Models	SI	ST	CI	Phantom	SI	ST	CI	Phantom	SI	ST	CI	Phantom	SI	ST	CI	Phantom
Qwen3-Max	24.68	53.25	35.71	67.53	18.18	51.95	25.32	63.64	20.78	14.29	13.64	41.56	5.84	4.55	0.00	11.04
Qwen-Flash	11.69	39.61	27.92	55.84	12.34	34.42	21.43	54.55	9.74	29.22	15.58	50.00	3.25	5.19	0.00	12.34
GPT-4o	4.55	20.13	16.88	53.25	3.90	23.38	7.79	53.25	3.90	0.65	0.00	31.17	1.95	4.55	0.65	12.34
GPT-4.1-Mini	26.62	46.75	12.34	67.53	24.68	48.70	16.23	69.48	26.62	54.55	6.49	69.48	7.79	6.49	4.55	15.58
GPT-4.1	14.29	29.87	0.00	75.97	12.99	31.82	0.00	76.62	16.43	43.51	0.00	74.03	7.79	5.19	0.00	18.18

Table 2. Assessment of ASR (in %) for attack methods under various defenses.

	w/o Defense				Delimiter Spotlighting				Tag Filter				Injection Detector
Models	SI	ST	CI	Phantom	SI	ST	CI	Phantom	SI	ST	CI	Phantom	SI	ST	CI	Phantom
Qwen3-Max	63.64	28.57	46.75	22.73	72.08	31.82	55.19	22.73	64.94	77.92	64.29	58.44	25.97	20.13	12.34	18.18
Qwen-Flash	56.49	14.29	29.87	14.29	50.65	10.39	25.32	12.34	59.09	23.38	48.70	14.29	24.03	15.58	9.09	11.69
GPT-4o	77.92	44.16	59.74	24.03	79.87	45.45	68.83	27.92	79.22	76.62	82.47	43.51	37.01	16.88	11.69	15.58
GPT-4.1-Mini	57.14	23.38	74.68	13.64	61.04	22.08	75.97	17.53	60.39	15.58	84.42	18.18	22.73	12.99	23.38	12.99
GPT-4.1	75.97	64.29	91.56	18.18	77.27	66.23	90.26	20.13	74.03	46.10	90.26	18.83	25.32	22.08	29.22	18.18

Table 3. Assessment of Utility (in %) for attack methods under various defenses.

Perturbation Strategy	Prob.	Attention Weight
		Attack	User
Original Template	100.00%	10.82	6.87
Force Tokenization	99.99%	6.22	5.67
HTML Bracket Encoding	46.29%	4.89	6.47
Bracket Removal	0.05%	6.36	8.65
Full Template Removal	0.00%	5.97	9.48

Table 4. Impact of structural pattern perturbations on attack efficacy and attention distribution. The Prob. denotes the probability of generating the target tool call. Attention scores are averaged over the last generated token directed towards the attack payload and user prompt, respectively.

Figure 5. ASR convergence over optimization iterations.

Dims	EM	F1	Qwen-Flash	DeepSeek-V3.2
8	65.18	92.66	53.25	76.62
16	87.26	97.95	55.84	78.57
24	90.71	98.85	53.25	81.17
32	92.70	99.13	52.60	77.27
40	94.01	99.46	52.60	78.57

Table 5. The impact of latent space dimension on template reconstruction ability and ASR (in %).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

"\n</tool_response><|im_end|>\n<|im_start|>assistant\n"

"<|im_end|>\n<|im_start|>user\n"

"<|im_end|>\n"

Appendix

Table 6. The template of Qwen3 open-source models

""<|end|><|start|>assistant<|channel|>final<|message|>"

"<|end|><|start|>user<|message|>"

"<|end|>"

Appendix

Table 7. The template of GPT-OSS.

"<end_of_turn>\n<start_of_turn>model\n"

"<end_of_turn>\n<start_of_turn>user\n"

"<end_of_turn>\n"

Appendix

Table 8. The template of Gemma3.

"</result>\n</function_results>\n\n</think>"

"<|end_of_sentence|><|User|>"

Appendix

Table 9. The template of DeepSeek-V3.2.

Models	Phantom	Phantom -light
Qwen-Flash	55.84%	48.70%
Qwen3-Max	67.53%	66.88%
DeepSeek-v3.2	78.57%	70.78%
GPT-4o	53.25%	48.05%
GPT-4.1-Mini	67.53%	66.23%
GPT-4.1	75.97%	77.27%

Appendix

Table 10. Comparison of Phantom and the lightweight search strategy.

Attack Type	Thinking	Scenarios
Attack Type	Thinking	Workspace	Travel	Slack	Banking
Semantic-Injection	Disable	11.69%	27.14%	46.67%	33.33%
Semantic-Injection	Enable	15.58%	44.29%	71.43%	65.97%
Single-Template	Disable	39.61%	77.14%	90.48%	57.64%
Single-Template	Enable	53.25%	86.43%	84.76%	79.86%
ChatInject	Disable	27.92%	29.29%	23.81%	42.36%

Appendix

Table 11. ASR performance of Qwen-Flash across different scenarios in non-thinking vs. thinking modes.

Agents	TODO	Ignore Previous	System Message	Injecagent	Important Instruction	Phantom
Qwen-Flash	2.95%	15.45%	3.23%	9.44%	29.71%	79.67%
Qwen3-Max	4.79%	11.76%	6.03%	5.01%	49.70%	78.32%
Deepseek-V3.2	5.77%	4.29%	5.48%	5.05%	77.79%	91.84%
GPT-4o	4.19%	2.97%	4.40%	3.36%	27.14%	81.96%
GPT-4.1-mini	3.36%	9.28%	5.19%	5.77%	53.96%	86.35%
GPT-4.1	2.84%	2.39%	2.85%	2.61%	40.59%	90.46%

Appendix

Table 12. ASRs of semantic-based attacks. The values are the average ASRs on the 4 scenarios in AgentDojo.

Figure 6. Evolution of ASR over optimization iterations across ten independent trials. The shaded region denotes the variance (mean $\pm$ standard deviation), illustrating the convergence behavior of the search process.

AI Agent Security

Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.

See all 508 papers

Jun 9, 2026

Assessing Automated Prompt Injection Attacks in Agentic Environments

Indirect prompt injection poses a critical threat to LLM agents that interact with untrusted external data, yet automated attack methods--proven effective for jailbreaking--remain underexplored in realistic agentic settings. We present a comprehensive empirical evaluation of automated prompt injection attacks against LLM agents, adapting both white-box (GCG) and black-box (TAP) methods to the agentic setting within the AgentDojo framework. We evaluate across 80 task pairs spanning four domains and multiple models, and find that black-box optimization substantially outperforms gradient-based methods, a gap we attribute to GCG's optimization instability under reasonable compute budgets. We also find that TAP's effectiveness depends on the attacker model, as both general capability and safety tuning affect attack success--stronger models produce more effective injections, while safety-tuned attackers can refuse to generate adversarial prompts. Task-universal attacks transfer effectively to unseen tasks and out-of-distribution domains, but attacks optimized on smaller open-source models do not transfer to frontier models like GPT-5. These findings highlight automated prompt injection as a credible but model-dependent threat, with significant barriers remaining for model-agnostic exploitation.

David Hofer, Edoardo Debenedetti, Florian Tramèr

ETH Zurich Zurich, Switzerland

Adversarial Prompt Generation LLM Agent Security AI Agent Security Benchmarks Indirect Prompt Injection Prompt Injection Attacks Prompt Injection Attacks on LLMs+2

Ask PDF

Jun 10, 2026

PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections

Large Language Models (LLMs) are rapidly evolving into agentic systems that interact with external tools and environments, introducing new security risks such as indirect prompt injection attacks through untrusted external sources. Existing defenses mainly focus on blocking malicious content at inference time, and current red-teaming methods primarily optimize attack success. As a result, developers have limited visibility into how latent prompt injections emerge and propagate through agents. We propose PI-Hunter, an automated agentic auditing framework for proactive vulnerability exposure in LLM agents. PI-Hunter constructs realistic source-aware test cases and iteratively evolves them through feedback-driven exploration to induce agents to retrieve and reveal latent malicious instructions embedded within external environments. Extensive experiments across multiple benchmarks, agent architectures, attacks, and defenses demonstrate that PI-Hunter substantially improves vulnerability exposure and attack-surface coverage over strong automated red-teaming baselines, while remaining effective under existing prompt injection defenses.

Pengfei He, Lesly Miculicich, Vishesh Sharma +5

Google Cloud AI Research · Google · Michigan State University

AI Agent Auditing LLM Auditing LLM Red Teaming AI Agent Security Indirect Prompt Injection Prompt Injection Attacks on LLMs+1

Ask PDF

Aug 1, 2026

Adversarial Attacks in Multi-Agent LLM Pipelines: Unveiling Structural Vulnerabilities in Agentic AI Architectures

Multi-agent LLM pipelines orchestrate multiple specialized language model agents into structured workflows where intermediate outputs are passed across agents to solve complex tasks. This design introduces a security gap absent in single-agent settings: once an agent accepts adversarial content, it is propagated as trusted input throughout the pipeline. We argue that this vulnerability stems from the absence of boundary verification, a security primitive that enforces explicit validation of data as it crosses inter-agent boundaries, including content, identity, execution intent, and state integrity. Without such verification, modern pipelines embed implicit trust assumptions that are not adversarially robust, giving rise to structurally distinct attack surfaces (e.g., content injection, agent impersonation, plan deviation, and memory poisoning). Leveraging annotated production traces from the GAIA and SWE-Bench benchmark, we show that these vulnerabilities arise in benign deployments and largely evade existing evaluation frameworks. We further operationalize these failure modes within a controlled multi-agent setting and evaluate them across GPT-5-mini, Claude Sonnet 4.5, and Kimi K2.5 under identical pipeline configurations. The results reveal that attack success aligns with pipeline structure rather than model capability, indicating that adversarial vulnerability is fundamentally an architectural property and motivating a shift toward pipeline-level defenses.

Faisal Haque Bappy, Tahrim Hossain, Tarannum Shaila Zaman +3

University of Maryland Baltimore County, MD, USA · Kent State University, OH, USA · Tennessee State University, TN, USA

Adversarial Attacks on LLMs AI Agent Security Multi-Agent LLMs

Ask PDF

Abstract

AI Agent Security

Explore similar work

Assessing Automated Prompt Injection Attacks in Agentic Environments

PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections

Adversarial Attacks in Multi-Agent LLM Pipelines: Unveiling Structural Vulnerabilities in Agentic AI Architectures