TRACER: Trace-Regularized Training Against Indirect Prompt Injection

Kshitij Mishra; Salem Lahlou; Nils Lukas

cs.LGOct 13, 2026

TRACER: Trace-Regularized Training Against Indirect Prompt Injection

Authors: Kshitij Mishra, Salem Lahlou, Nils Lukas

Organizations: Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates

Abstract

Indirect prompt injection uses untrusted retrieved content, tool outputs, or agent messages to override an LLM's intended behavior. We show that defense robustness is channel-conditioned: the same attack succeeds 18% of the time as a follow-up message but 53% when placed in retrieved context. We propose TRACER (Trace-Regularized Adversarial Channel Exposure with Reward-shaping), a channel-aware adversarial training method that samples presentation wrappers and combines an answer-anchored, evidence-gated reward with ATRC, which aligns attacked reasoning with clean evidence when an injection is invalid. On GSM8K with Qwen2.5-3B-Instruct, TRACER reduces RAG-framed ASR from 65.5% to 6.1% while preserving 78-82% clean accuracy. Joint RAG+tool training achieves 13.9% average ASR across framings at 3B scale, while a single epoch at 7B reaches 9.4%. These results demonstrate that robust defenses must explicitly train and evaluate across deployment channels.

Figures & tables

Figure 1: Left: the same malicious payload is injected through different untrusted-context channels. Middle: changing only the wrapper creates a framing gap; apparent authority/provenance cues are one possible explanation for the measured effect. Right: TRACER trains on channel-sampled three-turn games using an answer-anchored, evidence-gated reward and ATRC trace alignment, and evaluates both native robustness and cross-framing transfer.

Figure 2: TRACER : The method samples a task, payload, and channel; generates clean and post-attack traces; computes the composite reward; applies ATRC on committed clean spans when the gate is active; and updates

\pi_{\theta}

with the post-attack objective.

Figure 3: The framing gap. Left: the base model is much more vulnerable when the same attack appears inside a more formal wrapper, revealing an authority gradient. Right: six checkpoints trained only on follow-up attacks fail to transfer to RAG evaluation even when the attack payload is unchanged.

Figure 4: RAG training trajectory across model snapshots. Clean accuracy stays near 78–82% while robust accuracy rises from 38% to above 80%. The best intermediate model snapshot reaches 3.8% ASR.

Figure 5: Per-attack effects of RAG-framed training. Prompt injection is both the most dangerous attack and the one most reduced by TRACER .

(a) Attack generalization			(b) Objective ablation			(c) Valid evidence + injection
Setting	Base	TRACER	Variant	ATRC	ASR	Channel	Ev. acc.	Base flip	TRACER flip
Original	65.5	6.1	Base	no	65.5	RAG	83.0	58.1	13.8
Held-out paraphrase	62.7	10.0	Answer-only	no	21.3	Tool	82.5	52.5	17.5
Channel-aware	68.8	18.1	Composite reward	no	13.8	Agent	81.0	46.3	16.3
Defense-aware	70.0	23.8	Full TRACER	yes	6.1

Table 1: Stress tests and attribution (ASR in %). (a) Generalization beyond the original RAG templates. (b) Direct objective ablations under RAG evaluation. (c) Mixed-evidence evaluation, where useful channel content also contains an injection; “flip” is conditional ASR from a correct clean-evidence response.

Figure 6: Cross-framing transfer at 3B scale. Gold boxes mark native training framing(s). RAG-specific training yields the deepest native defense, tool-output training yields the flattest profile, and multi-framing training gives the best average performance.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Attack family	Goal	Train/evaluation relation
Reasoning rewrite	Replace the solution path with an alternative chain	Disjoint tasks + held-out paraphrases
Preemptive answer	Inject a false “verified” final answer	Disjoint tasks + held-out paraphrases
Prompt injection	Issue an explicit instruction override	Disjoint tasks + held-out paraphrases

Appendix

Table 2: Attack families and split. Families are shared to isolate channel transfer; task instances and held-out surface forms are disjoint.

RAG evaluation	Base	TRACER
Original templates	65.5%	6.1%
Held-out paraphrases	62.7%	10.0%
Channel-aware escalation	68.8%	18.1%
Defense-aware suite	70.0%	23.8%

Appendix

Table 3: Adaptive evaluation. The defense-aware suite explicitly targets the learned training behavior.

Signal	Used in	Implementation	Used by evaluation?	Audit
Evidence validity	Corrigibility, ATRC gate	Construction label	No	Exact
Step validity	Process reward	Verifier + rules	No	90.0% agreement
Anchor support/contradiction	Anchor reward	Verifier	No	91.0% agreement
Final-answer correctness	Accuracy, ASR	Answer extraction	Yes	99.5% agreement

Appendix

Table 4: Training and evaluation signals. Verifier audits report agreement on a manually checked sample; evaluation metrics do not depend on the training-only step or anchor verifiers.

Result	ASR	Successes/denom.	Wilson 95% CI
Follow-up eval. average	18.0%	173/960	[15.7, 20.6]
RAG eval. average	53.0%	509/960	[49.9, 56.2]
Base RAG	65.6%	99/151	[57.7, 72.7]
TRACER RAG	6.1%	10/164	[3.3, 10.9]

Appendix

Table 5: Conditional-ASR denominators and uncertainty. Minor differences between aggregated and plotted percentages arise from the reporting granularity of the corresponding evaluations.

Mixed-evidence channel	Evidence acc.	Base flip ASR	TRACER flip ASR
RAG document	83.0%	58.1%	13.8%
Tool output	82.5%	52.5%	17.5%
Agent message	81.0%	46.3%	16.3%

Appendix

Table 6: Valid evidence with an embedded injection. High evidence accuracy argues against blanket refusal or wholesale context ignoring.

Model	Follow-up	RAG	Tool	Agent	Spread
Qwen2.5-3B-Instruct	41.4%	65.5%	59.4%	46.5%	24.1 $\,\mathrm{pp}$
Qwen2.5-7B-Instruct	9.5%	34.3%	36.6%	16.8%	27.1 $\,\mathrm{pp}$
Llama-3.1-8B-Instruct	17.8%	42.5%	39.0%	24.0%	24.7 $\,\mathrm{pp}$
Gemma-2-9B-It	12.0%	31.5%	34.0%	19.0%	22.0 $\,\mathrm{pp}$
Mistral-7B-Instruct	24.5%	49.0%	45.5%	30.5%	24.5 $\,\mathrm{pp}$

Appendix

Table 7: Evaluation-only framing diagnostics across model families. Spread is the maximum minus minimum ASR across channels.

Figure 7: Complete RAG training evaluation (200 samples per checkpoint). Clean accuracy, robust accuracy, and ASR across 28 checkpoints. Most of the gain appears before the end of epoch 2.

Figure 8: Full transfer matrix visualization. The heatmap includes all 3B and 7B settings considered in the paper. The base model shows an authority gradient, while trained models differ in how concentrated or broad their robustness becomes.

Figure 9: Transfer asymmetry as reduction from the base model. Specialists reduce ASR unevenly across evaluation wrappers. RAG training gives the deepest native reduction, tool training gives the flattest profile, and multi-framing plus 7B training improve the overall shape.

Model	Followup	RAG	Tool-Output	Agent-Msg	Spread
3B Base (no LoRA)	41.4%	65.5%	59.4%	46.5%	24.1 $\,\mathrm{pp}$
3B RAG-trained	14.5%	6.1%	25.7%	26.5%	20.4 $\,\mathrm{pp}$
3B Tool-trained (epoch 1)	32.1%	33.1%	39.6%	40.3%	8.2 $\,\mathrm{pp}$
3B Tool-trained (final)	20.0%	20.2%	15.0%	15.8%	5.2 $\,\mathrm{pp}$
3B Multi-framing (RAG+tool)	11.3%	5.2%	21.7%	17.5%	16.5 $\,\mathrm{pp}$
3B Agent-trained	19.6%	15.7%	26.3%	18.4%	10.6 $\,\mathrm{pp}$

Appendix

Table 8: Complete cross-framing transfer matrix. Attack success rate (%) across training conditions and evaluation framings. Spread is the maximum minus minimum ASR across the four framings for a given trained model.

State	Epoch	Clean	Robust	ASR
Base	–	64.0%	24.5%	59.4%
step-250	1	66.0%	32.5%	57.6%
step-1750	1	72.0%	52.0%	39.6%
step-2500	2	73.0%	58.5%	32.2%
step-2750	2	69.5%	60.5%	28.8%
final	3	77.5%	75.5%	15.0%

Appendix

Table 9: Tool-output training trajectory. Three epochs of tool-output training improve native tool-output robustness from 59.4% to 15.0%.

Figure 10: Tool-output training curve. Tool-output training converges more gradually than RAG training but ultimately reaches 15.0% native ASR.

State	Followup	RAG	Tool	Agent	Avg
Base	41.4%	65.5%	59.4%	46.5%	53.2%
Epoch 1	21.0%	30.3%	43.3%	31.9%	31.6%
Epoch 2	12.4%	21.8%	20.1%	23.4%	19.4%
Final	19.6%	15.7%	26.3%	18.4%	20.0%

Appendix

Table 10: Agent-message training trajectory. Agent-message training improves average ASR, but its strongest final transfer is to RAG framing rather than native agent framing.

Condition	Sys	Wrapper	ASR	$\Delta$
Full	RAG	RAG	5.5%	—
Wrapper only	Default	RAG	10.0%	+4.5 $\,\mathrm{pp}$
Prompt only	RAG	None	11.3%	+5.8 $\,\mathrm{pp}$
Neither	Default	None	60.5%	+55.0 $\,\mathrm{pp}$

Appendix

Table 11: RAG ablation. Both the RAG wrapper and the RAG-specific system prompt matter. The wrapper contributes slightly more, but the strongest robustness appears only when both are present.

Model	Followup	RAG	Tool	Agent	Avg
7B Base	9.5%	34.3%	36.6%	16.8%	24.3%
7B RAG ep1	5.2%	10.4%	13.5%	8.5%	9.4%
7B RAG ep2	5.7%	18.0%	19.2%	6.5%	12.3%
7B RAG final	5.1%	13.6%	18.2%	6.2%	10.8%

Appendix

Table 12: 7B scaling results. Larger models are substantially more robust before training and reach even lower average ASR after framing-aware training.

Figure 11: 7B scaling. Larger models begin from a better robustness baseline and achieve their best cross-framing average after a single epoch of RAG-framed training.

Figure 12: Wrapper vs. system prompt. The document wrapper and the RAG-specific system prompt provide complementary gains.

Figure 13: Cross-domain robustness transfer. Framing-aware skepticism transfers across domains more readily than task-specific clean accuracy.

Explore similar work

Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment
Jun 13, 2026Lipeng He, Yihan Wang, Jiawen Zhang +1Adversarial Training AI Agent Safety
University of Waterloo Waterloo, Canada · Zhejiang University Hangzhou, China · KTH Royal Institute of Technology Stockholm, Sweden
RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents
Oct 5, 2026Mohamed Dhouib, Clement Elliker, Alexi Canesse +5Prompt Injection Defense LLM Agents
LIX, École polytechnique, Institut Polytechnique de Paris, CNRS · AMIAD (Agence Ministérielle pour l’IA de Défense) · IRT SystemX +1
LTBD: Learnable Trust-Boundary Delimiters for Prompt Injection Defense
Oct 8, 2026Luman Zhao, Minghui Xu, Yue Zhang +1LLM Security Prompt Injection Defense
Shandong University, Shandong Academy of Artificial Intelligence

Jun 13, 2026J/K move · Enter open · S save

Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment

Lipeng He, Yihan Wang, Jiawen Zhang, N. Asokan

University of Waterloo Waterloo, Canada · Zhejiang University Hangzhou, China · KTH Royal Institute of Technology Stockholm, Sweden

Adversarial Training AI Agent Safety Prompt Injection Defense Prompt Injection Attacks Prompt Injection Attacks on LLMs Prompt Injection Attacks on AI Agents

Open paper AskPDF arXiv

Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution. Existing defenses report near-zero attack success rate on static benchmarks, yet recent adaptive evaluations show that these results collapse once the attacker is allowed to optimize against the deployed defense. In this work, we trace this collapse to two failure modes. First, existing defense methods are confined to recognizing specific attack patterns, rather than assessing whether the intent of every embedded instruction is relevant to the user task. Second, training-based defenses, which otherwise offer the strongest safety-utility trade-off, assemble their adversarial examples from a handful of hand-crafted templates, and the resulting defender fails to generalize outside that narrow strategy distribution. To address these gaps, we propose RETA, a training-based method that grounds defense decisions on the user tasks rather than attacker-controlled data. At each tool-output step, the defender undertakes chain-of-thought reasoning verifying that its actions are consistent with the user task. Leveraging red-teaming, a simulated attacker synthesizes adversarial training data and receives a dictionary-learning diversity reward, achieving broad coverage of injection-reformulation strategies. Together, these allow the defender to be optimized via multi-objective reinforcement learning and achieve better safety-utility trade-off. Across six black-box adaptive attacks, RETA keeps every per-attack ASR below 10%, with average ASR of 2.92% and 3.75% on the two target models, while preserving most utility under attack and on clean inputs.

Abstract

Explore similar work

Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment

RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

LTBD: Learnable Trust-Boundary Delimiters for Prompt Injection Defense