Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role. Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we call role spoofing}. In this work, we seek to understand and mitigate role spoofing. We study how models track "who said what" in a conversation, showing that they use cues from both content (e.g., semantics and writing style) and structure (e.g., control tokens and formatting). We show that LLMs are highly susceptible to role spoofing attacks that combine both fake structure and content. Additionally, in a toy setting, we show that adversarial training which teaches the model to ignore content-based injections does mitigate content-based attacks but leaves the model susceptible to structure-based attacks. This can explain why frontier models, with increased resistance to many injection attacks, can still be susceptible to role spoofing. Finally, we propose a new method for detecting role spoofing using linear probes trained to predict roles in a conversation, and show that it is effective across different settings.
Figures & tables
Figure 1: (Left) A simplified example of role spoofing via a fake special token injection. The injection is shown in orange, with the real special tokens wrapping it, and forged special tokens within. (Right) LLM role spoofing results in the AgentDojo ‘workspace family’. The first two columns show the rate at which the model follows the malicious instructions, when sent by the user (control) or when injected in a tool call (injection). The third column shows the ratio of injection to control. Our setup, being focused on role spoofing in particular, differs from the AgentDojo benchmark defaults, see Appendix B for the full details.
Figure 2: Injection success rates on “Who Said What”, where models need to output the role assigned to a given message segment in the presence of either misleading content or fake structure.
Figure 3: Transformer attention patterns from our two-layer toy model ( Section 3.3 ) in the presence of a (toy) role spoofing attack. The squares represent message content tokens. The model on the left is attending to content tokens to determine the role of the previous message. The model on the right is attending to the most recent role token appearing at the beginning of a message.
Figure 4: Results from the toy setting. The dashed line shows the divider between instruction tuning and adversarial training. The titles refer to the test condition being measured. Individual seeds are shown in gray, while the mean and 95% confidence intervals are in color.
Figure 5: Comparison of the linear probes’ detection ability across a pool of held out domains and role-spoofing injections. The blue line is our role probe. The purple line is a role probe based on Ye et al. (2026) . The orange line represents probes trained to separate clean data from a variety of content-based injections, while the red line also mixes in fake special token based injections. The curves show the best true positive rates (y-axis) at given false positive rates (x-axis) between 0−1% .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The three rules share this two-layer architecture. The first head averages content scores and start-token indicators. The second head reads these values and the role tokens. Its attention and value weights determine which information contributes to the role score.
Mechanism
Clean
Content-only
Structure-only
Full
Content-based
Correct
Fooled
Correct
Fooled
Structure-based (real and fake)
Correct
Correct
Fooled
Fooled
Structure-based (real only)
Correct
Correct
Correct
Correct
Appendix
Table 1: Role decisions for the three parameter choices. The structural entries hold uniformly by theorem A.1 . Each content entry holds with probability at least 1−δ when the condition in theorem A.2 holds for that setting.
Condition
GPT-5.4 YAML
GPT-5.4 JSON
Opus 5 JSON
Control success
57.3% [53.9, 60.5]
57.5% [54.8, 60.2]
32.7% [31.1, 34.5]
Injection success
49.6% [43.9, 55.0]
47.5% [43.0, 51.6]
27.3% [24.8, 29.6]
Injection/control ratio
86.6% [76.7, 95.9]
82.6% [74.6, 90.4]
83.6% [75.6, 91.8]
Appendix
Table 2: AgentDojo tests with varying tool output formats. 95% confidence intervals show in brackets.
GPT
Claude
Gemini
Model
GPT-5.4
Opus 4.8
Opus 5
Sonnet 5
3.5 Flash
3.1 Pro
Injection success rate
100%
100%
100%
98.8%
98.5%
72.5%
Appendix
Table 3: Injection success rates in our Simple Tool Use Testbed. Only Gemini 3.1 Pro shows a non-negligible amount of resistance, but as can be seen in the right-hand table of fig. 1 , this doesn’t mean it’s always resistant.
Suite
Tasks
Tools
Envs
Bindings
Clean
Injected
Accounts
7
12
3
48
1,008
6,048
Calendar
5
8
3
48
720
2,880
Documents
4
8
3
24
288
864
Email
4
9
3
4
48
144
Weather
4
8
3
12
144
432
Total
24
45
15
–
2,208
10,368
Appendix
Table 4: Synthetic suites. Injected instances enumerate every ordered pair of distinct tasks in every environment and parameter binding; 3,420 of them have a security-sensitive adversary task.
Figure 7: Both charts measure how often models are fooled by misleading cues in the Who Said What setting. The chart on the left tests misleading content with no fake structure to back it up, while the chart on the right tests fake structure injections in data where message content is useless for guessing the sender. Full version of fig. 2
Figure 8: Illustration of the misleading content and fake structure distributions used in the “Who Said What” setting. The tokens highlighted in gray are special tokens.
Condition
Qwen3.5-27B
Content fine-tuned
Structure fine-tuned
Clean
100.0% [97.1, 100.0]
100.0% [97.1, 100.0]
100.0% [97.1, 100.0]
Content-Only Injection
0.0% [0.0, 2.0]
100.0% [98.0, 100.0]
0.0% [0.0, 2.0]
Native Fake Special Token Injection
0.0% [0.0, 3.8]
0.0% [0.0, 3.8]
100.0% [96.2, 100.0]
Other Fake Special Token Injection
0.0% [0.0, 0.3]
36.2% [33.7, 38.8]
100.0% [99.7, 100.0]
Appendix
Table 5: Percentage accuracy of Qwen3.5-27B and two fine-tuned versions of it, measured in four settings of the Who Said What task. Brackets show 95% confidence intervals; all values are percentages. One version is fine tuned to resist some content-only injections, and the other to resist some fake-special-token injections. The four settings we test are clean (uninjected) data, and three sets of injections (none of which were ever used in fine tuning).
Figure 9: Three layer version of the toy experiment results (as opposed to the two layer models in fig. 4 ).
Figure 10: Results for the two layer toy model on injections from tool messages impersonating user messages.
Figure 11: Results for the three layer toy model on injections from tool messages impersonating user messages.
Figure 12: Attention patterns of the two layer toy models (as in fig. 3 ) on clean data (top), a content injection (middle), and a structure injection (bottom). The left column shows the model at the end of instruction tuning, and the right column shows the same model after adversarial post-training.
Gemma 4 31B-IT
Qwen3.5-27B
Condition
Self
Role probe
Self
Role probe
Clean
0.0% [0.0, 0.4]
0.0% [0.0, 0.4]
0.0% [0.0, 0.4]
0.0% [0.0, 0.4]
Injection
100.0% [99.6, 100.0]
100.0% [99.6, 100.0]
100.0% [99.6, 100.0]
100.0% [99.6, 100.0]
Injection + prefix
0.0% [0.0, 0.4]
100.0% [99.6, 100.0]
25.0% [22.4, 27.8]
100.0% [99.6, 100.0]
Appendix
Table 6: Percentage of tool outputs with parts attributed as being not tool outputs (a proxy for detecting injections). Brackets show 95% confidence intervals; all values are percentages. Self attribution has the user ask the model whether a quoted span came from a tool or not; role probes flag a tool call if its output on any token exceeds the threshold (chosen to achieve 0.1% false positive rate on the data used to calculate the AUC curves in fig. 5 ). Higher is better for the injected rows, lower for the clean row.
Figure 13: The scores of one of the linear probes on each token in a randomly chosen test case. This is the same test case repeated four times, the ones on the left are injected, the ones on the right are a control where the fake special tokens are replaced with real ones, making the fake user real. This whole text is the payload returned by the attacked tool. Note that the probe doesn’t activate on parts representing tools, but does on user and assistant parts.
Figure 14: ROC curves for our probes for each model, with low false positive rates zoomed in on. Positive samples are messages containing injections, negative samples are clean (injectionless) messages. The curve shows the Pareto-frontier trade-off between high true positive rate and low false positive rate. This shows only the low false positive rate (less than 1%) section of the curve, the hardest regime. The plot shows that each model can achieve a greater than 99% true positive rate with zero false positives. The curves show the mean across random seeds; shading indicates the 95% confidence intervals.
Adversarial attacks on large language models probe how crafted inputs, internal changes, and agent interactions bypass safety or disrupt reliable behavior.
LLMs see the world as a single stream of text, partitioned into roles like <user> or <tool>. We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like <user> text, despite its <tool> label. We design role probes to measure how LLMs internally perceive "who is speaking," and find that injected text occupies the same representational space as the trusted role it imitates. We demonstrate this with CoT Forgery, a zero-shot attack that injects fabricated reasoning into user prompts and tool outputs. Models mistake the forgery for their own thoughts, yielding 60% attack success against frontier models with near-zero baselines. Strikingly, the degree of role confusion predicts attack success before a single token is generated. This mechanism generalizes beyond CoT Forgery to standard agent prompt injections, revealing prompt injection as a measurable consequence of role perception. To the model, sounding like a role is indistinguishable from being one. Project page and writeup: https://role-confusion.github.io
Charles Ye, Jasmine Cui, Dylan Hadfield-Menell
*Equal contribution 1Independent · 2Massachusetts Institute of Technology, Cambridge, MA, United States
Indirect prompt injection causes LLM agents to follow commands embedded in external data. A probe may distinguish instructions from data without identifying a state edit that changes the next action. We study this gap through counterfactual role probes, component-wise activation patching, and separate interventions on AgentDojo trajectories. Role decoding survives changes in content and format. In controlled Qwen tests, it precedes strong tool-choice effects from patches along an independently estimated role direction. On AgentDojo, directions estimated from hijacked and resisted training trajectories reduce attack success at pre-action and injected-span positions, but have little effect at random positions. In longer Qwen trajectories, single-position edits become less effective at later layers; span-wide and repeated edits reduce attack success on the same evaluation set. Removing the learned channel subspace preserves role decoding, yet effective intervention directions transfer poorly across the tested channels. These findings distinguish a readable role signal from an effective behavioral intervention: depth matters in controlled tool choice, while position and context also matter in attack trajectories.
Zhe Yu, Wenpeng Xing, Xingxing Yang +1
Zhejiang University · Binjiang Institute of Zhejiang University · Hong Kong Baptist University
Clarification-seeking behavior is widely regarded as a desirable property of LLM agents, enabling them to resolve ambiguity before acting on underspecified tasks. However, the security implications of this interaction pattern remain unexplored. We investigate whether the transition from standard execution to a clarification-seeking state increases an agent's susceptibility to prompt injection attacks. We introduce ASPI (Ambiguous-State Prompt Injection), a benchmark of 728 task-attack scenarios that isolates clarification as a distinct agent state and measures how this state transition affects vulnerability under controlled conditions. Each benchmark instance is evaluated under matched execution and clarification settings: in the execution setting, the agent acts on a fully specified instruction and encounters adversarial content only through tool-returned data; in the clarification setting, the agent must first request and incorporate additional user input before acting. We evaluate ten frontier LLMs and find that clarification-seeking consistently and substantially amplifies vulnerability. For instance, attack success rises from 1.8% to 34.0% for o3 and from 2.2% to 35.7% for Gemini-3-Flash. A decomposition analysis reveals that this gap reflects both a state-dependent shift in how models process incoming content and a channel-specific effect arising from the agent-solicited clarification interface. These findings demonstrate that standard execution-time security evaluation systematically underestimates the attack surface of interactive agents, and that robustness under fully specified tasks does not translate to robustness under ambiguity. For reproducibility, our data and source code are available at https://github.com/scaleapi/aspi.
Udari Madhushani Sehwag, Zhengyang Shan, Heming Liu +3
Scale AI · 2Boston University · 4Human Frontier Collective +3