cs.CROct 29, 2026
SaveUnderstanding and Mitigating Role Spoofing in LLMs
Organizations: University of Pennsylvania · Princeton University · Apple
Abstract
Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role. Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we call role spoofing}. In this work, we seek to understand and mitigate role spoofing. We study how models track "who said what" in a conversation, showing that they use cues from both content (e.g., semantics and writing style) and structure (e.g., control tokens and formatting). We show that LLMs are highly susceptible to role spoofing attacks that combine both fake structure and content. Additionally, in a toy setting, we show that adversarial training which teaches the model to ignore content-based injections does mitigate content-based attacks but leaves the model susceptible to structure-based attacks. This can explain why frontier models, with increased resistance to many injection attacks, can still be susceptible to role spoofing. Finally, we propose a new method for detecting role spoofing using linear probes trained to predict roles in a conversation, and show that it is effective across different settings.
Figures & tables
Figure 1: (Left) A simplified example of role spoofing via a fake special token injection. The injection is shown in orange, with the real special tokens wrapping it, and forged special tokens within. (Right) LLM role spoofing results in the AgentDojo ‘workspace family’. The first two columns show the rate at which the model follows the malicious instructions, when sent by the user (control) or when injected in a tool call (injection). The third column shows the ratio of injection to control. Our setup, being focused on role spoofing in particular, differs from the AgentDojo benchmark defaults, see Appendix B for the full details.
Figure 2: Injection success rates on “Who Said What”, where models need to output the role assigned to a given message segment in the presence of either misleading content or fake structure.
Figure 3: Transformer attention patterns from our two-layer toy model ( Section 3.3 ) in the presence of a (toy) role spoofing attack. The squares represent message content tokens. The model on the left is attending to content tokens to determine the role of the previous message. The model on the right is attending to the most recent role token appearing at the beginning of a message.
Figure 4: Results from the toy setting. The dashed line shows the divider between instruction tuning and adversarial training. The titles refer to the test condition being measured. Individual seeds are shown in gray, while the mean and 95% confidence intervals are in color.
Figure 5: Comparison of the linear probes’ detection ability across a pool of held out domains and role-spoofing injections. The blue line is our role probe. The purple line is a role probe based on Ye et al. (2026) . The orange line represents probes trained to separate clean data from a variety of content-based injections, while the red line also mixes in fake special token based injections. The curves show the best true positive rates (y-axis) at given false positive rates (x-axis) between .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The three rules share this two-layer architecture. The first head averages content scores and start-token indicators. The second head reads these values and the role tokens. Its attention and value weights determine which information contributes to the role score.
| Mechanism | Clean | Content-only | Structure-only | Full |
|---|---|---|---|---|
| Content-based | Correct | Fooled | Correct | Fooled |
| Structure-based (real and fake) | Correct | Correct | Fooled | Fooled |
| Structure-based (real only) | Correct | Correct | Correct | Correct |
Table 1: Role decisions for the three parameter choices. The structural entries hold uniformly by theorem A.1 . Each content entry holds with probability at least when the condition in theorem A.2 holds for that setting.
| Condition | GPT-5.4 YAML | GPT-5.4 JSON | Opus 5 JSON |
|---|---|---|---|
| Control success | 57.3% [53.9, 60.5] | 57.5% [54.8, 60.2] | 32.7% [31.1, 34.5] |
| Injection success | 49.6% [43.9, 55.0] | 47.5% [43.0, 51.6] | 27.3% [24.8, 29.6] |
| Injection/control ratio | 86.6% [76.7, 95.9] | 82.6% [74.6, 90.4] | 83.6% [75.6, 91.8] |
Table 2: AgentDojo tests with varying tool output formats. 95% confidence intervals show in brackets.
| GPT | Claude | Gemini | ||||
|---|---|---|---|---|---|---|
| Model | GPT-5.4 | Opus 4.8 | Opus 5 | Sonnet 5 | 3.5 Flash | 3.1 Pro |
| Injection success rate | 100% | 100% | 100% | 98.8% | 98.5% | 72.5% |
Table 3: Injection success rates in our Simple Tool Use Testbed. Only Gemini 3.1 Pro shows a non-negligible amount of resistance, but as can be seen in the right-hand table of fig. 1 , this doesn’t mean it’s always resistant.
| Suite | Tasks | Tools | Envs | Bindings | Clean | Injected |
|---|---|---|---|---|---|---|
| Accounts | 7 | 12 | 3 | 48 | 1,008 | 6,048 |
| Calendar | 5 | 8 | 3 | 48 | 720 | 2,880 |
| Documents | 4 | 8 | 3 | 24 | 288 | 864 |
| 4 | 9 | 3 | 4 | 48 | 144 | |
| Weather | 4 | 8 | 3 | 12 | 144 | 432 |
| Total | 24 | 45 | 15 | – | 2,208 | 10,368 |
Table 4: Synthetic suites. Injected instances enumerate every ordered pair of distinct tasks in every environment and parameter binding; 3,420 of them have a security-sensitive adversary task.
Figure 7: Both charts measure how often models are fooled by misleading cues in the Who Said What setting. The chart on the left tests misleading content with no fake structure to back it up, while the chart on the right tests fake structure injections in data where message content is useless for guessing the sender. Full version of fig. 2
Figure 8: Illustration of the misleading content and fake structure distributions used in the “Who Said What” setting. The tokens highlighted in gray are special tokens.
| Condition | Qwen3.5-27B | Content fine-tuned | Structure fine-tuned |
|---|---|---|---|
| Clean | 100.0% [97.1, 100.0] | 100.0% [97.1, 100.0] | 100.0% [97.1, 100.0] |
| Content-Only Injection | 0.0% [0.0, 2.0] | 100.0% [98.0, 100.0] | 0.0% [0.0, 2.0] |
| Native Fake Special Token Injection | 0.0% [0.0, 3.8] | 0.0% [0.0, 3.8] | 100.0% [96.2, 100.0] |
| Other Fake Special Token Injection | 0.0% [0.0, 0.3] | 36.2% [33.7, 38.8] | 100.0% [99.7, 100.0] |
Table 5: Percentage accuracy of Qwen3.5-27B and two fine-tuned versions of it, measured in four settings of the Who Said What task. Brackets show 95% confidence intervals; all values are percentages. One version is fine tuned to resist some content-only injections, and the other to resist some fake-special-token injections. The four settings we test are clean (uninjected) data, and three sets of injections (none of which were ever used in fine tuning).
Figure 9: Three layer version of the toy experiment results (as opposed to the two layer models in fig. 4 ).
Figure 10: Results for the two layer toy model on injections from tool messages impersonating user messages.
Figure 11: Results for the three layer toy model on injections from tool messages impersonating user messages.
Figure 12: Attention patterns of the two layer toy models (as in fig. 3 ) on clean data (top), a content injection (middle), and a structure injection (bottom). The left column shows the model at the end of instruction tuning, and the right column shows the same model after adversarial post-training.
| Gemma 4 31B-IT | Qwen3.5-27B | |||
|---|---|---|---|---|
| Condition | Self | Role probe | Self | Role probe |
| Clean | 0.0% [0.0, 0.4] | 0.0% [0.0, 0.4] | 0.0% [0.0, 0.4] | 0.0% [0.0, 0.4] |
| Injection | 100.0% [99.6, 100.0] | 100.0% [99.6, 100.0] | 100.0% [99.6, 100.0] | 100.0% [99.6, 100.0] |
| Injection + prefix | 0.0% [0.0, 0.4] | 100.0% [99.6, 100.0] | 25.0% [22.4, 27.8] | 100.0% [99.6, 100.0] |
Table 6: Percentage of tool outputs with parts attributed as being not tool outputs (a proxy for detecting injections). Brackets show 95% confidence intervals; all values are percentages. Self attribution has the user ask the model whether a quoted span came from a tool or not; role probes flag a tool call if its output on any token exceeds the threshold (chosen to achieve false positive rate on the data used to calculate the AUC curves in fig. 5 ). Higher is better for the injected rows, lower for the clean row.
Figure 13: The scores of one of the linear probes on each token in a randomly chosen test case. This is the same test case repeated four times, the ones on the left are injected, the ones on the right are a control where the fake special tokens are replaced with real ones, making the fake user real. This whole text is the payload returned by the attacked tool. Note that the probe doesn’t activate on parts representing tools, but does on user and assistant parts.
Figure 14: ROC curves for our probes for each model, with low false positive rates zoomed in on. Positive samples are messages containing injections, negative samples are clean (injectionless) messages. The curve shows the Pareto-frontier trade-off between high true positive rate and low false positive rate. This shows only the low false positive rate (less than 1%) section of the curve, the hardest regime. The plot shows that each model can achieve a greater than 99% true positive rate with zero false positives. The curves show the mean across random seeds; shading indicates the 95% confidence intervals.
More in this topic
Adversarial Attacks on LLMs
Adversarial attacks on large language models probe how crafted inputs, internal changes, and agent interactions bypass safety or disrupt reliable behavior.
See all 358 papers