cs.AIAug 24, 2026

Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors

Authors: Joshua Penman

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Everything a language model sees is tokens. The serving stack knows what each span is -- user input, tool output, instructions -- but the model must keep track of that itself, and can lose track or be confused: text can be written to read like anything. Prompt injection is a natural exploit of this phenomenon. By scrambling the model's understanding of span identity, an attacker can induce unwanted and dangerous actions. Adding a non-textual channel to the model's input -- a way to communicate span identity beyond text -- mitigates this class of attack. We thus introduce a general steering technique called Semantic Overlays: small learned adapters applied at chosen prefill positions to a frozen model's residual stream. Laying an overlay over a span creates an out-of-band annotation channel that cannot be replicated by tokens. Unlike steering vectors, Semantic Overlays are trained, adaptable, and selectively applied. An overlay can encode complex semantics that reshape how the model perceives the marked span: asked to copy a code snippet under an overlay asserting a different programming language, the model rewrites the snippet in the asserted language. Overlays compose, allow transparent reading of underlying content, and can carry complex payloads -- including imperatives the model will follow. An overlay which marks a span as "non-executable" defends against the broad class of prompt injections that add instructions in untrusted context. We report strong results on five prompt injection benchmarks: SEP separation rises from 24.3% to 99.0% with utility unchanged (our scoring rule; we correct a defect in the published grader), TensorTrust attack success falls from 34.8% to 6.2%, AlpacaFarm from 99.0% to 0%, and the overlay beats every published PIArena defense that leaves the model able to answer -- while marked spans stay readable, all at >95% character similarity to the original.

Figures & tables

Figure 1: The mark strips a span’s authority without touching its content. A retrieved passage answering a benign question also carries a planted instruction — visit a URL and enter your email and password. Because the passage is marked as non-executable using overlays, the frozen model answers the question and treats the instruction as inert text; unmarked, the same model obeys it and asks the user for their password. No token in the prompt distinguishes the two runs. This comparison can be run live at https://semantic-overlays.vercel.app .
Figure 2: Invisible marks, made visible. The prompt is the plain text of the note; the annotations show where six overlays were active during prefill — three underlines, two highlights, and a circle, stacked up to two deep. Asked to enumerate the underlined spans, the model returns exactly the three underlines, ignoring the other marks sharing their tokens. No token in the prompt marks any span.
 shared MLP embed.-cond.shared MLP code-cond.per-layer vectorsper-overlay MLP
params84M51M1.6M604M
non-overlapping spans
exact span retrieval99.5%96.1%96.6%63.3%
refusal when absent100%98.3%98.3%89.7%
stacked marks
verbalization99.0%100%96.2%98.1%
Table 1: Evaluation on held-out marks.
Figure 3: Python-colored reading. Snippet 1 of a five-snippet prompt is written in Racket; an overlay on its span asserts “This is in Python.” Asked to write out the Python snippet, the model selects the overlaid snippet and re-expresses the Racket program — same task, same structure — in Python.
 overlaidunoverlaid
selection (exact match)
language of snippet nn100%93%
which snippets are in XX100%100%
XX absent: “None of the snippets are in XX ”100%
mixed: one by overlay, one by code3/6
copying an overlaid snippet: rewrite into the asserted language (judge)
Table 2: Held-out language-overlay evaluation (about 60 questions per cell; copies scored by an LLM judge).
overlaycompliancejudge ceilingscored by
refusal (safety filter)100%—filter named, siblings answered
all caps89%—fully uppercased section
German81%—language detector
Spanish74%—language detector
nested bullets100%—depth ≥\geq 3, ≥\geq 5 bullets
explain like I’m five67%97%child register, judge
Table 3: Transform compliance on held-out prompts (served overlay set; 30–38 examples each).
 frozen modelwith overlay
probe in data ↓\downarrow71.8% (72.6)1.0% (2.6)
utility ↑\uparrow92.3% (93.1)92.6% (93.3)
SEP ↑\uparrow24.3% (23.9)99.0% (97.5)
Table 4: SEP, all 9,160 items, greedy decoding. “Probe in data”: answers the probe from the data span; “utility”: answers when the instruction asks; SEP: of items answered as an instruction, the fraction not also answered from data. Corrected rule with the published rule in parentheses; run-to-run noise ∼\sim 2pt (Appendix F ).
 frozen modelwith overlay
TensorTrust
hijacking ASR ↓\downarrow34.8%6.2%
hijacking Defense Validity ↑\uparrow100%96.1%
extraction ASR ↓\downarrow38.1%5.4%
AlpacaFarm / StruQ, four static attacks
attack success ↓\downarrow99.0%0.0%
Table 5: TensorTrust, AlpacaFarm, and Quadrat-IPI. TensorTrust is Defense Validity screened (Appendix F ); AlpacaFarm is the union of StruQ’s four static attacks, scored begin-with ; Quadrat-IPI is every verified injected document (Appendix I ).
 attack ↓\downarrowutility ↑\uparrow
no defense49.5%82.5%
prevention
PromptArmor48.0%84.0%
DataFilter30.0%80.0%
PISanitizer11.0%89.5%
SecAlign++3.5%68.5%
Table 6: PIArena, Direct attack, HotpotQA and NQ (Appendix H ). Defense rows are Table 2 of Geng et al. (2026) . † A quotation, not a compliance (footnote above). ‡ Utility with no attack present: what the detector costs benign traffic.
 attack ↓\downarrow
commercial, no defense
GPT-4o92%
Gemini-3-Flash88%
Gemini-3-Pro83%
GPT-4o-mini76%
GPT-570%
Table 7: PIArena, SQuAD v2, Direct attack. Commercial rows are Geng et al. (2026) ’s Table 3; the last two are ours. A no-injection control fires at 0% throughout.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

 positionsedit derived bylayers
ActAdd ( Turner et al., 2023 )selected prefill (first ℓ\ell tokens, fixed)activation difference of a prompt pairone layer
CAA ( Rimsky et al., 2024 )all decodemean difference over contrast pairsone layer
ITI ( Li et al., 2023 )all decode (prefill unstated)linear-probe directions48 heads across layers, one direction each
RepE ( Zou et al., 2023 )unstatedper-layer reading vectors from contrastsevery 3rd layer, that layer’s own vector
SAE clamping ( Templeton et al., 2024 )all prefill + decodefound SAE feature, clampedone layer
learned per-layer steering vectors (ours, § 4 )selected prefill (designated spans)trained end-to-end against behaviorall layers, one vector per layer
Appendix
Table A.1: Residual-stream editing methods: where the edit lands, how it is made, and how it varies with depth. “Unstated” means neither the paper nor its released code settles a single position policy.
overlaygold constructionvalidator
all capitalsprogrammaticfully uppercased
Spanishmodel rewritelanguage detector above 0.9
Germanmodel rewritelanguage detector above 0.9
haikumodel rewritethree lines, word bounds
nested bulletsmodel rewriteall lines bulleted, depth at least 3
declineedit, origin voicefilter named, target refused, siblings verbatim
Appendix
Table B.1: Gold construction and validators for the eight transform overlays.
instructionteacherforward-KLCE-on-edit
Spanish2/241/3635/36
decline1/241/3620/36
no-lists24/2436/3635/36
disclaimer22/2435/3629/36
Appendix
Table B.2: The teacher ceiling (§ 3.3 ). Held-out compliance counts. The in-context teacher ignores its own instruction most of the time, forward-KL distillation faithfully reproduces that ceiling, and cross-entropy toward a validated edit breaks past it.
armruleprobe in data ↓\downarrowutility ↑\uparrowSEP ↑\uparrowclean FP ↓\downarrow
frozen modelsubstring72.9% ±\pm 0.792.7% ±\pm 0.123.8% ±\pm 0.62.7% ±\pm 0.1
 corrected72.1% ±\pm 0.691.9% ±\pm 0.124.2% ±\pm 0.50.8% ±\pm 0.1
with overlaysubstring5.3% ±\pm 0.393.0% ±\pm 0.094.7% ±\pm 0.42.5% ±\pm 0.3
 corrected3.6% ±\pm 0.292.3% ±\pm 0.296.4% ±\pm 0.10.8% ±\pm 0.0
Appendix
Table G.1: SEP, all 9,160 items, temperature 0.7. Mean over three seeds ±\pm the range across them.
categoryfrozen basemarked
 injected ↓\downarrowclean ↓\downarrowinjected ↓\downarrowclean ↓\downarrow
infrastructure failure82.0%0.0%0.0%0.0%
phishing56.0%0.0%2.0%0.0%
content promotion54.0%0.0%0.0%0.0%
access denial20.0%0.0%0.0%0.0%
Appendix
Table G.2: PIArena, 50 items per category per condition. The clean column is the same item with no injection.
 defenseattack success ↓\downarrowutility ↑\uparrow
 none (undefended)49.5%82.5%
preventionPromptArmor48.0%84.0%
DataFilter30.0%80.0%
PISanitizer11.0%89.5%
SecAlign++3.5%68.5%
Semantic Overlays0.5%95.1% (retention)
Appendix
Table H.1: PIArena, Direct attack, mean of the HotpotQA and NQ RAG splits. Defense rows are Table 2 of Geng et al. (2026) ; the last row is this work on frozen Qwen3.5-9B. Utility is not commensurable down the column: theirs is absolute task accuracy, ours is retention (injected over clean), and the detection rows have none to report. It is shown because the two lowest-attack-success defenses are exactly the ones that pay for it.
 begins with marker ↓\downarrowmarker anywhere ↓\downarrow
attackfrozenoverlayfrozenoverlay
naive85.1%0.0%98.1%1.9%
ignore77.4%0.0%96.2%2.4%
escape separation92.8%0.0%99.0%1.9%
completion98.6%0.0%100%0.5%
any of the four99.0%0.0%100%3.8%
Appendix
Table I.1: AlpacaFarm, all 208 input-bearing items, frozen Qwen3.5-9B versus the same model with the input field marked. “Begins” is SecAlign’s criterion; “anywhere” is StruQ’s.
Figure J.1: Separation versus utility on SEP. Gray points are published operating points: open models with no defense, with prompt defenses, and with SEP fine-tuning ( Zverev et al., 2025 ) , and ASIDE on six models ( Zverev et al., 2026 ) . Fine-tuning buys 93–98% separation but pays for it in utility (17–77%); ASIDE’s strongest separation (Mistral, 92.1%) costs 20 points of utility. Two operating points combine both: GPT-4 under an engineered prompt, and the semantic overlay, shown on both of its base models (arrows): frozen Qwen3.5-9B, +70.8+70.8 points of separation at unchanged utility, and frozen Llama-3.1-8B (Appendix K ), +65.2+65.2 points at −1.7-1.7 .
defensemechanismserved base model
prompt defenses ( Zverev et al., 2025 )delimiters, warnings, re-stated instructionsunchanged; the defense is forgeable text
StruQ ( Chen et al., 2025 )fine-tune on structured promptsfine-tuned
ISE ( Wu et al., 2025 )trained segment embeddingsfine-tuned
ASIDE ( Zverev et al., 2026 )rotated data-token embeddingsfine-tuned
AIR ( Kariyappa and Suh, 2025 )privilege-indexed embedding at every decoder blockfine-tuned, jointly
V-Steer ( Zeng et al., 2026 )attribution-derived scaling of attention valuesfrozen; learned version named as future work
Appendix
Table J.1: Published prompt-injection defenses: mechanism, and what is served.
defensemodelseparationΔ\Delta seputilityΔ\Delta utilheadroom
ASIDEQwen3-8B45.3 →\to 71.4+26.1+26.158.9 →\to 66.3+7.4+7.448%
SEP fine-tuning ( Zverev et al., 2025 )seven open models93–98+22+22 to +84+8417–77−67-67 to +18+18—
engineered promptGPT-420.8 →\to 95.3+74.5+74.583.3 →\to 96.6+13.3+13.394%
Semantic Overlaysfrozen Qwen3.5-9B23.9 →\to 97.5+73.6+73.693.1 →\to 93.3+0.2+0.297%
corrected grader (Appendix F ) 24.3 →\to 99.0+74.7+74.792.3 →\to 92.6+0.3+0.399%
Appendix
Table J.2: SEP operating points. Paired cells are each method’s own baseline →\to defended; Δ\Delta columns are those paired differences, and headroom is the fraction of the baseline’s remaining separation recovered. Range rows aggregate models whose baselines differ; their deltas are per-model pairs. Foreign rows use each paper’s published scoring; the corrected-grader row rescores our own runs only.
 frozen modelwith overlay
SEP
probe in data ↓\downarrow66.8%3.5%
utility ↑\uparrow91.0%89.3%
SEP ↑\uparrow31.0%96.2%
TensorTrust
hijacking ASR ↓\downarrow68.9%4.4%
Appendix
Table K.1: The injection results of § 7 , replicated on frozen Llama-3.1-8B-Instruct . Metrics and scoring rules match the corresponding Qwen tables: SEP on 1,000 items under the published grader; TensorTrust on all 776 hijacking and 570 extraction rows, Defense Validity screened to rows the frozen model can answer; PIArena at 50 rows per family.

Explore similar work

CardsList
Apr 20, 2026

Beyond Pattern Matching: Seven Cross-Domain Techniques for Prompt Injection Detection

Current open-source prompt-injection detectors converge on two architectural choices: regular-expression pattern matching and fine-tuned transformer classifiers. Both share failure modes that recent work has made concrete. Regular expressions miss paraphrased attacks. Fine-tuned classifiers are vulnerable to adaptive adversaries: a 2025 NAACL Findings study reported that eight published indirect-injection defenses were bypassed with greater than fifty percent attack success rates under adaptive attacks. This work proposes seven detection techniques that each port a specific mechanism from a discipline outside large-language-model security: forensic linguistics, materials-science fatigue analysis, deception technology from network security, local-sequence alignment from bioinformatics, mechanism design from economics, spectral signal analysis from epidemiology, and taint tracking from compiler theory. Three of the seven techniques are implemented in the prompt-shield v0.4.1 release (Apache 2.0) and evaluated in a four-configuration ablation across six datasets including deepset/prompt-injections, NotInject, LLMail-Inject, AgentHarm, and AgentDojo. The local-alignment detector lifts F1 on deepset from 0.033 to 0.378 with zero additional false positives. The stylometric detector adds 11.1 percentage points of F1 on an indirect-injection benchmark. The fatigue tracker is validated via a probing-campaign integration test. All code, data, and reproduction scripts are released under Apache 2.0.

Thamilvendhan Munirathinam

Independent Researcher · prompt-shield project

Prompt Injection Attacks on LLMsPrompt Injection Attacks on AI AgentsPrompt Injection DetectionSequence Alignment
AskPDF
Aug 8, 2026

BASIS: Breach-Aware Selective Prompt Injection Shielding with Prefill Attention Probes

Prompt injection is a critical security threat in large language model (LLM) applications, where attackers hijack model behavior by embedding malicious instructions in user or external data. Existing detection methods only detect the presence of injection and refuse to respond upon detection, overlooking the fact that for many modern aligned models, well-crafted instructions can resist most injection attacks. This means that the injection robustness varies significantly across instructions and models. This leads to widespread unnecessary over-refusal: inputs containing injections that the model could have handled correctly are rejected incorrectly. To deal with this over-refusal issue, we propose BASIS (Robustness-Aware Prompt Injection Defense). This defense method uses the Attention Competition Ratio (ρρ) as features to train two sparse linear probes: an existence probe and a breach probe. Both probes make defense decisions through cascaded gating, which does not require additional LLM inference. BASIS comprises three stages: injection existence detection, per-sample breach prediction, and instruction robustness assessment; the online cascade refuses only when the model would actually be compromised and thus avoids over-refusal on robust instructions. Experiments across four tasks and six open-source LLMs show that BASIS maintains near-perfect injection detection while substantially reducing over-refusal on safe attack samples, especially under robust instruction templates.

Laiqiao Qin, Tianqing Zhu, Longxiang Gao +1

City University of Macau, Macau · Qilu University of Technology

Prompt Injection DefensePrompt Injection Attacks on LLMsRefusal Behavior in Language ModelsPrompt Injection DetectionPrompt Robustness
AskPDF
Feb 22, 2026

Prompt Injection as Role Confusion

LLMs see the world as a single stream of text, partitioned into roles like <user> or <tool>. We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like <user> text, despite its <tool> label. We design role probes to measure how LLMs internally perceive "who is speaking," and find that injected text occupies the same representational space as the trusted role it imitates. We demonstrate this with CoT Forgery, a zero-shot attack that injects fabricated reasoning into user prompts and tool outputs. Models mistake the forgery for their own thoughts, yielding 60% attack success against frontier models with near-zero baselines. Strikingly, the degree of role confusion predicts attack success before a single token is generated. This mechanism generalizes beyond CoT Forgery to standard agent prompt injections, revealing prompt injection as a measurable consequence of role perception. To the model, sounding like a role is indistinguishable from being one. Project page and writeup: https://role-confusion.github.io

Charles Ye, Jasmine Cui, Dylan Hadfield-Menell

*Equal contribution 1Independent · 2Massachusetts Institute of Technology, Cambridge, MA, United States

Adversarial Attacks on LLMsPrompt Injection AttacksPrompt Injection Attacks on LLMsPrompt Injection Attacks on AI Agents
AskPDF