cs.AIOct 14, 2026

Are You Reading the Right Thing? Rethinking Internal Signals for Prompt-Injection Detection

Authors: Sarthak Gupta, Priyanka Tembey

Organizations: Operant AI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Several recent studies propose detecting prompt injections from a language model's internal state. However, these methods do not measure a single security property. Residual-stream probes, attention scores, role projections, exposure probes, and concept readouts may capture injection presence, recognition, information routing, or behavioral influence. It is therefore unclear what differences in prompt-injection detection performance reveal about the internal signals these methods read. We study how five representation-reading methods vary under a common structured indirect prompt-injection setup on Qwen3.6-35B-A3B and Gemma-4-26B-A4B-it. We evaluate each method by how well it distinguishes injected prompts from benign controls. We then further analyze each detection method's ability to detect attacks in cases where they are successful and in cases where they are not. This helps us better assess what properties each detection method might rely on. Methods trained to detect injection presence or attack-related concepts generally separate unsuccessful injections more strongly from benign inputs, whereas attention diversion, exposure, and role-confusion-based methods separate successful injections more strongly on both models. Attention Tracker provides the strongest consistent successful injection separation, reaching 0.949 AUROC on Qwen and 0.887 on Gemma. These results suggest that signals closer to response formation may better encode information about an attack's success, and that detection performance can depend strongly on the prompt template. We share the data associated with this study at https://anonymous.4open.science/r/paper-evaluation-data-7877

Figures & tables

Figure 1: Overview of our evaluation. We compare five internal-state methods on the same structured inputs and separate attacks by behavioral outcome. The two signal groups show different patterns across successful and unsuccessful attacks.
DetectorQwenGemma
J-Lens0.811±0.0170.811\pm 0.0170.711±0.0180.711\pm 0.018
PIShield0.793±0.0160.793\pm 0.0160.935±0.009\mathbf{0.935\pm 0.009}
Attention Tracker0.794±0.0160.794\pm 0.0160.707±0.0200.707\pm 0.020
Role Confusion0.707±0.0190.707\pm 0.0190.770±0.0190.770\pm 0.019
Dong exposure probe0.836±0.015\mathbf{0.836\pm 0.015}0.683±0.0200.683\pm 0.020
Table 1: Overall prompt-injection detection AUROC.
DetectorSuccessfulUnsuccessfulΔ\Delta
Qwen3.6-35B-A3B
J-Lens0.753±0.0250.753\pm 0.0250.850±0.0170.850\pm 0.017−0.096±0.024-0.096\pm 0.024
PIShield0.637±0.0280.637\pm 0.0280.898±0.014\mathbf{0.898\pm 0.014}−0.261±0.028-0.261\pm 0.028
Attention Tracker0.949±0.011\mathbf{0.949\pm 0.011}0.691±0.0230.691\pm 0.023+0.259±0.022+0.259\pm 0.022
Role Confusion0.822±0.0200.822\pm 0.0200.631±0.0240.631\pm 0.024+0.190±0.022+0.190\pm 0.022
Dong exposure probe0.873±0.0180.873\pm 0.0180.812±0.0180.812\pm 0.018+0.061±0.025+0.061\pm 0.025
Table 2: Template-matched AUROC by attack outcome.
More in this topic

Prompt Injection Detection

Prompt injection detection identifies malicious instructions in prompts, retrieved content, webpages, images, or tool outputs before they hijack language models and agents.

See all 59 papers

Explore similar work

CardsList
May 26, 2026

Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals

Prompt injection poses a critical threat to the safe deployment of large language models, yet existing detection approaches are typically evaluated under limited settings that do not reflect real-world operating constraints. In this work, we present a deployment-aware evaluation of prompt injection detection using a multi-model and multi-regime experimental framework. We compare lexical, semantic, structural, and transformer-based detectors across multiple out-of-distribution settings, repeated data splits, and both ranking and thresholded deployment metrics. We introduce interpretable structural signals that capture hierarchy overrides, system prompt spoofing, role redefinition, and evasion patterns, and assess their contribution both within sparse models and in combination with strong encoder baselines. Our results show that detection performance is highly regime-dependent and sensitive to threshold selection, with no single model dominating across all settings. Transformer-based models achieve the strongest overall performance, while structural signals provide modest but consistent gains in certain regimes and improve low false positive rate behaviour in harder scenarios. These findings highlight the gap between ranking performance and deployment effectiveness and underscore the importance of evaluating prompt injection defences under realistic operational constraints. Code will be released.

Akindoyin Akinrele, Shreyank N Gowda

School of Computer Science, University of Nottingham, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, United Kingdom

Language Model Safety EvaluationPrompt Injection Attacks on LLMsPrompt Injection Detection
AskPDF
Oct 13, 2026

Prompt-Injection Probes Learn the Conversation, Not Just the Injection

Linear probes on a model's hidden states are used as cheap monitors for indirect prompt injection: they screen the tool outputs an agent reads and send suspicious ones to a slower second check, so every false alarm has a cost. But a probe learns whatever separates its training examples. The standard evaluation of injection probes can be passed without the model: conversation position and text lengths alone reach 0.91-0.98 AUROC. When each attack is compared with the same conversation without the injection, or with harmless text formatted like an injection, standard probes flag most of these harmless cases. We add such cases to the training data as matched negatives. Across six open models, in applications seen during training and at the default threshold, changing only the training data cuts false alarms on injection-free conversations from 41-80% to 4-27%, and on harmless look-alikes from 92-98% to 3-16%. With a threshold set on normal traffic, our probe still catches a median 95% of the attacks that fooled the agent (91% without our data). In new applications, false alarms stay low, but detection depends on the model and on how the threshold is set. The same data also helps a text-only filter.

Kaan Kale, Sriram Vishwanath

Georgia Institute of Technology

Prompt Injection DetectionIndirect Prompt InjectionHidden-State ProbingShortcut Learning
AskPDF
Apr 20, 2026

Beyond Pattern Matching: Seven Cross-Domain Techniques for Prompt Injection Detection

Current open-source prompt-injection detectors converge on two architectural choices: regular-expression pattern matching and fine-tuned transformer classifiers. Both share failure modes that recent work has made concrete. Regular expressions miss paraphrased attacks. Fine-tuned classifiers are vulnerable to adaptive adversaries: a 2025 NAACL Findings study reported that eight published indirect-injection defenses were bypassed with greater than fifty percent attack success rates under adaptive attacks. This work proposes seven detection techniques that each port a specific mechanism from a discipline outside large-language-model security: forensic linguistics, materials-science fatigue analysis, deception technology from network security, local-sequence alignment from bioinformatics, mechanism design from economics, spectral signal analysis from epidemiology, and taint tracking from compiler theory. Three of the seven techniques are implemented in the prompt-shield v0.4.1 release (Apache 2.0) and evaluated in a four-configuration ablation across six datasets including deepset/prompt-injections, NotInject, LLMail-Inject, AgentHarm, and AgentDojo. The local-alignment detector lifts F1 on deepset from 0.033 to 0.378 with zero additional false positives. The stylometric detector adds 11.1 percentage points of F1 on an indirect-injection benchmark. The fatigue tracker is validated via a probing-campaign integration test. All code, data, and reproduction scripts are released under Apache 2.0.

Thamilvendhan Munirathinam

Independent Researcher · prompt-shield project

Prompt Injection Attacks on LLMsPrompt Injection Attacks on AI AgentsPrompt Injection DetectionSequence Alignment
AskPDF