Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wild

Elad David; Max Fomin

cs.LGOct 1, 2026

Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wild

Authors: Elad David, Max Fomin

Organizations: Zenity Tel Aviv, Israel

Abstract

LLM agents increasingly rely on activation probes as runtime monitors for prompt injection, jailbreaks, and unsafe requests, reading the model's own hidden state to catch a harmful input before the agent acts on it. A cheap, increasingly common move, borrowed from LLM-as-judge prompting, is to append a short classification instruction after the user's turn and read the probe at that point, to sharpen it: the instruction asks the model to represent the incoming request as a class, concentrating the signal the probe must separate, at negligible serving cost. But does the wording of that suffix matter, and does its benefit hold in the wild, on attack types the probe never saw in training, the regime a deployed monitor faces? We test this with a controlled ladder of post-user suffixes under strict leave-one-dataset-out (LODO) evaluation across 13 safety benchmarks (jailbreak, injection, and benign chat) and three open-weight model families (Llama-3.1-8B, Qwen3.5-9B, Gemma-4-12B). On a single-position probe, a classification suffix consistently improves out-of-distribution detection over no suffix (up to ~4 AUC points); yet which suffix matters: prompting the model to classify the input, even into content-free labels, reliably wins; an off-topic or merely-attentive suffix helps little. The gain comes from the classification format, not the named criterion: a content-free suffix matches the real malicious/benign one, with the criterion adding precision only at strict thresholds. This is not an artifact of the single-position read: the benefit carries to the multi-position pooling probes used in production (attention, multi-max, MLP), though the best-performing suffix there is readout-dependent. Served through a KV-cache fork, it is a cheap drop-in for any activation-probe monitor, though not an automatic win: which suffix helps, and by how much, depends on the model and the readout.

Figures & tables

Suffix	Adds	Isolates
none	—	baseline
irrelevant	off-topic text	is any appended text enough?
reflect	instruction to attend	is attending (no decision) enough?
decide	a binary decision (prose)	does deciding help without a template?
classify_grammar/length	+ output template, unrelated criterion	does the criterion matter?
A/B	content-free labels	does criterion content matter?

Table 1: The suffix ladder (experimental design). Each suffix adds one ingredient; comparing adjacent suffixes isolates it.

Axis	Tillman & Mossing 2025	This work
What in the prompt matters	meaningful instruction (not decomposed)	format-vs-content ladder
Generalization test	limited within-task shift, 1 dataset	cross-dataset LODO, 13 benchmarks
Models	GPT-4o (one, closed)	3 open families (Llama/Qwen/Gemma)
Robustness	single layer/position	across layers and readout architectures (single-position to pooling)
Content sensitivity	—	content-tailoring overfits OOD
Deployment	prompt-side instruction	KV-fork suffix: single forward pass, no re-prefill

Table 2: Delta over Tillman and Mossing (2025) . Every row is backed in this paper. (T&M report AUROC on a single closed model, GPT-4o, with limited within-task distribution shift on one dataset.)

Panel of Figure 1 — Figure 1: The classification suffix on the KV-cache fork. A short classification instruction is appended after the user turn; the probe reads at end-of-suffix; the cache is reverted so generation proceeds unchanged. We adopt the KV-fork vehicle from David et al. (2026) and study what the suffix should say .

Figure 2: A classification suffix lifts OOD detection; format, not criterion, carries the ranking gain. $\Delta$ shared-benign AUC vs. no suffix (mean over 9 LODO datasets, $\pm$ SEM). A neutral suffix ( reflect ) barely moves ranking; asking the model to classify lifts it (Llama, Gemma), and content-free A/B matches named intent . Qwen is near-ceiling. Precision (the criterion’s low-FPR gain): App. A.1 .

Suffix	Llama	Qwen	Gemma
irrelevant (append only)	$-0.14$	$+0.07$	$\mathbf{+1.29^{*}}$
reflect (attend)	$-0.44$	$+0.76$	$+0.61$
decide (prose decision)	$+2.14$	$-0.25$	$+0.41$
classify_grammar	$+2.00$	$+0.85$	$+2.33$
classify_length	$-1.56$	$+1.22$	$\mathbf{+3.09^{*}}$
A/B (content-free)	$\mathbf{+3.61^{*}}$	$+1.18$	$\mathbf{+2.71^{*}}$

Table 3: Suffix ladder:

\Delta

shared-benign AUC vs. none (points; LODO,

N{=}2000

; 9 malicious-containing datasets; paired Wilcoxon

{}^{*}p{<}0.05

). Content-free A/B

\approx

named intent across models; on Llama the null suffixes sit at baseline, while Gemma responds to any post-user suffix (relocation, § 4 ). Qwen is near-ceiling.

Figure 3: The gain concentrates where the probe has headroom. Each point is one (model, dataset): no-suffix shared-benign AUC ( $x$ ) vs. the intent gain ( $y$ ). Pooled $r{=}{-}0.78$ (Llama $-0.77$ , Gemma $-0.92$ , Qwen $-0.29$ ); part of the slope is a ceiling effect (bounded AUC), so we read it as a headroom regularity, not a law.

Readout	Gemma-4	Qwen3.5	Llama-3.1
	(L47)	(L31)	(L31)
attention	$+1.7\ (+2.0)$	$+0.8\ (+1.3)$	$+2.3\ (+2.8)$
multi-max	$+4.5\ (+4.8)$	$+0.4\ (+0.7)$	$+2.0\ (+2.9)$
MLP	$+1.5\ (+1.9)$	$+0.9\ (+1.0)$	$+1.6\ (+1.8)$
mean	$+2.1\ (+2.4)$	$+0.6\ (+0.7)$	$+1.9\ (+2.2)$

Table 4: The suffix helps across readout architectures. Suffix gain over no-suffix in shared-benign LODO AUC (pp), primary layers, per pooling readout ( Kramár et al., 2026 ) . Each cell: mean over the five-rung ladder (best rung in parentheses). Averaged over the ladder, 22/24 model

\times

readout configs and 110/120 (readout

\times

rung) cells beat no-suffix (92%), needing no rung selection. Full grid: Table 14 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

dataset	none	reflect	A/B	intent
deepset	80.0	88.7	89.4	90.9
harmbench	80.9	88.4	90.0	91.1
yanismiraoui	89.2	85.9	94.6	94.0
jayavibhav	95.4	93.8	94.3	93.5
wildjailbreak	91.7	94.3	94.6	96.1
qualifire	92.3	95.2	97.7	95.3

Appendix

Table 5: Llama : per-dataset shared-benign AUC by suffix.

dataset	none	reflect	A/B	intent
deepset	90.3	92.5	90.1	89.1
harmbench	89.1	88.9	94.3	92.7
yanismiraoui	91.3	95.7	95.4	94.1
jayavibhav	93.7	95.0	96.3	95.6
wildjailbreak	95.4	97.4	95.9	95.8
qualifire	98.3	98.7	98.4	97.9

Appendix

Table 6: Qwen : per-dataset shared-benign AUC by suffix.

dataset	none	reflect	A/B	intent
deepset	84.4	84.8	88.5	90.8
harmbench	85.1	84.7	93.3	92.1
yanismiraoui	92.5	95.4	89.8	92.4
jayavibhav	92.6	92.8	96.1	96.4
wildjailbreak	96.0	–	96.8	97.4
qualifire	96.7	95.7	97.7	97.1

Appendix

Table 7: Gemma : per-dataset shared-benign AUC by suffix.

Panel of Figure 4 — Figure 4: Qwen (left) and Gemma (right) macro-ROC , malicious vs. shared-benign over the 9 LODO folds (cf. Llama, Fig. 7 ). Qwen curves bunch near a 0.94 baseline; Gemma separates like Llama on area.

Suffix	Llama	Qwen	Gemma
reflect (attend)	$-9.1$	$+4.7$	$+17.3$
A/B (content-free)	$-8.4$	$+4.2$	$+17.6$
intent (mal/benign)	$\mathbf{+19.6}$	$+4.5$	$\mathbf{+24.2}$

Appendix

Table 8: Suffix ladder:

\Delta

TPR@FPR

=

1% vs. none (pp). intent pulls ahead of content-free A/B on Llama/Gemma; A/B underperforms no-suffix on Llama; Qwen stays flat. The Gemma reflect figure averages 8 datasets ( wildjailbreak has no reflect run).

dataset	none	reflect	A/B	intent
deepset	5.9	10.3	5.4	30.5
harmbench	24.2	20.8	4.5	44.8
yanismiraoui	15.9	8.1	26.5	52.5
jayavibhav	33.9	17.9	11.2	30.5
wildjailbreak	18.0	14.3	10.5	42.7
qualifire	36.4	44.2	52.9	61.2

Appendix

Table 9: Llama : per-dataset TPR@1% (%) by suffix.

dataset	none	reflect	A/B	intent
deepset	8.4	8.4	9.4	4.4
harmbench	31.0	32.2	40.8	47.2
yanismiraoui	15.2	19.7	25.9	15.6
jayavibhav	12.4	23.7	26.3	24.0
wildjailbreak	12.4	30.8	13.3	19.3
qualifire	65.4	64.7	65.7	66.7

Appendix

Table 10: Qwen : per-dataset TPR@1% (%) by suffix.

dataset	none	reflect	A/B	intent
deepset	11.8	19.2	14.8	21.7
harmbench	16.8	37.2	46.8	62.0
yanismiraoui	17.9	49.4	23.0	27.5
jayavibhav	5.2	15.3	13.3	26.8
wildjailbreak	7.8	–	32.0	35.3
qualifire	54.1	54.8	63.4	68.7

Appendix

Table 11: Gemma : per-dataset TPR@1% (%) by suffix.

Benchmark	Type	benign	malicious
advbench	harmful instructions	—	520
harmbench	harmful behaviors	—	400
wildjailbreak	jailbreaks	—	2000
yanismiraoui	jailbreaks	—	1034
deepset	prompt injection	343	203
jayavibhav	prompt injection	996	1004

Appendix

Table 12: Evaluation benchmarks. Role and number of benign/malicious examples per benchmark in the primary panel.

Readout	Positions	Fit
single-position (linear)	last suffix token	L2-logistic, $C{=}0.1$
attention pooling	all	softmax-weighted, Adam
multi-max pooling	all	top- $k$ max, Adam
MLP pooling	all	per-position MLP + pool, Adam

Appendix

Table 13: Probe readouts. Token positions each readout uses and how it is fit.

Gemma-4 (L47)
Readout	attention	multi-max	MLP	mean
none	93.6	90.8	93.1	90.9
irrelevant	94.8	95.3	94.0	92.2
reflect	95.1	95.5	94.2	92.9
classify:length	95.6	95.0	95.0	93.2
classify:A/B	95.2	95.0	94.9	93.2

Appendix

Table 14: Full architecture

\times

suffix grid. Shared-benign LODO AUC (%),

N{=}2000

/dataset, 9 malicious datasets vs. shared benign pool. Each block is one model/layer; rows are the suffix ladder, columns the pooling readouts; none is the matched baseline, bold = best rung per column. Almost every column beats none where the baseline has headroom; the winning rung is readout-dependent.

Panel of Figure 5 — Figure 5: Where it has headroom, the suffix aligns the malicious representation (Gemma). Top: t-SNE of Gemma’s single-position readout across the ladder, colored malicious/benign; classes interleave with no suffix and separate under the classification suffixes. Bottom: leave-one-source-out transfer of a mean-difference malicious axis. Gemma climbs $0.74\to 0.93$ ; Llama and Qwen start already-aligned ( $0.93$ , $0.96$ ) and barely move.

Figure 6: The format/content dissociation. Each point is a suffix, by its mean change vs. no-suffix in shared-benign AUC ( $x$ , ranking) and TPR@1% ( $y$ , precision); one panel per model. A/B and intent share the same $x$ but intent is far higher on $y$ ( $\approx$ $28$ pp on Llama); neutral reflect sits near the origin (except Gemma, where it already lifts precision).

Figure 7: Llama macro-ROC (malicious vs. shared benign, 9 LODO folds). intent (0.936) and A/B (0.934) lift over no-suffix (0.898); reflect (0.893) does not. Curves cross near FPR $=$ 1% (dotted): A/B matches intent on area but drops below no-suffix at the strict operating point. $\pm 1\sigma$ $=$ cross-dataset spread.

Explore similar work

Oct 13, 2026

Prompt-Injection Probes Learn the Conversation, Not Just the Injection

Linear probes on a model's hidden states are used as cheap monitors for indirect prompt injection: they screen the tool outputs an agent reads and send suspicious ones to a slower second check, so every false alarm has a cost. But a probe learns whatever separates its training examples. The standard evaluation of injection probes can be passed without the model: conversation position and text lengths alone reach 0.91-0.98 AUROC. When each attack is compared with the same conversation without the injection, or with harmless text formatted like an injection, standard probes flag most of these harmless cases. We add such cases to the training data as matched negatives. Across six open models, in applications seen during training and at the default threshold, changing only the training data cuts false alarms on injection-free conversations from 41-80% to 4-27%, and on harmless look-alikes from 92-98% to 3-16%. With a threshold set on normal traffic, our probe still catches a median 95% of the attacks that fooled the agent (91% without our data). In new applications, false alarms stay low, but detection depends on the model and on how the threshold is set. The same data also helps a text-only filter.

Kaan Kale, Sriram Vishwanath

Georgia Institute of Technology

Prompt Injection Detection Indirect Prompt Injection Hidden-State Probing Shortcut Learning

Ask PDF

Apr 13, 2026

ProbeLogits: Kernel-Level LLM Inference Primitives for AI-Native Operating Systems

An OS kernel that runs LLM inference internally can read the model's own next-token logit distribution before any text is generated, and act on it as a governance primitive. I present ProbeLogits, a kernel-level operation that performs a single forward pass and reads specific token logits to classify an agent's action as safe or dangerous, with zero learned parameters. Because the probe reads a logit from the same base model the agent already runs, it removes the second model a fine-tuned guard requires: the marginal cost of a safety check becomes a single logit read. I evaluate ProbeLogits on three base models (Qwen2.5-7B, Llama-3-8B, Mistral-7B) across three external benchmarks (HarmBench, XSTest, ToxicChat). On HarmBench non-copyright, all three reach a 97-99% block rate. On ToxicChat (n=1,000), ProbeLogits attains F1 parity-or-better against Llama Guard 3: Qwen2.5-7B Safe/Dangerous reaches F1=0.812 (+13.7 pp, bootstrap 95% CIs disjoint), Llama-3 matches within CI (+0.4 pp), and Mistral exceeds by +4.4 pp. Classification is a measured 2.4-3.4x faster than Llama Guard 3 (332-556 ms vs. 851-1,142 ms), because it reads a single logit position instead of generating tokens. A calibration strength alpha acts as a deployment-time policy knob rather than a learned hyperparameter, trading recall for precision per operation class. I implement ProbeLogits within Anima OS, a bare-metal x86-64 kernel written in ~285,000 lines of Rust. Because agent actions must pass through kernel-mediated host functions, enforcement operates below the WASM sandbox boundary, making it substantially harder to circumvent than application-layer classifiers.

Daeyeon Son

Independent Researcher Republic of Korea

LLM Inference LLM Guardrails Language Model Introspection LLM Agent Security AI Control Safety Filtering+2

Ask PDF

Sep 16, 2026

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models

Autonomous systems increasingly rely on Large Language Models (LLMs) yet the safety infrastructure surrounding these models introduces latency and compute overhead. This limits utility in resource-constrained, time-critical deployments. Existing external guardrail models remain blind to the model's internal workings, creating a fundamental assurance gap. We ask: does the model already know when the content is harmful? We extract activations from LLaMA-3.1-8B and train lightweight MLP classifier probes (12.6M parameters) to detect harmful prompts. Evaluated on WildJailbreak, Beavertails, and AEGIS 2.0, our probes achieve F1 scores of 99%, 83%, and 84%, respectively competitive with 1000x larger guard models while cutting latency and compute costs.

Alizishaan Khatri, Chiquita Prabhu, Omkar Neogi

Wrynx Inc. · Independent Researcher

Safety Filtering LLM Safety Language Model Probing

Ask PDF