SALLIE: Generation-Free Hidden-State Detection of Jailbreaks and Prompt Injections Across Text and Vision

Guy Azov; Ofer Rivlin; Guy Shtar

cs.CRApr 6, 2026

SALLIE: Generation-Free Hidden-State Detection of Jailbreaks and Prompt Injections Across Text and Vision

Authors: Guy Azov, Ofer Rivlin, Guy Shtar

Organizations: Intuit

Abstract

Large Language Models (LLMs) and Vision-Language Models (VLMs) are vulnerable to jailbreaks and prompt injections delivered through text or images. Existing defenses often narrow threat coverage or add inference cost through input transformations, repeated model queries, generated responses, or threat-specific detectors. We present SALLIE (Single-Pass Activation Lookup for Layerwise Input Evaluation), a white-box, generation-free detector with a common architecture for textual and visual jailbreaks and prompt injections. Using one backbone forward pass, SALLIE extracts last-token residual-stream representations, applies layer-wise k-nearest-neighbor probes, and averages their scores across a contiguous layer range, without modifying the backbone or generating a response. To distinguish reusable detector configurations from deployment-specific tuning, we compare shared, threshold-only, and fully calibrated regimes across backbones and modalities. We evaluate SALLIE on compact open-weight VLMs (Gemma-3-4b-it, Phi-3.5-vision-instruct, SmolVLM2-2.2B-Instruct) using heterogeneous held-out test data spanning both threat families and modalities. Fully calibrated SALLIE-Phi achieves an aggregate F1 of 0.87 and higher aggregate balanced accuracy than RCS-KCD on all three backbones under a matched-calibration protocol. On Phi-3.5 visual inputs, threshold-only calibration yields an F1 of 0.99 with zero observed false positives, versus 0.54 for zero-shot Gemini-2.5-Flash-Lite. On Gemma-4-E2B-it, a backbone held out from representation-hyperparameter selection, the same procedure raises F1 from 0.62 to 0.79 on text and 0.68 to 0.98 on visual inputs. Representation hyperparameters can thus transfer across some backbones, while operating points remain backbone-, modality-, and distribution-dependent. We further report FPR/FNR tradeoffs, latency, and representation ablations.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of SALLIE . A single forward pass through the frozen backbone yields last-token activations from the layer range $\mathcal{L}_{m}$ , scored per layer by $k$ -NN probes against cached training activations and averaged into $\bar{p}$ , which is compared with the threshold $\tau_{m}$ . The same architecture serves both threat families and modalities.

	Gemma-3	Phi-3.5	SmolVLM2
$k_{m}$	3/11	5/11	11/9
$c_{m}$	64/None	512/None	128/None
$\tau_{m}$	0.61/0.95	0.65/0.06	0.85/0.96
$\mathcal{L}_{m}$	0-16/17-33	0-15/8-15	12-23/18-23
Val. FNR	0.04/0.23	0.03/0.15	0.26/0.29

Table 1: Fully calibrated SALLIE configurations selected on the validation split subject to

\mathrm{FPR}\leq 0.001

. Values are reported as text/vis.

c_{m}=\text{None}

denotes no PCA reduction, and

\mathcal{L}_{m}

denotes zero-indexed layer ranges. FNR is measured on validation data at the selected threshold.

	Deployment		F1			Bal. Acc.	FNR				FPR
Method	Proc. images	Gen.- free	Agg.	Text	Vis.	Agg.	Text JB	Text PI	Vis. JB	Vis. PI	Text	Vis.
Perplexity Filter	$\times$	✓	0.21	0.31	0.00 ‡	0.47	0.99	0.67	1.00	1.00	0.33	0.00
EEG-Defender	$\times$	✓	0.56	0.34	0.81 ‡	0.45	0.82	0.62	0.00	0.09	0.77	0.53
PIShield	$\times$	✓	0.70	0.75	0.63 ‡	0.45	0.03	0.00	0.00	0.30	1.00	1.00
JailGuard	✓	$\times$	0.61	0.58	0.64	0.68	0.12	0.80	0.61	0.44	0.08	0.13
HiddenDetect	✓	✓	0.47	0.65	0.11	0.65	0.17	0.70	1.00	0.91	0.02	0.00

Table 2: Held-out test performance at each method’s selected operating point. F1 denotes malicious-class F1. Aggregate balanced accuracy averages malicious recall and benign recall after pooling textual and visual test examples. For RCS-KCD, balanced accuracy is computed separately for each seed and then averaged. FNR is reported by threat category and FPR by modality. SALLIE uses fully calibrated configurations; RCS-KCD values are means over three seeds. Bold denotes the best overall result, and underline denotes the best open-weight result. Zero FPR or FNR values denote no observed errors of the corresponding type on the finite test set. † Proprietary autoregressive judges. ‡ Text-only methods ignore image content. s Per-seed and native-calibration RCS results are in Appendix G.

Backbone	Mod.	Shared	Threshold- only	Fully calibrated
Gemma-3	Text	0.78	0.86	0.85
	Vis.	0.71	0.71	0.83
Gemma-4 ⋆	Text	0.62	0.79	–
	Vis.	0.68	0.98	–
Phi-3.5	Text	0.73	0.73	0.79
	Vis.	0.50	0.99	0.96

Table 3: Held-out test F1 for the three calibration regimes on the same test set as Table 2 . ⋆ Gemma-4 is held out from representation-hyperparameter selection. † No SmolVLM2-text threshold satisfies the validation FPR constraint; the FPR-minimizing diagnostic value is shown.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Modality	Type	Split	Samples
( Taori et al. 2023 )	Textual	Benign	Training	20,000
( Shen et al. 2024 )	Textual	Jailbreak	Training	1,270
( Luo et al. 2024 )	Textual	Jailbreak	Training	414
( Zou et al. 2025 )	Textual	Benign	Training	10,000
( Zou et al. 2025 )	Textual	Prompt Injection	Training	10,000
( Luo et al. 2024 )	Visual	Jailbreak	Training	6,000

Appendix

Table 4: Training-split composition. § Samples drawn from a fixed split of the original dataset (validation split for Goyal et al. (2017) and Singh et al. (2019) , training split for Marino et al. (2019) ).

Dataset	Modality	Type	Split	Samples
( Hendrycks et al. 2021 )	Textual	Benign	Validation	14,042
( Röttger et al. 2024 )	Textual	Benign	Validation	250
( Chao et al. 2024 )	Textual	Jailbreak	Validation	637
( Andriushchenko et al. 2024 )	Textual	Jailbreak	Validation	689
( Jia et al. 2025 )	Visual	Benign	Validation	1,000
( Li et al. 2024b )	Visual	Jailbreak	Validation	1,168

Appendix

Table 5: Validation and test split composition. † Train and test samples are drawn from disjoint subsets; test samples consist of 200 examples each from the boolq, dolly, and hotelreview subsets (benign and malicious). ¶ We sample 200 benign data points. ∗ We put the text from Hazan et al. (2025) inside images. ∗∗ Sampled from the safebench data.

Parameter	Candidate values
Neighbor count $k$	$\{3,5,7,9,11\}$
PCA dimension $c$	$\{64,128,256,512,\mathrm{None}\}$
Relative layer range $\mathcal{L}$	$\{$ 0–25%, 0–50%, 25–50%, 25–75%, 25–100%, 50–100%, 75–100% $\}$

Appendix

Table 6: Complete configuration-search grid. Relative-depth intervals are mapped deterministically to inclusive, zero-indexed decoder-layer ranges using the rule above.

c=\mathrm{None}

denotes that no PCA reduction is applied.

Modality	$k$	$c$	$\mathcal{L}$	$\tau$
Text	3	256	0%-50%	0.70
Visual	11	512	25%-100%	0.90

Appendix

Table 7: Shared-regime configurations selected on validation data using Gemma-3 and Phi-3.5 as development backbones. Layer ranges are expressed as fractions of network depth and converted to zero-indexed layer ranges for each backbone.

Backbone	$\tau^{*}_{\text{text}}$	$\tau^{*}_{\text{vis}}$
Gemma-3-4b-it	0.608	0.899
Phi-3.5-vision-instruct	0.688	0.682
SmolVLM2-2.2B-Instruct	0.973 †	0.920
Gemma-4-E2B-it	0.589	0.613

Appendix

Table 8: Thresholds selected under the threshold-only regime (

\tau^{*}=\arg\min\mathrm{FNR}

subject to

\mathrm{FPR}\leq 0.001

on the validation split, per backbone and modality, with the shared representation configuration held fixed). † No threshold satisfies the constraint for SmolVLM2 on text; the FPR-minimizing value is reported and the corresponding results are diagnostic.

Method	Precision	Recall	F1
Perplexity Filter	0.50	0.13	0.21
EEG-Defender	0.56	0.57	0.56
PIShield	0.57	0.91	0.70
HiddenDetect	0.97	0.31	0.47
JailGuard	0.87	0.46	0.61
SALLIE (SmolVLM2)	1.00	0.35	0.52

Appendix

Table 9: Detection performance on the held-out test set, aggregated across modalities and attack types. SALLIE rows use fully calibrated configurations. All values are rounded to two decimal places.

Backbone	Modality	TP	FP	TN	FN	FPR	FNR	F1
Gemma-3	Text	701	36	564	212	0.060	0.232	0.850
Gemma-3	Visual	457	0	460	189	0.000	0.293	0.829
Gemma-3	Aggregate	1158	36	1024	401	0.034	0.257	0.841
Phi-3.5	Text	600	3	597	313	0.005	0.343	0.792
Phi-3.5	Visual	646	49	411	0	0.107	0.000	0.963
Phi-3.5	Aggregate	1246	52	1008	313	0.049	0.201	0.872

Appendix

Table 10: Fully calibrated SALLIE operating points on the held-out test set. TP/FP/TN/FN are reported separately for textual, visual, and aggregate inputs. FPR and FNR are realized test-set values, not validation constraints.

Backbone	Modality	$\boldsymbol{\tau}$	TP	FP	TN	FN	FPR	FNR	F1
Gemma-3	Text	0.608	708	26	574	205	0.043	0.225	0.860
	Visual	0.899	352	0	460	294	0.000	0.455	0.705
Gemma-4 ⋆	Text	0.589	609	14	586	304	0.023	0.333	0.793
	Visual	0.613	623	3	457	23	0.007	0.036	0.980
Phi-3.5	Text	0.688	529	3	597	384	0.005	0.421	0.732
	Visual	0.682	630	0	460	16	0.000	0.025	0.987

Appendix

Table 11: Held-out test operating points under the threshold-only regime. The shared values of

k

, PCA dimension, and relative layer range are retained, and only the decision threshold

\tau

is calibrated per backbone and modality using validation data. FPR, FNR, and malicious-class F1 are realized test-set values computed after pooling all held-out sources within each modality. Textual rows contain 913 malicious and 600 benign examples; visual rows contain 646 malicious and 460 benign examples. ⋆ Gemma-4 is held out from representation-hyperparameter selection. † No SmolVLM2-text threshold satisfies the validation FPR constraint; the FPR minimizing diagnostic operating point is shown.

Modality	Attack	Dataset	SALLIE		Other
Modality	Attack	Dataset	Phi-3.5	Gemma-3	JailGuard	HiddenDetect
Visual	PI	Astra	0.00	1.00	0.42	0.97
Visual	PI	VPI-Bench	0.00	0.01	0.44	0.89
Visual	JB	FigStep	0.00	0.24	0.61	1.00
Text	JB	StrongREJECT	0.39	0.20	0.12	0.17
Text	PI	BoolQ malicious	0.20	0.17	0.93	0.99

Appendix

Table 12: Per-source false negative rates on held-out attack datasets. SALLIE values use the fully calibrated configurations. Lower is better. PI denotes prompt injection and JB denotes jailbreak.

Modality	Dataset	SALLIE		Other
Modality	Dataset	Phi-3.5	Gemma-3	JailGuard	HiddenDetect
Visual	HallusionBench	0.09	0.00	0.07	0.00
Visual	LLaVA-Bench	0.37	0.00	0.13	0.00
Visual	MM-Vet	0.05	0.00	0.20	0.00
Text	boolq_clean	0.00	0.07	0.09	0.00
Text	dolly_clean	0.01	0.11	0.14	0.01

Appendix

Table 13: Per-source false positive rates on held-out benign datasets. SALLIE values use the fully calibrated configurations. Lower is better.

Backbone	Modality	Last token			Mean pool			$\Delta$ F1
		FPR	FNR	F1	FPR	FNR	F1
Gemma-3	Text	0.060	0.232	0.850	0.857	0.123	0.719	+0.131
Gemma-3	Visual	0.000	0.293	0.829	0.017	0.242	0.857	$-$ 0.028
Phi-3.5	Text	0.005	0.343	0.792	0.830	0.136	0.717	+0.075
Phi-3.5	Visual	0.107	0.000	0.963	0.400	0.000	0.875	+0.088

Appendix

Table 14: Token-representation ablation on held-out test data. The last-token representation yields higher F1 in three of the four evaluated backbone-modality settings; on text, mean pooling reduces FNR only at the cost of false positive rates above 0.8. The exception is Gemma-3 on visual inputs, where mean pooling attains slightly higher F1.

Panel of Figure 2 — Figure 2: Hyperparameter sensitivity under the constraint $\mathrm{FPR}\leq 0.001$ . For each model, the $k$ plot reports mean FNR as $k$ varies, averaged over all PCA dimensions, all candidate layer ranges, and both modalities. The $c$ plot reports mean FNR as the PCA dimension varies, averaged over all $k\in\{3,5,7,9,11\}$ values, all candidate layer ranges, and both modalities. $c=\mathrm{None}$ denotes the full dimensional representation.

Figure 3: Layer-range sensitivity under the constraint $\mathrm{FPR}\leq 0.001$ . For each model and candidate layer range, we report mean FNR averaged over all $k\in\{3,5,7,9,11\}$ values, all PCA dimensions $c\in\{64,128,256,512,\mathrm{None}\}$ , and both modalities. Lower values indicate stronger average detection performance across the remaining hyperparameter grid. Layer range has a substantial effect on FNR, and the preferred range differs across backbones.

Figure 4: PCA projection of last-token hidden-state activations at the midpoint of each backbone’s calibrated visual layer range, colored by input group. The visualization provides a qualitative view of the local structure used by SALLIE .

Figure 5: PCA projection of last-token hidden-state activations at the midpoint of each backbone’s calibrated visual layer range, separated by input modality.

		Aggregate F1 per seed			FPR (mean)
Backbone	Cal.	42	43	44	Text	Vis.
Phi-3.5	matched	0.955	0.935	0.847	0.32	0.09
	native	0.847	0.848	0.771	0.27	0.93
Gemma-3	matched	0.880	0.886	0.653	0.04	0.34
	native	0.843	0.860	0.616	0.11	0.94
SmolVLM2	matched	0.023	0.418	0.111	0.07	0.01

Appendix

Table 15: RCS-KCD per-seed aggregate F1 on the held-out test set under the matched calibration (min FNR s.t.

\mathrm{FPR}\leq 0.001

on validation) and the RCS-native calibration, with realized per-modality FPR averaged over seeds. The per-seed spread (e.g., Gemma-3 matched: 0.88/0.89/0.65) reflects the training-seed sensitivity of the learned per-layer projections.

Model	$\tau_{text}$	$\tau_{vis}$
Perplexity Filter	291.81	501.61
EEG-Defender	$-0.025$	$-0.042$
PIShield	0.0467	0.4962
HiddenDetect	0.305	0.401
JailGuard	0.02	0.025

Appendix

Table 16: Detection thresholds per modality for each baseline. JailGuard uses paper-default values and was not re-calibrated.

	Forward (ms)		$k$ -NN (ms)		Total (ms)
Backbone	100	1000	Text	Vis.	Text	Vis.
Gemma-3-4B	81	174	43	1024	217	1198
Phi-3.5	78	292	420	1743	712	2035
SmolVLM2-2.2B	44	216	44	322	260	538

Appendix

Table 17: Per-input decoder-plus-probe latency (ms; batch size 1, mean over 20 runs) for 100- and 1,000-token inputs under the fully calibrated configurations. Total sums the 1,000-token forward pass (with hidden-state extraction) and the

k

-NN overhead; these are partial visual-path measurements that exclude vision-encoder and host–device transfer time, not total request latency. High visual

k

-NN costs for Gemma-3, Phi-3.5, and SmolVLM2 reflect their no-PCA configurations under exact search.

Prompt Injection Detection

Prompt injection detection identifies malicious instructions in prompts, retrieved content, webpages, images, or tool outputs before they hijack language models and agents.

See all 59 papers

Feb 24, 2026

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks

Large Language Models (LLMs) are widely deployed in real-world systems. Given their broader applicability, prompt engineering has become an efficient tool for resource-scarce organizations to adopt LLMs for their own purposes. At the same time, LLMs are vulnerable to prompt-based attacks. Thus, analyzing this risk has become a critical security requirement. This work evaluates prompt-injection and jailbreak vulnerability using a large, manually curated dataset across multiple open-source LLMs, including Phi, Mistral, DeepSeek-R1, Llama 3.2, Qwen, and Gemma variants. We observe significant behavioural variation across models, including refusal responses and complete silent non-responsiveness triggered by internal safety mechanisms. Furthermore, we evaluated several lightweight, inference-time defence mechanisms that operate as filters without any retraining or GPU-intensive fine-tuning. Although these defences mitigate straightforward attacks, they are consistently bypassed by long, reasoning-heavy prompts.

Piyush Jaiswal, Aaditya Pratap, Shreyansh Saraswati +2

NIT Trichy, Tiruchirappalli, India · NCE Chandi, Nalanda, India · Bishop Cotton Boys’ School, Bengaluru, India +2

Prompt Injection Attacks on LLMs Language Model Safety Evaluation LLM Jailbreak Attacks Prompt Injection Defense

Ask PDF

Feb 23, 2026

Hiding in Plain Text: Detecting Concealed Jailbreaks via Activation Disentanglement

Large language models (LLMs) remain vulnerable to jailbreak prompts that are fluent and semantically coherent, and therefore difficult to detect with standard heuristics. A particularly challenging failure mode occurs when an attacker tries to hide the malicious goal of their request by manipulating its framing to induce compliance. Because these attacks maintain malicious intent through a flexible presentation, defenses that rely on structural artifacts or goal-specific signatures can fail. Motivated by this, we introduce a self-supervised framework for disentangling semantic factor pairs in LLM activations at inference. We instantiate the framework for goal and framing and construct GoalFrameBench, a corpus of prompts with controlled goal and framing variations, which we use to train Representation Disentanglement on Activations (ReDAct) module to extract disentangled representations in a frozen LLM. We then propose FrameShield, an anomaly detector operating on the framing representations, which improves model-agnostic detection across multiple LLM families with minimal computational overhead. Theoretical guarantees for ReDAct and extensive empirical validations show that its disentanglement effectively powers FrameShield. Finally, we use disentanglement as an interpretability probe, revealing distinct profiles for goal and framing signals and positioning semantic disentanglement as a building block for both LLM safety and mechanistic interpretability.

Amirhossein Farzam, Majid Behabahani, Mani Malek +2

Duke University · Princeton University · Morgan Stanley +2

LLM Security Representation Disentanglement Framing Effects in Language Models LLM Safety Benchmarks LLM Interpretability

Ask PDF

Oct 3, 2026

Reactivating Alignment: Defending LLMs from Jailbreaks via Intention-Aware Input-Output Matching

Large language models (LLMs) remain vulnerable to jailbreak attacks that conceal harmful intent within complex adversarial prompts. Existing defenses primarily rely on input perturbation or harmful-output suppression, but they rarely model where malicious intent resides, resulting in brittle protection and excessive over-refusal. We propose SENTINEL, a plug-and-play, generation-time jailbreak defense that reframes mitigation as an intent extraction problem. Our key insight is that instruction-tuned LLMs exhibit strong input--output semantic consistency: regardless of jailbreak complexity, generated outputs tend to align with the attacker's true intent. SENTINEL exploits this property by matching semantically aligned input--output regions to extract intention-revealing subsequences, scores these subsequences using refusal-direction projections to estimate harmfulness, and halts generation when necessary. Experiments on HarmBench across multiple LLMs show that SENTINEL reduces jailbreak success rates to close to 5% while maintaining low over-refusal. We further demonstrate robustness to adaptive attacks and provide a mechanistic interpretation: SENTINEL re-distributes jailbreak features from alignment blind spots to aligned regions.

Luoyu Chen, Weiqi Wang, Chenhan Zhang +2

University of Technology Sydney, Sydney, NSW, Australia · Xi’an Jiaotong University, Xi’an, Shaanxi, China · Southeast University, Nanjing, Jiangsu, China

Jailbreak Defense LLM Safety Alignment LLM Jailbreak Attacks

Ask PDF

Abstract

Prompt Injection Detection

Explore similar work

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks

Hiding in Plain Text: Detecting Concealed Jailbreaks via Activation Disentanglement

Reactivating Alignment: Defending LLMs from Jailbreaks via Intention-Aware Input-Output Matching