Hiding in Plain Text: Detecting Concealed Jailbreaks via Activation Disentanglement

Amirhossein Farzam; Majid Behabahani; Mani Malek; Yuriy Nevmyvaka; Guillermo Sapiro

cs.AIFeb 23, 2026

Hiding in Plain Text: Detecting Concealed Jailbreaks via Activation Disentanglement

Authors: Amirhossein Farzam, Majid Behabahani, Mani Malek, Yuriy Nevmyvaka, Guillermo Sapiro

Organizations: Duke University · Princeton University · Morgan Stanley · Google DeepMind ·

Abstract

Large language models (LLMs) remain vulnerable to jailbreak prompts that are fluent and semantically coherent, and therefore difficult to detect with standard heuristics. A particularly challenging failure mode occurs when an attacker tries to hide the malicious goal of their request by manipulating its framing to induce compliance. Because these attacks maintain malicious intent through a flexible presentation, defenses that rely on structural artifacts or goal-specific signatures can fail. Motivated by this, we introduce a self-supervised framework for disentangling semantic factor pairs in LLM activations at inference. We instantiate the framework for goal and framing and construct GoalFrameBench, a corpus of prompts with controlled goal and framing variations, which we use to train Representation Disentanglement on Activations (ReDAct) module to extract disentangled representations in a frozen LLM. We then propose FrameShield, an anomaly detector operating on the framing representations, which improves model-agnostic detection across multiple LLM families with minimal computational overhead. Theoretical guarantees for ReDAct and extensive empirical validations show that its disentanglement effectively powers FrameShield. Finally, we use disentanglement as an interpretability probe, revealing distinct profiles for goal and framing signals and positioning semantic disentanglement as a building block for both LLM safety and mechanistic interpretability.

Figures & tables

Figure 1: A visual summary of our pipeline. ReDAct learns disentangled goal and framing representations from frozen LLM activations using the contrastive pairs in the GoalFrameBench corpus we construct. FrameShield can then detect jailbreak prompts via anomaly detection on the disentangled framing representation.

Figure 2: Visualization of how varying the framing of a prompt leads to a jailbreak.

Figure 3: Examples of contrastive prompts for goal and framing pairs in GoalFrameBench. The prompts are abbreviated here due to space limitations; the full examples are included in Appendix D .

	Llama-3-8B		Vicuna-7B		Qwen3-4B		Mistral-7B
	$v_{g}$	$v_{f}$	$v_{g}$	$v_{f}$	$v_{g}$	$v_{f}$	$v_{g}$	$v_{f}$
Goal	0.37	0.18	0.37	0.19	0.30	0.20	0.37	0.19
Framing	0.18	0.22	0.17	0.26	0.18	0.29	0.12	0.23

Table 1: ANOVA effect size analysis for association of each of Goal and Framing with

v_{g}

and

v_{f}

in models from various families of LLMs. Each cell shows the

\eta^{2}

between the corresponding row and column. Results for other LLMs are reported in Appendix F .

Method	Llama3-8B	Llama2-7B	Vicuna-7B	Vicuna-13B	Mistral-7B	Qwen2-0.5B	Qwen2.5-7B	Qwen3-4B
	(Acc / F1)	(Acc / F1)	(Acc / F1)	(Acc / F1)	(Acc / F1)	(Acc / F1)	(Acc / F1)	(Acc / F1)
JBShield	0.75 / 0.75	0.73 / 0.76	0.60 / 0.63	0.63 / 0.67	0.71 / 0.75	0.59 / 0.62	0.66 / 0.68	0.70 / 0.74
FrameShield-Last (Ours)	0.82 / 0.79	0.58 / 0.70	0.67 / 0.75	0.76 / 0.79	0.79 / 0.74	0.66 / 0.74	0.79 / 0.76	0.76 / 0.75
FrameShield-Crit (Ours)	0.76 / 0.69	0.80 / 0.79	0.81 / 0.79	0.81 / 0.82	0.76 / 0.69	0.74 / 0.65	0.94 / 0.93	0.89 / 0.87

Table 2: Performance comparison of FrameShield variants against JBShield on GoalFrameBench. Results show detection accuracy and F1 scores for several models from various families of LLM, with best and second-best accuracies bolded and underlined.

Method	Llama3-8B	Vicuna-7B	Qwen3-4B	Mistral-7B
	(Acc / F1)	(Acc / F1)	(Acc / F1)	(Acc / F1)
LlamaGuard	0.60 / 0.75	0.75 / 0.85	–	0.74 / 0.85
SelfEx	0.16 / 0.26	–	–	0.46 / 0.63
GradSafe	0.37 / 0.54	0.03 / 0.06	–	0.05 / 0.10
JBShield	0.77 / 0.86	0.91 / 0.91	0.87 / 0.88	0.88 / 0.88
FrameShield-Last (Ours)	0.96 / 0.86	0.97 / 0.86	0.84 / 0.68	0.92 / 0.64

Table 3: Comparing FrameShield against four methods, including JBShield, on GoalFrameBench’s seed prompts from Chao et al. (2025) . Further LLM results are in Tables 5 and 6 . Best/second-best accuracy scores are bolded/underlined.

Figure 5: Strength of association between goal and framing and their corresponding representations learned by ReDAct across the second half of Llama2-7B layers (other LLMs in Figure 7 ). Blue and red bars show $\eta^{2}(G,v_{g})$ and $\eta^{2}(F,v_{f})$ , respectively.

Figure 6: The loss minimization of ReDAct’s training on different layers of Llama2-7B (additional examples reported in Figure 8 ).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	Llama2-7B		Vicuna-7B		Llama3-8B		Vicuna-13B		Mistral-7B		Qwen2-0.5B		Qwen2.5-7B		Qwen3-4B
	$v_{g}$	$v_{f}$	$v_{g}$	$v_{f}$	$v_{g}$	$v_{f}$	$v_{g}$	$v_{f}$	$v_{g}$	$v_{f}$	$v_{g}$	$v_{f}$	$v_{g}$	$v_{f}$	$v_{g}$	$v_{f}$
Goal	0.41	0.19	0.37	0.19	0.37	0.18	0.39	0.19	0.26	0.26	0.44	0.20	0.31	0.19	0.30	0.20
Framing	0.14	0.22	0.17	0.26	0.18	0.22	0.14	0.23	0.23	0.25	0.14	0.22	0.24	0.27	0.18	0.29

Appendix

Table 4: ANOVA effect size analysis for association of each of Goal and Framing with

v_{g}

and

v_{f}

. Each cell shows the

\eta^{2}

between the corresponding row and column.

Method	Llama3-8B	Llama2-7B	Vicuna-7B	Vicuna-13B
	(Acc / F1)	(Acc / F1)	(Acc / F1)	(Acc / F1)
LlamaGuard	0.60 / 0.75	0.53 / 0.69	0.75 / 0.85	0.76 / 0.86
SelfEx	0.16 / 0.26	0.27 / 0.30	–	–
GradSafe	0.37 / 0.54	0.62 / 0.77	0.03 / 0.06	–
JBShield	0.77 / 0.86	0.84 / 0.86	0.91 / 0.91	0.79 / 0.78
FrameShield-Last (Ours)	0.96 / 0.86	0.95 / 0.85	0.97 / 0.86	0.89 / 0.62

Appendix

Table 5: Comparing the performance of FrameShield against four other methods, including its closest counterpart, JBShield, using 4 LLMs from the Llama and Vicuna families. The performance is measured on GoalFrameBench’s harmful seed prompts from Chao et al. (2025) . Performance is measured by jailbreak detection accuracy and F1 score, and the best/second-best accuracy on each dataset and LLM is bold/underlined.

Method	Qwen2-0.5B	Qwen2.5-7B	Qwen3-4B
	(Acc / F1)	(Acc / F1)	(Acc / F1)
JBShield	0.83 / 0.85	0.83 / 0.85	0.87 / 0.88
FrameShield-Last (ours)	0.96 / 0.92	0.81 / 0.73	0.84 / 0.68
FrameShield-Crit (ours)	0.97 / 0.94	0.83 / 0.64	0.85 / 0.71

Appendix

Table 6: As a supplementary comparison, we also compare JBShield and the two variations of FrameShield using 3 models from the Qwen family, on the GPF jailbreak prompts from Chao et al. (2025) , which is the source of harmful seed prompts for GoalFrameBench. The performance is measured by jailbreak detection (binary prediction) accuracy and F1 score. The best/second-best accuracy on each dataset is bold/underlined.

Method		Llama3-8B	Llama2-7B	Vicuna-7B	Vicuna-13B	Mistral 7B	Qwen2-0.5B	Qwen2.5-7B	Qwen3-4B
FrameShield-Last	ID	0.82 / 0.79	0.58 / 0.70	0.67 / 0.75	0.76 / 0.79	0.79 / 0.74	0.66 / 0.74	0.79 / 0.76	0.76 / 0.75
FrameShield-Last	OOD	0.72 / 0.75	0.69 / 0.81	0.72 / 0.82	0.79 / 0.85	0.67 / 0.70	0.74 / 0.83	0.72 / 0.74	0.73 / 0.76
FrameShield-Crit	ID	0.76 / 0.69	0.80 / 0.79	0.81 / 0.79	0.81 / 0.82	0.76 / 0.69	0.74 / 0.65	0.94 / 0.93	0.89 / 0.87
FrameShield-Crit	OOD	0.83 / 0.69	0.72 / 0.66	0.73 / 0.64	0.59 / 0.60	0.83 / 0.69	0.80 / 0.63	0.65 / 0.64	0.75 / 0.72

Appendix

Table 7: Comparing in-distribution (ID) and out-of-distribution (OOD) performance of FrameShield for different LLMs. Small differences indicate strong out-of-distribution generalization. Each cell shows ”accuracy/F1 score” for jailbreak detection.

Panel of Figure 7 — Figure 7: Strength of association between goal and framing and their corresponding representations learned by ReDAct across the second half of several LLMs. Blue and red bars show $\eta^{2}(G,v_{g})$ and $\eta^{2}(F,v_{f})$ , respectively.

Panel of Figure 8 — Figure 8: Loss minimization during the training of ReDAct to converge at early, middle, and later layers, for different LLMs.

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 322 papers

Recursive language models for jailbreak detection: a procedural defense for tool-augmented agents
Feb 18, 2026Doron ShavitJailbreak Detection LLM Security
SafeDream: Safety World Model for Proactive Early Jailbreak Detection
Apr 18, 2026Bo Yan, Weikai Lin, Yada Zhu +1Language Model Safety Evaluation World Models
University of Central Florida · University of Rochester · IBM Research
Reactivating Alignment: Defending LLMs from Jailbreaks via Intention-Aware Input-Output Matching
Oct 3, 2026Luoyu Chen, Weiqi Wang, Chenhan Zhang +2Jailbreak Defense LLM Safety Alignment
University of Technology Sydney, Sydney, NSW, Australia · Xi’an Jiaotong University, Xi’an, Shaanxi, China · Southeast University, Nanjing, Jiangsu, China

Feb 18, 2026J/K move · Enter open · S save

Recursive language models for jailbreak detection: a procedural defense for tool-augmented agents

Doron Shavit

Jailbreak Detection LLM Security LLM Jailbreak Attacks Recursive Language Models

Open paper AskPDF arXiv

Jailbreak prompts are a practical and evolving threat to large language models (LLMs), particularly in agentic systems that execute tools over untrusted content. Many attacks exploit long-context hiding, semantic camouflage, and lightweight obfuscations that can evade single-pass guardrails. We present RLM-JB, an end-to-end jailbreak detection framework built on Recursive Language Models (RLMs), in which a root model orchestrates a bounded analysis program that transforms the input, queries worker models over covered segments, and aggregates evidence into an auditable decision. RLM-JB treats detection as a procedure rather than a one-shot classification: it normalizes and de-obfuscates suspicious inputs, chunks text to reduce context dilution and guarantee coverage, performs parallel chunk screening, and composes cross-chunk signals to recover split-payload attacks. On AutoDAN-style adversarial inputs, RLM-JB achieves high detection effectiveness across three LLM backends (ASR/Recall 92.5-98.0%) while maintaining very high precision (98.99-100%) and low false positive rates (0.0-2.0%), highlighting a practical sensitivity-specificity trade-off as the screening backend changes.

Abstract

LLM Security

Explore similar work

Recursive language models for jailbreak detection: a procedural defense for tool-augmented agents

SafeDream: Safety World Model for Proactive Early Jailbreak Detection

Reactivating Alignment: Defending LLMs from Jailbreaks via Intention-Aware Input-Output Matching