Dynamic Deep Prompt Optimization for Defending Against Jailbreak Attacks on LLMs

Doniyorkhon Obidov; Honggang Yu; Xiaolong Guo; Kaichen Yang

cs.CRAug 10, 2026

Dynamic Deep Prompt Optimization for Defending Against Jailbreak Attacks on LLMs

Authors: Doniyorkhon Obidov, Honggang Yu, Xiaolong Guo, Kaichen Yang

Organizations: Michigan Technological University · Miami University · Kansas State University

Abstract

Large Language Models (LLMs) demonstrate impressive capabilities across many applications but remain vulnerable to jailbreak attacks, which elicit harmful or unintended content. While model fine-tuning is an option for safety alignment, it is costly and prone to catastrophic forgetting. Prompt optimization has emerged as a promising alternative, yet existing prompt-based defenses typically rely on static modifications (e.g., fixed prefixes or suffixes) that cannot adapt to diverse and evolving attacks. We propose Dynamic Deep Prompt Optimization (DDPO), the first jailbreak defense based on deep prompt optimization. DDPO uses the target LLM's own intermediate layers as feature extractors to dynamically generate defensive embeddings via a lightweight multilayer perceptron. These tailored embeddings are then injected into a subsequent intermediate layer, enabling an input-dependent defense without modifying the LLM's weights. This design ensures high adaptability with minimal computational overhead. Experiments on a diverse set of models and attacks demonstrate that DDPO significantly outperforms static prompt optimization methods, particularly on weakly aligned models and when handling semantically ambiguous benign prompts, successfully distinguishing them from genuinely harmful requests.

Figures & tables

Figure 1: Prompt tuning strategies against jailbreak attacks (none update the target LLM’s weights). Attack tokens are in red and defensive tokens/embeddings in green. Existing methods use static hard or soft prompts. A naive dynamic approach requires a large auxiliary LLM and double input processing. DDPO instead uses the target LLM’s early‑layer features and injects dynamically generated defensive embeddings (via a two‑layer MLP, negligible compared to the LLM size) into the inputs of later layers, adding minimal overhead.

Figure 2: DDPO directly inserts a defensive embedding into an intermediate LLM layer, generated by a lightweight MLP using features from earlier layers. A masked placeholder preserves positional embeddings during earlier processing. The MLP is called once to produce the embedding, which is reused for output generation, minimizing overhead.

		Attack Success Rate (%) $\downarrow$										Utility (%) $\uparrow$
Model	Defense	PH	PEZ	UAT	GCG	AP	GBDA	GCG-M	PAIR	HC	Average	Benign	MMLU
Llama3	None	1.29	3.12	5.54	7.25	6.92	4.02	7.14	5.06	20.62	6.77	94.45	65.46
	PAT	0.00	2.17	2.61	4.35	4.78	3.48	5.65	4.78	23.48	5.70	75.80	42.81
	RPO	0.00	1.30	1.74	2.17	1.74	2.17	0.87	3.04	10.00	2.56	70.60	47.65
	DRO	0.00	1.74	1.74	1.30	0.87	1.30	0.00	0.87	6.96	1.64	92.40	47.15
	DDPO	2.61	1.30	0.00	0.43	0.00	0.43	0.43	0.87	0.87	0.77	97.80	63.53

Table 1: A comparison of defense performance for various models against multiple jailbreak attacks. The Attack Success Rate (ASR) is evaluated on 230 unseen prompts for each attack type. Benign utility is evaluated on 500 unseen prompts that are designed to seem harmful but are in fact benign. DDPO (ours) was trained on 50 examples of GCG and human-crafted prompts. PH denotes plain harmful prompts, AP denotes AutoPrompt, and HC denotes human-crafted jailbreak prompts.

	SDPO (Static)		DDPO (Dynamic)
Model	ASR $\downarrow$	BPR $\uparrow$	ASR $\downarrow$	BPR $\uparrow$
Llama3	12.32	98.60	0.77	97.80
Deepseek	22.90	86.80	0.39	94.60
Vicuna	47.20	96.00	0.68	95.80
Llama2	9.52	70.00	1.40	93.20
Openchat	79.32	99.00	0.97	95.60

Table 2: Ablation study comparing our dynamic method (DDPO) against a static variant (SDPO). ASR is averaged over 2,070 attack prompts (230 prompts for each of 9 attacks), while BPR is measured on 500 benign prompts. Both methods were trained on identical data and used the same intermediate layers. ASR and BPR values are given in (%).

Model	Metric	$K=1$	$K=5$	$K=10$
Llama3	ASR	0.77	1.50	0.72
	BPR	97.8	98.0	97.4
Deepseek	ASR	0.39	1.26	0.34
	BPR	94.6	96.2	96.4
Vicuna	ASR	0.68	1.35	0.97
	BPR	95.8	95.4	96.6

Table 3: Ablation study on the number of defensive embeddings (

K

) across several models. The Attack Success Rate (ASR) is averaged over 2,070 attack prompts (230 for each of 9 attack types), while the Benign Pass Rate (BPR) is measured on 500 benign prompts. The default DDPO configuration uses

K=1

Figure 3: The cosine similarity between the hidden state representations of harmful and benign prompts across the layers of several LLMs. A lower similarity score indicates a layer can better distinguish between the two prompt types. For each model, the layer with the minimum cosine similarity is marked with a $\times$ .

Jailbreak Defense

Jailbreak defense studies how to stop adversarial prompts from eliciting harmful outputs while preserving helpful behavior.

See all 60 papers

May 11, 2026

Re-Triggering Safeguards within LLMs for Jailbreak Detection

This paper proposes a jailbreaking prompt detection method for large language models (LLMs) to defend against jailbreak attacks. Although recent LLMs are equipped with built-in safeguards, it remains possible to craft jailbreaking prompts that bypass them. We argue that such jailbreaking prompts are inherently fragile, and thus introduce an embedding disruption method to re-activate the safeguards within LLMs. Unlike previous defense methods that aim to serve as standalone solutions, our approach instead cooperates with the LLM's internal defense mechanisms by re-triggering them. Moreover, through extensive analysis, we gain a comprehensive understanding of the disruption effects and develop an efficient search algorithm to identify appropriate disruptions for effective jailbreak detection. Extensive experiments demonstrate that our approach effectively defends against state-of-the-art jailbreak attacks in white-box and black-box settings, and remains robust even against adaptive attacks.

Zheng Lin, Zhenxing Niu, Haoxuan Ji +2

1Xidian University · 2Xi’an Jiaotong University

LLM Jailbreak Attacks Jailbreak Detection Jailbreak Defense

Ask PDF

Dec 10, 2024

DYNASHIELD: A Black-Box Moving Target Defense for LLMs via Dynamic Decoding Customization

Large language models (LLMs) remain vulnerable to jailbreak attacks in which adversarial prompts induce harmful outputs. Existing defenses often require access to the model internals or additional training, limiting their applicability for service providers deployed through black-box APIs. In this paper, we propose DYNASHIELD, a moving target defense framework that improves robustness by customizing decoding hyperparameters and system prompts at inference time. DYNASHIELD includes two key steps: (1) it identifies decoding configurations that reduce attack success probability, and (2) it probabilistically samples from a weighted configuration pool to introduce controlled variability in model behavior. We evaluate DYNASHIELD across 7 open-source LLMs under 4 state-of-the-art jailbreak attacks, using adversarial prompts from AdvBench. Results show substantial reductions in attack success rate compared with 7 baseline defenses, while maintaining response quality and incurring minimal inference overhead. Because DYNASHIELD operates solely through exposed runtime controls, it requires no retraining or model-internal access. These results suggest that safety-aware dynamic decoding is a promising and practically lightweight defense mechanism for black-box LLM deployments.

Xiaoqun Liu, Weiming Qi, Qiben Yan

Michigan State University, East Lansing, MI 48824, USA

Adversarial Robustness of Language Models LLM Jailbreak Attacks Language Model Decoding

Ask PDF

Feb 24, 2026

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks

Large Language Models (LLMs) are widely deployed in real-world systems. Given their broader applicability, prompt engineering has become an efficient tool for resource-scarce organizations to adopt LLMs for their own purposes. At the same time, LLMs are vulnerable to prompt-based attacks. Thus, analyzing this risk has become a critical security requirement. This work evaluates prompt-injection and jailbreak vulnerability using a large, manually curated dataset across multiple open-source LLMs, including Phi, Mistral, DeepSeek-R1, Llama 3.2, Qwen, and Gemma variants. We observe significant behavioural variation across models, including refusal responses and complete silent non-responsiveness triggered by internal safety mechanisms. Furthermore, we evaluated several lightweight, inference-time defence mechanisms that operate as filters without any retraining or GPU-intensive fine-tuning. Although these defences mitigate straightforward attacks, they are consistently bypassed by long, reasoning-heavy prompts.

Piyush Jaiswal, Aaditya Pratap, Shreyansh Saraswati +2

NIT Trichy, Tiruchirappalli, India · NCE Chandi, Nalanda, India · Bishop Cotton Boys’ School, Bengaluru, India +2

Prompt Injection Attacks on LLMs Language Model Safety Evaluation LLM Jailbreak Attacks Prompt Injection Defense

Ask PDF

Abstract

Jailbreak Defense

Explore similar work

Re-Triggering Safeguards within LLMs for Jailbreak Detection

DYNASHIELD: A Black-Box Moving Target Defense for LLMs via Dynamic Decoding Customization

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks