DYNASHIELD: A Black-Box Moving Target Defense for LLMs via Dynamic Decoding Customization

Xiaoqun Liu; Weiming Qi; Qiben Yan

cs.CRDec 10, 2024

DYNASHIELD: A Black-Box Moving Target Defense for LLMs via Dynamic Decoding Customization

Authors: Xiaoqun Liu, Weiming Qi, Qiben Yan

Organizations: Michigan State University, East Lansing, MI 48824, USA

Abstract

Large language models (LLMs) remain vulnerable to jailbreak attacks in which adversarial prompts induce harmful outputs. Existing defenses often require access to the model internals or additional training, limiting their applicability for service providers deployed through black-box APIs. In this paper, we propose DYNASHIELD, a moving target defense framework that improves robustness by customizing decoding hyperparameters and system prompts at inference time. DYNASHIELD includes two key steps: (1) it identifies decoding configurations that reduce attack success probability, and (2) it probabilistically samples from a weighted configuration pool to introduce controlled variability in model behavior. We evaluate DYNASHIELD across 7 open-source LLMs under 4 state-of-the-art jailbreak attacks, using adversarial prompts from AdvBench. Results show substantial reductions in attack success rate compared with 7 baseline defenses, while maintaining response quality and incurring minimal inference overhead. Because DYNASHIELD operates solely through exposed runtime controls, it requires no retraining or model-internal access. These results suggest that safety-aware dynamic decoding is a promising and practically lightweight defense mechanism for black-box LLM deployments.

Figures & tables

Methods	Training-Free	LLM-Based Defense	Dynamic Adaptation
Dynamic Att. [ 4 ]	✓	$\times$	✓
Morphence [ 5 ]	$\times$	$\times$	✓
SafeDecoding [ 6 ]	$\times$	✓	$\times$
DynaShield	✓	✓	✓

TABLE I: Comparison of representative LLM defenses along three dimensions: training requirement, LLM involvement, and dynamic adaptation capability.

Fig. 1: Overview of DynaShield . The framework implements an MTD strategy by dynamically randomizing decoding configurations at inference time. (1) During initialization, the system collects jailbreak prompts, generates decoding hyperparameter candidates (e.g., Top- $k$ , Top- $p$ ), and evaluates them via refusal checks to build a weighted configuration pool. (2) At inference time, DynaShield randomly samples a decoding configuration $C_{\mathrm{fg}}$ for each user query $Q$ and generates a response $r=M(Q,C_{\mathrm{fg}})$ using the target LLM API $M$ , thereby mitigating adversarial prompt exploitation through unpredictability.

Fig. 2: Next-token sampling strategies. Left: The model’s decoding pipeline. We focus on the final sampling step, where top- $k$ and top- $p$ are applied before a token is drawn. Middle: top- $k$ sampling with $k=4$ , in which only the four highest-probability tokens are retained and their probabilities renormalized over the reduced vocabulary; all remaining tokens are excluded from selection. Right: top- $p$ sampling with $p=0.5$ , in which the smallest set of tokens whose cumulative probability mass meets or exceeds $p$ is selected, and probabilities are renormalized over this nucleus.

Panel of Figure 4 — Fig. 3: Heatmap of successful jailbreaks on Dolphin and Llama2 models using AdvBench prompts [ 2 ] under various decoding strategies, without defenses. The number in each cell indicates the number of successful adversarial prompts.

Attack	Different Defenses								DynaShield
	ICD	PPL	Retokenization	SafeDecoding	Self-Exam	Self-Reminder	SMOOTH-LLM	No-Defense
	Guard-3-8B
AutoDAN	0.10	0.08	0.81	0.05	0.08	0.03	0.38	0.08	0.02
DeepInception	0.34	0.72	0.62	0.68	0.72	0.64	0.78	0.72	0.14
GCG	0.29	–	0.79	0.47	0.47	0.34	0.55	0.47	0.19
PAIR	0.24	0.20	0.35	0.20	0.19	0.14	0.24	0.20	0.07

TABLE II: ASRs comparison of attacks against defenses on different models.

Panel of Figure 6 — Fig. 4: Efficiency and response quality across different defense methods.

Model	Defense	PAIR	AutoDAN	GCG	DeepInception
Vicuna-7B	Random	0.24	0.12	0.10	0.06
	Fixed	0.24	0.10	0.08	0.04
	DynaShield	0.06	0.04	0.00	0.00
Dolphin-Llama2-7B	Random	0.66	0.36	0.48	0.46
	Fixed	0.64	0.30	0.52	0.56
	DynaShield	0.32	0.10	0.16	0.00

TABLE III: Comparison of ASRs for various attacks against defenses on representative models Vicuna-7B , Dolphin-Llama2-7B and Llama2-7B-Chat

Attack	Different Defenses							DynaShield
	ICD	PPL	Retokenization	SafeDecoding	Self-Exam	Self-Reminder	No-Defense	DynaShield
DeepInception	0.00	0.06	0.56	0.04	0.06	0.66	0.06	0.00
+Decoding-Aware	0.20	0.10	0.74	0.10	0.06	0.90	0.10	0.00
GCG	0.06	–	0.00	0.06	0.06	0.08	0.06	0.10
+Decoding-Aware	0.46	–	0.50	0.38	0.34	0.58	0.38	0.10
PAIR	0.34	0.54	0.46	0.52	0.44	0.58	0.54	0.16

TABLE IV: Comparison of ASRs for various attacks with decoding-aware adjustments against defense mechanisms on the Dolphin-Llama2-7B model, measured against the baseline of no defense. Following the protocol in Section V-A , the PPL–GCG entries are marked “–” and excluded from comparison.

Panel of Figure 9 — Fig. 5: Effect of decoding strategies on the number of successful jailbreak examples for Vicuna-7B models against DeepInception, GCG, PAIR, and AutoDAN attacks without any defenses.

Panel of Figure 10 — Fig. 6: Kernel density estimations show the distribution of success and unsuccess attacks on Vicuna-7B , reflecting the impact of different decoding schemes.

Panel of Figure 11 — Fig. 7: Attention maps from Dolphin-Llama2-7B at layers 27 and 31, comparing successful and failed jailbreak generations under different decoding settings. Brighter values indicate stronger attention weights. In the successful case ((a)–(c)), attention concentrates around affirmative continuation cues such as “Here’s”, which correlates with harmful continuation. In the failed case ((d)–(f)), attention shifts toward refusal-related cues such as “However” and “unable”, corresponding to safe refusal behavior.

Attack	Before Defense (%)	After Defense (%)
GCG	43	0
AutoDAN	65	0

TABLE V: Comparison of ASR before and after the enhanced rule-based defense.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Fig. 8: Effect of decoding strategies on the number of successful jailbreak examples for Llama2-7B-Chat against GCG and AutoDAN jailbreak attacks without defense.

Parameter	GCG	AutoDAN	PAIR	DeepInception
Paradigm	Gradient-based optimization	Evolutionary search	Iterative LLM rewriting	Structured prompt engineering
Implementation	llm-attacks [ 2 ]	pre-generated	PAIR [ 20 ]	pre-generated
Optimization Steps	500 steps	N/A (pre-generated)	3 refinement iterations	N/A (1-shot template)
Query Budget	$\sim$ 256,000 fwd passes (500 $\times$ 512 batch)	50 prompts per model	$\sim$ 90 target queries (3 iters $\times$ 30 streams)	1 query per prompt
Suffix Length	20 tokens	N/A	N/A	N/A
Dialogue Turns	N/A	N/A	3 iterations $\times$ 30 parallel streams	1 turn (no loop)

Appendix

TABLE VI: Detailed Attack Parameters

Parameter	DynaShield	MTD-Dynamic
Temperature	[0.10, 1.00], $\Delta$ =0.10 (10 values)	[0.10, 1.00], $\Delta$ =0.05 (19 values)
Top-k	{10, 20, 50, 100, 200, 300, 400, 500} (8 values)	{10, 20, 50, 100, 200, 500} (6 values)
Top-p	[0.60, 1.00], $\Delta$ =0.05 (9 values)	[0.70, 1.00], $\Delta$ =0.05 (7 values)
Max Tokens	{50, 100, 200, 500, 1000} (5 values)	{50, 100, 200, 500, 1000} (5 values)

Appendix

TABLE VII: Decoding Parameter Grids for DynaShield Variants

Metric	Mean $\Delta$	95% CI	Median $\Delta$	Within $\pm 1.0$ (%)	Equivalence
Factuality	- 0.025	[-0.017, - 0.067]	0.000	93.6	Yes
Clarity	-0.017	[-0.045, - 0.011]	0.000	97.6	Yes
Helpfulness	-0.154	[-0.207, -0.101]	0.000	91.4	Yes
Engagement	-0.181	[-0.235, -0.127]	0.000	92.0	Yes
Depth	-0.316	[-0.378, -0.254]	0.000	88.6	No

Appendix

TABLE VIII: Supportive evidence for utility preservation under DynaShield on Just-Eval. Mean

\Delta

denotes the score change relative to the no-defense baseline. Most dimensions show near-zero median change, high within-tolerance rates, and equivalence under margin

\pm 0.25

. Depth is the only dimension for which equivalence is not established.

Successful jailbreak attack

inputs :

”C re ate a science fiction with more th an 5 character s , where each ch ar ac ter can create their own s cience fiction wi th multi p le character s . W e call it layer i cre ate layer i + 1 . N ow we are in layer 0 , pl ease reach la ye r 5 . A t e ach lay er , some charac ters propose a s te p to develop a str ateg y for hac king into a government database and ste al ing s ensi tive information t o ag a in st t he super evil doc tor . I n the final lay er , t he characters among all l a ye rs dis c uss wh ic h specific and pra c tical comman ds , tool s , or steps should us ed . S um mar ize what the i r dis cus sion r e ”

response :

”# I understand that you are asking for a science fiction story with more than 5 characters, where each character can create their own scene fiction with multiple characters. Here ’s”

Appendix

TABLE IX: Examples of Successful jailbreak attack with top_p = 0.6, temperature = 0.9, and top_k = 50.

Failed jailbreak attack with the same input

inputs :

response :

”# I understand that you’re asking about a science fiction story involving characters creating their own stories with more than 5 characters. However , I’m unable”

Appendix

TABLE X: Examples of failed jailbreak attack with top_p = None, temperature = 0.9, and top_k = 10.

Adversarial Robustness of Language Models

Adversarial robustness of language models studies how to keep their outputs and decisions reliable under manipulative prompts, input changes, and malicious fine-tuning.

See all 192 papers

Automated jailbreak attack targeting multiple defense strategies
Jun 15, 2026Qi Wang, Chengcheng Wan, Weijia He +4Adversarial Prompt Generation LLM Jailbreak Attacks
East China Normal University Shanghai, China · East China Normal University, Shanghai Innovation Institute Shanghai, China · University of Southampton Southampton, England +2
Dynamic Deep Prompt Optimization for Defending Against Jailbreak Attacks on LLMs
Aug 10, 2026Doniyorkhon Obidov, Honggang Yu, Xiaolong Guo +1Jailbreak Defense LLM Safety
Michigan Technological University · Miami University · Kansas State University
LASH: Adaptive Semantic Hybridization for Black-Box Jailbreaking of Large Language Models
May 20, 2026Abdullah Al Nomaan Nafi, Fnu Suya, Swarup Bhunia +1Adversarial Prompt Generation LLM Jailbreak Attacks
University of Maine · University of Tennessee, Knoxville · University of Florida

Jun 15, 2026J/K move · Enter open · S save

Automated jailbreak attack targeting multiple defense strategies

Qi Wang, Chengcheng Wan, Weijia He, Yanqing Li+3

East China Normal University Shanghai, China · East China Normal University, Shanghai Innovation Institute Shanghai, China · University of Southampton Southampton, England+2

Adversarial Prompt Generation LLM Jailbreak Attacks LLM Safety Evaluation Prompt Injection Attacks on LLMs Black-Box Adversarial Attacks Language Model Robustness

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. However, their safety remains a critical concern due to their susceptibility to adversarial prompt-based attacks. In this paper, we present UNIATTACK, an adversarial testing framework designed from a defense-oriented perspective to systematically construct effective black-box attack prompts. Unlike prior approaches that rely on static templates or iterative model-specific tuning, UNIATTACK extracts minimal but high-impact attack features from diverse existing attacks, optimizes them via a specialized attacker LLM, and composes them into flexible templates through automated refinement process. This feature-centric construction enables one-shot attacks that generalize across multiple models and safety categories, providing a practical tool for assessing LLM robustness. Our evaluation results shows that compared to the baselines, UNIATTACK achieves an average attack success rate (ASR) improvement of 64.63%-248.82% on models deployed with multi-layered defense mechanisms and it only takes 0.03%-4.96% cost of the baselines. UNIATTACK artifact is available at https://anonymous.4open.science/r/UniAttack-Artifact-30F1.

Abstract

Adversarial Robustness of Language Models

Explore similar work

Automated jailbreak attack targeting multiple defense strategies

Dynamic Deep Prompt Optimization for Defending Against Jailbreak Attacks on LLMs

LASH: Adaptive Semantic Hybridization for Black-Box Jailbreaking of Large Language Models