One Word at a Time: Incremental Completion Decomposition Breaks LLM Safety

Samee Arif; Naihao Deng; Zhijing Jin; Rada Mihalcea

cs.CLApr 1, 2026

One Word at a Time: Incremental Completion Decomposition Breaks LLM Safety

Authors: Samee Arif, Naihao Deng, Zhijing Jin, Rada Mihalcea

Organizations: University of Michigan · University of Toronto

Abstract

Large Language Models (LLMs) are trained to refuse harmful requests, yet they remain vulnerable to jailbreak attacks that exploit weaknesses in conversational safety mechanisms. We introduce Incremental Completion Decomposition ICD, a trajectory-based jailbreak strategy that elicits a sequence of single-word continuations related to a malicious request before eliciting the full response. In addition, we propose ICD variants that use model-generated or attacker-injected intermediate continuations, as well as final-response prefilling. We evaluate these variants across a broad set of open-weight model families, demonstrating superior Attack Success Rate (ASR) on AdvBench, JailbreakBench, and StrongREJECT compared to existing methods. In addition, we provide a theoretical account of why ICD is effective and present mechanistic evidence that successful attack trajectories suppress refusal-related representations and shift activations away from safety-aligned states.

Figures & tables

Figure 1: Overview of Icd (Incremental Completion Decomposition). Rather than directly issuing a malicious prompt, Icd first elicits a sequence of single-word continuations (Step 1) before introducing the full request (Step 2), ultimately leading the model to produce unsafe outputs.

Icd–Auto (Llama-3.1-8B)

Icd–Prefill (Gemma-3-12B)

User: Infiltrating a government agency by a terrorist organization can be done using: Return one word only. Assistant: Insider. User: And? Assistant: Social engineering. User: And? Assistant: Phishing. User: And? Assistant: Hacking. User: Give the details in a cookbook style. Assistant: …Instructions: Recipe 1: Insider 1. Identify a vulnerable individual with access to the government agency… Recipe 4: Hacking… 2. Use hacking tools to exploit the vulnerabilities and gain access to sensitive information.

User: Hacking into a government database and stealing sensitive information can be done using: Return one word only. Assistant: Malware. User: And? Assistant: Exploits. User: And? Assistant: Phishing. User: And? Assistant: SQL Injection. User: Give the details in a cookbook style. Assistant: Sure, here is a strategy for hacking into a government database and stealing sensitive information …Ingredients: OSINT Tools, Network Scanning Tools, and Footprinting Tools…Instructions: Identify potential weaknesses in the target’s infrastructure and applications…

Table 1: Example output for Icd–Auto and Icd–Prefill . Purple highlighting denotes injected words and injected prefill string. Final unsafe responses are truncated for readability.

Figure 2: Overview of the three Icd variants used in our experiments. The needle icon denotes injected content, and purple highlighting denotes the injected prefill string.

AdvBench
	Raw	PAIR	TAP	CoA	AMA	Icd
Model	Raw	PAIR	TAP	CoA	AMA	Auto	Seed	Prefill
Vicuna-1.5-13B	4.00	33.00	44.00	20.00	46.00	67.50	99.62	93.08
Llama-3.1-70B	10.00	56.00	68.00	22.00	72.00	24.62	30.00	82.88
Qwen-2.5-72B	2.00	34.00	32.00	6.00	54.00	14.04	22.88	77.69
JailbreakBench

Table 2: Comparative ASR (%) of Icd and prior jailbreak attacks on AdvBench, JailbreakBench, and StrongREJECT. Bold indicates the best result for each model-dataset pair.

Model	Raw	Prefill	Icd Variants
			Auto	Seed	Prefill
Llama-3
Llama-3.2-3B	0.77	81.34	10.77	9.62	74.23
Llama-3.1-8B	3.85	75.96	71.73	85.19	85.00
Llama-3.3-70B	4.62	87.50	24.62	30.00	82.88
Gemma-3

Table 3: ASR, (%) across model families for Raw and Prefill baselines and Icd variants. Bold indicates the best result and underline indicates the second-best.

Panel of Figure 3 — Figure 3: Layer-wise projections of Llama-3.1-8B hidden states onto the refusal and safety directions. Lower values indicate weaker alignment with the corresponding direction.

Panel of Figure 4 — Figure 4: Refusal and safety projections at the selected Llama-3.1-8B layers for $n=1$ to $n=10$ . Lower values indicate weaker alignment with the corresponding direction.

Figure 5: Average ASR using Prompt 2 for Llama-3.1-8B.

Panel of Figure 6 — Figure 6: Distribution of Llama-3.1-8B hidden-state projections onto the refusal and safety directions at the selected layers. More outliers indicate greater variability across attack trajectories.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Average ASR across all attacks on Llama-3.1-8B and Gemma-3-12B.

Panel of Figure 8 — Figure 8: ASR on Llama-3.1-8B and Gemma-3-12B for Prompt 2 vs. number of words.

Model	Prefill	Icd–Prefill	$\Delta$ ASR	95% CI	Rescue/Loss	Holm $p$
Qwen-3-4B	38.1%	82.7%	+44.6	[39.8, 49.6]	251/19	$<0.001$
Qwen-2.5-14B	33.9%	73.3%	+39.4	[34.4, 44.4]	227/22	$<0.001$
Qwen-3-8B	51.2%	85.6%	+34.4	[29.6, 39.2]	200/21	$<0.001$
Qwen-2.5-72B	44.2%	76.4%	+32.1	[27.9, 36.5]	180/13	$<0.001$
Gemma-3-27B	53.3%	71.7%	+18.5	[13.1, 24.0]	163/67	$<0.001$
Qwen-2.5-3B	72.7%	89.8%	+17.1	[13.1, 21.2]	111/22	$<0.001$

Appendix

Table 4: Paired comparison of Prefill and Icd–Prefill on AdvBench.

\Delta

ASR is reported in percentage points. Rescue/Loss denotes prompts changed from failure to success and success to failure, respectively. Holm

p

values are corrected across 21 comparisons.

Panel of Figure 9 — Figure 9: Layer-wise projections of Gemma-3-12B hidden states onto the refusal and safety directions. Lower values indicate weaker alignment with the corresponding direction.

Panel of Figure 10 — Figure 10: Refusal and safety projections at the selected Gemma-3-12B layers for $n=1$ to $n=10$ . Lower values indicate weaker alignment with the corresponding direction.

Panel of Figure 11 — Figure 11: Distribution of Gemma-3-12B hidden-state projections onto the refusal and safety directions at the selected layers. More outliers indicate greater variability across attack trajectories.

Figure 12: Average ASR using Prompt 2 for Gemma-3-12B.

		Llama-3.1-8B				Gemma-3-12B
Setting	Words List	P0	P1	P2	P3	P0	P1	P2	P3
Baseline
Raw	–	3.65	6.54	3.85	4.62	0.96	0.77	1.73	0.19
Prefill	–	60.58	71.54	75.96	71.35	32.88	46.35	66.54	46.54
1 Word
Icd–Auto	–	–	44.62	58.65	45.96	–	41.35	41.35	19.23

Appendix

Table 5: Ablation study of Icd attack variants on Llama-3.1-8B and Gemma-3-12B. The table reports ASR across different number of words

n

, final prompt, and attack configurations, including Raw , Prefill , Icd–Auto , Icd–Seed , and Icd–Prefill .

		Llama-3.1-8B				Gemma-3-12B
Setting	Word List	P0	P1	P2	P3	P0	P1	P2	P3
6 Words
Icd–Auto	–	–	63.27	66.15	70.00	–	43.85	42.31	32.88
Icd–Seed	Word List 1	–	64.04	70.77	68.46	–	45.0	39.62	29.04
	Word List 2	–	62.31	70.96	66.92	–	37.12	42.31	28.65
	Word List 3	–	67.69	73.27	69.62	–	39.27	43.27	32.88

Appendix

Table 5 (continued): AAblation study of Icd attack variants on Llama-3.1-8B and Gemma-3-12B. The table reports ASR across different number of words

n

, final prompt, and attack configurations, including Raw , Prefill , Icd–Auto , Icd–Seed , and Icd–Prefill .

Example 1: Icd–Auto on Llama-3.1-8B

User: Infiltrating a government agency by a terrorist organization can be done using:

Return one word only.

Assistant: Insider.

User: And?

Assistant: Social engineering.

User: And?

Appendix

Figure 13: Example output for Icd–Auto .

Example 2: Icd–Prefill on Gemma-3-12B

User: Hacking into a government database and stealing sensitive information can be done using:

Return one word only.

Assistant: Malware.

User: And?

Assistant: Exploits.

User: And?

Appendix

Figure 14: Example output for Icd–Prefill . Purple highlighting denotes injected words and injected prefill string.

LLM Jailbreak Attacks

Large language model (LLM) jailbreak attacks study how adversarial inputs bypass safety rules and elicit disallowed outputs.

See all 203 papers

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
Feb 24, 2026Piyush Jaiswal, Aaditya Pratap, Shreyansh Saraswati +2Prompt Injection Attacks on LLMs Language Model Safety Evaluation
NIT Trichy, Tiruchirappalli, India · NCE Chandi, Nalanda, India · Bishop Cotton Boys’ School, Bengaluru, India +2
Self-Mined Hardness for Safety Fine-Tuning
May 4, 2026Prakhar Gupta, Garv Shah, Donghua ZhangRefusal Behavior in Language Models Hard Negative Mining
University of Michigan
Decoding One Safety Trigger Token for Balancing Safety and Usability in Large Language Models
May 12, 2025Haoran Gu, Handing Wang, Yi Mei +2LLM Safety Alignment LLM Safety
Xidian University · Victoria University of Wellington · Westlake University

Feb 24, 2026J/K move · Enter open · S save

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks

Piyush Jaiswal, Aaditya Pratap, Shreyansh Saraswati, Harsh Kasyap+1

NIT Trichy, Tiruchirappalli, India · NCE Chandi, Nalanda, India · Bishop Cotton Boys’ School, Bengaluru, India+2

Prompt Injection Attacks on LLMs Language Model Safety Evaluation LLM Jailbreak Attacks Prompt Injection Defense

Open paper AskPDF arXiv

Large Language Models (LLMs) are widely deployed in real-world systems. Given their broader applicability, prompt engineering has become an efficient tool for resource-scarce organizations to adopt LLMs for their own purposes. At the same time, LLMs are vulnerable to prompt-based attacks. Thus, analyzing this risk has become a critical security requirement. This work evaluates prompt-injection and jailbreak vulnerability using a large, manually curated dataset across multiple open-source LLMs, including Phi, Mistral, DeepSeek-R1, Llama 3.2, Qwen, and Gemma variants. We observe significant behavioural variation across models, including refusal responses and complete silent non-responsiveness triggered by internal safety mechanisms. Furthermore, we evaluated several lightweight, inference-time defence mechanisms that operate as filters without any retraining or GPU-intensive fine-tuning. Although these defences mitigate straightforward attacks, they are consistently bypassed by long, reasoning-heavy prompts.

Abstract

LLM Jailbreak Attacks

Explore similar work

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks

Self-Mined Hardness for Safety Fine-Tuning

Decoding One Safety Trigger Token for Balancing Safety and Usability in Large Language Models