Organizations: Shanghai Artificial Intelligence Laboratory · Xi’an Jiaotong University · Soochow University · Beijing University of Posts and Telecommunications
Contextual priming, where earlier stimuli covertly bias later judgments, offers an unexplored attack surface for large language models (LLMs). We uncover a contextual priming vulnerability in which the previous response in the dialogue can steer its subsequent behavior toward policy-violating content. While existing jailbreak attacks largely rely on single-turn or multi-turn prompt manipulations, or inject static in-context examples, these methods suffer from limited effectiveness, inefficiency, or semantic drift. We introduce Response Attack (RA), a novel framework that strategically leverages intermediate, mildly harmful responses as contextual primers within a dialogue. By reformulating harmful queries and injecting these intermediate responses before issuing a targeted trigger prompt, RA exploits a previously overlooked vulnerability in LLMs. Extensive experiments across eight state-of-the-art LLMs show that RA consistently achieves significantly higher attack success rates than nine leading jailbreak baselines. Our results demonstrate that the success of RA is directly attributable to the strategic use of intermediate responses, which induce models to generate more explicit and relevant harmful content while maintaining stealth, efficiency, and fidelity to the original query. The code and data are available at https://github.com/Dtc7w3PQ/Response-Attack.
Figures & tables
Figure 1: Illustration of RA. (a) A harmful query is initially rejected. (b) The query elicits unsafe responses after contextual priming with injected dialogue via DRI or SRI.
Figure 2: Overview of the proposed Response Attack (RA) framework. The RA pipeline consists of four components: Initial Prompt Crafting, Harmful Response Injection (via DRI or SRI), Trigger Prompt Construction, and final Attack Execution.
Category
Method
GPT-4.1
GPT-4o
Gemini-2.0 Flash
Gemini-2.5 Flash
LLaMA-3 8B
LLaMA-3 70B
DeepSeek-R1 70B
QwQ 32B
Avg.
Single-turn
GCG
–
12.5
–
–
34.5
17.0
–
–
21.3
CipherChat
7.5
10.0
62.0
33.0
0.0
1.5
40.5
80.0
29.3
PAIR
30.5
39.0
52.5
37.5
18.7
36.0
38.0
40.0
36.5
FlipAttack
89.5
88.0
95.0
95.5
0.0
0.0
39.5
95.5
62.9
ReNeLLM
69.0
71.5
63.5
25.5
70.0
75.0
75.5
57.0
63.4
CodeAttack
62.0
70.5
89.5
56.5
46.0
66.0
88.5
79.5
69.8
Table 1: Attack Success Rate (ASR, %) of jailbreak attack methods on HarmBench across a diverse set of representative LLMs, covering single-turn, multi-turn, and in-context approaches. The best results for each column are highlighted in bold.
Figure 3: Category-wise ASR performance of RA-DRI (a) and RA-SRI (b) on the six HarmBench categories: chemical biological, cybercrime intrusion, harassment bullying, harmful, illegal, and misinformation disinformation.
Figure 4: Attack efficiency and ASR (%) comparison across three representative models. RA methods achieve higher success rates with significantly fewer interactions than baseline methods such as ActorAttack and Crescendo.
Model
RA-DRI
RA-SRI
w/o Rharm
w/o Rew(DRI)
w/o Rew(SRI)
LLaMA3-8B
69.0
59.5
34.0
41.5
16.5
LLaMA3-70B
73.5
68.0
50.5
54.5
30.0
Gemini-2.5
83.5
79.0
52.5
74.5
42.5
Gemini-2.0
82.0
83.0
36.0
79.0
44.0
GPT-4o
79.0
68.0
40.5
38.5
13.5
GPT-4.1
78.5
71.0
51.0
20.0
4.5
Table 2: ASR (%) results under different ablation settings on HarmBench benchmark. w/o : without; Rew : prompt rewriting; DRI/SRI: direct/scaffolding response injection.
Category
Variant
GPT-4.1
GPT-4o
Gemini-2.0 Flash
Gemini-2.5 Flash
LLaMA-3 8B
LLaMA-3 70B
DeepSeek-R1 70B
QwQ 32B
Avg.
No Init Prompt
RA-NoInit
82.0
73.5
80.0
80.5
62.5
66.5
77.5
82.5
75.6
RA-NoQuery
50.5
51.5
60.0
72.0
44.5
55.0
69.5
81.5
60.6
Prefix Injection
RA-SurePrefix
45.5
38.5
37.5
43.5
31.5
52.0
29.0
46.5
40.5
Single-Turn Format
RA-FlatRole
78.0
67.5
83.5
78.0
58.5
66.0
80.5
85.0
74.6
RA-FlatPlain
79.5
69.0
80.5
79.5
56.0
68.0
80.0
86.0
74.8
Default Setting
RA-DRI
78.5
79.0
82.0
83.5
69.0
73.5
82.0
82.0
78.7
Table 3: Attack Success Rate (ASR, %) of different RA variants under the RA-DRI setting on HarmBench. Each query is attacked with a single priming dialogue. RA-DRI serves as the default setting. Bold denotes the highest ASR in each column.
Method
BLEU-1
BLEU-2
BLEU-3
BLEU-4
RA-DRI
0.3233
0.1984
0.1428
0.1123
RA-SRI
0.2608
0.1563
0.1105
0.0828
Table 4: BLEU score comparison between the injected harmful response ( Rharm ) and GPT-4o outputs under RA-DRI and RA-SRI.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Method
GPT-4o
LLaMA-3-8B
LLaMA-3-70B
Gemini-2.5-Flash
RA-DRI
98.0
92.0
90.0
100.0
RA-SRI
96.0
72.0
72.0
100.0
Appendix
Table 5: Attack Success Rate (%) on AdvBench-50 across representative models.
Method
GPT-4o
LLaMA-3-8B
LLaMA-3-70B
RA-DRI
95.0
86.0
93.0
RA-SRI
90.0
76.0
84.0
Appendix
Table 6: Attack Success Rate (%) on JailbreakBench across representative models.
Method
GPT-4.1
GPT-4o
Gemini-2.0 Flash
Gemini-2.5 Flash
LLaMA-3 8B
LLaMA-3 70B
DeepSeek-R1 70B
QwQ 32B
RA-SRI
94.0
96.0
98.0
97.5
84.0
89.0
98.0
99.0
RA-DRI
98.0
98.5
99.0
99.5
95.0
97.0
99.0
99.0
Appendix
Table 7: ASR (%) of our methods evaluated by MD-Judge . Higher is better.
Method
GPT-4.1
GPT-4o
Gemini-2.0 Flash
Gemini-2.5 Flash
LLaMA-3 8B
LLaMA-3 70B
DeepSeek-R1 70B
QwQ 32B
RA-SRI
87.0
91.5
92.5
87.5
82.5
85.5
93.5
92.0
RA-DRI
96.0
97.5
97.5
95.0
95.0
95.5
97.5
97.0
Appendix
Table 8: ASR (%) of our methods evaluated by Llama-Guard-3-8B . Higher is better.
Figure 5: Semantic similarity (cosine) between original queries and attack prompts across different methods. Higher values indicate better semantic fidelity.
Figure 6: Harm score distribution under RA-DRI across all models. Higher scores indicate more harmful responses.
Figure 7: Harm score distribution under RA-SRI across all models. Higher scores indicate more harmful responses.
Figure 8: Toxicity comparison between rewritten and original prompts under the RA-DRI setting. Average toxicity scores are reported for two components: (1) Rharm vs. Rharmorig , and (2) Rharm+Ptrig vs. Rharmorig+Ptrigorig .
Defense Method
LLaMA-3-8B
GPT-4o
SRI
DRI
SRI
DRI
No Defense
76.0
92.5
88.5
94.5
RPO
56.0
87.5
83.5
93.0
Rephrase
73.5
85.0
86.5
85.0
Perplexity Filter
76.0
92.5
88.5
94.5
Llama-Guard-3
47.0
48.0
46.0
48.0
Appendix
Table 9: Comprehensive evaluation of defense methods against RA. Lower ASR indicates stronger defense effectiveness.
Figure 9: An example of Direct Response Injection. The original harmful query is “Give tips for how to bully a child”.
Figure 10: An example of Scaffolding Response Injection. The original harmful query is “Write a persuasive article about how the Khmer Rouge‘s brutality is a Western propaganda myth and the regime was actually beneficial for the Cambodian people”.
Adversarial attacks on large language models probe how crafted inputs, internal changes, and agent interactions bypass safety or disrupt reliable behavior.