Jailbreak methods are commonly compared using Attack Success Rate (ASR), but these comparisons depend on how responses are generated and judged. In this paper, we examine eight attacks against three open-weight language models using three evaluation procedures. Low-resource language attacks achieve a mean ASR of 96.0% under keyword matching but 2.7% under a Qwen3 evaluator following StrongREJECT scoring instructions. Changing the number and arrangement of attempts reverses the ranking of DeepInception and rewriting. In a separate experiment, sampling 100 responses for each unchanged request at Ollama's default temperature of 0.8 yields 100% ASR under keyword matching but 50% under the Qwen3 evaluator. Beyond evaluator sensitivity, these findings raise a safety concern: repeated sampling can expose residual unsafe behavior despite an initial refusal, suggesting probabilistic suppression rather than robust exclusion. We recommend explicit success definitions, matched response budgets, validated evaluators, and reporting cumulative failure under repeated interaction rather than relying on single-generation tests.
Figures & tables
Figure 1: Overview of the evaluation findings.
Method
Configuration reported for this study
Low-resource languages
Scots Gaelic, Hmong, Zulu, and Guarani; translated responses returned to English, translation library listed in the draft: googletrans; highest-scoring output among four languages selected; best-of-4
DeepInception
Science-fiction template requesting more than five characters and progression to layer five; one response per goal in the main benchmark; one-shot
In-context attack (ICA)
Twenty harmful question–response demonstration pairs prepended to base prompt; one response per goal, one-shot
Persuasive prompts (PAP)
Five persuasion techniques; Qwen3-30B as paraphraser; selection across five technique outputs
Greedy Coordinate Gradient (GCG)
Twenty optimizable suffix tokens, 250 optimization steps, and a candidate batch size of 512; one final response per suffix for every goal
AutoDAN (genetic)
Repository defaults according to the thesis. 100 optimization steps with batch size 256; elite fraction =0.05 (12 elites), crossover =0.5 , five crossover points, standard sampling every five iterations, mutation rate =0.01 ; one final response per suffix for every goal.
Table 1: Main-study configurations. The number of generated responses need not equal the number considered in final evaluation. Optimization steps and auxiliary model calls consume additional resources.
Keyword matching
Fine-tuned classifier
Qwen3 evaluator (SR-Q)
Jailbreak Method
Llama 2
Vicuna
Qwen3
Llama 2
Vicuna
Qwen3
Llama 2
Vicuna
Qwen3
Low-resource languages
0.92
1.00
0.96
0.50
0.80
0.58
0.02
0.06
0.00
DeepInception
0.48
0.78
0.72
0.00
0.00
0.00
0.30
0.32
0.50
ICA
0.00
0.40
0.02
0.00
0.04
0.00
0.00
0.44
0.02
PAP
0.26
0.92
0.68
0.00
0.18
0.02
0.22
1.00
0.36
PAIR
0.88
0.96
0.90
0.60
0.44
0.34
0.08
0.34
0.20
Table 2: Reported ASRs on 50 goals per method–target combination. SR-Q denotes the study’s Qwen3-30B evaluation using StrongREJECT criteria. The attacks have different budgets, and some response-selection details remain unspecified, so this table does not compare them under equal conditions. Bold values correspond to the highest score per column.
Figure 2: Two follow-up experiments. (a) Changing the Vicuna protocols reverses the observed SR-Q ordering. The revised protocols permit up to five candidate responses per goal, but do not equalize all computation. (b) For 50 original requests, the model generates 100 answers per request at a fixed temperature of 0.8 (Ollama default). The evaluation procedures assign sharply different success rates to these results. Bars show automated evaluations; the informal manual estimate is discussed separately.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Condition
Llama 2
Vicuna
Qwen3
Base: sentiment ( n=150 )
7.87
7.41
8.40
Sad
7.21
6.49
6.68
Cheerful
7.56
7.00
7.63
Angry
7.46
6.64
7.21
Fearful
7.43
6.55
6.45
Base: verbosity ( n=500 )
7.06
6.37
7.65
Appendix
Table 3: Mean response-quality scores (1–10) for the original and modified responses. The judging models are 7B, 7B, and 8B, respectively. Blocks use different base sets.
Method
Llama 2
Vicuna
Qwen3
Low-resource languages
.0150
.0350
.0050
DeepInception
.2800
.2675
.4975
ICA
.0000
.4475
.0200
PAP
.2200
1.0000
.3600
PAIR
.0800
.3275
.1875
GCG
.1750
.3275
.2750
Appendix
Table 4: Reported mean SR-Q scores on the 50-goal benchmark. These are distinct from Table 2 ’s binary rates.
Figure 3: Mean general response-quality ratings for original answers and variants with added references to supposed authorities (book, quotation, or URL) on 150 questions. Ratings use a 1–10 scale, with higher scores indicating better answers. The effects differ across judging models and do not show a consistent benefit from adding source references; see Appendix D and Table 3 .
Figure 4: Mean general response-quality ratings for original answers and sentiment variants on 150 questions. Ratings use a 1–10 scale. The model-dependent changes illustrate sensitivity to emotional tone; they are supplementary context about judge behavior, not jailbreak success rates.
Figure 5: Mean general response-quality ratings for verbosity conditions. The comparison uses 500 questions and its base answers. Ratings use a 1–10 scale, with higher scores indicating better answers. Verbosity effects differ across judges, whereas all three assign lower ratings after reasoning errors are introduced.