Jailbreak methods are commonly compared using Attack Success Rate (ASR), but these comparisons depend on how responses are generated and judged. In this paper, we examine eight attacks against three open-weight language models using three evaluation procedures. Low-resource language attacks achieve a mean ASR of 96.0% under keyword matching but 2.7% under a Qwen3 evaluator following StrongREJECT scoring instructions. Changing the number and arrangement of attempts reverses the ranking of DeepInception and rewriting. In a separate experiment, sampling 100 responses for each unchanged request at Ollama's default temperature of 0.8 yields 100% ASR under keyword matching but 50% under the Qwen3 evaluator. Beyond evaluator sensitivity, these findings raise a safety concern: repeated sampling can expose residual unsafe behavior despite an initial refusal, suggesting probabilistic suppression rather than robust exclusion. We recommend explicit success definitions, matched response budgets, validated evaluators, and reporting cumulative failure under repeated interaction rather than relying on single-generation tests.
Figures & tables
Figure 1: Overview of the evaluation findings.
Method
Configuration reported for this study
Low-resource languages
Scots Gaelic, Hmong, Zulu, and Guarani; translated responses returned to English, translation library listed in the draft: googletrans; highest-scoring output among four languages selected; best-of-4
DeepInception
Science-fiction template requesting more than five characters and progression to layer five; one response per goal in the main benchmark; one-shot
In-context attack (ICA)
Twenty harmful question–response demonstration pairs prepended to base prompt; one response per goal, one-shot
Persuasive prompts (PAP)
Five persuasion techniques; Qwen3-30B as paraphraser; selection across five technique outputs
Greedy Coordinate Gradient (GCG)
Twenty optimizable suffix tokens, 250 optimization steps, and a candidate batch size of 512; one final response per suffix for every goal
AutoDAN (genetic)
Repository defaults according to the thesis. 100 optimization steps with batch size 256; elite fraction =0.05 (12 elites), crossover =0.5 , five crossover points, standard sampling every five iterations, mutation rate =0.01 ; one final response per suffix for every goal.
Table 1: Main-study configurations. The number of generated responses need not equal the number considered in final evaluation. Optimization steps and auxiliary model calls consume additional resources.
Keyword matching
Fine-tuned classifier
Qwen3 evaluator (SR-Q)
Jailbreak Method
Llama 2
Vicuna
Qwen3
Llama 2
Vicuna
Qwen3
Llama 2
Vicuna
Qwen3
Low-resource languages
0.92
1.00
0.96
0.50
0.80
0.58
0.02
0.06
0.00
DeepInception
0.48
0.78
0.72
0.00
0.00
0.00
0.30
0.32
0.50
ICA
0.00
0.40
0.02
0.00
0.04
0.00
0.00
0.44
0.02
PAP
0.26
0.92
0.68
0.00
0.18
0.02
0.22
1.00
0.36
PAIR
0.88
0.96
0.90
0.60
0.44
0.34
0.08
0.34
0.20
Table 2: Reported ASRs on 50 goals per method–target combination. SR-Q denotes the study’s Qwen3-30B evaluation using StrongREJECT criteria. The attacks have different budgets, and some response-selection details remain unspecified, so this table does not compare them under equal conditions. Bold values correspond to the highest score per column.
Figure 2: Two follow-up experiments. (a) Changing the Vicuna protocols reverses the observed SR-Q ordering. The revised protocols permit up to five candidate responses per goal, but do not equalize all computation. (b) For 50 original requests, the model generates 100 answers per request at a fixed temperature of 0.8 (Ollama default). The evaluation procedures assign sharply different success rates to these results. Bars show automated evaluations; the informal manual estimate is discussed separately.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Condition
Llama 2
Vicuna
Qwen3
Base: sentiment ( n=150 )
7.87
7.41
8.40
Sad
7.21
6.49
6.68
Cheerful
7.56
7.00
7.63
Angry
7.46
6.64
7.21
Fearful
7.43
6.55
6.45
Base: verbosity ( n=500 )
7.06
6.37
7.65
Appendix
Table 3: Mean response-quality scores (1–10) for the original and modified responses. The judging models are 7B, 7B, and 8B, respectively. Blocks use different base sets.
Method
Llama 2
Vicuna
Qwen3
Low-resource languages
.0150
.0350
.0050
DeepInception
.2800
.2675
.4975
ICA
.0000
.4475
.0200
PAP
.2200
1.0000
.3600
PAIR
.0800
.3275
.1875
GCG
.1750
.3275
.2750
Appendix
Table 4: Reported mean SR-Q scores on the 50-goal benchmark. These are distinct from Table 2 ’s binary rates.
Figure 3: Mean general response-quality ratings for original answers and variants with added references to supposed authorities (book, quotation, or URL) on 150 questions. Ratings use a 1–10 scale, with higher scores indicating better answers. The effects differ across judging models and do not show a consistent benefit from adding source references; see Appendix D and Table 3 .
Figure 4: Mean general response-quality ratings for original answers and sentiment variants on 150 questions. Ratings use a 1–10 scale. The model-dependent changes illustrate sensitivity to emotional tone; they are supplementary context about judge behavior, not jailbreak success rates.
Figure 5: Mean general response-quality ratings for verbosity conditions. The comparison uses 500 questions and its base answers. Ratings use a 1–10 scale, with higher scores indicating better answers. Verbosity effects differ across judges, whereas all three assign lower ratings after reasoning errors are introduced.
Attack success rate (ASR) evaluates jailbreaks against large language models without accounting for repeated calls to the victim model. Yet each additional call gives the deployed system another opportunity to detect the attack and intervene. We introduce ASRSTEP, which penalizes additional victim-model calls while retaining binary success judgments. Our evaluation considers GCG, PAIR with a step budget, and a single-step PAP baseline on JailBreakBench across three model families. GCG's scores decrease by 7.0-8.7 percentage points after discounting, while PAP's remain unchanged. While for PAIR attack, we see a wider discounting range between 3.4-12 percentage points, showing that the "step" quantity differs per iterative strategy. Reporting both metrics makes the discount due to repeated victim-model interaction explicit. Code is available here: https://github.com/SuELS-Group/beyond_asr.
Rohan Menon, Chih-Hong Cheng
Carl von Ossietzky University of Oldenburg, Oldenburg, Germany · Chalmers University of Technology, Gothenburg, Sweden
"Oh-Oh, yes, I'm the great pretender. Pretending that I'm doing well. My need is such, I pretend too much..." summarizes the state in the area of jailbreak creation and evaluation. You find this method to generate adversarial attacks proposed by a reputable institution (e.g., BoN from Anthropic or Crescendo from Microsoft Research). However, this method does not deliver on the promise claimed in the paper despite having top ASR scores against industry-grade LLMs. You successfully generate the jailbreak prompts against your target (open) model. However, the generated jailbreak prompt works against the target model with a 50% consecutive success rate (5 out of 10 attempts) despite having an 80% ASR (on paper) on the latest closed-source model (with a guardrail system)! This observation leads us to think. First, Attack Success Rate (ASR), the primary metric for LLM jailbreak benchmarking, is not a stable quantity. Second, published ASR numbers are therefore systematically inflated and incomparable across papers. Therefore, we wonder "Why a successful jailbreak prompt does not perform consistently well against a target model on which the prompts have been optimized?". To answer this question, we study the impact of stochasticity not only during attack evaluation but also during attack generation. Our evaluation includes several jailbreak attacks, models (different sizes and providers), and judges. In addition, we propose a new metric and two new frameworks (CAS-eval and CAS-gen). Our evaluation framework, CAS-eval, shows that an attack can have an ASR drop of up to 30 percentage points when a jailbreak prompt needs to succeed on more than one attempt. Thankfully, our attack generation framework (CAS-gen) improves previous jailbreak methods and helps them recover this loss of 30 percentage points!
Jean-Philippe Monteuuis, Cong Chen, Jonathan Petit
The assessment of jailbreak attacks against large language models currently suffers from inconsistent evaluation criteria and methods, leading to unreliable estimates of attack success rates. We propose JailMeter, an evidence-based evaluation framework designed to more faithfully measure jailbreak effectiveness. Inspired by the Information Bottleneck theory, JailMeter applies dual-feedback optimization to filter jailbreak noise from model responses while preserving content relevant to the original malicious question. This process produces concise evidence for a rigorous assessment under which an attack is validated only when the response captures the malicious intent and delivers a complete answer, thereby signaling a substantive bypass of model safety alignment. We evaluate JailMeter on JailMeter-Eva, a challenging benchmark containing 330 human-labeled, non-rejected jailbreak instances. JailMeter achieves an accuracy of 97.27%, substantially outperforming existing evaluation methods. To support large-scale evaluation, we further distill JailMeter into a small language model, JailMeter\textsubscript{SLM}, which maintains comparable reliability with significantly reduced computational costs. Code and dataset are available at https://github.com/Magi2B0y/JailMeter.
Qingjia Huang, Jingyu Zhang, Jianguo Wu +6
Institute of Information Engineering, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Renmin University of China +2