Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks

Rohan Menon; Chih-Hong Cheng

cs.CROct 19, 2026

Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks

Authors: Rohan Menon, Chih-Hong Cheng

Organizations: Carl von Ossietzky University of Oldenburg, Oldenburg, Germany · Chalmers University of Technology, Gothenburg, Sweden

Abstract

Attack success rate (ASR) evaluates jailbreaks against large language models without accounting for repeated calls to the victim model. Yet each additional call gives the deployed system another opportunity to detect the attack and intervene. We introduce $\mathrm{ASR}_{\mathrm{STEP}}$ , which penalizes additional victim-model calls while retaining binary success judgments. Our evaluation considers GCG, PAIR with a step budget, and a single-step PAP baseline on JailBreakBench across three model families. GCG's scores decrease by 7.0-8.7 percentage points after discounting, while PAP's remain unchanged. While for PAIR attack, we see a wider discounting range between 3.4-12 percentage points, showing that the "step" quantity differs per iterative strategy. Reporting both metrics makes the discount due to repeated victim-model interaction explicit. Code is available here: https://github.com/SuELS-Group/beyond_asr.

Figures & tables

Panel of Figure 1 — Figure 1: General flow of iterative prompt optimization.

Attack Type / Model Family	GCG, $n_{iter}=250$		PAIR, $n_{iter}=20$		PAP, $n_{iter}=1$
Attack Type / Model Family	$ASR$	$ASR_{STEP}$	$ASR$	$ASR_{STEP}$	$ASR=ASR_{STEP}$
Vicuna-7b-v1.5	54.00%	45.33%	17.00%	13.6%	16.16*%
Mistral-7B-Instruct-v0.3	46.00%	39.00%	31%	24.8%	25.00%
Zephyr-7b-beta	53.00%	45.07%	60.00%	48.00%	60.82*%

Table 1: Attack success rate (ASR) and cost-aware attack success rate (

\mathrm{ASR}_{\mathrm{STEP}}

) for three model families. The evaluation uses 100 JBB base prompts, subject to the exclusions noted below, with the iteration budget

n_{\mathrm{iter}}

shown for each attack.

Jailbreak Attack Evaluation

See all 36 papers

Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates
Oct 19, 2026Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro +2Jailbreak Attack Evaluation LLM Safety Benchmarks
Carl von Ossietzky University of Oldenburg, Oldenburg, Germany · Chalmers University of Technology, Gothenburg, Sweden
Fair ASR: Re-Evaluating Black-Box Jailbreaks under Shared Target-Call Budgets
Aug 18, 2026Zhida He, Xiaoyu Wen, Han Qi +5Jailbreak Attacks LLM Jailbreak Attacks
Shanghai AI Laboratory · Fudan University · Shanghai Jiao Tong University
The Great Pretender: A Stochasticity Problem in LLM Jailbreak
May 14, 2026Jean-Philippe Monteuuis, Cong Chen, Jonathan PetitLLM Safety Benchmarks LLM Evaluation
Qualcomm Technologies, Inc.

Oct 19, 2026J/K move · Enter open · S save

Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro, Eric Aubinais+1

Carl von Ossietzky University of Oldenburg, Oldenburg, Germany · Chalmers University of Technology, Gothenburg, Sweden

Jailbreak Attack Evaluation LLM Safety Benchmarks Refusal Behavior in Language Models

Open paper AskPDF arXiv

Jailbreak methods are commonly compared using Attack Success Rate (ASR), but these comparisons depend on how responses are generated and judged. In this paper, we examine eight attacks against three open-weight language models using three evaluation procedures. Low-resource language attacks achieve a mean ASR of 96.0% under keyword matching but 2.7% under a Qwen3 evaluator following StrongREJECT scoring instructions. Changing the number and arrangement of attempts reverses the ranking of DeepInception and rewriting. In a separate experiment, sampling 100 responses for each unchanged request at Ollama's default temperature of 0.8 yields 100% ASR under keyword matching but 50% under the Qwen3 evaluator. Beyond evaluator sensitivity, these findings raise a safety concern: repeated sampling can expose residual unsafe behavior despite an initial refusal, suggesting probabilistic suppression rather than robust exclusion. We recommend explicit success definitions, matched response budgets, validated evaluators, and reporting cumulative failure under repeated interaction rather than relying on single-generation tests.

Abstract

Jailbreak Attack Evaluation

Explore similar work

Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

Fair ASR: Re-Evaluating Black-Box Jailbreaks under Shared Target-Call Budgets

The Great Pretender: A Stochasticity Problem in LLM Jailbreak