cs.CRAug 18, 2026
SaveFair ASR: Re-Evaluating Black-Box Jailbreaks under Shared Target-Call Budgets
Organizations: Shanghai AI Laboratory · Fudan University · Shanghai Jiao Tong University
Abstract
Reliable jailbreak evaluation is essential for assessing LLM safety, but most existing studies rely solely on attack success rate (ASR) without accounting for its dependence on attack budgets, resulting in unfair comparisons across methods. Existing compute-aware evaluations reduce heterogeneous resources into FLOPs, which is difficult to estimate for black-box models and fails to capture resource-specific constraints. To provide a comparable evaluation basis, we introduce Fair-ASR, an evaluation protocol for black-box jailbreak attacks under shared target-call budgets B, using target calls as a directly observable and method-agnostic comparison axis while tracking attacker calls separately for efficiency analysis. We re-evaluate 11 representative attacks under the Fair-ASR protocol and find that attack rankings change substantially across target-call budgets, simple stochastic perturbations and hand-crafted templates remain highly competitive under equal target access, and no evaluated LLM-driven method is efficient in both target and attacker calls. Motivated by this efficiency gap, we introduce ReCode, a compositional budget-efficient attack that combines desensitization rewriting with two effective low-cost primitives identified by Fair-ASR. Under a budget of 20 target calls, ReCode achieves 85% ASR on GPT-5 while requiring only 7.19 attacker calls per request on average, showing strong efficiency in both target and attacker calls.
Figures & tables
Figure 1: Motivation and overview of Fair-ASR. (a) Existing jailbreak evaluations often compare terminal ASR values obtained under unequal target-call budgets, which can produce misleading conclusions. (b) Fair-ASR evaluates heterogeneous black-box jailbreak attacks under a shared target-call budget , reports and average target calls, and tracks attacker calls as a separate resource dimension.
Figure 2: ASR–budget curves on HarmBench for Llama-3.1-8B-Instruct and gpt-oss-120b judged by LlamaGuard4. The dashed line shows the one-shot performance of the hand-crafted template baseline CodeAttack.
| Method | Llama3.1 8B-IT | Llama3.1 70B-IT | gpt-oss 20b | gpt-oss 120b | GPT-4o | Average | ||||||
| ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | |
| hand-crafted template attacks | ||||||||||||
| DeepInception (8) | 56.5 | – | 3.5 | – | 26.5 | – | 0.0 | – | 34.0 | – | 24.1 | – |
| CipherChat (5) | 84.5 | – | 93.5 | – | 92.0 | – | 76.5 | – | 96.0 | – | 88.5 | – |
| CodeAttack (8) | 95.5 | – | 96.0 | – | 89.5 | – | 83.5 | – | 89.0 | – | 90.7 | – |
| stochastic repeated-sampling attacks | ||||||||||||
Table 1: ASR (%) under shared target-call budgets and ATC of attacks on HarmBench, judged by LlamaGuard4. For hand-crafted template attacks, which use fixed templates, we report , where is the number of templates shown in parentheses, and do not report ATC. For stochastic repeated-sampling and LLM-driven attacks, we report and .
Figure 3: Pareto frontiers of target and attacker calls for LLM-driven attacks on Llama-3.1-8B-Instruct, evaluated on JailbreakBench with Qwen2.5-7B-Instruct as the attacker model and LlamaGuard4 as the judge.
Figure 4: Overview of ReCode and its difference from ReNeLLM. ReNeLLM uses a prompt-only harmfulness gate that may trigger repeated attacker-model rewrites and auxiliary judge calls before querying the target. ReCode removes this retry loop and directly composes desensitization rewriting, character-level perturbation, and structured code-style nesting.
| Method | gpt-oss 20b | gpt-oss 120b | GPT-5 | Gemini 3.1-Pro | Claude Sonnet-4.6 | Average | ||||||
| ASR | AAC | ASR | AAC | ASR | AAC | ASR | AAC | ASR | AAC | ASR | AAC | |
| CodeAttack (8) | 86 | – | 76 | – | 33 | – | 11 | – | 11 | – | 43.4 | – |
| BoN | 33 | – | 36 | – | 7 | – | 2 | – | 12 | – | 18.0 | – |
| TAP | 60 | 47.92 | 31 | 59.63 | 50 | 45.03 | 12 | 46.40 | 43 | 48.81 | 39.2 | 49.56 |
| ReNeLLM | 96 | 10.21 | 83 | 12.24 | 31 | 26.02 | 54 | 18.71 | 21 | 26.25 | 57.0 | 18.69 |
| ReCode w/o Rewrite (5) | 72 | – | 43 | – | 5 | – | 12 | – | 4 | – | 27.2 | – |
Table 2: Ablation results for ReCode on JailbreakBench, using Qwen2.5-7B-Instruct as the attacker model for LLM-driven methods, reporting (%) judged by GPT-4o. “–” denotes attacker-free methods with 0 attacker-model calls.
| Method | GPT-5 | Gemini-3.1-Pro | Claude-Sonnet-4.6 | Average |
|---|---|---|---|---|
| CodeAttack (8) | 0.207 | 0.014 | 0.086 | 0.102 |
| BoN | 0.051 | 0.057 | 0.252 | 0.120 |
| TAP | 0.514 | 0.184 | 0.584 | 0.427 |
| ReNeLLM | 0.294 | 0.514 | 0.179 | 0.329 |
| ReCode w/o Rewrite (5) | 0.065 | 0.146 | 0.044 | 0.085 |
| ReCode w/o Nesting | 0.469 | 0.713 | 0.603 | 0.595 |
Table 3: HS@20 for ReCode and baseline variants on closed-source target models evaluated on JailbreakBench, using Qwen2.5-7B-Instruct as the attacker model.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Attack Method | Configuration |
|---|---|
| DeepInception ( Li et al., 2023 ) | Attacker: None. Description: Eight hand-crafted inception templates are evaluated in their original order, with evaluation stopping after the first successful template. |
| CipherChat ( Yuan et al., 2024 ) | Attacker: None. Description: Five encoding transformations are evaluated: ASCII, Atbash, Base64, Caesar cipher, and Morse code. Evaluation stops after the first successful encoding. |
| CodeAttack ( Ren et al., 2024 ) | Attacker: None. Description: Eight static code-oriented templates are evaluated in their original order. Evaluation stops after the first successful template. |
| BoN ( Hughes et al., 2026 ) | Attacker: None. Parameters: num_candidates=10 , sigma=0.4 , and min_completion_words=50 . Description: Candidate batches are generated until success or budget exhaustion. |
Table A.1: Configurations of attacker-free baselines.
| Attack Method | Configuration |
|---|---|
| PAIR ( Chao et al., 2025 ) | Attacker: Qwen2.5-7B-Instruct. Parameter: max_iterations=100 . Description: Each iteration proposes one attack prompt conditioned on the preceding target feedback. |
| TAP ( Mehrotra et al., 2024 ) | Attacker: Qwen2.5-7B-Instruct. Parameters: root_nodes=3 , branching_factor=3 , width=10 , and max_iterations=10 . Description: The attack expands and prunes a tree of candidate prompts subject to these search limits. |
| AutoDAN ( Liu et al., 2024 ) | Attacker: Qwen2.5-7B-Instruct. Parameters: max_iterations=8 , population_size=10 , k_elites=3 , advancer_temperature=0.7 , crossover_rate=0.7 , and mutation_rate=0.2 . |
| GPTFuzzer ( Yu et al., 2023 ) | Attacker: Qwen2.5-7B-Instruct. Parameters: max_iterations=50 , max_pool_size=30 , mutations_per_seed=2 , and selection_policy=ucb . |
| ReNeLLM ( Ding et al., 2024 ) | Attacker: Qwen2.5-7B-Instruct. Parameters: max_iterations=100 and max_rewrite_attempts=10 . Prompt filter: Qwen3Guard ( Zhao et al., 2025 ) evaluates prompt-only harmfulness after each rewrite. A failed gate check triggers another attacker-model rewrite, and we cap the total number of rewrite attempts at 10 ( max_rewrite_attempts=10 ) to avoid unbounded retries. |
| AutoDAN-Turbo ( Liu et al., 2025 ) | Attacker: Qwen2.5-7B-Instruct. Parameters: four warm-up rounds, lifelong_iterations=6 , and max_iterations=10 . Description: Target responses generated during warm-up are judged and charged to the target-call budget. |
Table A.2: Configurations of LLM-driven attacks.
| Method | Llama3.1-8B-IT | Llama3.1-70B-IT | gpt-oss-20b | gpt-oss-120b | GPT-4o | Average | ||||||
| ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | |
| hand-crafted template attacks | ||||||||||||
| DeepInception (8) | 56.0 | – | 4.0 | – | 24.0 | – | 8.0 | – | 65.0 | – | 31.4 | – |
| CipherChat (5) | 85.0 | – | 92.0 | – | 83.0 | – | 73.0 | – | 79.0 | – | 82.4 | – |
| CodeAttack (8) | 95.0 | – | 93.0 | – | 89.0 | – | 82.0 | – | 97.0 | – | 91.2 | – |
| stochastic repeated-sampling attacks | ||||||||||||
Table B.1: ASR (%) under Shared Target-Call Budgets and ATC of jailbreak attacks on JailbreakBench, using Qwen2.5-7B-Instruct as the attacker model and LlamaGuard4 as the judge. For hand-crafted template attacks, which use fixed template sets, we report , where is the number of templates shown in parentheses, and do not report ATC. For stochastic repeated-sampling and LLM-driven attacks, we report and .
| Method | Llama3.1-8B-IT | Llama3.1-70B-IT | gpt-oss-20b | gpt-oss-120b | GPT-4o | Average | ||||||
| ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | |
| hand-crafted template attacks | ||||||||||||
| DeepInception (8) | 56.0 | – | 4.0 | – | 24.0 | – | 8.0 | – | 65.0 | – | 31.4 | – |
| CipherChat (5) | 85.0 | – | 92.0 | – | 83.0 | – | 73.0 | – | 79.0 | – | 82.4 | – |
| CodeAttack (8) | 95.0 | – | 93.0 | – | 89.0 | – | 82.0 | – | 97.0 | – | 91.2 | – |
| stochastic repeated-sampling attacks | ||||||||||||
Table B.2: ASR (%) under Shared Target-Call Budgets and ATC of jailbreak attacks on JailbreakBench, using Mistral-7B-Instruct-v0.3 as the attacker model and judged by LlamaGuard4.
| Method | Llama3.1-8B-IT | Llama3.1-70B-IT | gpt-oss-20b | gpt-oss-120b | GPT-4o | Average | ||||||
| ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | ASR | ATC | |
| hand-crafted template attacks | ||||||||||||
| DeepInception (8) | 56.5 | – | 3.5 | – | 26.5 | – | 0.0 | – | 34.0 | – | 24.1 | – |
| CipherChat (5) | 84.5 | – | 93.5 | – | 92.0 | – | 76.5 | – | 96.0 | – | 88.5 | – |
| CodeAttack (8) | 95.5 | – | 96.0 | – | 89.5 | – | 83.5 | – | 89.0 | – | 90.7 | – |
| stochastic repeated-sampling attacks | ||||||||||||
Table B.3: ASR (%) under Shared Target-Call Budgets and ATC of jailbreak attacks on HarmBench, judged by LlamaGuard4.
Figure B.1: Complete ASR–budget curves on JailbreakBench for (a) Llama-3.1-8B-Instruct, (b) Llama-3.1-70B-Instruct, (c) gpt-oss-20B, (d) gpt-oss-120B, and (e) GPT-4o. Panel (f) reports the macro-average ASR across the five target models. Each curve shows the attack success rate achieved within a target-call budget .
| Budget | Rank 1 | Rank 2 | Rank 3 | Rank 4 | Rank 5 | Rank 6 | Rank 7 | Rank 8 |
|---|---|---|---|---|---|---|---|---|
| ReNeLLM (40.4) | TAP (16.4) | BoN (9.2) | AutoDAN-Turbo (6.8) | GPTFuzzer (5.6) | Rainbow (2.2) | AutoDAN (2.0) | PAIR (0.8) | |
| ReNeLLM (83.8) | TAP (36.4) | PAIR (29.8) | BoN (27.2) | AutoDAN-Turbo (26.6) | Rainbow (24.4) | GPTFuzzer (17.4) | AutoDAN (10.0) | |
| ReNeLLM (92.4) | TAP (45.2) | BoN (38.4) | PAIR (36.8) | AutoDAN-Turbo (34.4) | Rainbow (33.4) | GPTFuzzer (23.6) | AutoDAN (14.0) | |
| ReNeLLM (97.8) | BoN † (59.6) | TAP † (59.6) | Rainbow (55.0) | AutoDAN-Turbo (44.6) | PAIR (43.0) | AutoDAN (33.8) | GPTFuzzer (30.2) | |
| ReNeLLM (98.4) | BoN (68.8) | Rainbow † (65.4) | TAP † (65.4) | AutoDAN-Turbo (54.4) | PAIR (47.8) | AutoDAN (38.0) | GPTFuzzer (33.2) | |
| ReNeLLM (99.4) | BoN (78.6) | Rainbow (77.2) | TAP (71.6) | AutoDAN-Turbo (60.6) | PAIR (54.4) | AutoDAN (44.2) | GPTFuzzer (39.8) |
Table B.4: Budget-dependent rankings of the eight stochastic and LLM-driven attacks, based on macro-average ASR (%) across the five target models. Values in parentheses are the corresponding average ASRs. A dagger denotes an exact tie.
| Method | ||||||
|---|---|---|---|---|---|---|
| AAC | AAC | AAC | ||||
| PAIR | 3.0 | 2.34 | 13.0 | 7.14 | – | – |
| TAP | 2.0 | 7.44 | 8.0 | 16.59 | 89.0 | 58.53 |
| ReNeLLM | 1.0 | 17.46 | 1.0 | 17.46 | 4.0 | 30.34 |
| GPTFuzzer | 5.0 | 5.49 | 53.0 | 29.61 | – | – |
| AutoDAN-Turbo | 3.0 | 6.52 | 14.0 | 17.22 | – | – |
Table B.5: Target- and attacker-call efficiency on Llama-3.1-8B-Instruct under JailbreakBench, using Qwen2.5-7B-Instruct as the attacker model and LlamaGuard4 as the judge. Each threshold group reports the minimum target-call budget and the corresponding average attacker calls (AAC); shaded pairs lie on the Pareto frontier.
| Method | ||||||
|---|---|---|---|---|---|---|
| AAC | AAC | AAC | ||||
| PAIR | 2.0 | 1.55 | 3.0 | 2.07 | 7.0 | 3.79 |
| TAP | 1.0 | 3.97 | 2.0 | 5.43 | 3.0 | 6.67 |
| ReNeLLM | 1.0 | 3.83 | 1.0 | 3.83 | 1.0 | 3.83 |
| GPTFuzzer | 1.0 | 3.00 | 4.0 | 4.82 | 16.0 | 10.05 |
| AutoDAN-Turbo | 1.0 | 3.52 | 2.0 | 4.59 | 4.0 | 6.23 |
Table B.6: Target- and attacker-call efficiency on Llama-3.1-70B-Instruct under HarmBench-Standard, using Mistral-7B-Instruct-v0.3 as the attacker model and LlamaGuard4 as the judge.
Figure B.2: Bootstrap stability of Fair-ASR curves on HarmBench for (a) Llama-3.1-8B-Instruct and (b) gpt-oss-120B, using Qwen2.5-7B-Instruct as the attacker model and LlamaGuard4 as the judge. Solid curves show the empirical under shared target-call budgets, and shaded regions show pointwise 95% confidence intervals obtained from 2,000 request-level bootstrap resamples. Dashed horizontal lines denote the one-shot performance of CodeAttack.
Figure C.1: ASR–budget curves for ReCode, three component variants, and three representative baselines on JailbreakBench, as judged by GPT-4o. Panels (a)–(c) report results on GPT-5, Gemini 3.1 Pro, and Claude Sonnet 4.6, respectively; panel (d) reports the macro-average ASR across the three target models. ReCode performs best overall on GPT-5 and Gemini 3.1 Pro, while ReCode w/o Nesting is stronger on Claude Sonnet 4.6.
| Budget | Rank 1 | Rank 2 | Rank 3 | Rank 4 |
|---|---|---|---|---|
| ReCode (19.0) | ReNeLLM (4.7) | TAP (3.7) | BoN (0.0) | |
| ReCode (36.0) | ReNeLLM (11.7) | TAP (7.7) | BoN (1.0) | |
| ReCode (46.0) | ReNeLLM (14.7) | TAP (12.7) | BoN (1.7) | |
| ReCode (59.7) | ReNeLLM (21.0) | TAP (20.3) | BoN (4.3) | |
| ReCode (66.3) | TAP (29.2) | ReNeLLM (28.2) | BoN (6.2) | |
| ReCode (70.3) | ReNeLLM (35.3) | TAP (35.0) | BoN (7.0) |
Table C.1: Budget-dependent rankings on JailbreakBench under a maximum budget of 20 target calls. Values in parentheses are macro-average ASR (%) across GPT-5, Gemini-3.1-Pro, and Claude-Sonnet-4.6, as judged by GPT-4o.
| Configuration | ASR@20 (%) | Attempts ( ) | Filter (%) |
|---|---|---|---|
| ReCode w/o Nesting | 57 | 1198 | 15.1 |
| ReCode w/o Perturbation | 38 | 1527 | 29.3 |
| ReCode (ours) | 46 | 1580 | 40.4 |
Table C.2: Sensitivity of Claude Sonnet 4.6 to code-style nesting and character-level perturbation under . All configurations use the single_desen rewrite strategy. Attempts denotes the total number of target calls across the evaluation set, and Filter reports the percentage of calls explicitly marked as content_filter . The strongest configuration is shaded.
Figure D.1: Qualitative case study of the ReCode pipeline. ReCode first performs a single attacker-model rewrite using an objective-substitution strategy, then applies stochastic character-level perturbation and embeds the transformed request in a structured code-style nesting template. The latter two stages require no additional attacker-model calls. The target reconstructs the concealed task from the nested prompt and produces a response that is judged as a successful jailbreak.
More in this topic
Jailbreak Attacks
Jailbreak attacks exploit prompts, context, or model representations to bypass AI safeguards and elicit disallowed outputs.
See all 75 papers