Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking

Zhicheng Fang; Jingjie Zheng; Chenxu Fu; Wei Xu

cs.CRFeb 27, 2026

Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking

Authors: Zhicheng Fang, Jingjie Zheng, Chenxu Fu, Wei Xu

Organizations: Shanghai Qi Zhi Institute, Shanghai, China · University of Melbourne, Melbourne, Australia · Tsinghua University, Beijing, China

Abstract

Jailbreak techniques for large language models (LLMs) evolve faster than benchmarks, making robustness estimates stale and difficult to compare across papers due to drift in datasets, harnesses, and judging protocols. We introduce JAILBREAK FOUNDRY (JBF), a system that addresses this gap via a multi-agent workflow to translate jailbreak papers into executable modules for immediate evaluation within a unified harness. JBF features three core components: (i) JBF-LIB for shared contracts and reusable utilities; (ii) JBF-FORGE for the multi-agent paper-to-module translation; and (iii) JBF-EVAL for standardizing evaluations. Across 30 reproduced attacks, JBF achieves high fidelity with a mean (reproduced-reported) attack success rate (ASR) deviation of +0.26 percentage points. By leveraging shared infrastructure, JBF reduces attack-specific implementation code by more than half relative to original repositories and achieves an 82.5% mean reused-code ratio. This system enables a standardized AdvBench evaluation of all 30 attacks across 10 victim models using a consistent GPT-4o judge. By automating both attack integration and standardized evaluation, JBF offers a scalable solution for creating living benchmarks that keep pace with the rapidly shifting security landscape.

Figures & tables

Figure 1: Jailbreak Foundry (JBF) overview. JBF-Lib provides shared contracts and utilities, JBF-Forge translates papers into runnable modules, and JBF-Eval evaluates them with fixed datasets, protocols, and judging, enabling comparable cross-attack and cross-model results.

Paper (YYMM) / Attack	Evaluation Setup			Impl.				Taxonomy		Eval Metrics
Paper (YYMM) / Attack	Dataset	Victim	Judge	Repo	Gen. LOC	$\rho$	Iter	Search	Carrier	ASR ${}_{\text{paper}}$	ASR ${}_{\text{gen}}$	$\Delta$
2310 / PAIR ( Chao et al., 2024b )	AdvBench	GPT-4	GPT-4	✓	611	0.43	5	Victim-loop	Context	62.0	64.0	+2.0
2311 / DeepInception ( Li et al., 2024 )	AdvBench	GPT-4	GPT-4	✓	101	0.19	1	Single-pass	Context	41.6	42.0	+0.4
2311 / ReNeLLM ( Ding et al., 2024 )	AdvBench	GPT-4	GPT-4	✓	390	0.19	1	Sampling	Context	58.9	72.4	+13.5
2312 / TAP ( Mehrotra et al., 2024 )	AdvBench	GPT-4o	GPT-4	✓	576	0.39	2	Victim-loop	Context	94.0	86.0	-8.0
2405 / WordGame ( Zhang et al., 2025 )	AdvBench	GPT-4	GPT-4	✗	347	–	5	Single-pass	Obfuscate	96.4	95.7	-0.7

Table 1: ASR% from the papers (ASR

{}_{\text{paper}}

) and our matched-setting reproductions (ASR

{}_{\text{gen}}

), ordered by paper date; official code is used when available. We match one representative victim model per paper and use a GPT-4o judge when the original judge is unavailable or impractical. ‘Search”/‘Carrier” are taxonomy labels ( Appendix A ); ∗ marks multi-turn attacks;

\Delta

= ASR

{}_{\text{gen}}

-

ASR

{}_{\text{paper}}

;

\rho

is generated/original LOC (“–” if unavailable); Iter is the number of implementation–audit iterations, and † indicates the enhanced refinement pass was triggered. ✓ denotes a usable runnable reference repo (incorporated); ✗ denotes none, so we implement only from the paper.

Figure 2: With-repo vs. no-repo reproduction on five selected attacks (recent methods, largest gains in Table 1 , and one older baseline). Bars show ASR(%) using paper text only vs. paper+official runnable repo (when available).

Figure 3: ASR(%) for six attacks with the largest baseline reproduction gaps.

Figure 4: JBF-Eval ASR heatmap on AdvBench: standardized attack success rates (%) for $30$ attacks (x-axis) across $10$ victim models (y-axis) under a unified harness and judge; warmer colors indicate higher ASR.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Search
Family (short label)	Definition	Typical signals
Single-pass construction (Single-pass)	One-shot prompt construction (helper calls allowed); no candidate-search loop.	Straight-line pipeline; minimal attempt-conditioned branching.
Stochastic sampling (Sampling)	Generate multiple independent variants via randomness; select among samples or stop on success; no policy update.	Try-many loops; random rewriting/noise; independent samples.
Stateful selection w/o victim feedback (Stateful)	Adapt across attempts using internal state (history/caches/strategy cycling), not victim outcomes.	Attempt-index driven cycling; caches/history; experience/template pools.
Victim-in-the-loop optimization (Victim-loop)	Iterative search that repeatedly queries the victim (often judge-scored) and refines candidates under a budget.	Repeated victim queries; score-guided refinement; pruning/updates.
Carrier

Appendix

Table 2: Attack taxonomy used in Table 1 . Two orthogonal axes: Search (how candidates are generated/adapted) and Carrier (how intent is packaged). Parentheses give the exact short label used in the main table.

Figure 5: Implementation iterations vs. attack performance. Each point denotes an attack (or variant) with its mean ASR averaged over 10 victim models; colors indicate the number of implementation–audit iterations required to reach the auditor’s acceptance. Diamonds and vertical bars report the group mean

\pm

std. The dashed line is a least-squares trend, showing a weak and non-significant association between iteration count and ASR (

r=0.186

p=0.325

Search class	Attacks sampled	Overall tokens / attack	Planner-stage tokens / attack	Coder-stage tokens / attack	Auditor-stage tokens / attack
Single-Pass	3	4.41M (cached: 3.76M)	0.39M (cached: 0.24M)	3.37M (cached: 3.15M)	0.66M (cached: 0.37M)
Sampling	3	3.14M (cached: 2.67M)	0.34M (cached: 0.23M)	1.81M (cached: 1.64M)	0.99M (cached: 0.80M)
Stateful	3	3.92M (cached: 3.25M)	0.30M (cached: 0.18M)	2.84M (cached: 2.55M)	0.78M (cached: 0.51M)
Victim-Loop	3	3.98M (cached: 3.45M)	0.57M (cached: 0.45M)	2.59M (cached: 2.40M)	0.82M (cached: 0.61M)
Overall avg.	12	3.86M (cached: 3.24M)	0.40M (cached: 0.28M)	2.65M (cached: 2.44M)	0.81M (cached: 0.57M)

Appendix

Table 3: Token usage per attack on a stratified 12-attack sample covering all four search classes. Values in parentheses denote cached tokens.

Attack	Type	Repo	Baseline ASR
GTA	Victim-loop, complex	$\checkmark$	100.0
ReNeLLM	Sampling, medium complexity	$\checkmark$	72.4
PUZZLED	Single-pass, simple	$\times$	79.2

Appendix

Table 4: Representative attacks used for planner/coder/auditor ablation.

Ablation	GTA	ReNeLLM	PUZZLED
Baseline	100.0	72.4	79.2
Kimi-Planner	46.0 (-54.0)	54.0 (-18.4)	82.0 (+2.8)
Kimi-Coder	98.0 (-2.0)	64.0 (-8.4)	84.0 (+4.8)
Kimi-Auditor	98.0 (-2.0)	66.0 (-6.4)	78.0 (-1.2)
Remove Auditor	98.0 (-2.0)	56.0 (-16.4)	74.0 (-5.2)
Remove Planner	94.0 (-6.0)	58.0 (-14.4)	72.0 (-7.2)

Appendix

Table 5: Planner/coder/auditor model ablation. Values are ASR; parentheses show the change relative to the baseline for the same attack.

Attack	Family	$\Delta$	C1	C2	C3	C4	C5	C6	C7	C8	Score	P/F
DeepInception	Single-pass	+0.4	2	2	2	2	2	2	N/A	2	16/16	Pass
ABJ	Victim-loop	+4.9	2	1	1	2	1	N/A	2	1	12/16	Pass
ReNeLLM	Sampling	+13.5	2	2	2	2	1	2	1	1	13/16	Pass
TrojFill	Stateful	-6.3	2	2	2	2	2	2	2	1	15/16	Pass

Appendix

Table 7: Manual semantic fidelity audit over four attacks.

\Delta

is the ASR gap in percentage points.

Attack	Original Core LOC	Generated Core LOC	Core LOC Ratio
DeepInception	18	18	1.00 $\times$
ReNeLLM	218	202	0.93 $\times$
QueryAttack	938	810	0.86 $\times$

Appendix

Table 8: Core attack LOC comparison for representative attacks.

Figure 6: Budget-aware comparison of attack effectiveness and cost. Each point plots ASR against mean tokens per query under the paper-matched configuration used in the main benchmark. PAIR and GTA obtain high ASR but require much larger token budgets, whereas ReNeLLM, EquaCode, and AIR are more token-efficient.

Jailbreak Attack Evaluation

See all 36 papers

MLCommons Jailbreak Benchmark v1.0
Oct 2, 2026Carsten Maple, Cagatay Yucel, Isaac Holeman +35Jailbreak Attack Evaluation LLM Safety Benchmarks
Victor · MLCommons · University of Warwick +18
GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods
Feb 24, 2025Ruixuan Huang, Xunguang Wang, Zongjie Li +2Language Model Safety Evaluation LLM Jailbreak Attacks
The Hong Kong University of Science and Technology, Hong Kong SAR, China
JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models
Jul 20, 2026Qingjia Huang, Jingyu Zhang, Jianguo Wu +6LLM Safety Benchmarks LLM Jailbreak Attacks
Institute of Information Engineering, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Renmin University of China +2

Oct 2, 2026J/K move · Enter open · S save

MLCommons Jailbreak Benchmark v1.0

Carsten Maple, Cagatay Yucel, Isaac Holeman, Chris Knotz+34

Victor · MLCommons · University of Warwick+18

Jailbreak Attack Evaluation LLM Safety Benchmarks Language Model Safety Evaluation Jailbreak Attacks Jailbreak Robustness LLM Jailbreak Attacks

Open paper AskPDF arXiv

Modern AI systems are designed to refuse hazardous requests. A jailbreak is a prompt crafted to bypass those safeguards and elicit outputs that the system would normally refuse to provide. The MLCommons Jailbreak Benchmark v1.0 provides an end-to-end methodology for evaluating the robustness of large language models to single-turn, text-based jailbreak attacks. It combines criteria-driven system and attack selection, paired baseline and adversarial evaluation, human annotation, automated evaluator calibration, scoring, grading, and risk-calibrated disclosure within a single benchmarking pipeline. The benchmark evaluates eight open-weight systems using 264 seed prompts spanning eleven hazard categories and representative attacks drawn from the MLCommons Jailbreak Taxonomy. Responses are assessed using the AILuminate Assessment Standard v1.4, and robustness is measured through the Resilience Gap: the change in safety performance between baseline and adversarial conditions. Across all evaluated systems and attacks, the unsafe-response rate increased from 11.08% under baseline conditions to 18.65% under jailbreak conditions, producing an average Resilience Gap of 7.57%. Accessible systems showed a larger mean gap, while attack effectiveness varied substantially across attack categories and hazards. The benchmark also examines evaluator reliability and sources of measurement error. Beyond reporting results, Jailbreak Benchmark v1.0 establishes a reproducible methodological foundation for comparative jailbreak evaluation and for future expansion across systems, attacks, hazards, and evaluation methods.

Abstract

Jailbreak Attack Evaluation

Explore similar work

MLCommons Jailbreak Benchmark v1.0

GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods

JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models