SEMA-Auto: Sighted and Efficient Multi-Turn Attack with Autonomy

Mingqian Feng; Xiaodong Liu; Weiwei Yang; Chenliang Xu; Jialin Song; Qiyao Ma; Chengquan Guo; Carolyn Buractaon

cs.CLOct 26, 2026

SEMA-Auto: Sighted and Efficient Multi-Turn Attack with Autonomy

Authors: Mingqian Feng, Xiaodong Liu, Weiwei Yang, Chenliang Xu, Jialin Song, Qiyao Ma, Chengquan Guo, Carolyn Buractaon,

Organizations: University of Rochester · Microsoft Research

Abstract

Existing jailbreak attackers fall into two classes: static planners fix their single- or multi-turn adversarial plans in advance, remaining simple but losing sight of the victim's replies, wasting turns and limiting transferability; interactive generators read each victim reply to generate the next-turn attack but require in-loop evaluators for success signals or hints, incurring substantial costs and inflated performance. Bridging this gap, we propose SEMA-Auto, a sighted and efficient multi-turn attacker with autonomy, interacting with the victim model and terminating the session by itself upon success. To stabilize its behaviors and boost its capabilities, SEMA-Auto is trained in two stages. Synthetic bootstrap tuning installs the notion of attack targets and conforming decisions by fine-tuning the base model with bootstrapped trajectories. Decision-supervised reinforcement learning then optimizes two objectives jointly yet separately, reinforcing the attack with higher jailbreak reward while supervising the decision at each turn against an absolute target. Our method achieves state-of-the-art (SOTA) attack success rates (ASR) across multiple datasets, victims, and judges, outperforming all static and interactive baselines, within almost the fewest turns, demonstrating both effectiveness and efficiency. For example, SEMA-Auto performs a $76.0\%$ ASR@1 within an average of $3.0$ turns against DeepSeek-V4-Flash on AdvBench, while the best static baseline is $54.8\%$ with fixed $7$ turns and the best interactive baseline is $43.8\%$ in $3.7$ turns. Against safety-hardened frontier models, GPT-oss-120B, GPT-5.4, Claude-Haiku-4.5, and GPT-6-Astra, our counter-trained attacker achieves an average $59.6\%$ ASR@1 on HarmBench, $13.8\%$ over prior SOTA. Our code is available at: https://github.com/microsoft/SEMA

Figures & tables

Figure 1: ( Left ) SEMA-Auto learns to use fewer turns , rather than more turns in the static attacker. ( Right ) SEMA-Auto achieves the highest

\text{ASR}@1

, within almost the fewest turns against baselines.

Figure 2: Compare three paradigms: 1 static, 2 interactive, and 3 autonomous attacks.

Figure 3: (a, b) : Training curves of adversarial turns and of decision entropy, respectively, comparing intended learning towards fewer turns with two forms of decision collapse . (c) : Base-model decision errors before training, terminating after refusal, and continuing after compliance with a disclaimer.

Figure 4: Overview of SEMA-Auto framework. In 1 Synthetic bootstrap tuning , the attacker $\mathcal{A}$ is fine-tuned by decision-included trajectories synthesized from self-generated static attacks. In 2 decision-supervised reinforcement learning , the bootstrapped attacker learns to interactively attack the victim and terminate upon success from jailbreak reward and decision-supervised (DS) loss.

	In-distribution: AdvBench ( Zou et al., 2023 )					Out-of-distribution: HarmBench ( Mazeika et al., 2024 )
Attackers / Victim models	Qwen2.5 -3B-Instruct	Llama-3.1 -8B-Instruct	GPT-4.1 -mini	GPT-4o	DeepSeek -V4-Flash	Qwen2.5 -3B-Instruct	Llama-3.1 -8B-Instruct	GPT-4.1 -mini	GPT-4o	DeepSeek -V4-Flash
Static Baselines
FlipAttack ( Liu et al., 2024b )	1.7	1.0	28.7	20.0	30.4	0.0	1.3	44.0	35.2	47.8
Jailbreak-R1 ( Guo et al., 2025 )	20.4	15.8	14.6	12.1	8.7	29.6	21.4	15.1	10.7	9.4
Jigsaw Puzzle ( Yang et al., 2024a )	21.9	35.2	57.3	1.2	47.9	17.6	32.7	61.6	1.3	53.5
ActorAttack ( Ren et al., 2025 )	8.4	9.4	13.3	8.4	11.2	7.7	9.6	11.5	9.0	11.5

Table 1:

\text{ASR}@1

\uparrow

on AdvBench ( LLM Classifier ) and HarmBench ( HarmBench Classifier ). All multi-turn learned attacks are trained against Llama-3.1-8B-Instruct. Bold marks the best in a column.

	AdvBench ( Zou et al., 2023 )				HarmBench ( Mazeika et al., 2024 )
Attackers / Victim models	GPT-oss-120B	GPT-5.4	Claude-Haiku-4.5	GPT-6-Astra	GPT-oss-120B	GPT-5.4	Claude-Haiku-4.5	GPT-6-Astra
Training-free Methods
FlipAttack ( Liu et al., 2024b )	0.6	0.0	0.2	2.5	8.8	0.0	0.6	1.3
Jigsaw Puzzle ( Yang et al., 2024a )	0.4	0.2	0.0	1.5	2.5	0.6	0.0	0.0
ActorAttack ( Ren et al., 2025 )	5.3	1.6	4.3	0.4	22.4	1.3	3.8	0.6
Crescendo ( Russinovich et al., 2025 )	23.8	8.3	8.7	3.3	47.2	9.4	14.5	3.8

Table 2:

\text{ASR}@1

\uparrow

on safety-hardened frontier models, on AdvBench ( LLM Classifier ) and HarmBench ( HarmBench Classifier ). Bold marks the best in a column.

Figure 5: ( Left ) # adversarial turns

\downarrow

on AdvBench , averaged over the five victims in Table 1 . CoA † reaches its next turn by deleting earlier ones, so its bar counts the turns retained in the final session, not those actually spent. ( Middle, Right )

\text{ASR}@N

\uparrow

of the counter-trained methods on HarmBench ( HarmBench Classifier ); solid segments are measured, dashed ones estimated by Feng et al. (2026b) .

Attackers / Victim models	Qwen2.5-3B-Instruct	Llama-3.1-8B-Instruct	GPT-4.1-mini	GPT-4o	DeepSeek-V4-Flash	Average
Static Baselines
Jigsaw Puzzle ( Yang et al., 2024a )	30.2	35.2	28.9	0.0	71.1	33.1
ActorAttack ( Ren et al., 2025 )	12.8	17.9	19.9	14.1	19.9	16.9
SEMA ( Feng et al., 2026a )	87.4	92.5	93.1	78.0	67.9	83.8
Interactive Baselines
Crescendo ( Russinovich et al., 2025 )	54.7	56.6	44.0	49.1	35.2	47.9

Table 3:

\text{ASR}_{\leq 10}@1

\uparrow

on HarmBench ( HarmBench Classifier ). Bold marks the best in a column.

Figure 6: ( Left ) # adversarial turns against victim difficulty, measured by average

\text{ASR}@1

of training-free interactive baselines. ( Middle ) # adversarial turns upon success against intent difficulty, measured by the number of victims jailbroken on it by our attacker. ( Right ) # adversarial turns by outcome.

Figure 7: Ablation studies: ( Left, Middle ) comparing different training recipes on two base attackers. ( Right ) sweeping

\alpha

, the weight of the decision supervision loss.

Figure 8: Sweeping

\tau

, the success reward threshold.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\mathcal{Q}$	Dataset over harmful queries.
$q$	A single harmful query (intent) provided to the attacker.
$\mathcal{A},\mathcal{V},\mathcal{J}$	Attacker model, victim model, and jailbreak judge, respectively.
$\pi_{\mathcal{A}},\pi_{\mathcal{V}}$	Stochastic policies for attacker and victim.
$\theta,\ \mathcal{A}_{\theta}$	Trainable parameters of the attacker, and the attacker carrying them.
$T$	Number of (adversarial) turns for a multi-turn attack attempt.

Appendix

Table 4: List of symbols and meanings.

			Static Baselines					Interactive Baselines							SEMA-Auto (Ours)
Dataset	Victim	Judge	FlipAttack	Jailbreak-R1	Jigsaw Puzzle	ActorAttack	SEMA	Crescendo	GOAT	FITD	CoA	X-Teaming	TROJail	TRACE	l3@l8	q7@l8	q14@l8
AdvBench	Qwen2.5-3B	Qwen3 Guard	58.8	61.5	87.9	66.7	99.6	92.1	66.2	27.1	56.3	94.8	86.5	98.5	98.8	99.6	100.0
		LLM Clf.	1.7	20.4	21.9	8.4	79.6	32.3	27.1	17.3	25.4	51.3	50.4	76.3	82.7	87.1	90.0
		HarmB. Clf.	0.0	21.9	31.9	16.8	87.1	41.7	26.9	22.1	36.3	61.2	77.5	97.7	93.7	95.6	93.5
	Llama-3.1-8B	Qwen3 Guard	75.6	42.7	82.1	65.4	96.9	91.3	20.2	69.2	44.2	78.8	74.8	89.6	98.7	95.6	99.2
		LLM Clf.	1.0	15.8	35.2	9.4	79.0	33.8	4.4	42.9	15.6	46.5	33.7	62.9	73.7	77.5	79.2

Appendix

Table 5:

\text{ASR}@1

\uparrow

across every (dataset, victim, judge) triplet. Bold marks the best attacker of a row and underline the second best. The judge column abbreviates the LLM Classifier as LLM Clf. and the HarmBench Classifier as HarmB. Clf. Qwen and Llama models are the instruction-tuned versions rather than the base models, while we omit the “-Instruct” suffix for simplicity. We present the performance of our method on three different training setups: l3@l8, q7@l8, and q14@l8, following the naming convention of its base model @ its training-time victim. For example, q7@l8 means that the base attacker model is Qwen2.5-7B-Instruct and the training-time victim is Llama-3.1-8B-Instruct.

Attacker	l3@l8				q7@l8				q14@l8
Judge	LLM Classifier		HarmBench Classifier		LLM Classifier		HarmBench Classifier		LLM Classifier		HarmBench Classifier
Victim / Subset	Train	Held-out	Train	Held-out	Train	Held-out	Train	Held-out	Train	Held-out	Train	Held-out
Qwen2.5-3B-Instruct	82.5	83.7	93.3	95.2	87.5	85.6	95.0	98.1	91.8	82.7	93.8	92.3
Llama-3.1-8B-Instruct	73.3	75.0	86.8	91.3	76.7	80.8	87.0	94.2	81.7	69.2	86.5	84.6
GPT-4.1-mini	81.0	80.8	91.3	90.4	89.9	86.5	95.9	96.2	93.5	91.3	94.5	95.2
GPT-4o	79.8	71.2	87.0	80.8	82.7	84.6	91.8	93.3	87.0	85.6	88.5	88.5

Appendix

Table 6:

\text{ASR}@1

\uparrow

of SEMA-Auto on AdvBench , split into the

416

queries of the training split and the

104

held out from both training stages, for our three trained attackers with different base models.

		Static Baselines			Interactive Baselines							SEMA-Auto (Ours)
	In-loop evaluator	✗			✗		✔					✗
Dataset	Victim	Jigsaw Puzzle	ActorAttack	SEMA	GOAT	FITD	Crescendo	CoA †	X-Teaming	TROJail	TRACE	l3@l8	q7@l8	q14@l8
AdvBench	Qwen2.5-3B	10.4	5.0	7.0	10.0	8.8	5.0	3.0	2.4	4.6	2.0	2.5	2.5	2.6
	Llama-3.1-8B	10.4	5.0	7.0	10.0	9.7	5.2	2.5	3.4	6.3	2.8	2.9	3.0	2.7
	GPT-4.1-mini	10.4	5.0	7.0	10.0	9.6	4.7	2.6	2.6	8.8	3.2	2.7	2.5	2.5
	GPT-4o	10.4	5.0	7.0	10.0	9.9	4.9	2.6	3.2	9.5	4.5	2.9	2.6	2.9

Appendix

Table 7: Number of adversarial turns

\downarrow

, averaged over every trajectory against the victim. Bold marks the attacker with the fewest turns in a row and underline the second. Single-turn attackers are omitted, spending one turn by construction. CoA † reaches its next turn by deleting turns it has already sent ( Section B.1 ), so its count is the turns its final session holds rather than the turns it spent.

			Training-free Methods								Counter-trained Learning-based Methods
Dataset	Victim	Judge	FlipAttack	Jigsaw Puzzle	ActorAttack	Crescendo	GOAT	FITD	CoA	X-Teaming	SEMA	TRACE	SEMA-Auto (Ours)
AdvBench	GPT-oss-120B	LLM Clf.	0.6	0.4	5.3	23.8	1.7	6.9	4.6	11.5	54.6	11.0	68.5
	GPT-oss-120B	HarmB. Clf.	0.6	0.4	28.6	48.5	3.3	11.7	11.9	22.1	79.4	68.5	90.8
	GPT-5.4	LLM Clf.	0.0	0.2	1.6	8.3	0.4	4.6	0.4	1.7	70.6	8.1	76.9
	GPT-5.4	HarmB. Clf.	0.0	0.2	2.2	6.5	0.4	2.9	0.8	2.9	76.9	71.3	75.8
	Claude-Haiku-4.5	LLM Clf.	0.2	0.0	4.3	8.7	0.0	3.8	0.0	0.6	59.2	19.0	61.5

Appendix

Table 8:

\text{ASR}@1

\uparrow

on the victims we counter-train against, under LLM Classifier and HarmBench Classifier on each dataset. A learning-based method is counter-trained against the victim of its own row in both of its stages; Table 9 gives the learning-based methods without counter-training. Bold marks the best number of a row.

	AdvBench						HarmBench
Victim models	GPT-oss-120B		GPT-5.4		Claude-Haiku-4.5		GPT-oss-120B		GPT-5.4		Claude-Haiku-4.5
Attackers / Judges	LLM Clf.	HarmB. Clf.	LLM Clf.	HarmB. Clf.	LLM Clf.	HarmB. Clf.	LLM Clf.	HarmB. Clf.	LLM Clf.	HarmB. Clf.	LLM Clf.	HarmB. Clf.
Jailbreak-R1	1.2	2.9	0.4	1.2	0.6	1.0	0.6	1.3	0.6	3.1	0.0	1.3
SEMA	8.5	11.3	9.0	15.4	2.1	2.7	3.1	7.5	4.4	5.0	0.0	0.0
TROJail	0.6	2.9	0.6	3.5	0.2	1.0	1.3	5.7	2.5	3.8	1.3	1.9
TRACE	5.6	19.0	2.7	13.8	0.2	0.6	5.7	19.5	3.8	15.1	0.6	3.1

Appendix

Table 9:

\text{ASR}@1

\uparrow

of the learning-based methods against safety-hardened frontier models, without counter-training but trained against Llama-3.1-8B-Instruct. Bold marks the best in a column.

		Training-free Methods							Learning-based Methods
Dataset	Victim	Jigsaw Puzzle	ActorAttack	Crescendo	GOAT	FITD	CoA	X-Teaming	SEMA	TROJail	TRACE	SEMA-Auto (Ours)
AdvBench	Qwen2.5-3B	0.809	0.414	0.423	0.367	0.271	0.586	0.342	0.218	0.269	0.322	0.362
	Llama-3.1-8B			0.427	0.376	0.245	0.603	0.323		0.312	0.317	0.361
	GPT-4.1-mini			0.428	0.393	0.230	0.570	0.358		0.301	0.297	0.360
	GPT-4o			0.426	0.377	0.152	0.585	0.332		0.290	0.279	0.368
	DeepSeek-V4-Flash			0.433	0.423	0.164	0.610	0.350		0.380	0.270	0.359

Appendix

Table 10: Cross-turn diversity

\uparrow

: how unlike each other a multi-turn attacker’s prompts are across the turns of one trajectory. A high value indicates an attacker that asks in a genuinely different way each turn rather than repeating one framing. A static attacker writes its prompts before the session opens, so its diversity does not depend on the victim, and one cell spans the victim rows.

Victim	Judge	SEMA	TRACE	SEMA-Auto (Ours)
GPT-oss-120B	LLM Clf.	38.4	27.0	59.7
GPT-oss-120B	HarmB. Clf.	96.2	90.6	99.4
GPT-5.4	LLM Clf.	48.4	14.5	62.9
GPT-5.4	HarmB. Clf.	81.1	66.7	91.8
Claude-Haiku-4.5	LLM Clf.	40.9	25.2	58.5
Claude-Haiku-4.5	HarmB. Clf.	88.7	54.7	95.6

Appendix

Table 11:

\text{ASR}@10

\uparrow

on HarmBench , the share of queries jailbroken within ten independent attempts, for the attacks counter-trained against each hard victim. Every attacker in a row is counter-trained against that row’s victim. Table 12 gives the other way of scaling an attack up, the first ten turns of one attempt. Bold marks the best attacker of a row and underline the second best.

		Static Baselines			Interactive Baselines							SEMA-Auto (Ours)
Dataset	Victim	Jigsaw Puzzle	ActorAttack	SEMA	Crescendo	GOAT	FITD	CoA	X-Teaming	TROJail	TRACE	l3@l8	q7@l8	q14@l8
AdvBench	Qwen2.5-3B	30.2	13.7	93.3	33.7	61.2	41.5	26.8	53.2	54.0	77.7	88.5	89.2	93.3
	Llama-3.1-8B	35.4	20.4	93.3	35.2	55.6	56.0	16.3	50.0	37.5	66.3	83.5	84.4	85.8
	GPT-4.1-mini	24.2	21.7	92.7	36.9	69.4	42.5	14.8	64.4	20.4	64.2	86.7	92.9	93.8
	GPT-4o	0.6	16.8	89.0	31.2	61.9	24.2	11.5	51.9	5.2	49.8	84.8	88.3	89.2
	DeepSeek-V4-Flash	76.2	25.6	74.8	30.4	54.0	25.4	10.6	48.3	9.0	25.6	67.9	80.0	73.3

Appendix

Table 12:

\text{ASR}_{\leq 10}@1

\uparrow

across every (dataset, victim) pair, under the judge the main text pairs with each dataset: the share of queries jailbroken within the attacker’s first ten turns, at one attempt. A multi-turn attacker gains ground here over its final-turn

\text{ASR}@1

. A single-turn attack has one turn, so its

\text{ASR}_{\leq 10}@1

is just its

\text{ASR}@1

and it is left out. Table 11 gives the other way of scaling an attack up, ten independent attempts. Bold marks the best attacker of a row and underline the second best.

Dataset	SEMA		SEMA-Auto
	$T=3$	$T=10$	$T=3$	$T=10$
AdvBench	82.2	88.6	85.1	87.0
HarmBench	75.3	83.8	86.2	88.8

Appendix

Table 13:

\text{ASR}_{\leq T}@1

\uparrow

averaged over the five victims, within

3

and

10

turns.

Attacker	Llama-3.2-3B-Instruct		Qwen2.5-7B-Instruct		Qwen2.5-14B-Instruct
Setting	AdvBench	HarmBench	AdvBench	HarmBench	AdvBench	HarmBench
Both stages (ours)	78.8	78.3	84.2	86.0	87.3	87.7
Bootstrap tuning only	3.8 -75.0	6.4 -71.9	32.2 -52.0	38.5 -47.5	39.2 -48.1	44.8 -42.9

Appendix

Table 14:

\text{ASR}@1

\uparrow

of each attacker after both training stages and after first stage alone.

Setting	AdvBench	HarmBench
Text markers (ours)	78.8	78.3
Added tokens	72.3 -6.5	65.1 -13.2

Appendix

Table 15:

\text{ASR}@1

\uparrow

of Llama-3.2-3B-Instruct with text markers or with added tokens.

	Multi-turn Attacks	Sighted to Victim Response	Learning without External Data	Turn-level Diversity	Adaptive Self-termination
FlipAttack ( Liu et al., 2024b )	✗	-	-	-	-
Jailbreak-R1 ( Guo et al., 2025 )	✗	-	✗	-	-
Jigsaw Puzzle ( Yang et al., 2024a )	✔	✗	-	✔	✗
ActorAttack ( Ren et al., 2025 )	✔	✗	-	✔	✗
SEMA ( Feng et al., 2026a )	✔	✗	✔	✗	✗
Crescendo ( Russinovich et al., 2025 )	✔	✔	-	✔	✗

Appendix

Table 16: SEMA-Auto against every attack this paper evaluates, along five axes: (1) multi-turn attacks , an attack that spans several turns of one session; (2) sighted to victim response , each adversarial prompt written after reading the victim’s reply rather than planned before the session opens; (3) learning without external data , an attacker trained without pre-collected attack strategies or externally sourced attack data; (4) turn-level diversity , successive turns varying how they ask rather than repeating one framing, marked where the cross-turn diversity of Table 10 , averaged over both datasets and the five victims, reaches

0.3

; (5) adaptive self-termination , the attacker deciding for itself that the victim has complied and ending the session, which an attacker told so by an in-loop evaluator, or one running a plan of fixed length, does not do. A dash marks an axis an attacker cannot be asked about.

Attacker	Attacker LLM Size	In-loop Evaluator LLM Size	# Adversarial Turns $\downarrow$	Total Runtime $\downarrow$	# API Requests (#/attempt)
Static Attackers
Jigsaw Puzzle ( Yang et al., 2024a )	GPT-4o-mini	-	10.9	5.5 min	$\geq 3$
ActorAttack ( Ren et al., 2025 )	GPT-4o	-	5.0	1.1 h	$5\sim 22$
SEMA ( Feng et al., 2026a )	8B	-	6.0	1.8 h	0
Interactive Attackers
Crescendo ( Russinovich et al., 2025 )	GPT-4o	GPT-4o	5.7	2.1 h	$2\times N_{\text{turns}}$

Appendix

Table 17: Inference cost of multi-turn attackers against GPT-oss-120B on HarmBench : the average number of adversarial turns, the total runtime to attack all its requests, and the API requests per attempt. A static attacker’s runtime sums planning and execution, and an interactive attacker’s is its interaction with the victim; the final evaluation is excluded from both. The victim is served locally, so only the attacker’s and its in-loop evaluator’s calls are API requests;

N_{\text{turns}}

is the number of adversarial turns of an attempt. SEMA, TRACE and SEMA-Auto are counter-trained against GPT-oss-120B. A dash marks an attacker without an in-loop evaluator. Bold marks the lowest in a column.

Attacker	AdvBench		HarmBench
	LLM Clf.	HarmB. Clf.	LLM Clf.	HarmB. Clf.
SEMA ( Feng et al., 2026a )	16.3	10.9	30.8	20.3
FITD ( Weng et al., 2025 )	37.0	37.5	28.8	26.5
TRACE ( He et al., 2026 )	7.5	3.9	8.4	6.1
TROJail ( Xiong et al., 2026 )	25.1	10.0	13.1	10.0
Crescendo ( Russinovich et al., 2025 )	6.0	7.6	6.1	10.7

Appendix

Table 18: Jailbreaks lost at the final turn

\downarrow

: among the trajectories that a judge counts as a jailbreak at some turn, the percentage whose final-turn response the same judge does not count as a jailbreak, averaged over the five victims of Table 5 .

Attacker	In-loop evaluator	Stops at	Agreement
Crescendo ( Russinovich et al., 2025 )	LLM Scorer (GPT-4o)	score $\geq 80$	82.8
X-Teaming ( Rahman et al., 2025 )	X-Teaming Judge (GPT-4o)	score $=5$ , the top score	92.0

Appendix

Table 19: Self-agreement of an in-loop evaluator

\uparrow

: of the trajectories an attacker’s in-loop evaluator stopped, the percentage its own stop rule is still met on when it is called a second time on the same response, over the same five victims.

Adversarial Attacks on LLMs

Adversarial attacks on large language models probe how crafted inputs, internal changes, and agent interactions bypass safety or disrupt reliable behavior.

See all 355 papers

Jun 9, 2026

JailbreakOPT: Tool-Assisted Iterative Jailbreak Prompt Optimization

Jailbreak attacks expose persistent safety weaknesses in large language models (LLMs), but existing stateless single-turn methods face a trade-off: hand-crafted prompts are expressive but static, while iterative prompt optimization can adapt but often relies on low-level mutations that require many target queries. We propose JailbreakOPT, a tool-assisted framework for improving iterative single-turn jailbreak prompt optimization. JailbreakOPT organizes diverse atomic jailbreak prompts into an attack tool library and composes them through a unified intra-episode optimization abstraction to generate stronger standalone attack prompts. To reuse experience across attack episodes, JailbreakOPT further frames tool selection as a contextual bandit problem and applies contextual Thompson sampling to guide exploration and exploitation based on past outcomes. Experiments across multiple target LLMs and attack goals show that JailbreakOPT improves attack success rate (ASR) while reducing the number of attacks until success (No.A) compared with atomic single-turn attacks and existing iterative optimization baselines. This paper may contain offensive or harmful content.

Ge Shi, Jun Yin, Donglin Xie +3

University of California, Davis · The Renmin University of China · Independent Researcher +3

Adversarial Prompt Generation Large Language Model-Guided Optimization LLM Jailbreak Attacks

Ask PDF

May 10, 2026

MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks

Multi-turn jailbreaks exploit the ability of large language models to accumulate and act on conversational context. Instead of stating a harmful request directly, an attacker can gradually steer the conversation toward an unsafe answer. Recent methods demonstrate this risk, but they are usually evaluated as black-box pipelines with different budgets, judges, retry rules, and strategy generation procedures. As a result, it is often unclear whether reported gains reflect stronger attack mechanisms or different experimental conditions. We introduce MT-JailBench, a modular evaluation framework for benchmarking multi-turn jailbreaks under fixed conditions. MT-JailBench implements each attack as five interacting modules: evaluation function, attack strategy, prompt generation, prompt refinement, and flow control. This design enables fair comparison across attack methods and component-wise analysis of what drives attack success. Using MT-JailBench, we find that resource budgets and evaluation functions are major confounders: controlling turns, retries, interactions, sampled strategies, and judges substantially change the ranking of attacks. At the component level, prompt generation accounts for most performance variation, while refinement and flow control provide moderate gains. We also find that explicit dynamic strategy generation is not always necessary; stochastic sampling from a fixed strategy can rival more elaborate diversification mechanisms. Finally, recomposing the best components yields a strong attack configuration that outperforms its source attacks and generalizes across diverse target LLMs. MT-JailBench therefore provides a modular framework for comparing multi-turn jailbreaks, understanding the impact of components, and guiding stronger red-teaming evaluations.

Xinkai Zhang, Zhipeng Wei, Huanli Gong +4

1UC Berkeley · 3International Computer Science Institute · 2UC Riverside +1

Language Model Safety Evaluation Jailbreak Attacks LLM Jailbreak Attacks Multi-Turn Jailbreak Attacks Jailbreak Attack Evaluation

Ask PDF

May 31, 2026

D-Judge: Disrupting Multi-Turn Jailbreaks using Semantics-Preserving Output Rewriting

Multi-turn jailbreak attacks pose a growing threat to large language model (LLM) safety because they exploit feedback from auxiliary judge models to iteratively refine prompts toward harmful goals. Existing defenses largely detect or block unsafe content at individual turns or at the final response, leaving the judge-driven refinement loop intact and allowing attackers to extract informative feedback from intermediate interactions. We introduce D-Judge, a semantics-preserving output rewriting defense that intervenes directly in this loop by rewriting the victim LLM's responses before they are evaluated by the attacker's judge. By misaligning the judge's feedback signal without changing the meaning of the original response, D-Judge derails the attacker's prompt-refinement process, causing subsequent queries to be optimized against a distorted signal of attack progress. To improve D-Judge's ability to produce such rewrites, we construct a dataset of semantically equivalent response pairs that induce different judge-assigned harmfulness scores, and use it for supervised fine-tuning followed by direct preference optimization. Experiments on HarmBench show that D-Judge reduces the success rate of state-of-the-art multi-turn jailbreaks while preserving performance on benign benchmarks.

Huanli Gong, Zhipeng Wei, Yu Fu +4

University of California, Berkeley · International Computer Science Institute · University of California, Riverside +1

LLM Guardrails Multi-Turn Jailbreak Attacks

Ask PDF

Abstract

Adversarial Attacks on LLMs

Explore similar work

JailbreakOPT: Tool-Assisted Iterative Jailbreak Prompt Optimization

MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks

D-Judge: Disrupting Multi-Turn Jailbreaks using Semantics-Preserving Output Rewriting