$M^3$: A Multi-LLM System for Multi-Turn Jailbreaking as a Markov Decision Process

Yuqing Huang; Jack Ji; Jithin Pradeep; Sagarika Suresh; Jekaterina Novikova

cs.AIOct 22, 2026

$M^3$ : A Multi-LLM System for Multi-Turn Jailbreaking as a Markov Decision Process

Authors: Yuqing Huang, Jack Ji, Jithin Pradeep, Sagarika Suresh, Jekaterina Novikova

Organizations: The Vanguard Group∗

Abstract

Modern Large Language Models (LLMs) remain vulnerable to carefully orchestrated multi-turn interactions that progressively bypass safety mechanisms. However, many existing red-teaming approaches rely on heuristics or impose strong assumptions that limit their applicability in fully black box settings. We address this gap by introducing $M^3$ , an automated red-teaming framework that formalizes multi-turn jailbreaking as a finite-horizon Markov Decision Process (MDP). In $M^3$ , an attacker LLM interacts with a target LLM while operating over a compact internal state produced by a state LLM. Across two benchmarks and six target models, $M^3$ consistently outperforms state-of-the-art multi-turn red-teaming methods in Attack Success Rates (ASR), achieving improvements of up to 53% in ASR@5. Even with lightweight attacker, $M^3$ attains gains up to 26% in ASR@1 and 34% in ASR@5, compared to baselines that use higher capacity attackers. Beyond improved effectiveness, $M^3$ exhibits steeper risk escalation over conversational turns and achieves more consistent performance across risk categories (up to 4.82 improvement in Gini index) without relying on predefined risk taxonomy, providing a scalable and principled framework for multi-turn red-teaming and safety evaluation.

Figures & tables

Figure 1: $M^{3}$ framework (left) and an example representing the process (right). At each turn $i$ , an attacker generates action $A_{i}$ and interacts with a target. The state maintainer updates a compact state $S_{i}$ and the evaluator produces reward $R_{i}$ ensuring risk escalation over turns.

Figure 2: Progression of attack success measured by ASR@5 over conversational turns for GPT - 4–family target LLMs on HarmBench . Lines show ASR@5, with shaded regions indicating standard error across tasks. Solid lines with darker shading denote GPT-4o mini , while dashed lines with lighter, hatched shading denote GPT-4o .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: ASR@5 progression over conversational turns for GPT - 4 family target LLMs on Crescendo (left) and HarmBench (right). Lines show ASR@5, with shaded regions indicating standard error across tasks. Solid lines with darker shading denote GPT-4o mini , while dashed lines with lighter, hatched shading denote GPT-4o .

Panel of Figure 4 — Figure 4: ASR@5 progression over conversational turns for Claude family target LLMs on Crescendo (left) and HarmBench (right). Lines show ASR@5, with shaded regions indicating standard error across tasks. Solid lines with darker shading denote Claude 3 Haiku , while dashed lines with lighter, hatched shading denote Claude 3.5 Sonnet .

Panel of Figure 5 — Figure 5: ASR@5 progression over conversational turns for Llama family target LLMs on Crescendo (left) and HarmBench (right). Lines show ASR@5, with shaded regions indicating standard error across tasks. Solid lines with darker shading denote Llama3-8B-IT , while dashed lines with lighter, hatched shading denote Llama3-70B-IT .

Figure 6: Performance across risk categories for ActorAttack-LC and ActorAttack-HC , measured as ASR@5 $\uparrow$ across 6 target models ( GPT-4o , GPT-4o mini , Claude 3.5 Sonnet , Claude 3 Haiku , Llama3-70B-IT , Llama3-8B-IT ) and 12 risk categories. Higher scores/darker fields indicate better jailbreaking performance of ActorAttack .

Figure 7: Performance across risk categories for GOAT-LC and GOAT-HC , measured as ASR@5 $\uparrow$ across 6 target models ( GPT-4o , GPT-4o mini , Claude 3.5 Sonnet , Claude 3 Haiku , Llama3-70B-IT , Llama3-8B-IT ) and 12 risk categories. Higher scores/darker fields indicate better jailbreaking performance of GOAT .

Figure 8: Performance across risk categories for Crescendo-LC and Crescendo-HC , measured as ASR@5 $\uparrow$ across 6 target models ( GPT-4o , GPT-4o mini , Claude 3.5 Sonnet , Claude 3 Haiku , Llama3-70B-IT , Llama3-8B-IT ) and 12 risk categories. Higher scores/darker fields indicate better jailbreaking performance of Crescendo .

Figure 9: Performance across risk categories for $M^{3}$ -LC and $M^{3}$ -HC , measured as ASR@5 $\uparrow$ across 6 target models ( GPT-4o , GPT-4o mini , Claude 3.5 Sonnet , Claude 3 Haiku , Llama3-70B-IT , Llama3-8B-IT ) and 12 risk categories. Higher scores/darker fields indicate better jailbreaking performance of $M^{3}$ .

Adversarial Attacks on LLMs

Adversarial attacks on large language models probe how crafted inputs, internal changes, and agent interactions bypass safety or disrupt reliable behavior.

See all 355 papers

ATTCK-R: Knowledge-Aware Long-Horizon Jailbreaking for Protected LLMs
Oct 12, 2026Md Mushfiqur Rahman, Tobias Rohde, Rahul Tripathi +2LLM Jailbreak Attacks LLM Red Teaming
George Mason University · Amazon
AMT-X: Phase-Structured Multi-Turn Red-Teaming with Checklist-Gated Evaluation
Jul 13, 2026Yi Ting Shen, Kentaroh Toyoda, Alex LeungLLM Safety Benchmarks Adversarial Attacks on LLMs
Vulcan Research, AIFT, Singapore
MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks
May 10, 2026Xinkai Zhang, Zhipeng Wei, Huanli Gong +4Language Model Safety Evaluation Jailbreak Attacks
1UC Berkeley · 3International Computer Science Institute · 2UC Riverside +1

Oct 12, 2026J/K move · Enter open · S save

ATTCK-R: Knowledge-Aware Long-Horizon Jailbreaking for Protected LLMs

Md Mushfiqur Rahman, Tobias Rohde, Rahul Tripathi, Khalil Elkhalil+1

George Mason University · Amazon

LLM Jailbreak Attacks LLM Red Teaming Multi-Turn Jailbreak Attacks Jailbreak Attack Evaluation LLM Safety Evaluation Black-Box Adversarial Attacks

Open paper AskPDF arXiv

As large language models are deployed behind increasingly sophisticated safety mechanisms, evaluating their robustness under realistic adversarial conditions becomes increasingly difficult and critical. Existing jailbreaking evaluations often rely on static prompts, limited interaction horizons, or lightly protected systems, failing to reflect protected deployment settings with prompt-based safety protections. We show that the introduction of a prompt-based safety protection reduces the average attack success rate of a strong multi-turn baseline from 0.896 to 0.608 (32.1% relative), demonstrating that unprotected evaluations substantially overestimate real-world vulnerabilities. We propose Adaptive Traceback and Tactic Curation for Knowledge-based Red-teaming (ATTCK-R), a planning-driven multi-turn red teaming framework designed for protected black-box systems. ATTCK-R models jailbreaking as a long-horizon trajectory optimization problem using a structured belief state, introduces optimization-based tactic curation to abstract reusable strategies from prior successes and failures, and employs informed traceback to resume attacks from strategically meaningful states rather than restarting. Evaluated across 9 widely used LLMs configured with a single prompt-based safety protection layer, ATTCK-R consistently outperforms strong recent multi-turn jailbreaking frameworks. Ablation results confirm that long-horizon planning and tactic curation improve attack effectiveness, while adaptive traceback improves efficiency through structured recovery from conversational failure.

Abstract

Adversarial Attacks on LLMs

Explore similar work

ATTCK-R: Knowledge-Aware Long-Horizon Jailbreaking for Protected LLMs

AMT-X: Phase-Structured Multi-Turn Red-Teaming with Checklist-Gated Evaluation

MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks