There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning

Roman Belaire; Arunesh Sinha; Pradeep Varakantham

cs.LGAug 6, 2025

There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning

Authors: Roman Belaire, Arunesh Sinha, Pradeep Varakantham

Organizations: Singapore Management University · Rutgers University

Abstract

Red teaming is essential for securing Large Language Models, yet current automated methods remain limited by templates and single-turn attacks. To simulate the complex, interactive nature of real-world adversarial attacks, we introduce a novel red teaming paradigm designed to maximize expected cumulative harm through strategic interaction. By formalizing red teaming as a Markov Decision Process in a hierarchical reinforcement learning framework, we navigate the challenges of sparse rewards and long-horizon planning. Our generative agent learns diverse, multi-turn attacks using a token-level harm reward, consistently uncovering vulnerabilities that bypass baselines. This approach achieves a new state of the art and reframes LLM red teaming as a principled, trajectory-based process.

Figures & tables

Figure 1: An adversarial conversation between the red team agent and the target LLM. The dialogue state consists of alternating Target and Low-Level Policy utterances.

Figure 2: Overview of red teaming as an HRL problem. The high-level policy $\pi_{1}$ learns a strategy over a dialogue trajectory and provides guides to the low-level policy. The low-level policy $\pi_{2}$ generates the action to send to the target model based on the state and guide.

Method	Llama-3.1-8B-Instruct				GPT-5.4-mini
	Myopic		Context-Aware		Myopic		Context-Aware
	$\uparrow$ ASR	$\downarrow$ TtS	$\uparrow$ ASR	$\downarrow$ TtS	$\uparrow$ ASR	$\downarrow$ TtS	$\uparrow$ ASR	$\downarrow$ TtS
LLM-HL (Ours)	75.2	3.66	62.5	3.8	35.0	2.52	41.0	4.68
Ferret [ 9 ]	31.25	3.87	23.8	4.45	5.0	4.75	0.0	-
WildTeaming [ 22 ]	65.0	3.53	10.3	4.36	4.5	4.66	0.0	-
HARM [ 55 ]	10.2	4.97	17.5	4.97	24.7	4.9	20.0	4.95

Table 1: Our method outperforms baselines on Harmbench data, attacking open source and closed source reasoning models, under few-shot (ASR@5) settings described in Section 5 . We report the average turns to success (TtS); methods with zero success have TtS of nil (-).

Method	Llama-3.1-8b-Instruct			Llama-3.1-70b-Instruct			GPT-4o	——–Average——–
	$\uparrow$ HB	$\uparrow$ WB	$\uparrow$ JB	$\uparrow$ HB	$\uparrow$ WB	$\uparrow$ JB	$\uparrow$ HB	$\downarrow$ PPL	$\downarrow$ TtS	$\uparrow$ Div
LLM-HL (Ours)	95.0	76.0	77.5	86.5	78.0	66.0	55.0	13.3	4.1	0.92
Template-HL (Ours)	96.0	74.5	75.0	89.1	72.0	63.0	43.5	7.2	4.4	0.85
Low-level Only (Ours)	35.0	27.0	19.5	24.5	18.5	13.0	12.5	19.3	5.8	0.55
Ferret [ 9 ]	82.5	63.0	68.5	81.7	39.5	58.0	18.7	12.3	7.5	0.58
Wildteaming [ 22 ]	76.1	77.5	40.0	45.7	45.0	22.0	15.0	8.1	6.2	0.65

Table 2: Experiments in a 30-step, context-aware setting (ASR@30) against open and closed-source models of varying size, using WildBench (WB), JailbreakBench (JB), and the validation set of Harmbench (HB). We also report mean perplexity (PPL), turns to success (TtS), and pairwise cosine distance for attack embeddings, i.e., Diversity (Div).

Table 3: Sensitivity testing to model architectures. In 3 3(a) , we ablate guard models used in training as the immediate reward. The ability of the red teamer is roughly correlated with the strength of the guard model. In 3 3(b) , we show the effect of freezing each level during training. While each frozen method outperforms baselines, indicating the advantages of hierarchy, the comprehensive method is required for SOTA results. We use Harmbench data and attack Llama-3.1-8b-Instruct in these tables.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Llama-3.1-8b-Instruct			Llama-3.1-70b-Instruct			Mixtral-8x22b			GPT-4o
	$\uparrow$ HB	$\uparrow$ WB	$\uparrow$ JB	$\uparrow$ HB	$\uparrow$ WB	$\uparrow$ JB	$\uparrow$ HB	$\uparrow$ WB	$\uparrow$ JB	$\uparrow$ HB
Template-HL (Ours)	97.0	74.5	75.0	89.1	72.0	63.0	90.0	79.5	66.5	43.5
Low-level Only (Ours)	35.0	27.0	19.5	24.5	18.5	13.0	38.0	29.0	21.5	12.5
LLM-HL (Ours)	97.0	76.0	77.5	87.0	78.0	66.0	90.0	82.5	69.5	55.0
Ferret [ 9 ]	82.5	63.0	68.5	81.7	39.5	58.0	50.5	37.0	46.0	18.7
Wildteaming [ 22 ]	76.1	77.5	40.0	45.7	45.0	22.0	55.0	61.5	27.5	15.0

Appendix

Table 4: Experimental results measuring ASR in a 30-step, context-aware setting (ASR@30) against open and closed source models covering a range of model sizes. Seed prompts are procured from WildBench (WB) [ 28 ] , JailbreakBench (JB) [ 5 ] , and the validation set of Harmbench (HB) [ 32 ] .

Figure 4: We track the transition matrix denoting the frequency that one action follows another to illustrate the existence of a distribution over strategies.

Panel of Figure 5 — Figure 5: Comparison of critic and policy model losses under different reward attribution methods.

LLM Red Teaming

Large language model red teaming probes how models and agents can be induced to violate safety, security, or operational constraints.

See all 59 papers

Training a General Purpose Automated Red Teaming Model
Apr 24, 2026Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea +1Adversarial Attacks on LLMs LLM Red Teaming
NVIDIA
ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing
Sep 8, 2026Yi Ting Shen, Kentaroh Toyoda, Alex LeungLLM Safety Benchmarks Language Model Safety Evaluation
Vulcan Research, AIFT Singapore
STAR-Teaming: A Strategy-Response Multiplex Network Approach to Automated LLM Red Teaming
Apr 21, 2026MinJae Jung, YongTaek Lim, Chaeyun Kim +3LLM Security LLM Jailbreak Attacks
DATUMO INC.

Apr 24, 2026J/K move · Enter open · S save

Training a General Purpose Automated Red Teaming Model

Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea, Christopher Parisien

NVIDIA

Adversarial Attacks on LLMs LLM Red Teaming LLM Post-Training Automated Red Teaming

Open paper AskPDF arXiv

Automated methods for red teaming LLMs are an important tool to identify LLM vulnerabilities that may not be covered in static benchmarks, allowing for more thorough probing. They can also adapt to each specific LLM to discover weaknesses unique to it. Most current automated red teaming methods are intended for tackling safety and content moderation. Thus, they make use of content safety models as evaluators and optimize for circumventing them, and as such, have not been tested with other adversarial intents not typically captured by these. We propose a pipeline for training a red teaming model that can generalize to arbitrary adversarial goals, including objectives it has not been directly trained on, and that does not depend on the existence of a pre-existing evaluator available at training time. We demonstrate that finetuning small models, such as Qwen3-8B, using this pipeline results in a substantial improvement in their ability to generate attacks for both in and out of domain adversarial goals.

Abstract

LLM Red Teaming

Explore similar work

Training a General Purpose Automated Red Teaming Model

ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing

STAR-Teaming: A Strategy-Response Multiplex Network Approach to Automated LLM Red Teaming