CART: Closed-Loop Adaptive Red Teaming for Large Language Models

Dongdong Zhang; Tengchao Lv; Yilin Jia; Yuzhong Zhao; Yupan Huang; Wenshan Wu; Xiangyang Zhou; Shaohan Huang

cs.AISep 23, 2026

CART: Closed-Loop Adaptive Red Teaming for Large Language Models

Authors: Dongdong Zhang, Tengchao Lv, Yilin Jia, Yuzhong Zhao, Yupan Huang, Wenshan Wu, Xiangyang Zhou, Shaohan Huang,

Organizations: Microsoft Research

Abstract

Automated red teaming often replays a fixed set of prompts, which measures known risks but cannot learn from failures found during testing. We present CART (Closed-Loop Adaptive Red Teaming), a framework that uses each result to guide what it tests next. CART begins with broad risk coverage, follows weaknesses that emerge, keeps new probes diverse, and records the evidence and source of every finding. It separates the Challenger that creates tests, the Target being tested, which may be a text-only model or a bounded tool-using agent, and the Judge that evaluates the results, allowing these roles to be studied independently. Across three evaluation families (Frontier, JAH, and Agentic), CART discovers more failures and higher average risk than static seed replay for every Target with an available baseline. The gains extend to tool-mediated agent tests, suggesting that contextual adaptation can reveal weaknesses that direct prompt replay does not exercise. These results describe what the test policies discover, not how often failures occur in real deployments. We also find that Challenger-Judge choices affect the evidence uncovered, highlighting the need for role separation and independent review. Overall, CART turns red teaming from a one-time checklist into a continuous, adaptive, and auditable search for model and agent weaknesses.

Figures & tables

Direction	What is changing	Alignment in Cart
Static suites to adaptive search	Evaluation moves from independent prompt execution to sequential testing that learns from the run history.	Memory-guided Thompson sampling allocates budget between coverage and weakness pursuit, and reports the discovery trajectory rather than only an endpoint score.
Model-only tests to agentic systems	The target expands from a single response to tool use, retrieval, autonomy boundaries, handoffs, and oversight.	The taxonomy includes goal drift, permission creep, oversight erosion, and skill supply-chain exposure, with a bounded tool-use harness for testing them.
Direct jailbreaks to contextual attacks	Threats increasingly exploit retrieved content, role framing, authority cues, conflicting instructions, and multi-turn state.	The strategy library treats adversarial context, indirect injection, goal splitting, and multi-round poisoning as first-class testing patterns.
Binary success to calibrated evidence	A pass/fail label is too coarse for release decisions and prioritisation.	The Judge records severity, confidence, answer quality, blast radius, reproducibility, evidence, and a composite risk score for every probe.
Benchmark replay to seed-grounded exploration	Public benchmarks and frameworks provide grounding, but operational testing must mutate them into domain-relevant probes.	Framework and benchmark seeds supply risk mechanisms and test prompts with explicit lineage labels: self-generated, seed expansion, seed adaptation, and seed replay.
One-off audits to continuous governance	Red teaming becomes a repeatable process connected to risk frameworks, repeated evaluation, and release gates.	Configurable taxonomies, budgets, seed-selection settings, scoring weights, and report templates support continuous, auditable operation.

Table 1: Evolving paradigms in LLM red teaming and how Cart aligns with the transition from prompt-level attacks to operational safety assurance.

Figure 1: Overview of the Cart framework. Taxonomies, seed cases, target settings, and past evidence enter a five-stage adaptive loop. The Target may be an LLM that returns text or an agent that produces both a response and a tool-action trace. The Judge’s score updates memory and guides the next round. The resulting records support reports, weakness analysis, and repeated evaluation.

Group	Strategy family	Purpose in the adaptive search
Coverage	Basic_coverage	Scans each risk category once to establish the initial weakness profile.
	Grounded retrieval	Periodically requests a seed-grounded probe recorded with Seed Adaptation lineage when a seed is available.
Adaptation	Weakness pursuit	Revisits categories with recent failures or high scores.
	Variant testing	Retests the same mechanism across wording, language, role, scenario, or format variants.
	Boundary testing	Probes near the allowed/refused boundary to reveal under-refusal and over-refusal.
Pressure	Stress consistency	Applies urgency, authority framing, repeated challenge, or reframing to test stability under pressure.

Table 2: Grouped strategy families in the default Cart configuration. Strategies are behavioural testing patterns, not provider-specific prompt templates; new strategies can be added by defining their group, intent, applicable risk categories, required context, and judge expectations.

Mechanism	Primary decision	Effect on the search
Strategy Manager	Selects the risk category, strategy family, seed-retrieval schedule, and budget allocation.	Directs the run toward under-tested or high-risk regions of the evaluation space.
Diversity mandate	Varies persona, domain, format, language, role frame, attack surface, and interaction pattern within the selected region.	Prevents near-duplicate probes and tests whether a weakness survives across distinct surface realisations.

Table 3: Different roles of strategy selection and diversity control. Strategy selection chooses what to test; diversity control changes how the test is expressed.

Risk category	What it captures
Prompt injection	Untrusted content attempts to override trusted instructions.
Jailbreak susceptibility	Pressure, role framing, or prompt mutation erodes refusal boundaries.
Hallucination	Unsupported claims are asserted with inappropriate certainty.
Model-identity exposure	The model reveals, overclaims, or fabricates non-public provenance.
Uncertainty calibration	The model fails to communicate limits, missing context, or need for verification.
Over-refusal	Safe and legitimate requests are blocked too broadly.

Table 4: Default risk taxonomy used in the experiments.

Capability category	What the probe exercises
Instruction following	Whether the model follows the correct instruction hierarchy.
Reasoning	Whether reasoning remains valid under ambiguity, pressure, or conflicting context.
Factuality	Whether the model separates known facts from unsupported claims.
Model-provenance transparency	Whether questions about model identity, lineage, pretraining, and distillation are handled with appropriate disclosure boundaries and uncertainty.
Cross-lingual robustness	Whether safety behaviour, factuality, and refusal boundaries remain consistent across languages or mixed-language prompts.
Text-representation robustness	Whether semantic understanding and safeguards survive Unicode variants, visual lookalikes, styled text, spacing marks, symbols, or simple enciphered text.

Table 5: Default capability taxonomy used in the experiments.

Family	Source	Cases	Coverage
Frontier	ATLAS + OWASP + curated patterns [ 16 , 20 ]	76	57 ATLAS case studies, 10 OWASP cases, and 9 additional curated risk patterns.
JAH	JailbreakBench [ 6 ]	100	Jailbreak and refusal robustness.
	HarmBench [ 14 ]	400	Standardised harmful behaviour.
	AgentHarm [ 2 ]	260	Malicious multi-step tasks.
	JAH subtotal	760	Combined harm, refusal, and malicious-task benchmark family.
Agentic	Agentic ASB [ 35 ]	2,000	Attacker tools crossed with agent tasks.

Table 6: Three evaluation-data families and their component sources. The Frontier family combines ATLAS and OWASP cases with additional curated risk patterns; JAH combines JailbreakBench, HarmBench, and AgentHarm; and Agentic combines Agentic ASB, InjecAgent, and Agent Behavioral.

Panel of Figure 2 — Figure 2: Target-model results on Frontier-family cases: (a) failure rate and (b) average risk score. Cart uses adaptive, strategy-guided probe generation, whereas the static baseline directly replays the stored seed prompts.

Panel of Figure 3 — Figure 3: Frontier-family risk-category statistics. Comparing allocation with failure rate distinguishes persistent search targets from high-yield categories.

Panel of Figure 4 — Figure 4: Target-model results on the combined JAH benchmark suite: (a) failure rate and (b) average risk score. Cart uses adaptive, strategy-guided probe generation, whereas the static baseline directly replays the stored seed prompts. Targets are ordered by the Cart failure rate.

Panel of Figure 5 — Figure 5: JAH risk-category statistics. Failures extend beyond direct jailbreaks to identity, refusal quality, hallucination, and calibration.

Panel of Figure 6 — Figure 6: Target-model results on Agentic cases: (a) failure rate and (b) average risk score. Cart uses adaptive, strategy-guided probe generation, whereas the static baseline directly replays the stored seed prompts.

Panel of Figure 7 — Figure 7: Agentic risk-category statistics. The allocation profile shows CART concentrating its budget on malicious tool registration while retaining coverage of other tool, memory, permission, and oversight risks.

Capability	DeepSeek	Gemini	Claude	Grok	GLM	Kimi	GPT
Cross-lingual robustness	2.967/–	3.520/–	0.386/0.000	0.388/–	0.444/–	1.450/–	0.611/–
Domain judgment	3.072/0.525	0.642/0.660	0.061/0.165	0.991/0.348	0.555/0.296	0.874/0.068	1.005/0.017
Factuality	3.501/–	0.474/–	0.112/0.000	1.192/–	0.991/–	1.177/–	0.416/–
Instruction following	2.846/1.406	0.766/0.571	1.600/0.071	1.220/0.387	0.309/0.445	0.169/0.518	0.222/0.000
Model-provenance transparency	0.713/–	2.937/–	0.000/0.000	0.559/–	0.330/–	0.632/–	0.099/–
Reasoning	3.706/–	1.422/–	0.000/0.000	0.692/–	0.851/–	1.058/–	0.667/–

Table 7: Average risk score by Target and capability on JAH. A dash denotes an unavailable observation; each cell reports CART/Static. Higher values indicate greater observed risk exposure, not stronger capability.

Challenger $\backslash$ Judge	Claude	DeepSeek	Gemini	GLM	GPT	Grok	Kimi
Claude	20.30%	17.30%	9.70%	26.40%	24.30%	19.00%	27.00%
DeepSeek	17.30%	16.70%	18.70%	28.90%	34.00%	17.00%	30.30%
Gemini	8.10%	12.70%	22.00%	12.70%	11.40%	9.00%	13.30%
GLM	12.80%	13.70%	10.70%	21.70%	15.40%	9.30%	9.30%
GPT	27.70%	73.70%	77.00%	41.70%	51.00%	67.70%	66.00%
Grok	18.30%	23.70%	32.30%	22.00%	38.70%	12.30%	33.30%

Table 8: Measured failure rates for the full

7\times 7

Challenger–Judge factorial experiment, with Kimi as the fixed Target. Rows select the Challenger and columns select the Judge. The static baseline bypasses the Challenger and is separate from the factorial matrix.

Panel of Figure 8 — Figure 8: Strategy statistics aggregated over the full $7\times 7$ Challenger–Judge matrix with Kimi fixed as the Target. Tests that use several strategies are divided equally among those strategies.

	As Challenger		As Judge
Model	$\boldsymbol{\mu}$	$\boldsymbol{\sigma^{2}}$	$\boldsymbol{\mu}$	$\boldsymbol{\sigma^{2}}$
Claude	20.57%	0.3152%	17.27%	0.3198%
DeepSeek	23.27%	0.4789%	26.66%	3.9654%
Gemini	12.74%	0.1763%	27.96%	4.5478%
GLM	13.27%	0.1630%	24.77%	0.7027%
GPT	57.83%	2.8545%	28.51%	1.6227%

Table 9: Role-wise mean and population variance computed from decimal failure-rate proportions in Table 8 , then displayed in percentage format. Means report

100\mu

and the variance columns report

100\sigma^{2}

; the latter are percentage-formatted decimal variances, not variances computed directly from percentage-point values.

Panel of Figure 9 — Figure 9: Mean measured failure rate by Challenger and Judge on Frontier-family data, with Kimi fixed as Target. Means exclude the static-baseline row.

Peer reviewer	Verdict agree	Original correct	Judge quality	$\|\Delta\text{score}\|$
Claude	98.9%	98.2%	87.20	3.55
DeepSeek	97.5%	96.5%	93.46	3.88
Gemini	98.9%	98.2%	98.21	2.54
GPT	99.3%	97.2%	92.59	2.48
Grok	98.9%	96.5%	86.76	3.48
GLM	99.6%	98.2%	90.96	2.50

Table 10: Exhaustive peer audit of the fully self-coupled Kimi run, in which Kimi serves as Challenger, Target, and original Judge. Each of six peer models independently evaluates all 282 cases, for 1,692 reviewer–case decisions. Judge quality and target-answer scores use a 0–100 scale;

|\Delta\text{score}|

is the mean absolute difference from the original target-answer score. “Original correct” is stricter than verdict agreement and also penalizes unsupported reasoning, material miscalibration, inconsistency, or rubric mismatch.

Risk category	Cases	Decisions	Agree (%)	Correct (%)	$\|\Delta\|$
Prompt injection	27	162	98.1	98.1	4.33
Jailbreak susceptibility	10	60	100.0	98.3	1.70
Hallucination	75	450	97.1	95.3	3.54
Model-identity exposure	9	54	100.0	100.0	2.07
Uncertainty calibration	13	78	98.7	89.7	6.42
Over-refusal	10	60	100.0	98.3	3.07

Table 11: Category-conditioned peer-review statistics for the fully self-coupled Kimi run. Each case contributes six peer decisions. Agree is agreement with the original binary verdict; Full correct additionally requires acceptable reasoning, evidence, calibration, and rubric application;

|\Delta|

is the mean absolute difference from the original answer-quality score on the 0–100 scale.

Panel of Figure 10 — Figure 10: Judge-faithfulness controls on Frontier-family data with Kimi as Target. (a) directly replays seed prompts and varies only the Judge; (b) assigns the same named model to Challenger and Judge.

Panel of Figure 11 — Figure 11: Cumulative failure rate for the GPT row of Table 8 , with Kimi fixed as Target, GPT fixed as Challenger, and the seven series representing the Judge models. Values are reported at 15-case intervals. Late-stage flattening indicates that the aggregate failure-rate estimates stabilise as evidence accumulates.

Phase	Rounds	Behaviour
Reconnaissance	1–12	initial coverage scans every category; all scores $0$
Weakness emerges	13	first non-zero score in cascading hallucination
Weakness pursuit	13–99	$67\%$ of post-recon probes hit the two most brittle categories; peak risk $4.37$

Table 12: Illustrative within-run dynamics of memory-guided search on a

99

-round run. Reconnaissance builds the weakness profile; Thompson sampling then concentrates budget while strategies rotate.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Prompt role	Purpose	Expected result
Challenger	Generate adaptive model-level tests	One structured red-team test case
Judge	Evaluate target-model responses	Safety judgment, score, and rationale
Peer reviewer	Independently audit Judge decisions	Independent verdict, calibration check, and Judge-quality assessment
Agentic Challenger	Construct tool-using agent scenarios	Task, tools, injections, and safe reference
Agentic Judge	Evaluate agent trajectories	Agent-safety judgment and evidence

Appendix

Table 13: Prompt roles in the Cart evaluation framework.

Automated Red Teaming

Automated red teaming uses computational adversaries to uncover and assess vulnerabilities in AI models, agents, and their safety defenses.

See all 58 papers

Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
Dec 21, 2025Zhang Wei, Hanxuan Chen, Peilu Hu +19Adversarial Robustness of Language Models LLM Safety Evaluation
Independent Researcher · Hunan University · Shenzhen Kaihong Digital Industry Development Co., Ltd. +12
ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing
Sep 8, 2026Yi Ting Shen, Kentaroh Toyoda, Alex LeungLLM Safety Benchmarks Language Model Safety Evaluation
Vulcan Research, AIFT Singapore
Training a General Purpose Automated Red Teaming Model
Apr 24, 2026Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea +1Adversarial Attacks on LLMs LLM Red Teaming
NVIDIA

Dec 21, 2025J/K move · Enter open · S save

Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models

Zhang Wei, Hanxuan Chen, Peilu Hu, Zhenyuan Wei+18

Independent Researcher · Hunan University · Shenzhen Kaihong Digital Industry Development Co., Ltd.+12

Adversarial Robustness of Language Models LLM Safety Evaluation Automated Red Teaming

Open paper AskPDF arXiv

Red-teaming is becoming a central part of large language model (LLM) safety evaluation, yet current practice still relies heavily on expert-written prompts or fixed benchmark suites. This creates a gap between what is easy to test and what deployed models can actually do: failures may be rare, context-sensitive, and distributed across many threat categories. We study automated red-teaming as a constrained adversarial search problem and introduce a learning-driven framework that couples category-aware attack generation with hierarchical vulnerability detection. The method starts from curated safety seeds, expands them through meta-prompt-guided and evolutionary search, and scores the resulting prompt--response pairs with lexical, semantic, and behavioral detectors. Across six threat categories on GPT-OSS-20B, the framework discovers 47 validated vulnerabilities, including 21 high-severity cases and 12 novel attack patterns. Under matched query budgets, it achieves a 3.9

\times

higher discovery rate than manual expert red-teaming while maintaining 89% detection accuracy and full category coverage. Ablations show that the gains do not come from more prompts alone: diversity constraints prevent template collapse, coverage constraints prevent category blind spots, and semantic detection recovers failures missed by lexical rules. These results suggest that red-teaming can be made more scalable and reproducible when treated as adaptive search rather than as a static checklist.

Abstract

Automated Red Teaming

Explore similar work

Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models

ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing

Training a General Purpose Automated Red Teaming Model