Attacker Diversity Reduction in Adaptive LLM Red Teaming
Abstract
Reinforcement learning can make automated red-teaming more successful while narrowing the attacks it produces. Co-training an attacker and a defender with group-relative policy optimization (GRPO) has been reported to reduce attacker diversity. This experiment reproduces attack diversity reduction using three runs with a Qwen2.5-14B attacker and a Phi-4-mini defender trained by direct preference optimization (DPO). The attacker receives a success-only reward and uses a length-unbiased loss, bounded updates, warm-up and ratio clipping. The attacker is evaluated on 26 AdvBench behaviors. Training ended after 12 rounds, following a decision based on development checkpoints, so the reported intervals are nominal. Attack Diversity Retention (ADR) compares the within-behavior of the trained attacker spread with the untrained attacker. In order for an attack diversity to be consider as having been reduced, a criterion requires the ADR to be below a 0.75 target with a 95% confidence interval together with continued attacker effectiveness. Therefore, if an attacker narrows its attack vector by 25% consistently its considered less diverse. After 24 attacker and 12 defender LORA-based updates, ADR for the three runs were 0.63, 0.61 and 0.63, respectively. Each run met the criterion at round 12; the earlier checks at rounds 4 and 8 fell short. Despite attacks being less diverse, attack success continued to improve as the attacker was updated after each training run.
Figures & tables
| Setting | Value |
|---|---|
| Attacker | Qwen2.5-14B-Instruct [ 10 ] , low-rank adaptation (LoRA) [ 11 ] rank 16, |
| Defender | Phi-4-mini-instruct [ 12 ] , LoRA rank 16, , direct preference optimization (DPO) [ 13 ] |
| Judge | HarmBench-Llama-2-13b classifier [ 14 ] |
| Embedder | bge-large-en-v1.5 [ 15 ] |
| Attacker optimizer | AdamW [ 16 ] , no weight decay, learning rate , gradient-norm clip 1.0, eight rows per step |
| Kullback–Leibler (KL) weight | 0.005 to the base model |
| Change | What it does | Why |
|---|---|---|
| Length-unbiased loss | Each completion token loss is divided by the constant (Eq. ( 4 )). The group-mean baseline and standard-deviation scaling is retained. Liu et al. [ 8 ] also remove the scaling and updates to the model keeps it. | A per-completion mean gives tokens in long responses less weight and biases GRPO toward longer responses, especially incorrect ones. A constant normalizer removes this weighting bias [ 8 ] . |
| Bounded update | A job with fewer than 64 signal rows is recorded as skipped. A larger job uses a seeded uniform subsample of at most 512 rows, visited once in groups of eight: at most 64 optimizer steps. | Uniform-reward groups have zero advantage [ 9 ] , leaving few rows to carry an update. The floor avoids training on a handful of rows; the cap limits optimization per rollout. |
| Warm-up | The learning rate rises linearly over the first four optimizer steps of a job and then stays at . | A job initializes its own AdamW state [ 16 ] . Warm-up reduces the variance of early adaptive steps [ 18 ] . |
| Ratio clipping | The surrogate clips the probability ratio of each token against the starting policy for the job to . The KL term uses the base model (Eq. ( 4 )). | The surrogate limits incentives for large probability changes across minibatch steps [ 19 , 7 ] . It is applied throughout the job sequence of up to 64 steps. |
| Round | Run | ADR (95% CI) | Length-matched ADR (95% CI) | ASR@1: untrained check | Over-refusal increase | Shows reduction |
|---|---|---|---|---|---|---|
| 4 | 1 | 1.08 (0.99–1.18) | 1.29 (1.17–1.39) | 0.04 0.42 | +0.000 | no: ADR above 0.75 |
| 4 | 2 | 0.84 (0.77–0.93) | 0.86 (0.75–0.97) | 0.00 0.04 | +0.000 | no: ADR above 0.75; ASR gain 0.04 |
| 4 | 3 | 0.79 (0.71–0.87) | 0.99 (0.89–1.10) | 0.00 0.35 | +0.000 | no: ADR above 0.75 |
| 8 | 1 | 0.67 (0.59–0.754) | 0.755 (0.65–0.87) | 0.04 0.92 | +0.018 | no: upper bound 0.754 |
| 8 | 2 | 0.77 (0.69–0.88) | 0.89 (0.80–1.01) | 0.00 0.38 | +0.036 | no: ADR above 0.75 |
| 8 | 3 | 0.79 (0.70–0.89) | 0.99 (0.86–1.12) | 0.00 0.58 | +0.000 | no: ADR above 0.75 |
| Family (%) | Untrained | First 3 | Latest 3 |
|---|---|---|---|
| successes ( ) | 20 | 541 | 5,616 |
| hypothetical framing | 0.0 | 14.4 | 63.2 |
| format transformation | 65.0 | 53.2 | 15.7 |
| instruction override | 30.0 | 16.8 | 7.7 |
| contextual manipulation | 0.0 | 10.4 | 11.1 |
| role play | 0.0 | 3.9 | 2.2 |
| Quantity | Run 1 | Run 2 | Run 3 |
|---|---|---|---|
| ADR at the last checkpoint (95% CI) | 0.64 (0.57–0.72) | 0.60 (0.54–0.68) | 0.62 (0.56–0.69) |
| Checkpoints with ADR below 0.75: value / whole interval (of 24) | 10 / 4 | 8 / 3 | 10 / 4 |
| Behaviors less diverse than untrained / with ratio below 0.75 (of 25) | 22 / 16 | 24 / 19 | 24 / 20 |
| Diversity, untrained last (mean cosine distance) | 0.221 0.143 | 0.228 0.137 | 0.235 0.143 |
| ASR@1, untrained last | 0.04 0.73 | 0.00 0.58 | 0.00 0.81 |
| ASR@8, untrained last | 0.15 1.00 | 0.23 1.00 | 0.19 1.00 |
Automated Red Teaming
Automated red teaming uses computational adversaries to uncover and assess vulnerabilities in AI models, agents, and their safety defenses.
See all 66 papers