cs.CRNov 15, 2026

Attacker Diversity Reduction in Adaptive LLM Red Teaming

Authors: Chris Tava

Abstract

Reinforcement learning can make automated red-teaming more successful while narrowing the attacks it produces. Co-training an attacker and a defender with group-relative policy optimization (GRPO) has been reported to reduce attacker diversity. This experiment reproduces attack diversity reduction using three runs with a Qwen2.5-14B attacker and a Phi-4-mini defender trained by direct preference optimization (DPO). The attacker receives a success-only reward and uses a length-unbiased loss, bounded updates, warm-up and ratio clipping. The attacker is evaluated on 26 AdvBench behaviors. Training ended after 12 rounds, following a decision based on development checkpoints, so the reported intervals are nominal. Attack Diversity Retention (ADR) compares the within-behavior of the trained attacker spread with the untrained attacker. In order for an attack diversity to be consider as having been reduced, a criterion requires the ADR to be below a 0.75 target with a 95% confidence interval together with continued attacker effectiveness. Therefore, if an attacker narrows its attack vector by 25% consistently its considered less diverse. After 24 attacker and 12 defender LORA-based updates, ADR for the three runs were 0.63, 0.61 and 0.63, respectively. Each run met the criterion at round 12; the earlier checks at rounds 4 and 8 fell short. Despite attacks being less diverse, attack success continued to improve as the attacker was updated after each training run.

Figures & tables

Explore similar work

CardsList
  1. Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO

    Jun 8, 2026Blake Bullwinkel, Eugenia Kim, Amanda Minnich +1Adversarial TrainingAdversarial Attacks on LLMs

  2. CoFiBreak: A Coarse-to-Fine Diversity Reward for Effective Multi-Turn Jailbreak

    Oct 29, 2026Jialin Song, Xiaodong Liu, Weiwei Yang +7LLM Red TeamingJailbreak Attacks

  3. Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models

    Jun 8, 2025Ren-Jian Wang, Ke Xue, Zeyu Qin +7LLM Red TeamingAdversarial Prompt Generation