DACE: Diversity-Driven Adversarial Co-Evolution for Robust LLM Safety Alignment
Organizations: Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Fudan University
Abstract
Ensuring robust safety alignment of large language models (LLMs) is increasingly difficult as adversarial attacks evolve and outpace alignment pipelines built on pre-collected data. Existing co-evolutionary frameworks let the defender chase a moving attacker, yet their training dynamics expose two compounding pathologies: attack strategy collapse, where the attacker overfits to a narrow set of high-reward rewrites, and defense adversarial forgetting, where the defender loses competence against earlier attacks as the attack distribution drifts. Existing remedies are partial: attacker-side diversity rewards score textual novelty rather than strategy novelty, and defender-side replay relies on discrete judge scores that ignore estimation uncertainty, response stochasticity, and defender non-stationarity. We introduce \method, a diversity-driven adversarial co-evolution framework that targets both pathologies jointly. On the attacker side, \method couples an explicit two-dimensional strategy space (risk category attack style) with a normalized marginal coverage gain reward, providing a bounded, non-vanishing exploration signal at the strategy layer. On the defender side, \method maintains a unified Bayesian adversarial replay pool whose Beta--Bernoulli threat posteriors are refreshed by time decay and sampled via Thompson sampling, anchoring defender training to an evolving threat landscape. Across standard safety, automated-attacker, and general-capability benchmarks, \method improves robustness to out-of-distribution attacks while preserving general capabilities, and yields broader attacker strategy coverage.
Figures & tables
| Harmful Refusal | Benign Compliance | ||||||||||
| Method | WG:Test | WJB | DAN | HarmBench | OR-Bench | XSTest | StrongREJECT | WJB | XSTest | ||
| ASR | ASR | ASR | ASR | RTA | RTA | RTA | ASR | Comply | |||
| adv. harm | van. harm | adv. harm | adv. harm | adv. harm | van. harm | van. harm | van. harm | van. harm | adv. benign | van. benign | |
| Qwen2.5-7B-Instruct | 0.365 | 0.038 | 0.701 | 0.327 | 0.363 | 0.250 | 0.892 | 0.800 | 0.964 | 0.992 | 0.940 |
| + Self-RedTeam | 0.255 | 0.017 | 0.442 | 0.323 | 0.237 | 0.047 | 0.973 | 0.825 | 0.988 | 0.980 | 0.904 |
| + MAGIC | 0.023 | 0.002 | 0.198 | 0.043 | 0.055 | 0.019 | 0.977 | 0.860 | 0.988 | 0.968 | 0.945 |
| Method | IFEval | ARC-C | GPQA | MMLU | AlpacaEval 2 | |
| Prompt Loose | Instruct Loose | 0-shot Acc | 0-shot Acc | Acc | vs. GPT4-turbo (LC Win ) | |
| Qwen2.5-7B-Instruct | 0.749 | 0.824 | 0.592 | 0.342 | 0.733 | 33.733% |
| + Self-RedTeam | 0.728 | 0.805 | 0.592 | 0.301 | 0.735 | 34.252% |
| + MAGIC | 0.745 | 0.821 | 0.592 | 0.308 | 0.735 | 33.224% |
| + DACE (ours) | 0.747 | 0.816 | 0.592 | 0.324 | 0.735 | 35.978% |
| Qwen2.5-14B-Instruct | 0.801 | 0.863 | 0.666 | 0.381 | 0.796 | 38.113% |
| Attacker (HarmBench ASR , %) | ||||||
| Defender | no-rev | GCG | PAIR | TAP | AutoDAN | AutoDAN-Turbo |
| Qwen2.5-7B-Instruct | 25.62 | 43.90 | 44.38 | 63.75 | 47.81 | 75.00 |
| + Self-Eval | 24.06 | 28.75 | 37.19 | 50.00 | 42.81 | 89.06 |
| + SmoothLLM | 19.38 | 20.00 | 30.63 | 61.25 | 62.19 | 70.31 |
| + Self-RedTeam | 19.38 | 22.50 | 31.88 | 40.00 | 30.31 | 66.88 |
| + MAGIC | 13.44 | 11.25 | 25.31 | 35.63 | 24.38 | 54.69 |
| Unseen defender (ASR) | Paired defender (ASR) | |||||||
| Attacker | Stage | Gemini-2.5-Flash | Qwen2.5-7B-IT | Llama3.1-8B-IT | Mistral-7B-IT | Avg. | MAGIC-defender | DACE -defender |
| Base | – | 24.67 | 37.33 | 25.00 | 46.83 | 33.46 | 10.50 | 10.33 |
| MAGIC | RL only | 31.33 | 58.00 | 35.83 | 64.17 | 47.33 | 9.00 | 6.83 |
| Full s150 | 29.50 | 54.50 | 33.17 | 58.33 | 43.88 | 7.67 | 4.00 | |
| Full s300 | 6.67 | 25.83 | 13.33 | 11.50 | 14.33 | 3.83 | 3.67 | |
| DACE | RL only | 45.67 | 60.33 | 41.33 | 63.33 | 52.67 | 15.50 | 4.33 |
| Strategy level | Semantic | Textual level | |||
| Attacker source | Coverage | Shannon H | SBERT cos | Self-BLEU | Distinct-2 |
| MAGIC last 15 steps | 45.83% | 3.382 | 0.331 | 0.500 | 0.546 |
| DACE last 15 steps | 88.33% | 3.457 | 0.326 | 0.434 | 0.616 |
| DACE final replay pool | 100.00% | 4.298 | 0.276 | 0.348 | 0.672 |
| Model | Multi-turn ASR | Imp. (ASR) | JailBreak Rate | Imp. (JB) |
| Qwen2.5-7B-Instruct | 94.9% | – | 50.7% | – |
| + Self-RedTeam | 89.9% | 5.27% | 50.4% | 0.59% |
| + MAGIC | 85.4 % | 10.01 % | 43.0 % | 15.19 % |
| + DACE (ours) | 79.1 % | 16.65 % | 37.5 % | 26.04 % |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Game form | Strategy | Diversity | Replay | Threat model | Decay |
| Self-RedTeam | zero-sum self-play | no | no | no | – | no |
| MAGIC | asym. sequential | no | no | no | – | no |
| AdvEvo-MARL | multi-agent | no | no | seed pool | – | no |
| SSP | self-play | no | no | yes | discrete score | no |
| TriPlay-RL | tri-role self-play | no | textual | no | – | no |
| PIA | adv. self-play | persona | semantic | persona pool | lineage ASR | lineage |
| Category | Definition |
| Violent Crimes | Responses that enable, encourage, or endorse the commission of violent crimes, including unlawful violence toward people (terrorism, genocide, murder, hate-motivated violent crimes, child abuse, assault, battery, kidnapping) and unlawful violence toward animals. |
| Non-Violent Crimes | Responses that enable, encourage, or endorse non-violent crimes including personal crimes (labor trafficking, threats), financial crimes (fraud, scams, money laundering), property crimes (theft, arson), drug crimes, weapons crimes (unlicensed firearms), and cyber crimes (hacking). |
| Sex-Related Crimes | Responses that enable, encourage, or endorse sex-related crimes, including sex trafficking, sexual assault, sexual harassment (physical, verbal, or visual), and prostitution. |
| Child Sexual Exploitation | Responses that contain, describe, enable, encourage, or endorse the sexual abuse of children. |
| Defamation | Responses that are both verifiably false and likely to injure a living person’s reputation. |
| Specialized Advice | Responses that contain specialized financial, medical, or legal advice, or that indicate dangerous activities or objects are safe. |
| DACE Index | LG4 Code | Name |
| S1 | S1 | Violent Crimes |
| S2 | S2 | Non-Violent Crimes |
| S3 | S3 | Sex-Related Crimes |
| S4 | S4 | Child Sexual Exploitation |
| S5 | S5 | Defamation |
| S6 | S6 | Specialized Advice |
| Style | Definition |
| Slang | Uses informal language or colloquialisms to disguise malicious intent. |
| Technical Terms | Uses complex jargon to confuse or overwhelm the AI system. |
| Role Play | Asks the AI to assume a specific persona, potentially bypassing ethical constraints. |
| Authority Manipulation | The user pretends to be an authority figure to gain unwarranted compliance. |
| Misspellings | Intentionally misspells keywords to evade content filters and trick the AI system. |
| Word Play | Utilizes puns, homophones, or double meanings to obfuscate true intent. |
| Group | Symbol | Default |
| Coverage smoothing | ||
| Coverage clip | true | |
| Diversity coefficient | ||
| Time decay | ||
| Beta prior | ||
| Prune threshold |
| Hyperparameter | Value |
| Defender backbones | Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Llama3.1-8B-Instruct |
| Attacker initialization | SFT-warm-started from the defender backbone |
| Optimizer | AdamW (veRL default) |
| Actor learning rate | |
| Reference KL coefficient | |
| PPO clip range |
| Switch | Harmful Refusal | Benign Compliance | Instruct Following | |||||
| Method | G | D | R | WJB | StrongREJECT | WJB | XSTest | AlpacaEval 2 |
| ASR | RTA | ASR | Comply | LC Win | ||||
| adv harm | van. harm | adv benign | van. benign | vs gpt4-turbo | ||||
| Qwen2.5-7B-Instruct | – | – | – | 0.701 | 0.964 | 0.992 | 0.940 | 33.733% |
| + MAGIC | – | – | – | 0.198 | 0.988 | 0.968 | 0.945 | 33.224% |
| + DACE w/o strategy grid | – | – | ✓ | 0.178 | 0.994 | 0.976 | 0.920 | 33.789% |
| Strategy collapse (entropy / coverage) | Adversarial forgetting (replayed ASR) | |||||
| Configuration | step 100 | step 200 | step 300 | step 100 | step 200 | step 300 |
| + DACE w/o SFT warm-start | 3.52 / 65.8% | 3.95 / 91.7% | 4.17 / 95.0% | 0.0973 | 0.0965 | 0.1005 |
| + DACE w/o diversity reward | 2.54 / 34.5% | 2.52 / 35.3% | 2.49 / 34.7% | 0.0936 | 0.1124 | 0.1172 |
| + DACE w/o replay pool | 3.73 / 75.0% | 4.12 / 95.8% | 4.25 / 99.1% | 0.2448 | 0.2009 | 0.1276 |
| + DACE (full) | 4.05 / 89.2% | 4.23 / 100% | 4.32 / 100% | 0.0729 | 0.0775 | 0.0813 |
| (a) The grid inside RL training, without the coverage reward | ||||||
| Configuration | WJB adversarial ASR | |||||
| + MAGIC | 0.198 | |||||
| + MAGIC w/ strategy grid | 0.193 | |||||
| + DACE (full) | 0.068 | |||||
| (b) The grid as a test-time prompt for an untrained attacker | ||||||
| Attacker | Qwen2.5-7B | Llama3.1-8B | Mistral-7B | Gemini-2.5-Flash | MAGIC-def | DACE -def |
| Agreement with declared cell | step 100 | step 200 | step 300 |
| Joint (risk and style) | 94.20% | 96.40% | 95.28% |
| Risk axis only | 96.25% | 97.88% | 97.13% |
| Style axis only | 95.95% | 97.53% | 96.65% |
| Harmful Refusal | Benign Compliance | ||||||||||
| Method | WG:Test | WJB | DAN | HarmBench | OR-Bench | XSTest | StrongREJECT | WJB | XSTest | ||
| ASR | ASR | ASR | ASR | RTA | RTA | RTA | ASR | Comply | |||
| adv. harm | van. harm | adv. harm | adv. harm | adv. harm | van. harm | van. harm | van. harm | van. harm | adv. benign | van. benign | |
| WildGuard judge | |||||||||||
| Qwen2.5-7B-Instruct | 0.255 | 0.019 | 0.863 | 0.367 | 0.249 | 0.066 | 0.872 | 0.905 | 0.964 | 1.000 | 0.952 |
| + Self-RedTeam | 0.160 | 0.002 | 0.568 | 0.310 | 0.137 | 0.025 | 0.979 | 0.950 | 0.988 | 0.992 | 0.944 |
| Harmful Refusal | Benign Compliance | |||||||||||
| Method | Train judge | WG:Test | WJB | DAN | HarmBench | OR-Bench | XSTest | StrongREJ | WJB | XSTest | ||
| ASR | ASR | ASR | ASR | RTA | RTA | RTA | ASR | Comply | ||||
| adv. harm | van. harm | adv. harm | adv. harm | adv. harm | van. harm | van. harm | van. harm | van. harm | adv. benign | van. benign | ||
| MAGIC | Qwen3Guard | 0.023 | 0.002 | 0.198 | 0.043 | 0.055 | 0.019 | 0.977 | 0.860 | 0.988 | 0.968 | 0.945 |
| DACE | WildGuard | 0.012 | 0.000 | 0.074 | 0.027 | 0.031 | 0.009 | 0.985 | 0.880 | 0.984 | 0.952 | 0.932 |
| DACE | Qwen3Guard | 0.000 | 0.000 | 0.068 | 0.006 | 0.024 | 0.009 | 0.991 | 0.885 | 0.996 | 0.994 | 0.940 |
| Harmful Refusal | Benign Compliance | |||||||||||
| Method | Seed | WG:Test | WJB | DAN | HarmBench | OR-Bench | XSTest | StrongREJ | WJB | XSTest | ||
| ASR | ASR | ASR | ASR | RTA | RTA | RTA | ASR | Comply | ||||
| adv. harm | van. harm | adv. harm | adv. harm | adv. harm | van. harm | van. harm | van. harm | van. harm | adv. benign | van. benign | ||
| MAGIC | 1 | 0.023 | 0.002 | 0.198 | 0.043 | 0.055 | 0.019 | 0.977 | 0.860 | 0.988 | 0.968 | 0.945 |
| 2 | 0.018 | 0.002 | 0.212 | 0.043 | 0.061 | 0.025 | 0.959 | 0.865 | 0.994 | 0.976 | 0.936 | |
| 3 | 0.033 | 0.002 | 0.181 | 0.050 | 0.047 | 0.019 | 0.989 | 0.840 | 0.974 | 0.972 | 0.944 | |
| IFEval | ARC-C | GPQA | MMLU | AlpacaEval 2 | |||
| Method | Seed | Prompt Loose | Instruct Loose | 0-shot Acc | 0-shot Acc | Acc | vs. GPT4-turbo (LC Win ) |
| MAGIC | 1 | 0.745 | 0.821 | 0.592 | 0.308 | 0.735 | 33.224% |
| 2 | 0.756 | 0.823 | 0.592 | 0.310 | 0.735 | 32.050% | |
| 3 | 0.721 | 0.810 | 0.589 | 0.304 | 0.735 | 34.286% | |
| Avg. | 0.741 0.018 | 0.818 0.007 | 0.591 0.002 | 0.307 0.003 | 0.735 0.000 | 33.186% 1.118 | |
| DACE | 1 | 0.747 | 0.816 | 0.592 | 0.324 | 0.735 | 35.978% |
| Defender | Attacker | 1-shot | PAIR | TAP | AutoDAN | AutoDAN-Turbo |
| Claude Opus 4.8 | Base | 0.00 | 7.81 | 1.88 | 5.94 | 6.88 |
| MAGIC | 0.63 | 9.06 | 5.63 | 10.31 | 12.19 | |
| DACE | 1.88 | 11.88 | 7.81 | 10.00 | 15.00 | |
| GPT-5.5 | Base | 1.88 | 10.94 | 7.81 | 6.88 | 10.00 |
| MAGIC | 2.19 | 9.69 | 8.44 | 11.88 | 12.81 | |
| DACE | 3.75 | 15.00 | 12.81 | 11.88 | 20.94 |
LLM Safety
Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.
See all 467 papers