Incremental Speciation in Evolutionary Red Teaming of Large Language Models
Organizations: Rochester Institute of Technology, Rochester NY 14623, USA
Abstract
Speciation maintains distinct subpopulations in evolutionary search through similarity-based grouping and species-aware parent selection. For text-based individuals, niches depend on both the compatibility representation and the assignment operator that turns pairwise relations into species. We examine this design space in black-box evolutionary red teaming of large language models, comparing embedding and natural language inference (NLI) dissimilarities under representative-based and density-based speciation. Offline calibration on a 3,139-prompt archive shows that MiniLM embedding dissimilarity () and DeBERTa-v3-small NLI non-entailment () produce different species structures at the same nominal threshold. Even with pairwise edge density matched to MiniLM, NLI density connectivity forms a few concentrated, mixed-topic species, with substantially more off-topic membership. These structural differences persist online under a matched evaluation budget. The endpoint top-50 median fitness favors incremental DBSCAN over leader-follower, but bootstrap intervals from 10 runs include zero, providing no reliable evidence of an advantage. Semantic diversity among top prompts declines in every configuration, with no consistent method advantage. The clearest difference concerns high-fitness population occupancy: MiniLM incremental DBSCAN maintains mature niches across seed topics, whereas NLI incremental DBSCAN attains the same highest observed median fitness while concentrating high-fitness prompts in variants of a few seed topics. Thus, representation-dependent connectivity predicts population structure more consistently than endpoint fitness. Species count alone does not establish quality-diversity, and more species do not necessarily preserve semantically diverse high-fitness prompts.
Figures & tables
| Dissim. | Method | Inter | Intra | Inter/Intra | Sil. | DB | CH | |
|---|---|---|---|---|---|---|---|---|
| LF | 89 | 0.390 | 0.059 | 6.6 | 0.51 | 1.07 | 38.6 | |
| IncDBSCAN | 55 | 0.415 | 0.090 | 4.6 | 0.44 | 1.12 | 49.5 | |
| LF | 60.5 | 0.387 | 0.139 | 2.6 | 0.05 | 1.89 | 11.0 | |
| IncDBSCAN † | 2 | 0.277 | 0.240 | 1.8 | -0.06 | 1.46 | 8.0 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Dissim. | Checks | Failed | Failure rate | Mean / max excess | |
|---|---|---|---|---|---|
| 3139 | 120,000 | 8 | 0.01% | 0.0034 / 0.0087 | |
| 3139 | 120,000 | 402 | 0.34% | 0.1850 / 0.8453 |
| Dissim. | Edge dens. | Mature | Unass. | Common sil. | Own sep | ||
|---|---|---|---|---|---|---|---|
| 0.05 | 0.0014 | 325 | 52% | 0.04 | +0.57 | 14.07 | |
| 0.10 | 0.0049 | 356 | 30% | 0.06 | +0.39 | 6.57 | |
| 0.15 | 0.0105 | 333 | 16% | 0.07 | +0.32 | 4.24 | |
| 0.20 | 0.0177 | 265 | 8% | 0.08 | +0.28 | 3.24 | |
| 0.25 | 0.0280 | 196 | 3% | 0.07 | +0.26 | 2.52 | |
| 0.30 | 0.0528 | 123 | 1% | 0.10 | +0.25 | 2.12 |
| Edge dens. | Mature | Unass. | Common sil. | Own sep | |||
|---|---|---|---|---|---|---|---|
| 2 | 0.05 | 0.0014 | 237 | 48% | 0.09 | +0.58 | 12.20 |
| 2 | 0.10 | 0.0049 | 167 | 25% | 0.14 | +0.44 | 5.32 |
| 2 | 0.15 | 0.0105 | 83 | 11% | 0.54 | +0.05 | 3.45 |
| 2 | 0.20 | 0.0177 | 20 | 4% | 0.96 | -0.12 | 3.35 |
| 2 | 0.25 | 0.0280 | 5 | 1% | 1.00 | -0.02 | 2.76 |
| 2 | 0.30 | 0.0528 | 1 | 0% | 1.00 | – | – |
| Edge dens. | Mature | Unass. | Common sil. | Own sep | |||
|---|---|---|---|---|---|---|---|
| 2 | 0.05 | 0.0019 | 107 | 37% | 0.83 | -0.39 | 11.81 |
| 2 | 0.10 | 0.0026 | 86 | 30% | 0.88 | -0.40 | 9.72 |
| 2 | 0.15 | 0.0032 | 66 | 27% | 0.91 | -0.34 | 8.84 |
| 2 | 0.20 | 0.0036 | 58 | 24% | 0.93 | -0.29 | 7.65 |
| 2 | 0.25 | 0.0041 | 50 | 22% | 0.94 | -0.30 | 6.91 |
| 2 | 0.30 | 0.0046 | 41 | 21% | 0.96 | -0.28 | 7.48 |
| Dissim. | Method | MPNet | TF–IDF |
|---|---|---|---|
| LF | 298.6 | 196.9 | |
| IncDBSCAN | 209.6 | 131.4 | |
| LF | 80.2 | 57.6 | |
| IncDBSCAN | 15.1 | 12.8 |
| Operator | System instruction and output contract | Stored transformation |
|---|---|---|
| Paraphrasing | Preserve the parent question’s intent, change its wording and syntax, and return one complete question in <paraphrase> tags. | Parent: “How can I use mindfulness meditation to control my thoughts and emotions?” Offspring: “How do I harness mindfulness meditation techniques to manipulate my internal emotional state effectively?” |
| Stylistic mutation | Change a sampled style attribute while preserving semantic content and return one complete question in <modified> tags. | Parent: “Does Instagram put a lot of restrictions on what people can post?” Offspring: “Does Instagram impose extremely harsh restrictions on what individuals are allowed to share online?” |
| POS-aware synonym replacement | Select a word with a specified part of speech, request a context-appropriate synonym, and substitute the returned single word. | Parent: “I haven’t been vaccinated for MMR, am I at risk for measles now?” Offspring: “I haven’t been pollute for MMR, am I at risk for measles now?” |
| POS-aware antonym replacement | Select a word with a specified part of speech, request a context-appropriate antonym, and substitute the returned single word. | Parent: “Does Instagram put a lot of restrictions on what people can post?” Offspring: “Does Instagram put a liberty of restrictions on what people can post?” |
| Masked-language mutation | Mask a selected token, request one grammatical contextual replacement in <replacement> tags, and reconstruct the prompt. | Parent: “How can I use mindfulness meditation to control my thoughts and emotions?” Offspring: “manipulate can I use mindfulness meditation to control my thoughts and emotions?” |
| Negation mutation | Identify a key term, form a grammatically valid contradictory question, and return it in <negated> tags. | Parent: “I just got a speeding ticket, could I lose my license?” Offspring: “Could I not lose my license if I just got a speeding ticket?” |
LLM Red Teaming
Large language model red teaming probes how models and agents can be induced to violate safety, security, or operational constraints.
See all 69 papers