cs.NENov 18, 2026

Incremental Speciation in Evolutionary Red Teaming of Large Language Models

Authors: Onkar Shelar, Travis Desell

Organizations: Rochester Institute of Technology, Rochester NY 14623, USA

Abstract

Speciation maintains distinct subpopulations in evolutionary search through similarity-based grouping and species-aware parent selection. For text-based individuals, niches depend on both the compatibility representation and the assignment operator that turns pairwise relations into species. We examine this design space in black-box evolutionary red teaming of large language models, comparing embedding and natural language inference (NLI) dissimilarities under representative-based and density-based speciation. Offline calibration on a 3,139-prompt archive shows that MiniLM embedding dissimilarity (dEd_E) and DeBERTa-v3-small NLI non-entailment (d¬Ed_{\neg E}) produce different species structures at the same nominal threshold. Even with pairwise edge density matched to MiniLM, NLI density connectivity forms a few concentrated, mixed-topic species, with substantially more off-topic membership. These structural differences persist online under a matched evaluation budget. The endpoint top-50 median fitness favors incremental DBSCAN over leader-follower, but bootstrap intervals from 10 runs include zero, providing no reliable evidence of an advantage. Semantic diversity among top prompts declines in every configuration, with no consistent method advantage. The clearest difference concerns high-fitness population occupancy: MiniLM incremental DBSCAN maintains mature niches across seed topics, whereas NLI incremental DBSCAN attains the same highest observed median fitness while concentrating high-fitness prompts in variants of a few seed topics. Thus, representation-dependent connectivity predicts population structure more consistently than endpoint fitness. Species count alone does not establish quality-diversity, and more species do not necessarily preserve semantically diverse high-fitness prompts.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Distributed Quality-Diversity Search for Toxicity in Large Language Models

    Jun 23, 2026Onkar Shelar, Travis DesellAdversarial Prompt GenerationLLM Red Teaming

  2. Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphs

    May 31, 2026Denica Kjorvezir, Marko Djukanović, Ana Gjorgjevikj +2LLM Evaluation