LLM Red Teaming

LLM: Large Language Model

Overview

LLM red teaming studies how adversarial prompts and interactions expose failures in language models, from harmful text generation to unsafe tool use by agents. Approaches include jailbreak prompts, multi-turn dialogue, prompt-injection attacks, reinforcement learning, evolutionary search, and human-guided testing, with evaluations spanning general safety, coding, health, finance, and cybersecurity. As systems gain tools and longer workflows, tests increasingly track executed actions, attack trajectories, severity, and transfer across models rather than relying only on single-turn pass/fail scores. A central tension is that adaptive attacks uncover failures missed by static benchmarks, while limited coverage, judge disagreement, and uneven transfer make it difficult to infer real-world safety from a clean evaluation.

Latest papers

CardsList

69 papers7 papers in the last four weeks, up 17% on the four weeks before. 0.1% of all new papers.

  1. PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents

    Nov 18, 2026Lipeng He, Yihan Wang, Jiawen Zhang +1Prompt Injection Attacks on AI AgentsPrompt Injection Defense

  2. Incremental Speciation in Evolutionary Red Teaming of Large Language Models

    Nov 18, 2026Onkar Shelar, Travis DesellLLM Red TeamingEvolutionary Optimization

  3. Stress Testing Chain-of-Thought Monitoring Against Covert Misalignment

    Nov 5, 2026Yao Huang, Yifan Wang, Yitong Sun +2LLM Agent SafetyCoT Monitoring

  4. On the Framing Effect of AI Red-Teaming Instructions: Evidence from Prior Exercises and Controlled Simulations

    Oct 30, 2026Michael Feffer, Alice Qian, Anusha Sinha +6LLM Red TeamingAI Safety Evaluation

  5. CoFiBreak: A Coarse-to-Fine Diversity Reward for Effective Multi-Turn Jailbreak

    Oct 29, 2026Jialin Song, Xiaodong Liu, Weiwei Yang +7LLM Red TeamingJailbreak Attacks

  6. Red-Teaming Large Language Models in Low-Resource African Languages: A Case Study of Yoruba and Nigerian Pidgin

    Oct 29, 2026Paula Chikaodinaka Anyanwu, Valdini Douglace Lemofouet, Blessing Ngozi Uzor +7Language Model Safety EvaluationLLM Red Teaming

  7. The Target Agent Is Its Own Best Critic but a Poor Proposer: Self-Improved Red Teaming

    Oct 26, 2026Mintong Kang, Stephen Rawls, Melissa Kazemi Rad +2LLM Red TeamingAutomated Red Teaming

  8. Black-Box Red-Teaming of Multi-Agent Systems via Reinforcement Learning

    Oct 20, 2026Yuqi Jia, Hangsheng Zhang, Chenxi Liu +2LLM Red TeamingBlack-Box Adversarial Attacks

  9. Budgeted Attack Scheduling for LLM Red-Teaming: The Roles of Static Priors, Semantic Guidance, and Online Feedback

    Oct 19, 2026Zhicheng Fang, Han Sun, Jiachen Lei +5LLM Red TeamingLanguage Model Safety Evaluation

  10. RedDarwin: Evolving Jailbreak Skills against Frontier LLMs

    Oct 14, 2026Zhouyuan Ma, Yutao Wu, Hanxun Huang +3LLM Jailbreak AttacksLLM Red Teaming

  11. ATTCK-R: Knowledge-Aware Long-Horizon Jailbreaking for Protected LLMs

    Oct 12, 2026Md Mushfiqur Rahman, Tobias Rohde, Rahul Tripathi +2LLM Jailbreak AttacksLLM Red Teaming

  12. No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents

    Sep 30, 2026Ayan Javeed Shaikh, Arunesh Sinha, Nathaniel D. Bastian +1LLM Red TeamingAI Agent Evaluation

  13. Climbing the Hill: Prompt Injection Red-Teaming Against Frontier Models with Curriculum Reinforcement Learning

    Sep 27, 2026Chenlong Yin, Xiaolong Jin, Wei Zou +2Adversarial Attacks on LLMsLLM Red Teaming

  14. SkillDRE: Dual-Stage Red-Team Evolution of Agent Skills via Pre-Execution and Runtime Feedback

    Sep 26, 2026Pengyu Zhu, Jingyi Yang, Yi Liu +2Automated Red TeamingLLM Red Teaming

  15. REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems

    Aug 11, 2026Zixing Chen, Xingyuan Liu, Jie Zhu +6LLM Red TeamingLLM Agents

  16. Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)

    Aug 5, 2026Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi +6Autonomous Cyber DefenseLLM Red Teaming

  17. GPT-Red: Automated Red Teaming via Self-Play at Scale

    Jul 28, 2026Eric Wallace, Christopher A. Choquette-Choo, Nikhil Kandpal +15Adversarial TrainingLLM Red Teaming

  18. AMT-X: Phase-Structured Multi-Turn Red-Teaming with Checklist-Gated Evaluation

    Jul 13, 2026Yi Ting Shen, Kentaroh Toyoda, Alex LeungLLM Safety BenchmarksAdversarial Attacks on LLMs

  19. CONTRA: Red-Teaming Configurations of Personalizable Agents

    Jul 3, 2026Jonathan Nöther, Adish Singla, Goran RadanovicLLM Red TeamingAI Agent Security

  20. MIRROR: Novelty-Constrained Memory-Guided MCTS Red-Teaming for Agentic RAG

    Jun 25, 2026Inderjeet Singh, Andrés Murillo, Motoyoshi Sekiya +2Multimodal RAGTool-Augmented Language Model Agents

  21. Distributed Quality-Diversity Search for Toxicity in Large Language Models

    Jun 23, 2026Onkar Shelar, Travis DesellAdversarial Prompt GenerationLLM Red Teaming

  22. OTTER: A Red-Teaming System for Toxicity-Evading Jailbreak Prompt Optimization

    Jun 19, 2026Jerry Wang, Hsin-Ling Hsu, Yi-Cheng Lai +2Adversarial Prompt GenerationAdversarial Attacks on LLMs

  23. NRT-Bench: Benchmarking Multi-Turn Red-Teaming of LLM Operator Agents in Safety-Critical Control Rooms

    Jun 18, 2026Hanwool Lee, Dasol Choi, Bokyeong Kim +2Multi-Agent LLM SystemsSafety-Critical Control

  24. FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teaming

    Jun 18, 2026Daeyoung Park, Chaeyun Kim, Junghwan Kim +4LLM Safety BenchmarksFinancial Services