Automated Red Teaming

Ask about this topic

Overview

Automated red teaming studies how to generate adversarial tests that reveal harmful, insecure, or unreliable behavior in AI systems, including language models, multimodal and web agents, and multi-agent workflows. Common approaches use adaptive prompt search, reinforcement learning, self-play, evolutionary strategies, and agents that learn reusable attack skills from execution feedback; tests increasingly include multi-turn interactions, tool use, memory, and realistic environments. Benchmarks and evaluation methods measure attack success, risk severity, coverage, and whether attacks transfer across models or evade monitors, while adversarial testing is also used to improve defenses and harden benchmarks. A central challenge is making evaluations realistic and reproducible while ensuring that discovered attacks generalize, represent meaningful harm, and keep pace with defenses that can themselves adapt.

Latest papers

CardsList

60 papers14 papers in the last four weeks, up 75% on the four weeks before. 0.1% of all new papers.

  1. Program-Level Adversarial Co-Adaptive Self-Play for Robotic Security

    Oct 26, 2026Leo Benaharon, Chi Zhang, Wei Zheng +2Robot Policy AdaptationSelf-Evolving Agents

    The authors are with Isuzu Technical Center of America, Plymouth, MI 48170 USA.

  2. The Target Agent Is Its Own Best Critic but a Poor Proposer: Self-Improved Red Teaming

    Oct 26, 2026Mintong Kang, Stephen Rawls, Melissa Kazemi Rad +2LLM Red TeamingAutomated Red Teaming

    University of Illinois Urbana-Champaign · Capital One

  3. CyberDual: AI Attackers versus AI Code Reviewers in Real Software Maintenance

    Oct 25, 2026Chenyuan Yang, Hwiwon Lee, Jongseong Kim +4AI Agent Security BenchmarksAutomated Code Review

  4. SEEK: Skill Extraction with Evolving Knowledge against Frontier Science Agents

    Oct 25, 2026Cheng Gui, Xiang Zheng, Yutao Wu +5AI Agent SecurityAgent Skill Security Auditing

    City University of Hong Kong, Hong Kong · Hong Kong Institute of AI for Science, Hong Kong · Deakin University, Australia +2

  5. M3M^3: A Multi-LLM System for Multi-Turn Jailbreaking as a Markov Decision Process

    Oct 22, 2026Yuqing Huang, Jack Ji, Jithin Pradeep +2Adversarial Attacks on LLMsMulti-Turn Jailbreak Attacks

    The Vanguard Group∗

  6. AutoRedClaw: Automated Red-Teaming for Claw-Style Agents

    Oct 19, 2026Hao Li, Tongyan Hu, Weidi Luo +8AI Agent SecurityAutomated Red Teaming

    Washington University in St. Louis · Johns Hopkins University · National University of Singapore +3

  7. Red-Teaming Multi-Agent Attacks for Loss-of-Control Risks

    Oct 16, 2026John Kitaoka, Thao Amelia Pham, Carlos Rafael Giudice +1AI Agent SafetyMulti-Agent System Security

    MATS Research · Redwood Research · Independent

  8. Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming

    Oct 14, 2026Gosha Kozhevnikov, Tyler TracyLLM Red TeamingAutomated Red Teaming

    Redwood Research

  9. CoRedSkill: Stealthy Skill Composition Attacks via AST-Guided Injection

    Oct 14, 2026Qichen Zhao, Shengfang Zhai, Qingni Shen +2AI Agent SecurityAutomated Red Teaming

    Peking University · National University of Singapore

  10. RedDarwin: Evolving Jailbreak Skills against Frontier LLMs

    Oct 14, 2026Zhouyuan Ma, Yutao Wu, Hanxun Huang +3LLM Jailbreak AttacksLLM Red Teaming

    Fudan University · Deakin University · University of Melbourne +2

  11. X-AgentRed: Red-Teaming Personal Agents in Evolving Multi-Service Ecosystems

    Oct 9, 2026Yeonsung Jung, Zhiyuan Wang, Zifan Wang +5Automated Red TeamingAI Agent Security

    Meta Reality Labs Research · Meta Superintelligence Labs · Meta +1

  12. Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution

    Oct 7, 2026Wanjing Han, Levi Taiji Li, Mu Zhang +2AI Agent SecurityTool-Using Vision-Language Agents

    University of Utah

  13. Red-TTT: Test-Time Training for Automated Jailbreaking Large Language Models

    Oct 4, 2026Tongyan Hu, Hao Li, Xiaogeng Liu +8Test-Time TrainingLLM Jailbreak Attacks

    Johns Hopkins University · National University of Singapore · Washington University in St. Louis +2

  14. Penumbra: Sample-Efficient Adversarial Search for Regulatory Obligations

    Oct 3, 2026Anthony RhodesAutomated Red TeamingAI Regulatory Compliance

    Confidential Core AI

  15. HASTE: Evolving Agent Harnesses Against Emerging Attacks Using Sparse Evidence

    Oct 2, 2026Xiqiao Xiong, Moxin Li, Zhixin Ma +4LLM Agent SafetyAgent Harness Optimization

    University of Science and Technology of China · National University of Singapore · Singapore Management University

  16. No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents

    Sep 30, 2026Ayan Javeed Shaikh, Arunesh Sinha, Nathaniel D. Bastian +1LLM Red TeamingAI Agent Evaluation

    Indiana University Bloomington, IN, USA · Rutgers University Piscataway, NJ, USA · Johns Hopkins University Baltimore, MD, USA

  17. SkillDRE: Dual-Stage Red-Team Evolution of Agent Skills via Pre-Execution and Runtime Feedback

    Sep 26, 2026Pengyu Zhu, Jingyi Yang, Yi Liu +2Automated Red TeamingLLM Red Teaming

    Beijing University of Posts and Telecommunications · North China Electric Power University · Chongqing University of Posts and Telecommunications

  18. AgentXploit: Autonomous Repository-to-Runtime Red-Teaming for AI Agents

    Sep 25, 2026Weida Liang, Shi Qiu, Zhun Wang +6AI Agent AuditingAutomated Red Teaming

    National University of Singapore · University of North Carolina at Chapel Hill · University of California, Berkeley +2

  19. Stealth Apart, Harm Together: Skill Cascading Attacks on Skill-Based Agent Systems

    Sep 24, 2026Zihao Zhu, Siwei Lyu, Adel Bibi +1Adversarial AttacksLLM Agent Security

    The Chinese University of Hong Kong, Shenzhen · University at Buffalo, SUNY · University of Oxford

  20. CART: Closed-Loop Adaptive Red Teaming for Large Language Models

    Sep 23, 2026Dongdong Zhang, Tengchao Lv, Yilin Jia +9Automated Red TeamingLanguage Model Safety Evaluation

    Microsoft Research

  21. Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents

    Sep 17, 2026Alex Remedios, Simon Storf, Fabien Roger +1AI Agent SecurityAI Agent Monitoring

    Anthropic Fellows Program · Anthropic

  22. Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery

    Sep 9, 2026Divyanshu Kumar, Nitin Aravind Birur, Tanay Baswa +2AI Agent SecurityAI Agent Security Benchmarks

    Enkrypt AI

  23. EvoSkill Injection: Red-Teaming Autonomous Skill Generation and Evolution in Self-Evolving Agents

    Aug 31, 2026Doyun Kim, Chanwoo Kim, Sugyeong Eo +2AI Agent SecurityAI Agent Security Benchmarks

    Soongsil University · Yonsei University · Jeju National University

  24. SIR: Self-improving Red-teaming for Compute Use Agents

    Aug 31, 2026Chen Xiong, Zhiyuan He, Pin-Yu Chen +2Computer-Use AgentsIndirect Prompt Injection

    The Chinese University of Hong Kong · IBM Research · University of Zagreb, FER & Radboud University +1

  25. RedEvoAgent: Automatic Red-Teaming Agent with Experience-Driven Skill Evolution

    Aug 27, 2026Junjie Zhang, Hui Liu, Kecheng Chen +3Automated Red TeamingLLM Agent Security

    City University of Hong Kong · Shenzhen MSU-BIT University

  26. JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills

    Aug 17, 2026Xiaoyu Wen, Jiajia Li, Zhida He +11LLM Jailbreak AttacksAutomated Red Teaming

    Shanghai AI Laboratory · Shanghai Jiao Tong University · Northwestern Polytechnical University +2

  27. IO Factory: Simulating AI-Enabled Influence Campaigns at Scale

    Aug 11, 2026Lukasz Olejnik, Wenchao Dong, Jonas R. Kunst +4Agent-Based ModelingAutomated Red Teaming

    Department of War Studies, King’s College London, London, United Kingdom · Independent researcher · Max Planck Institute for Security and Privacy, Bochum, Germany +4

  28. Generating Attacks for LLMs with GFlowNets

    Aug 10, 2026Berkay Ozcam, Irem Onen, Mehmet Fatih Amasyali +1Adversarial Prompt GenerationAdversarial Attacks on LLMs

    Cybersecurity R&D Turkcell Istanbul, Turkey · Department of Computer Engineering Yıldız Technical University Istanbul, Turkey

  29. Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)

    Aug 5, 2026Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi +6Autonomous Cyber DefenseLLM Red Teaming

    University of California, Irvine · Northeastern University · Johns Hopkins University

  30. Salami Attack: Stealthy Collusive Memory Poisoning against OpenClaw

    Aug 3, 2026Zheng Lin, Yuzhe Huang, Zhenxing Niu +2Agent Memory PoisoningAutomated Red Teaming

    Xidian University Xi’an, Shaanxi, China