LLM Red Teaming

LLM: Large Language Model

LLM red teaming studies how adversarial prompts and interactions expose failures in language models, from harmful text generation to unsafe tool use by agents. Approaches include jailbreak prompts, multi-turn dialogue, prompt-injection attacks, reinforcement learning, evolutionary search, and human-guided testing, with evaluations spanning general safety, coding, health, finance, and cybersecurity. As systems gain tools and longer workflows, tests increasingly track executed actions, attack trajectories, severity, and transfer across models rather than relying only on single-turn pass/fail scores. A central tension is that adaptive attacks uncover failures missed by static benchmarks, while limited coverage, judge disagreement, and uneven transfer make it difficult to infer real-world safety from a clean evaluation.

Latest papers

63 papers6 papers in the last four weeks, up 50% on the four weeks before. 0.0% of all new papers.

RedCoder: Automated Multi-Turn Red Teaming for Code LLMs
Jun 25, 2025Wenjie Jacky Mo, Qin Liu, Xiaofei Wen +5Adversarial Prompt Generation LLM Red Teaming
University of California, Davis · University of Southern California
Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models
Jun 8, 2025Ren-Jian Wang, Ke Xue, Zeyu Qin +7LLM Red Teaming Adversarial Prompt Generation
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · The Hong Kong University of Science and Technology, Hong Kong, SAR, China +3
Learning diverse attacks on large language models for robust red-teaming and safety tuning
May 28, 2024Seanie Lee, Minsu Kim, Lynn Cherif +8Adversarial Attacks on LLMs LLM Red Teaming
KAIST · Mila – Québec AI Institute · Université de Montréal +3

Jun 25, 2025J/K move · Enter open · S save

RedCoder: Automated Multi-Turn Red Teaming for Code LLMs

Wenjie Jacky Mo, Qin Liu, Xiaofei Wen, Dongwon Jung+4

University of California, Davis · University of Southern California

Adversarial Prompt Generation LLM Red Teaming Automated Red Teaming Code Language Models LLM Code Generation

Open paper AskPDF arXiv

Large Language Models (LLMs) for code generation (i.e., Code LLMs) have demonstrated impressive capabilities in AI-assisted software development and testing. However, recent studies have shown that these models are prone to generating vulnerable or even malicious code under adversarial settings. Existing red-teaming approaches rely on extensive human effort, limiting their scalability and practicality, and generally overlook the interactive nature of real-world AI-assisted programming, which often unfolds over multiple turns. To bridge these gaps, we present RedCoder, a red-teaming agent that engages victim models in multi-turn conversation to elicit vulnerable code. The pipeline to construct RedCoder begins with a multi-agent gaming process that simulates adversarial interactions, yielding a set of prototype conversations and an arsenal of reusable attack strategies. We then fine-tune an LLM on these prototype conversations to serve as the backbone of RedCoder. Once deployed, RedCoder autonomously engages Code LLMs in multi-turn conversations, dynamically retrieving relevant strategies from the arsenal to steer the dialogue toward vulnerability-inducing outputs. Experiments across multiple Code LLMs show that our approach outperforms prior single-turn and multi-turn red-team methods in inducing vulnerabilities in code generation, offering a scalable and effective tool for evaluating the security boundaries of modern code-generation systems.

Latest papers

RedCoder: Automated Multi-Turn Red Teaming for Code LLMs

Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models

Learning diverse attacks on large language models for robust red-teaming and safety tuning