Red-Teaming

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 71

All topics

Red-TTT: Test-Time Training for Automated Jailbreaking Large Language Models
Oct 4, 2026Tongyan Hu, Hao Li, Xiaogeng Liu +8Large Language Model Jailbreaks Red-Teaming
Johns Hopkins University · National University of Singapore · Washington University in St. Louis +2
No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents
Sep 30, 2026Ayan Javeed Shaikh, Arunesh Sinha, Nathaniel D. Bastian +1Red-Teaming Cybersecurity
Indiana University Bloomington, IN, USA · Rutgers University Piscataway, NJ, USA · Johns Hopkins University Baltimore, MD, USA
Climbing the Hill: Prompt Injection Red-Teaming Against Frontier Models with Curriculum Reinforcement Learning
Sep 27, 2026Chenlong Yin, Xiaolong Jin, Wei Zou +2Attacker Large Language Model Red-Teaming
The Pennsylvania State University · Purdue University
Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents
Sep 17, 2026Alex Remedios, Simon Storf, Fabien Roger +1Red-Teaming Coding Agents
Anthropic Fellows Program · Anthropic
Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Sep 10, 2026Divyanshu Kumar, Nitin Aravind Birur, Tanay Baswa +2Red-Teaming Agentic Evaluations
Enkrypt AI
EvoFlint: An Evolutionary Atlas of Multi-Turn LLM Vulnerabilities
Aug 31, 2026Feitong Qiao, Liren Peng, Shiming Ren +7Attacker Large Language Model Red-Teaming
Reinforce Labs USA
SIR: Self-improving Red-teaming for Compute Use Agents
Aug 31, 2026Chen Xiong, Zhiyuan He, Pin-Yu Chen +2Indirect Prompt Injection Red-Teaming
The Chinese University of Hong Kong · IBM Research · University of Zagreb, FER & Radboud University +1
REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems
Aug 11, 2026Zixing Chen, Xingyuan Liu, Jie Zhu +6Red-Teaming Large Language Model Agents
Fudan University · The Hong Kong University of Science and Technology · Qwen DianJin Team, Alibaba Cloud Computing +1
Generating Attacks for LLMs with GFlowNets
Aug 10, 2026Berkay Ozcam, Irem Onen, Mehmet Fatih Amasyali +1Attacker Large Language Model Large Language Model Safety
Cybersecurity R&D Turkcell Istanbul, Turkey · Department of Computer Engineering Yıldız Technical University Istanbul, Turkey
From Inaudible Inputs to Model Failures: Low-Frequency Safety Risks in LALMs
Aug 10, 2026Yuanhe Zhang, Weiliu Wang, Jie Ren +7Large Audio Language Models Audio Understanding
Beijing University of Posts and Telecommunications · Institute of Information Engineering, Chinese Academy of Sciences · Nanyang Technological University +3
Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)
Aug 5, 2026Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi +6Red-Teaming Large Language Model Safety
University of California, Irvine · Northeastern University · Johns Hopkins University
OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution
Aug 1, 2026Yunhao Chen, Xin Wang, Yixu Wang +6Red-Teaming Artificial Intelligence Safety
Fudan University · Shanghai Artiﬁcial Intelligence Laboratory · Work done during Yunhao Chen’s internship at Shanghai AI Lab. +1
Symbolic Attack Chain Generation from Atomic Red Team Techniques: An Empirical Study of Predicate Representation Granularity
Jul 31, 2026Ramya VarunsegarRed-Teaming Symbolic Predicates
GPT-Red: Automated Red Teaming via Self-Play at Scale
Jul 28, 2026Eric Wallace, Christopher A. Choquette-Choo, Nikhil Kandpal +15Red-Teaming Attacker Large Language Model
OpenAI
What AI Red-Team Evaluations Can and Cannot Prove
Jul 23, 2026Bandana KaurArtificial Intelligence Evaluation Safety Benchmarks
APIsec Research Labs
Code Monitor Red Teaming for Public-Test-Passing Code
Jul 23, 2026Junchi Liao, Jiawen Deng, Fuji RenCode Quality Red-Teaming
University of Electronic Science and Technology of China Chengdu, China
An Early Warning of Emerging Biosecurity Risks in Frontier LLMs
Jul 20, 2026Zhida He, Xia Hu, Baichen Le +20Large Language Model Safety Life Sciences Research
Shanghai Artificial Intelligence Laboratory
Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming
Jul 13, 2026Xutao Mao, Xiang Zheng, Cong WangRed-Teaming Large Language Model Agents
City University of Hong Kong
AMT-X: Phase-Structured Multi-Turn Red-Teaming with Checklist-Gated Evaluation
Jul 13, 2026Yi Ting Shen, Kentaroh Toyoda, Alex LeungRed-Teaming Attack-Success Rate
Vulcan Research, AIFT, Singapore
Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI Safety
Jul 8, 2026Yujiao ChenRed-Teaming Artificial Intelligence Safety
Massachusetts Institute of Technology · Cambridge, MA 02139
Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents
Jul 8, 2026Harry Owiredu-AshleyRed-Teaming Attack-Success Rate
Independent Researcher New Jersey, USA
CONTRA: Red-Teaming Configurations of Personalizable Agents
Jul 3, 2026Jonathan Nöther, Adish Singla, Goran RadanovicRed-Teaming Malicious Agents
Max Planck Institute for Software Systems, Germany
Online Safety Monitoring for LLMs
Jul 2, 2026Mona Schirmer, Metod Jazbec, Alexander Timans +3Large Language Model Safety Red-Teaming
UvA Bosch-Delta Lab, University of Amsterdam · University of Wisconsin Madison · Johns Hopkins University
A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
Jun 24, 2026Abrar Alotaibi, Raed Mughus, Moataz AhmedLarge Language Model Reliability Red-Teaming
Information and Computer Science Department, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia. · College of Computer Science and Information Technology, Imam Abdulrahman Bin Faisal University, Dammam, 31441, Saudi Arabia. · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia.
RIFT-Bench: Dynamic Red-teaming For Agentic AI Systems
Jun 22, 2026Yarin Yerushalmi Levi, Roy Betser, Amit Giloni +5Red-Teaming Cyberattacks
1Fujitsu Research of Europe (FRE) · 2Fujitsu Research of India Pvt. Ltd. (FRIPL)
REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs
Jun 22, 2026Yifei Zhao, Qian Lou, Mengxin ZhengAdversarial Evaluation Red-Teaming
University of Central Florida
OTTER: A Red-Teaming System for Toxicity-Evading Jailbreak Prompt Optimization
Jun 19, 2026Jerry Wang, Hsin-Ling Hsu, Yi-Cheng Lai +2Toxicity Red-Teaming
1National Chengchi University, Taipei, Taiwan
NRT-Bench: Benchmarking Multi-Turn Red-Teaming of LLM Operator Agents in Safety-Critical Control Rooms
Jun 18, 2026Hanwool Lee, Dasol Choi, Bokyeong Kim +2Red-Teaming Safety Benchmarks
AIM Intelligence · Republic of Korea · KAERI
FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teaming
Jun 18, 2026Chaeyun Kim, Daeyoung Park, Junghwan Kim +4Red-Teaming Large Language Model Safety
DATUMO INC., Republic of Korea · Korea Advanced Institute of Science and Technology (KAIST), Republic of Korea · Financial Security Institute (FSI), Republic of Korea
Learning Red Agent Policy from Observations for Neurosymbolic Autonomous Cyber Agents
Jun 16, 2026Ankita Samaddar, Sandeep Neema, Daniel Balasubramanian +1Red-Teaming Neuro-Symbolic Framework
Departmaent of Computer Science Vanderbilt University Nashville, TN
A Red-Team Study of Anthropic Fable 5 & Opus 4.8 Models
Jun 16, 2026Nicola FrancoJailbreak Attacks Red-Teaming
Head of AI Security Lab The Italian Institute of Artificial Intelligence (AI4I) Corso Castelfidardo 22, 10129 Turin, Italy
MAStrike: Shapley-Guided Collusive Red-Teaming on Multi-Agent Systems
Jun 11, 2026Chejian Xu, Zhaorun Chen, Jingyang Zhang +5Multi-Agent Communication Red-Teaming
University of Illinois, Urbana-Champaign · Virtue AI · University of Chicago +3
PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections
Jun 10, 2026Pengfei He, Lesly Miculicich, Vishesh Sharma +5Prompt-Injection Detectors Attacker Large Language Model
Google Cloud AI Research · Google · Michigan State University
Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO
Jun 8, 2026Blake Bullwinkel, Eugenia Kim, Amanda Minnich +1Red-Teaming LLM Defense Mechanisms
Microsoft AI Red Team · Microsoft Azure
Diffuse AI Control on Fuzzy Tasks
Jun 8, 2026Mikhail Terekhov, Caglar Gulcehre, Vivek Hebbar +1Artificial Intelligence Safety Sabotage
Anthropic Fellows Program (via MATS) · EPFL · Redwood Research +1
Beyond Pass/Fail: Using Process Mining to Understand How LLMs Resist (and Fail) Red Team Attacks
Jun 5, 2026Zvi TopolRed-Teaming Attacker Large Language Model
MuyVentive, LLC
FoeGlass: Simple In-Context Learning Is Enough for Red Teaming Audio Deepfake Detectors
Jun 3, 2026Sepehr Dehdashtian, Jacob H Seidman, Vishnu N Boddeti +1Deepfake Detection Audio Understanding
Michigan State University · Reality Defender
Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety
Jun 3, 2026Catherine Ge-Wang, Tyler Crosse, Benjamin Hadad +3Artificial Intelligence Safety Artificial Intelligence Control
University of Oxford · Mentorship for Alignment Research Students (MARS), CAISH · Georgia Institute of Technology +3
Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming
Jun 3, 2026Nicholas SabanRed-Teaming Attack-Success Rate
Patronus AI · University of California, Berkeley
AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations
Jun 1, 2026Hiskias Dingeto, William LeeneyRed-Teaming Indirect Prompt Injection
StackOne Technologies
Cross-Generational Transfer of Adversarial Attacks Reveals Non-Monotonic Safety Alignment in LLMs
May 30, 2026Subhadip MitraSafety Alignment Red-Teaming
Research Lead, Rota Labs
Quality-Diversity Evolution for Discovering Diverse Vulnerabilities in LLM Safety
May 30, 2026Subhadip MitraAttacker Large Language Model Large Language Model Safety
Research Lead, Rota Labs
Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South
May 18, 2026Charvi Rastogi, Mukul Bhutani, Minsuk Kahng +13Multilingual Safety Red-Teaming
Google DeepMind, USA · Yonsei University, Republic of Korea · Imperial College, UK +4
Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems
May 12, 2026Zhaojiacheng ZhouRed-Teaming Skills
Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai 200240, China
IPI-proxy: An Intercepting Proxy for Red-Teaming Web-Browsing AI Agents Against Indirect Prompt Injection
May 12, 2026Chia-Pei, Chen, Kentaroh Toyoda +2Indirect Prompt Injection Web Agents
Janet · Vulcan Research, AIFT
Persona-Conditioned Adversarial Prompting: Multi-Identity Red-Teaming for Adversarial Discovery and Mitigation
May 12, 2026Cristian Morasso, Anisa Halimi, Muhammad Zaid Hameed +1Red-Teaming Adversarial Evaluation
IBM Research · Trinity College Dublin, Ireland
Beyond Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers
May 11, 2026Nikita Kezins, Urbas Ekka, Pascal Berrang +1Large Language Model Safety Toxicity
Delft University of Technology · University of Birmingham & Zeroth Research
Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw
May 11, 2026Hongwei Yao, Yiming Liu, Yiling He +1Openclaw Security Evaluation
Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · University College London, London, United Kingdom +1
MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring
May 10, 2026Monika Jotautaitė, Maria Angelica Martinez, Ollie Matthews +1Red-Teaming Coding Agents
Independent · Redwood Research
The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
May 9, 2026Ismail Hossain, Tanzim Ahad, Md Jahangir Alam +3Jailbreak Attacks Jailbreaks
Department of Computer Science, University of Texas at El Paso, TX, USA 79902 · School of Computing, Southern Illinois University Carbondale, IL, USA 62901 · University of Illinois Urbana-Champaign, IL, USA
OTora: A Unified Red Teaming Framework for Reasoning-Level Denial-of-Service in LLM Agents
May 9, 2026Xinyu Li, Ronghui Mu, Lin Li +2LLM Reasoning Strategies Distributed Denial-Of-Service
Department of Computer Science, University of Exeter · Department of Computer Science, University of Oxford · Department of Mathematics and Computer Science, Eindhoven University of Technology
LoopTrap: Termination Poisoning Attacks on LLM Agents
May 7, 2026Huiyu Xu, Zhibo Wang, Wenhui Zhang +4Large Language Model Agents Poisoning
The State Key Laboratory of Blockchain and Data Security Zhejiang University Hangzhou, Zhejiang, P. R. China · School of Cyber Science and Engineering Southeast University Nanjing, Jiangsu, P. R. China
PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AI
May 7, 2026Wesley Hanwen Deng, Mingxi Yan, Sunnie S. Y. Kim +5Red-Teaming Human-Ai Collaboration
Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Apple Seattle, Washington, USA · Apple Cupertino, California, USA +1
DecodingTrust-Agent Platform (DTap): A Controllable and Interactive Red-Teaming Platform for AI Agents
May 6, 2026Zhaorun Chen, Xun Liu, Haibo Tong +14Red-Teaming Artificial Intelligence Agents
Virtue AI · University of Chicago · University of Illinois, Urbana-Champaign +4
Redefining AI Red Teaming in the Agentic Era: From Weeks to Hours
May 5, 2026Raja Sekhar Rao Dheekonda, Will Pearce, Nick LandersRed-Teaming Open-Source
1Dreadnode, USA
ContextualJailbreak: Evolutionary Red-Teaming via Simulated Conversational Priming
May 4, 2026Mario Rodríguez Béjar, Francisco J. Cortés-Delgado, S. Braghin +1Large Language Model Jailbreaks Red-Teaming
Universidad de Murcia · IBM Research
Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balance
May 1, 2026Minchan Kwon, Sunghyun Baek, Minseo Kim +3Generative Flow Networks Large Language Model Safety
1KAIST · 2Naver AI Lab.
Training a General Purpose Automated Red Teaming Model
Apr 24, 2026Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea +1Red-Teaming Attacker Large Language Model
NVIDIA
RedVLA: Physical Red Teaming for Vision-Language-Action Models
Apr 24, 2026Yuhao Zhang, Borong Zhang, Jiaming Fan +4Red-Teaming Unsafe
Institute for AI, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University
AutoRISE: Agent-Driven Strategy Evolution for Red-Teaming Large Language Models
Apr 23, 2026Tanmay Gautam, Alireza Bahramali, Sandeep AtluriRed-Teaming Attacker Large Language Model
Responsible AI, Microsoft

Oct 4, 2026, cs.CLJ/K move · Enter open · S save

Red-TTT: Test-Time Training for Automated Jailbreaking Large Language Models

Tongyan Hu, Hao Li, Xiaogeng Liu, Ruida Wang+7

Johns Hopkins University · National University of Singapore · Washington University in St. Louis+2

Large Language Model Jailbreaks Red-Teaming

Open paper AskPDF arXiv

Large language models remain vulnerable to jailbreaks, and automated red teaming is the standard way to find jailbreaks in large language models at scale. Current methods either draw more samples at test time through search, rewriting, and tree expansion, or train a stronger attacker offline with reinforcement learning. Both share a limitation: once an attack on a specific target behavior begins, the attacker's weights are frozen. Any signal it gathers about the behavior stays in its context window and is discarded afterward. The attacker never adapts its proposal distribution mid-attack, so success depends almost entirely on the sampling budget, and under a budget affordable at scale, many behaviors remain unbroken. We propose Red-TTT, which updates the attacker's parameters during the attack on each behavior. At each round, the attacker samples a group of candidates, scores them against the victim's replies, and takes a policy-gradient step before drawing the next group, so what it discovers about the current victim is consolidated into weights rather than accumulated as context. We also adapt the training objective to red teaming, where success is judged by the single best sample rather than the average. Red-TTT requires only sampling access to the victim and integrates into existing attack pipelines with no other changes. Against the Best-of-N baseline, Red-TTT raises attack success rate from 55.9% to 72.4% on average at a budget of 120 samples, improving over the baseline in every configuration and cracking many behaviors previous method cannot. The code is available at https://github.com/SaFo-Lab/Red-TTT

Momentum

Related topics

Latest papers 71