Automated Red Teaming

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 35

All topics

Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution
Oct 7, 2026Wanjing Han, Levi Taiji Li, Mu Zhang +2AI Agent Security Tool-Using Vision-Language Agents
University of Utah
Red-TTT: Test-Time Training for Automated Jailbreaking Large Language Models
Oct 4, 2026Tongyan Hu, Hao Li, Xiaogeng Liu +8Test-Time Training LLM Jailbreak Attacks
Johns Hopkins University · National University of Singapore · Washington University in St. Louis +2
No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents
Sep 30, 2026Ayan Javeed Shaikh, Arunesh Sinha, Nathaniel D. Bastian +1LLM Red Teaming AI Agent Evaluation
Indiana University Bloomington, IN, USA · Rutgers University Piscataway, NJ, USA · Johns Hopkins University Baltimore, MD, USA
Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents
Sep 17, 2026Alex Remedios, Simon Storf, Fabien Roger +1AI Agent Security AI Agent Monitoring
Anthropic Fellows Program · Anthropic
Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Sep 10, 2026Divyanshu Kumar, Nitin Aravind Birur, Tanay Baswa +2AI Agent Security AI Agent Security Benchmarks
Enkrypt AI
EvoSkill Injection: Red-Teaming Autonomous Skill Generation and Evolution in Self-Evolving Agents
Aug 31, 2026Doyun Kim, Chanwoo Kim, Sugyeong Eo +2AI Agent Security AI Agent Security Benchmarks
Soongsil University · Yonsei University · Jeju National University
SIR: Self-improving Red-teaming for Compute Use Agents
Aug 31, 2026Chen Xiong, Zhiyuan He, Pin-Yu Chen +2Computer-Use Agents Indirect Prompt Injection
The Chinese University of Hong Kong · IBM Research · University of Zagreb, FER & Radboud University +1
IO Factory: Simulating AI-Enabled Influence Campaigns at Scale
Aug 11, 2026Lukasz Olejnik, Wenchao Dong, Jonas R. Kunst +4Agent-Based Modeling Automated Red Teaming
Department of War Studies, King’s College London, London, United Kingdom · Independent researcher · Max Planck Institute for Security and Privacy, Bochum, Germany +4
Generating Attacks for LLMs with GFlowNets
Aug 10, 2026Berkay Ozcam, Irem Onen, Mehmet Fatih Amasyali +1Adversarial Prompt Generation Adversarial Attacks on LLMs
Cybersecurity R&D Turkcell Istanbul, Turkey · Department of Computer Engineering Yıldız Technical University Istanbul, Turkey
Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)
Aug 5, 2026Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi +6Autonomous Cyber Defense LLM Red Teaming
University of California, Irvine · Northeastern University · Johns Hopkins University
Salami Attack: Stealthy Collusive Memory Poisoning against OpenClaw
Aug 3, 2026Zheng Lin, Yuzhe Huang, Zhenxing Niu +2Agent Memory Poisoning Automated Red Teaming
Xidian University Xi’an, Shaanxi, China
GPT-Red: Automated Red Teaming via Self-Play at Scale
Jul 28, 2026Eric Wallace, Christopher A. Choquette-Choo, Nikhil Kandpal +15Adversarial Training LLM Red Teaming
OpenAI
Automatic Hard Example Synthesis with Multi-Level Agentic Data Curation
Jul 15, 2026Genglin Liu, Muye Zhang, Krishnamurthy Viswanathan +5Multimodal Robustness Adversarial Scenario Generation
1UCLA · *Work done as a student researcher at Google. · 2Google
Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming
Jul 13, 2026Xutao Mao, Xiang Zheng, Cong WangLLM Agent Security Automated Red Teaming
City University of Hong Kong
Mako: A Self-Evolving Agentic Operating System (SE-AOS) for Autonomous Web Exploitation
Jul 13, 2026Praneeth Narisetty, Shiva Nagendra Babu KoreAutomated Penetration Testing Automated Red Teaming
LaunchSafe
RIFT-Bench: Dynamic Red-teaming For Agentic AI Systems
Jun 22, 2026Yarin Yerushalmi Levi, Roy Betser, Amit Giloni +5AI Agent Security AI Agent Security Benchmarks
1Fujitsu Research of Europe (FRE) · 2Fujitsu Research of India Pvt. Ltd. (FRIPL)
MAStrike: Shapley-Guided Collusive Red-Teaming on Multi-Agent Systems
Jun 11, 2026Chejian Xu, Zhaorun Chen, Jingyang Zhang +5AI Agent Security Benchmarks Shapley Value Attribution
University of Illinois, Urbana-Champaign · Virtue AI · University of Chicago +3
Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops
Jun 8, 2026Ziqian Zhong, Ivgeni Segal, Ivan Bercovich +3Reward Hacking AI Agent Security Benchmarks
Carnegie Mellon University · Fewshot Corp · Independent Researcher
Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
May 12, 2026Hao Wang, Hanchen Li, Qiuyang Mang +3Computer-Use Agent Benchmarks Reward Hacking
UC Berkeley
Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems
May 12, 2026Zhaojiacheng ZhouLLM Agent Security Automated Red Teaming
Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai 200240, China
IPI-proxy: An Intercepting Proxy for Red-Teaming Web-Browsing AI Agents Against Indirect Prompt Injection
May 12, 2026Chia-Pei, Chen, Kentaroh Toyoda +2AI Agent Security AI Agent Security Benchmarks
Janet · Vulcan Research, AIFT
Persona-Conditioned Adversarial Prompting: Multi-Identity Red-Teaming for Adversarial Discovery and Mitigation
May 12, 2026Cristian Morasso, Anisa Halimi, Muhammad Zaid Hameed +1Adversarial Training Adversarial Prompt Generation
IBM Research · Trinity College Dublin, Ireland
Comment and Control: Hijacking Agentic Workflows via Context-Grounded Evolution
May 11, 2026Neil Fendley, Zhengyu Liu, Aonan Guan +2Adversarial Attacks on LLMs Agentic Workflows
Johns Hopkins University Maryland, USA · Wyze Labs Washington, USA
MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring
May 10, 2026Monika Jotautaitė, Maria Angelica Martinez, Ollie Matthews +1Software Engineering Agents AI Agent Security Benchmarks
Independent · Redwood Research
PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AI
May 7, 2026Wesley Hanwen Deng, Mingxi Yan, Sunnie S. Y. Kim +5Adversarial Prompt Generation LLM Red Teaming
Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Apple Seattle, Washington, USA · Apple Cupertino, California, USA +1
DecodingTrust-Agent Platform (DTap): A Controllable and Interactive Red-Teaming Platform for AI Agents
May 6, 2026Zhaorun Chen, Xun Liu, Haibo Tong +14AI Agent Evaluation AI Agent Security
Virtue AI · University of Chicago · University of Illinois, Urbana-Champaign +4
Redefining AI Red Teaming in the Agentic Era: From Weeks to Hours
May 5, 2026Raja Sekhar Rao Dheekonda, Will Pearce, Nick LandersAdversarial Attacks on LLMs LLM Red Teaming
1Dreadnode, USA
Training a General Purpose Automated Red Teaming Model
Apr 24, 2026Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea +1Adversarial Attacks on LLMs LLM Red Teaming
NVIDIA
AutoRISE: Agent-Driven Strategy Evolution for Red-Teaming Large Language Models
Apr 23, 2026Tanmay Gautam, Alireza Bahramali, Sandeep AtluriLarge Language Model-Guided Optimization Program Synthesis
Responsible AI, Microsoft
Adaptive Instruction Composition for Automated LLM Red-Teaming
Apr 22, 2026Jesse Zymet, Andy Luo, Swapnil Shinde +2Adversarial Prompt Generation LLM Red Teaming
Capital One, AI Foundations

Oct 7, 2026, cs.CRJ/K move · Enter open · S save

Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution

Wanjing Han, Levi Taiji Li, Mu Zhang, Yue Jiang+1

University of Utah

AI Agent Security Tool-Using Vision-Language Agents Automated Red Teaming Adversarial Attacks on VLMs Visual Grounding

Open paper AskPDF arXiv

Modern web agents built on large vision-language models process webpages, select relevant UI elements, and translate model outputs into browser actions. Existing visual red-teaming approaches use adversarial visual content to manipulate this process. However, they primarily target model inference and do not explicitly account for structured input processing or action post-processing. Consequently, model-level success does not establish control over browser execution and cannot reliably characterize end-to-end agent robustness. To address this gap, we formulate red teaming for vision-grounded web agents as an end-to-end grounding-to-execution problem, and introduce WebMirage, a framework that crafts localized visual perturbations that cause agents to select attacker-controlled content and execute the corresponding browser action across varying webpage renderings. It uses a role-slot abstraction and webpage recomposition to capture competition among webpage elements, and dataflow analysis to align optimization with action post-processing. We evaluate WebMirage across four agent configurations and six VLM backbones on 2,250 tasks covering 13 public websites and a sandbox benchmark. WebMirage achieves an average attack success rate of 91.9%, compared with 17.4% for the strongest baseline, and remains effective against three agent-level defenses.

Momentum

Related topics

Latest papers 35