Black-Box Adversarial Attacks

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 115

All topics

Assessing Automated Prompt Injection Attacks in Agentic Environments
Jun 9, 2026David Hofer, Edoardo Debenedetti, Florian TramèrAdversarial Prompt Generation LLM Agent Security
ETH Zurich Zurich, Switzerland
What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks
Jun 8, 2026Qin Yang, Lu Malloy, Joshua Lee +4Language Model Safety Evaluation Black-Box Adversarial Attacks
University of Connecticut · University of Tennessee · University of California, Santa Barbara +1
SHIELD-IDS: Structurally Heterogeneous Ensemble with Integrated Layered Defense for Intrusion Detection Systems
Jun 5, 2026Maryam Zaman, Muhammad Khuram ShahzadEnsemble Learning Network Intrusion Detection
School of Electrical Engineering and Computing(SEECS) · 2National University of Sciences and Technology, Islamabad, Pakistan
Beyond Waveform Robustness: Robust Feature-Vocoder Adversarial Attacks on Automatic Speech Recognition
Jun 4, 2026Yifan Liao, Zongmin Zhang, Zhen Sun +3Adversarial Attacks Transfer-Based Adversarial Attacks
The Hong Kong University of Science and Technology (Guangzhou) · Wuhan University
FoeGlass: Simple In-Context Learning Is Enough for Red Teaming Audio Deepfake Detectors
Jun 3, 2026Sepehr Dehdashtian, Jacob H Seidman, Vishnu N Boddeti +1Audio Deepfake Detection Adversarial Robustness
Michigan State University · Reality Defender
Black-box, Adaptive, Efficient, Transferable, Harmful, Applicable... Attacks Are All You Need to Break LLMs
Jun 2, 2026Vincent Limbach, Jonas Dornbusch, David Lüdke +2LLM Jailbreak Attacks LLM Safety Evaluation
Department of Computer Science, Technical University of Munich, Germany · Munich Data Science Institute · Munich Center for Machine Learning +1
ATLAS: A Large-Scale Evaluation Benchmark for Adversarial LiDAR Perception
Jun 1, 2026Mellon M. Zhang, Siddhant Panse, Zimo Fan +3Lidar-Based 3D Object Detection Adversarial Attacks
Georgia Institute of Technology
MaskForge: Structure-Aware Adaptive Attacks for Jailbreaking Diffusion Large Language Models
Jun 1, 2026Yingzi Ma, Zhengyue Zhao, Xiaogeng Liu +3Masked Diffusion Models Adversarial Attacks on LLMs
University of Wisconsin-Madison · Johns Hopkins University · Responsible AI Research (RAIR) Centre, The University of Adelaide +1
DiscourseFlip: An Oblique Discourse-Level Opinion Manipulation Attack against Black-box Retrieval-Augmented Generation
May 31, 2026Yuyang Gong, Miaokun Chen, Jiawei Liu +5RAG Poisoning Attacks RAG Security
Wuhan University · East China Normal University · Nanyang Technological University +1
Latent Geometric Chords for Query-Efficient Decision-Based Adversarial Attacks
May 29, 2026Ei Hmue Khine, Yao Li, Jiebao Sun +3Adversarial Attacks Adversarial Perturbations
School of Mathematics, Harbin Institute of Technology, Harbin 150001, China
EvoDefense: Co-Evolving Black-Box Defense with Large Language Models
May 29, 2026Yu Li, Yuenan Hou, Yingmei Wei +2Evolutionary Optimization Adversarial Attacks on LLMs
National University of Defense Technology · Shanghai AI Laboratory
Audio Pirates: Black-box Audio Watermark Removal via Diffusion Priors
May 28, 2026Lingfeng Yao, Xincong Zhong, Chenpei Huang +6Watermark Removal Audio Watermarking
University of Houston · Waseda University · UC Berkeley +3
PAST2HARM: A Simple Adaptive Past Tense Attack for Jailbreaking Multimodal AI
May 26, 2026Snehasis MukhopadhyayAdversarial Prompt Generation Multimodal Jailbreak Attacks
Indian Institute of Information Technology, Kalyani
Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
May 26, 2026Xiang Fang, Wanlong Fang, Changshuo WangVLM Robustness Adversarial Attacks on VLMs
School of Software Engineering, Huazhong University of Science and Technology · Nanyang Technological University, Singapore · University College London
Erased but Exploitable: Black-box Embedding-Aware Prompting Against Unlearned Text-to-Image Diffusion Models
May 25, 2026Arian Komaei Koma, Seyed Amir Kasaei, AmirMahdi Sadeghzadeh +1Adversarial Prompt Generation Adversarial Attacks
Department of Computer Engineering Sharif University of Technology
Opportunistic Target Selection: Early Directional Commitment for Query-Efficient Black-Box Adversarial Attacks
May 25, 2026Florent Tariolle, Florian YgerAdversarial Attacks Black-Box Adversarial Attacks
INSA Rouen Normandy · LITIS
SEP-Attack: A Simple and Effective Paradigm for Transfer-Based Textual Adversarial Attack
May 24, 2026Han Liu, Zhi Xu, Xiaotong Zhang +5Determinantal Point Process Transfer-Based Adversarial Attacks
Dalian University of Technology Dalian, China · Peking University Beijing, China · Macao Polytechnic University Macao, China +1
IterInject: Indirect Prompt Injection Against LLM Agents via Feedback-Guided Iterative Optimization
May 23, 2026Zixuan Chen, Jiaxiang Chen, Li Luo +4LLM Agent Security Indirect Prompt Injection
1Shanghai Jiao Tong University · 2The University of Hong Kong
LASH: Adaptive Semantic Hybridization for Black-Box Jailbreaking of Large Language Models
May 20, 2026Abdullah Al Nomaan Nafi, Fnu Suya, Swarup Bhunia +1Adversarial Prompt Generation LLM Jailbreak Attacks
University of Maine · University of Tennessee, Knoxville · University of Florida
Learning to Look Benign: Targeted Evasion of Malware Detectors via API Import Injection
May 18, 2026Juozas Dautartas, Olga Kurasova, Juozapas Rokas Čypas +1Adversarial Attacks Malware Classification
Institute of Data Science and Digital Technologies, Faculty of Mathematics and Informatics, Vilnius University, LT-08412 Vilnius, Lithuania
Attacking the First-Principle: A Black-Box, Query-Free Targeted Mimicry Attack on Binary Function Classifiers
May 18, 2026Gabriel Sauger, Jean-Yves Marion, Sazzadur Rahaman +3Adversarial Attacks Malware Classification
UL, CNRS, LORIA, Inria · ∗Universit´e de Lorraine, CNRS, LORIA, F-54000 Nancy, France · CNRS, UL, LORIA, Inria +2
Whispers in the Noise: Surrogate-Guided Concept Awakening via a Multi-Agent Framework
May 18, 2026Mengyu Sun, Ziyuan Yang, Zunlong Zhou +3Adversarial Attacks Black-Box Adversarial Attacks
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · Lee Kong Chian School of Medicine, Nanyang Technological University · School of Cyber Science and Engineering, Sichuan University
An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments
May 18, 2026Hongjang Yang, Hyunsik Na, Daeseon ChoiData Leakage Privacy Leakage in Language Models
Department of Information Security Soongsil University Seoul, Korea · AI Safety Center Soongsil University Seoul, Korea · Department of AI Software Soongsil University Seoul, Korea
Threats to Arabic Handwriting Recognition: Investigating Black-Box Adversarial Attacks on embedded ConvNet models
May 18, 2026Mohsine EL Khayati, Abdelillah Semma, Abdelaziz Courr +1Adversarial Robustness Black-Box Adversarial Attacks
Systems theory and informatics laboratory Moulay Ismail University of Meknes Meknes, Morocco · Department of Computer Science EST of Sidi Bennour Chouaib Doukkali University Eljadida, Morocco · Faculty of Education Sciences University Mohammed V Rabat, Morocco +1
Babel: Jailbreaking Safety Attention via Obfuscation Distribution Optimized Sampling
May 18, 2026Ziwei Wang, Jing Chen, Ruichao Liang +6LLM Safety LLM Jailbreak Attacks
Wuhan University · Nanyang Technological University · Southeast University +1
MoCo-EA: Exploiting Adversarial Mode Connectivity for Efficient Evolutionary Attacks
May 18, 2026Hyo Seo Kim, Gang Luo, Can Chen +3Evolutionary Optimization Adversarial Attacks
Illinois Institute of Technology · University of North Carolina at Chapel Hill · Singapore Management University
Escaping the Linearity Trap: Manifold Detours for Black-Box Adversarial Attacks on Singing Audio Deepfake Detection
May 18, 2026Yifan Liao, Yule Liu, Zhen Sun +5Audio Deepfake Detection Transfer-Based Adversarial Attacks
Wuhan University · The Hong Kong University of Science and Technology (Guangzhou)
Towards Universal Physical Adversarial Attacks via a Joint Multi-Objective and Multi-Model Optimization Framework
May 18, 2026Ziyang Liu, Hongyuan Wang, Zijian Wang +4Transfer-Based Adversarial Attacks Black-Box Adversarial Attacks
Research Center for Space Optical Engineering, Harbin Institute of Technology, Harbin 150001, China · Zhengzhou Research Institute, Harbin Institute of Technology, Zhengzhou 450000, China
SoK: A Comprehensive Analysis of the Current Status of Neural Tangent Generalization Attacks with Research Directions
May 12, 2026Thushari Hapuarachchi, Kaiqi XiongAdversarial Attacks Adversarial Robustness
University of South Florida Tampa, USA
Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw
May 11, 2026Hongwei Yao, Yiming Liu, Yiling He +1AI Agent Security LLM Agent Security
Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · University College London, London, United Kingdom +1

Jun 9, 2026, cs.CRJ/K move · Enter open · S save

Assessing Automated Prompt Injection Attacks in Agentic Environments

David Hofer, Edoardo Debenedetti, Florian Tramèr

ETH Zurich Zurich, Switzerland

Adversarial Prompt Generation LLM Agent Security AI Agent Security Benchmarks Indirect Prompt Injection Prompt Injection Attacks Prompt Injection Attacks on LLMs+2

Open paper AskPDF arXiv

Indirect prompt injection poses a critical threat to LLM agents that interact with untrusted external data, yet automated attack methods--proven effective for jailbreaking--remain underexplored in realistic agentic settings. We present a comprehensive empirical evaluation of automated prompt injection attacks against LLM agents, adapting both white-box (GCG) and black-box (TAP) methods to the agentic setting within the AgentDojo framework. We evaluate across 80 task pairs spanning four domains and multiple models, and find that black-box optimization substantially outperforms gradient-based methods, a gap we attribute to GCG's optimization instability under reasonable compute budgets. We also find that TAP's effectiveness depends on the attacker model, as both general capability and safety tuning affect attack success--stronger models produce more effective injections, while safety-tuned attackers can refuse to generate adversarial prompts. Task-universal attacks transfer effectively to unseen tasks and out-of-distribution domains, but attacks optimized on smaller open-source models do not transfer to frontier models like GPT-5. These findings highlight automated prompt injection as a credible but model-dependent threat, with significant barriers remaining for model-agnostic exploitation.

Momentum

Related topics

Latest papers 115