Black-Box Adversarial Attacks

Ask about this topic

Overview

Black-box adversarial attacks ask how an attacker can cause an AI system to misclassify, produce unsafe output, or take an unwanted action without access to its internal parameters or gradients. Methods use feedback from model queries—such as labels, scores, or responses—with decision-boundary search, finite differences, bandits, evolutionary or reinforcement-learning optimization; others craft perturbations on surrogate models and rely on transfer to unseen targets. Work spans images and physical patches, text and jailbreak prompts, audio, graphs, malware detectors, and tool-using or multimodal agents, often under constraints such as limited queries, imperceptibility, and preserving input meaning or function. Key tensions include whether attacks transfer across architectures and real-world transformations, how to compare them under fair query budgets, and whether defenses can resist adaptive attacks without sacrificing clean performance or utility.

Latest papers

CardsList

175 papers16 papers in the last four weeks, down 6% on the four weeks before. 0.1% of all new papers.

  1. Led to Mislead: Adversarial Content Injection for Attacks on Neural Ranking Models

    May 2, 2026Amin Bigdeli, Amir Khosrojerdi, Radin Hamidi Rad +3Learning to RankAdversarial Attacks

    University of Waterloo, Canada · University of Toronto, Canada · Mila – Quebec AI Institute, Canada +1

  2. Low Rank Adaptation for Adversarial Perturbation

    Apr 30, 2026Han Liu, Shanghao Shi, Yevgeniy Vorobeychik +2Adversarial AttacksLow-Rank Approximation

    Washington University in St. Louis

  3. Agentic Adversarial Rewriting Exposes Architectural Vulnerabilities in Black-Box NLP Pipelines

    Apr 26, 2026Mazal Bethany, Kim-Kwang Raymond Choo, Nishant Vishwamitra +1Fake News DetectionAdversarial Prompt Generation

    Department of Information Systems and Cybersecurity, University of Texas at San Antonio, San Antonio, TX 78249 USA

  4. Transient Turn Injection: Exposing Stateless Multi-Turn Vulnerabilities in Large Language Models

    Apr 23, 2026Naheed Rayhan, Sohely JahanLanguage Model Safety EvaluationAdversarial Attacks on LLMs

    Jagannath University · University of Barishal

  5. EvoJail: Evolutionary Diverse Jailbreak Prompt Generation for Large Language Models

    Apr 22, 2026Rui Tang, Kaiyu Xu, Pengsen Cheng +3Adversarial Prompt GenerationLLM Jailbreak Attacks

    aSchool of Cyber Science and Engineering, Sichuan University, Chengdu 610065, China · dCollege of Computer Science, Sichuan University, Chengdu 610065, China · aSchool of Cyber Science and Engineering, Sichuan University, Chengdu 610065, China bKey Laboratory of Data Protection and Intelligent Management (Sichuan University), Ministry of Education, Chengdu 610065, China

  6. Beyond Explicit Refusals: Soft-Failure Attacks on Retrieval-Augmented Generation

    Apr 20, 2026Wentao Zhang, Yan Zhuang, ZhuHang Zheng +3Retrieval-Augmented GenerationRAG Security

    University of Electronic Science and Technology of China, Chengdu, China · Shenzhen Institute for Advanced Study, UESTC, Shenzhen, China

  7. When Background Matters: Breaking Medical Vision Language Models by Transferable Attack

    Apr 19, 2026Akash Ghosh, Subhadip Baidya, Sriparna Saha +1VLM RobustnessAdversarial Attacks on VLMs

    Indian Institute of Technology Patna1 · Work done as Visiting Researcher at MBZUAI · MBZUAI3 +1

  8. Route to Rome Attack: Directing LLM Routers to Expensive Models via Adversarial Suffix Optimization

    Apr 16, 2026Haochun Tang, Yuliang Yan, Jiahua Lu +2Adversarial Prompt GenerationCost-Aware Inference

    1Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin University · 2The Hong Kong University of Science and Technology (Guangzhou)

  9. Physically-Induced Atmospheric Adversarial Perturbations: Enhancing Transferability and Robustness in Remote Sensing Image Classification

    Apr 16, 2026Weiwei Zhuang, Wangze Xie, Qi Zhang +9Remote Sensing Image UnderstandingTransfer-Based Adversarial Attacks

    School of Computer and Information Engineering, Xiamen University of Technology, Xiamen, 361000, China · Faculty of Data Science, City University of Macau, Macau SAR, China · Dundee International Institute, Central South University, Changsha, China +7

  10. Fragile Reconstruction: Adversarial Vulnerability of Reconstruction-Based Detectors for Diffusion-Generated Images

    Apr 14, 2026Haoyang Jiang, Mingyang Yi, Shaolei Zhang +4Adversarial Attacks on AI-Generated Image DetectorsAI-Generated Image Detection

    Renmin University of China, Beijing, China · Tencent Inc., Shenzhen, China

  11. HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models

    Apr 14, 2026Han Liu, Jiaqi Li, Zhi Xu +5Adversarial Attacks on VLMsBlack-Box Adversarial Attacks

    Dalian University of Technology, Dalian, China · The Pennsylvania State University, Pennsylvania, USA · Shenzhen University, Shenzhen, China

  12. Beyond A Fixed Seal: Adaptive Stealing Watermark in Large Language Models

    Apr 13, 2026Shuhao Zhang, Yuli Chen, Jiale Han +2Adversarial Attacks on LLMsLLM Watermarking

    State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications · Hong Kong University of Science and Technology

  13. Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection

    Apr 9, 2026Wenkui Yang, Chao Jin, Haisu Zhu +7GUI AgentsAI Agent Security

    School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · 2MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences · 4ShanghaiTech University +2

  14. Reading Between the Pixels: An Inscriptive Jailbreak Attack on Text-to-Image Models

    Apr 7, 2026Zonghao Ying, Haowen Dai, Lianyu Hu +5Black-Box Adversarial AttacksSafe T2I Generation

    Beihang University Beijing, China · University of Nottingham Ningbo China Ningbo, China · Nanyang Technological University Singapore +2

  15. Adversarial Robustness of Deep State Space Models for Forecasting

    Apr 3, 2026Sribalaji C. Anand, George J. PappasAdversarial RobustnessState Space Models

    Department of Electrical and Systems Engineering, University of Pennsylvania, United States · KTH Royal Institute of Technology, Sweden

  16. Low-Effort Jailbreak Attacks Against Text-to-Image Safety Filters

    Apr 2, 2026Ahmed B Mustafa, Zihan Ye, Yang Lu +2Adversarial Attacks on VLMsSafe T2I Generation

    School of Computer Science, University of Nottingham, Wollaton Road, NG8 1BB · Department of Intelligent Science, Xi’an Jiaotong-Liverpool University, China, 215123 · School of Informatics, Xiamen University, Xiamen, 361005, China

  17. From Pixels to Reality: Physical-Digital Patch Attacks on Real-World Camera

    Mar 30, 2026Victoria Leonenkova, Ekaterina Shumitskaya, Dmitriy Vatolin +1Adversarial AttacksAdversarial Patch Attacks

    Lomonosov MSU, Moscow, Russia · AI Center, Lomonosov MSU, IAI MSU, Moscow, Russia

  18. DTP-Attack: A decision-based black-box adversarial attack on trajectory prediction

    Mar 27, 2026Jiaxiang Li, Jun Yan, Daniel Watzenig +1Black-Box Adversarial AttacksAdversarial Robustness

    College of Electronics and Information Engineering, Tongji University, Shanghai 201804, China. · Graz University of Technology and the Virtual Vehicle Research, Graz 8010, Austria.

  19. PEANUT: Perturbations by Eigenvector Alignment for Attacking Graph Neural Networks Under Topology-Driven Message Passing

    Mar 27, 2026Bhavya Kohli, Biplab SikdarAdversarial AttacksGraph Neural Networks

    National University of Singapore, Singapore

  20. Invisible Threats from Model Context Protocol: Generating Stealthy Injection Payload via Tree-based Adaptive Search

    Mar 25, 2026Yulin Shen, Xudong Pan, Geng Hong +1Prompt Injection Attacks on AI AgentsMCP Security

    School of Computer Science, Fudan University · Shanghai Innovation Institute

  21. AdvSplat: Adversarial Attacks on Feed-Forward Gaussian Splatting Models

    Mar 24, 2026Yiran Qiao, Yiren Lu, Yunlai Zhou +4Adversarial Perturbations3D Gaussian Splatting

    Case Western Reserve University, Cleveland OH 44106, USA

  22. Targeted Adversarial Traffic Generation : Black-box Approach to Evade Intrusion Detection Systems in IoT Networks

    Mar 24, 2026Islam Debicha, Tayeb Kenaza, Ishak Charfi +3Adversarial AttacksIoT Intrusion Detection

  23. Adversarial Vulnerabilities in Neural Operator Digital Twins: Gradient-Free Attacks on Nuclear Thermal-Hydraulic Surrogates

    Mar 23, 2026Samrendra Roy, Kazuma Kobayashi, Souvik Chakraborty +2Adversarial RobustnessNeural Operators

    Department of Nuclear, Plasma & Radiological Engineering, University of Illinois Urbana-Champaign, Urbana, IL 61801, USA · Department of Applied Mechanics, Indian Institute of Technology Delhi, New Delhi 110016, India · National Center for Supercomputing Applications, University of Illinois Urbana-Champaign, Urbana, IL 61801, USA

  24. JANUS: A Lightweight Framework for Jailbreaking Text-to-Image Models via Distribution Optimization

    Mar 22, 2026Haolun Zheng, Yu He, Tailun Chen +6Adversarial Attacks on VLMsBlack-Box Adversarial Attacks

    The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou HighTech Zone (Binjiang) Blockchain and Data Security Research Institute, China · Alibaba Group

  25. Can LLMs Fool Graph Learning? Exploring Universal Adversarial Attacks on Text-Attributed Graphs

    Mar 22, 2026Zihui Chen, Yuling Wang, Pengfei Jiao +4Adversarial Text AttacksText-Attributed Graph Learning

    School of Cyberspace, Hangzhou Dianzi University, Hangzhou, China · Hangzhou Dianzi University, Hangzhou, China · Beihang University, Beijing, China +1

  26. OmniPatch: A Universal Adversarial Patch for ViT-CNN Cross-Architecture Transfer in Semantic Segmentation

    Mar 21, 2026Aarush Aggarwal, Akshat Tomar, Amritanshu Tiwari +1Adversarial Patch AttacksAdversarial Attacks

    Data Science Group, Indian Institute of Technology Roorkee, India

  27. REFORGE: Multi-modal Attacks Reveal Vulnerable Concept Unlearning in Image Generation Models

    Mar 17, 2026Yong Zou, Haoran Li, Fanxiao Li +5Adversarial AttacksAdversarial Robustness

    Yunnan University · Northeastern University · National University of Singapore

  28. AI Evasion and Impersonation Attacks on Facial Re-Identification with Activation Map Explanations

    Mar 16, 2026Noe Claudel, Weisi Guo, Yang XingAdversarial AttacksAdversarial Patch Attacks

    All authors are with the Centre for Assured and Connected Autonomy, Cranfield University, Bedford, UK.

  29. PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses

    Mar 13, 2026Chenlong Yin, Runpeng Geng, Yanting Wang +1Prompt Injection Attacks on LLMsLLM Red Teaming

    The Pennsylvania State University

  30. PivotAttack: Rethinking the Search Trajectory in Hard-Label Text Attacks via Pivot Words

    Mar 11, 2026Yuzhi Liang, Shiliang Xiao, Jingsong Wei +2Adversarial Text AttacksBlack-Box Adversarial Attacks

    Guangdong University of Foreign Studies, China