Adversarial Prompt Generation

Ask about this topic

Overview

Adversarial prompt generation studies how to construct inputs that bypass safeguards, induce failures, or reveal weaknesses in language models, agents, and image-generation systems. Common approaches include evolutionary and reinforcement-learning search, gradient or embedding optimization, black-box feedback, and compositional or multi-turn attacks; some use semantic obfuscation, personas, or tool-use trajectories to find less obvious vulnerabilities. Researchers evaluate attacks on benchmarks such as HarmBench and JailbreakBench, as well as in agent and multimodal settings, using automated judges and measures of success, diversity, and transfer. A central tension is that reported success depends strongly on the target model, query budget, judge, and stopping rule, while realistic and repeatable attacks must also transfer across systems and withstand adaptive defenses.

Latest papers

CardsList

67 papers4 papers in the last four weeks, down 20% on the four weeks before. 0.0% of all new papers.

  1. Target-Aware Curation of Hallucination Stress Tests for Large Language Models

    Oct 19, 2026Jiho Lee, Bingquan Zhang, Lin Ting Liang +2LLM EvaluationLanguage Model Generation Evaluation

    Dartmouth College · Oakland University

  2. REPRO: Reflective Evolutionary Prompt Optimizer for Automated LLM Red-Teaming

    Oct 11, 2026Abrar Alotaibi, Moataz AhmedAdversarial Prompt GenerationAdversarial Attacks on LLMs

    King Fahd University of Petroleum & Minerals, Information and Computer Science Department, Dhahran, Saudi Arabia · Imam Abdulrahman Bin Faisal University, College of Computer Science and Information Technology, Dammam, Saudi Arabia · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals, Dhahran, Saudi Arabia

  3. FinRT: Distilling Adaptive Red-Teaming Strategies into Reusable Adversarial Generators in Consumer Finance

    Sep 29, 2026Rikhiya Ghosh, Himanshu Kumar, Sriram Venkatapathy +3Adversarial Prompt GenerationAdversarial Attacks on LLMs

    AI Foundations, Capital One

  4. RISE: Red-teaming via Iterative Strategy Evolution for Modern Text-to-Image Models

    Sep 28, 2026Dmitrii Kharlapenko, Sergei Bratchikov, Konstantin Korolev +1Adversarial Prompt GenerationT2I Generation Evaluation

    White Circle

  5. ENDOPROMPT: Victim-Side Pseudo-References for Utility Degradation

    Sep 24, 2026Qingyu Wu, Zeyu Feng, Yongda Yu +2Adversarial Prompt GenerationPrompt Injection Attacks

    Defense Innovation Institute, Academy of Military Science, Beijing, China · Nanjing University, Nanjing, China

  6. Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions

    Sep 23, 2026Tiantong Wu, Wei Yang Bryan LimAdversarial Prompt GenerationPrompt Injection Attacks

    Nanyang Technological University Singapore

  7. Automatic Red Teaming for Implicit Vulnerabilities of Text-to-Image Models

    Sep 5, 2026Chang Ma, Junlin Han, Shuo Chen +3Safe T2I GenerationAdversarial Prompt Generation

    Columbia University, USA · University of Oxford, UK · LMU Munich, Germany +2

  8. ASCII Attack: Recontextualising Harmful Requests as Artistic Critique in Large Language Models

    Sep 2, 2026Da Cheng Gu, Yifei Dong, Xinghao Yang +2LLM Jailbreak AttacksLLM Safety Evaluation

    Faculty of Engineering and Information Technology, University of Technology Sydney, Ultimo, NSW, 2007, Australia · China University of Petroleum, China · Shandong University, China

  9. Breadth Beats Depth: Improving GCG-Based Jailbreak Optimization with Breadth-Oriented Suffix Search

    Sep 2, 2026Shiliang Xiao, Jingsong Wei, Yuzhi Liang +3LLM Jailbreak AttacksAdversarial Prompt Generation

    School of Information Science and Technology Guangdong University of Foreign Studies

  10. InjecMEM: Memory Injection Attack on LLM Agent Memory Systems

    Aug 24, 2026Hanling Tian, Gengyu Zhang, Zeyang Sha +5LLM Agent MemoryPrompt Injection Attacks on AI Agents

    Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University · Ant Group

  11. Generating Attacks for LLMs with GFlowNets

    Aug 10, 2026Berkay Ozcam, Irem Onen, Mehmet Fatih Amasyali +1Adversarial Prompt GenerationAdversarial Attacks on LLMs

    Cybersecurity R&D Turkcell Istanbul, Turkey · Department of Computer Engineering Yıldız Technical University Istanbul, Turkey

  12. Mind the Gap: Zero-Query Jailbreaks via Filter-Generator Discrepancy in Text-to-Image Systems

    Aug 2, 2026Wanguang Li, Zhaoxin Wang, Handing WangAdversarial Prompt GenerationT2I Generation

    School of Artificial Intelligence, Xidian University, Shaanxi, China

  13. Adversarial Prompts for Acceptance Collapse in Speculative Decoding

    Jul 23, 2026Run Wang, Chaoyi Zhou, Xi Liu +7Adversarial Prompt GenerationSpeculative Decoding

    Clemson University · Wayne State University · University of Washington

  14. Dynamic Defense Profiling Enables Cognitive Jailbreak of Text-to-Image Models

    Jul 20, 2026Dongdong Yang, Deyue Zhang, Zhao Liu +5Adversarial Prompt GenerationAdversarial Examples

    AI Security Lab, Beijing, China · School of Artificial Intelligence, Beihang University, Beijing, China

  15. Adversarial Prompting Framework for AI Safety Assessment

    Jul 15, 2026Yash Bhatnagar, Kunal Banerjee, Anirban ChatterjeeAdversarial Prompt GenerationAdversarial Attacks on LLMs

    Microsoft · Bengaluru, Karnataka, India · Walmart Global Tech

  16. Seduced by the Narrative: Assessing Rule Adherence in Semi-Open Textual Sandboxes

    Jul 2, 2026Weiying Chen, Junlong Shen, Zhanyuan Guo +1LLM Safety BenchmarksAdversarial Prompt Generation

    University of Alberta, Edmonton, AB, Canada · N-Dice Association · Qilu University of Technology, Shandong, China

  17. AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability

    Jun 23, 2026Khanak KhandelwalAdversarial Prompt GenerationAdversarial Attacks on LLMs

    Indian Institute of Technology Jodhpur

  18. Distributed Quality-Diversity Search for Toxicity in Large Language Models

    Jun 23, 2026Onkar Shelar, Travis DesellAdversarial Prompt GenerationLLM Red Teaming

    Rochester Institute of Technology, Rochester NY 14623, USA

  19. Adversarial Domain Prompt Tuning and Generation for Single Domain Generalization

    Jun 19, 2026Zhipeng Xu, De Cheng, Xinyang Jiang +3Synthetic Data AugmentationAdversarial Prompt Generation

    Xidian University · Microsoft Research Asia · Chongqing University of Posts and Telecommunications

  20. OTTER: A Red-Teaming System for Toxicity-Evading Jailbreak Prompt Optimization

    Jun 19, 2026Jerry Wang, Hsin-Ling Hsu, Yi-Cheng Lai +2Adversarial Prompt GenerationAdversarial Attacks on LLMs

    1National Chengchi University, Taipei, Taiwan

  21. Automated jailbreak attack targeting multiple defense strategies

    Jun 15, 2026Qi Wang, Chengcheng Wan, Weijia He +4Adversarial Prompt GenerationLLM Jailbreak Attacks

    East China Normal University Shanghai, China · East China Normal University, Shanghai Innovation Institute Shanghai, China · University of Southampton Southampton, England +2

  22. GAS-Leak-LLM: Genetic Algorithm-Based Suffix Optimization for Black-Box LLM Jailbreaking

    Jun 14, 2026Aman Anifer, Vignesh Kumar Kembu, Vishnu M +4Adversarial Prompt GenerationLLM Safety

    Department of Computer Applications, Cochin University of Science and Technology, Kerala, India. · Department of Electrical, Computer and Biomedical Engineering, University of Pavia, A. Ferrata, 5, Pavia, 27100, Italy.

  23. Greedy Coordinate Diffusion: Effective and Semantically Coherent Adversarial Attacks via Diffusion Guidance

    Jun 14, 2026Bohdan Turbal, Blossom Metevier, Max Springer +1Adversarial Prompt GenerationAdversarial Attacks

    Princeton University, Princeton, NJ, USA.

  24. JailbreakOPT: Tool-Assisted Iterative Jailbreak Prompt Optimization

    Jun 9, 2026Ge Shi, Jun Yin, Donglin Xie +3Adversarial Prompt GenerationLarge Language Model-Guided Optimization

    University of California, Davis · The Renmin University of China · Independent Researcher +3

  25. Assessing Automated Prompt Injection Attacks in Agentic Environments

    Jun 9, 2026David Hofer, Edoardo Debenedetti, Florian TramèrAdversarial Prompt GenerationLLM Agent Security

    ETH Zurich Zurich, Switzerland

  26. Diffuse AI Control on Fuzzy Tasks

    Jun 8, 2026Mikhail Terekhov, Caglar Gulcehre, Vivek Hebbar +1Adversarial Prompt GenerationLanguage Model Safety Evaluation

    Anthropic Fellows Program (via MATS) · EPFL · Redwood Research +1

  27. CHASE: Adversarial Red-Blue Teaming for Improving LLM Safety using Reinforcement Learning

    Jun 4, 2026Rahul Markasserithodi, Aditya Joshi, Yuekang Li +3Adversarial TrainingAdversarial Prompt Generation

    University of New South Wales, Australia

  28. Voices Across Registers: Corpus-Conditioned Vernacular Jailbreaks against Aligned LLMs via Fanfiction Subgenres

    Jun 3, 2026Zhongze Luo, Ruihe Shi, Zhenshuai Yin +3Adversarial Prompt GenerationLanguage Model Safety Evaluation

    School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen) · School of Microelectronics, Xi’an Jiaotong University · 2The Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen) +1

  29. Dive into Ambiguity: A*-Inspired Multi-Agents Commonsense Obfuscation Attack on LLM Prompts

    May 31, 2026Boxuan Wang, Zhuoyun Li, Xiaowei Huang +1Adversarial Prompt GenerationAdversarial Attacks

    University of Liverpool Merseyside, United Kingdom

  30. Quality-Diversity Evolution for Discovering Diverse Vulnerabilities in LLM Safety

    May 30, 2026Subhadip MitraAdversarial Prompt GenerationAdversarial Attacks on LLMs

    Research Lead, Rota Labs