Adversarial Prompt Generation

Follow this topic Ask about this topic

Adversarial prompt generation studies how to construct inputs that bypass safeguards, induce failures, or reveal weaknesses in language models, agents, and image-generation systems. Common approaches include evolutionary and reinforcement-learning search, gradient or embedding optimization, black-box feedback, and compositional or multi-turn attacks; some use semantic obfuscation, personas, or tool-use trajectories to find less obvious vulnerabilities. Researchers evaluate attacks on benchmarks such as HarmBench and JailbreakBench, as well as in agent and multimodal settings, using automated judges and measures of success, diversity, and transfer. A central tension is that reported success depends strongly on the target model, query budget, judge, and stopping rule, while realistic and repeatable attacks must also transfer across systems and withstand adaptive defenses.

Latest papers

67 papers4 papers in the last four weeks, down 20% on the four weeks before. 0.0% of all new papers.

Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Search
Feb 26, 2026Xun Huang, Simeng Qin, Xiaoshuang Jia +6LLM Jailbreak Attacks Adversarial Prompt Generation
Nanyang Technological University, Singapore · BraneMatrix AI, China · Nanjing University of Science and Technology, China +6
FERRET: Framework for Expansion Reliant Red Teaming
Feb 17, 2026Ninareh Mehrabi, Vitor Albiero, Maya Pavlova +1Automated Red Teaming Adversarial Scenario Generation
Meta Superintelligence Labs (MSL)
"Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewers
Nov 3, 2025Qin Zhou, Zhexin Zhang, Zhi Li +1Adversarial Prompt Generation Indirect Prompt Injection
Institute of Information Engineering, CAS, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · The Conversational AI (CoAI) group, DCST, Tsinghua University
NonTextual Target Attack
Oct 3, 2025Xinzhe Huang, Wenjing Hu, Tianhang Zheng +6Adversarial Prompt Generation Adversarial Attacks on LLMs
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou 310007, China · Hangzhou High-Tech Zone, Binjiang Institute of Blockchain and Data Security, Hangzhou 310053, China · School of Cyberspace Security, Nanjing University of Science and Technology, Nanjing 210094, China +3
RedCoder: Automated Multi-Turn Red Teaming for Code LLMs
Jun 25, 2025Wenjie Jacky Mo, Qin Liu, Xiaofei Wen +5Adversarial Prompt Generation LLM Red Teaming
University of California, Davis · University of Southern California
Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models
Jun 8, 2025Ren-Jian Wang, Ke Xue, Zeyu Qin +7LLM Red Teaming Adversarial Prompt Generation
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · The Hong Kong University of Science and Technology, Hong Kong, SAR, China +3
Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Models
Mar 8, 2025Thomas Winninger, Boussad Addad, Katarzyna KapustaAdversarial Prompt Generation Adversarial Attacks on LLMs
*Work started during internship at Thales.

Feb 26, 2026J/K move · Enter open · S save

Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Search

Xun Huang, Simeng Qin, Xiaoshuang Jia, Ranjie Duan+5

Nanyang Technological University, Singapore · BraneMatrix AI, China · Nanjing University of Science and Technology, China+6

LLM Jailbreak Attacks Adversarial Prompt Generation Black-Box Adversarial Attacks

Open paper AskPDF arXiv

As Large Language Models (LLMs) are increasingly used, their security risks have drawn increasing attention. Existing research reveals that LLMs are highly susceptible to jailbreak attacks, with effectiveness varying across language contexts. This paper investigates the role of classical Chinese in jailbreak attacks. Owing to its conciseness and obscurity, classical Chinese can partially bypass existing safety constraints, exposing notable vulnerabilities in LLMs. Based on this observation, this paper proposes a framework, CC-BOS, for the automatic generation of classical Chinese adversarial prompts based on multi-dimensional fruit fly optimization, facilitating efficient and automated jailbreak attacks in black-box settings. Prompts are encoded into eight policy dimensions-covering role, behavior, mechanism, metaphor, expression, knowledge, trigger pattern and context; and iteratively refined via smell search, visual search, and cauchy mutation. This design enables efficient exploration of the search space, thereby enhancing the effectiveness of black-box jailbreak attacks. To enhance readability and evaluation accuracy, we further design a classical Chinese to English translation module. Extensive experiments demonstrate that effectiveness of the proposed CC-BOS, consistently outperforming state-of-the-art jailbreak attack methods.