cs.CLNov 5, 2026

RL-ATTACK: A Generalized Black-Box Adversarial Attack on Language Models via Reinforcement Learning

Authors: Taikun Chen, Yue Zhang, Wei Qian, Hui Fang

Organizations: Financial Services Analytics University of Delaware, Newark, DE, USA · Department of Electrical and Computer Engineering University of Delaware, Newark, DE, USA · Department of Applied Economics and Statistics University of Delaware, Newark, DE, USA

Abstract

Large language models (LLMs) exhibit vulnerabilities to adversarial perturbations that can induce unintended responses. These vulnerabilities can be identified via intentional attacks and subsequently used to improve model safety. In black-box settings where gradients are inaccessible, existing methods tend to guess adversarial transformations using external knowledge, ignoring the information provided by the target model. To mitigate this, we propose a trainable black-box framework based on reinforcement learning, which learns from the target model to generate tailored adversarial transformations. We demonstrate that our approach achieves lower post-attack accuracy, greater query efficiency and better semantic preservation on both natural language inference and generative Q&A tasks. On natural language inference, it drives post-attack accuracy to 2% within 20 queries. Against generative language models, our trained attacker consistently achieves lower post-attack accuracy than the untrained attacker and wins in 4 out of 6 settings when compared to a strong optimization-based baseline. We further experiment with multiple reward designs and training settings, and we find that no design or setting is consistently dominant. Our code is available at https://github.com/taikunudel/RL-ATTACK.

Figures & tables

Explore similar work

CardsList
  1. Reward Stealing Attack on Large Language Models

    Oct 5, 2026Jiaming Qian, Pengyang Zhou, Jiahe Xu +1LLM AlignmentModel Extraction Attacks

  2. Automated jailbreak attack targeting multiple defense strategies

    Jun 15, 2026Qi Wang, Chengcheng Wan, Weijia He +4Adversarial Prompt GenerationLLM Jailbreak Attacks

  3. Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Models

    Mar 8, 2025Thomas Winninger, Boussad Addad, Katarzyna KapustaAdversarial Prompt GenerationAdversarial Attacks on LLMs