Organizations: Financial Services Analytics University of Delaware, Newark, DE, USA · Department of Electrical and Computer Engineering University of Delaware, Newark, DE, USA · Department of Applied Economics and Statistics University of Delaware, Newark, DE, USA
Large language models (LLMs) exhibit vulnerabilities to adversarial perturbations that can induce unintended responses. These vulnerabilities can be identified via intentional attacks and subsequently used to improve model safety. In black-box settings where gradients are inaccessible, existing methods tend to guess adversarial transformations using external knowledge, ignoring the information provided by the target model. To mitigate this, we propose a trainable black-box framework based on reinforcement learning, which learns from the target model to generate tailored adversarial transformations. We demonstrate that our approach achieves lower post-attack accuracy, greater query efficiency and better semantic preservation on both natural language inference and generative Q&A tasks. On natural language inference, it drives post-attack accuracy to 2% within 20 queries. Against generative language models, our trained attacker consistently achieves lower post-attack accuracy than the untrained attacker and wins in 4 out of 6 settings when compared to a strong optimization-based baseline. We further experiment with multiple reward designs and training settings, and we find that no design or setting is consistently dominant. Our code is available at https://github.com/taikunudel/RL-ATTACK.
Figures & tables
Fig. 2: Overview of the RL-ATTACK framework.
Variant
radv(x′,x)
Purpose
A (default)
C(ytrue∣x)−C(ytrue∣x′)
the raw drop in the target’s confidence in the true label (Eq. 2 )
B
sigmoid(C(ytrue∣x)−C(ytrue∣x′))
a bounded, monotone (sigmoid) reshaping of the drop into (0,1) : larger drops earn more reward and confidence increases are penalized, on a common, outlier-robust scale
C
C(ytrue∣x)(C(ytrue∣x)−C(ytrue∣x′))
the drop weighted by the target’s original confidence C(ytrue∣x) , so the same drop counts for more on documents the target was already very confident about—the hardest cases
TABLE I: The three adversarial-reward variants. Unless stated otherwise, we use variant A.
Dataset
Target Model
Method
Post-attack accuracy (%)
# of queries
USE
% of perturbation
SNLI
BERT
RL-ATTACK
2.03 ( ± 0.54)
13
0.65
5.1
BERT (untrained)
5.73
18
0.60
6.0
TextFooler [ 11 ]
4.00
60
0.45
18.5
BERT-ATTACK [ 8 ]
5.73
16
0.40
12.4
SSPAttack (hard-label) [ 27 ]
1.30
1000
0.54
15.9
HQA-Attack (hard-label) [ 28 ]
1.30
1000
0.54
15.9
TABLE II: Adversarial attack results on three Natural Language Inference datasets . Post-attack accuracy is the percentage of adversarial documents still assigned their correct labels. Parenthetical values are standard errors computed from 10 validation folds.
Judge
Target Model
Method
Post-attack accuracy
# of queries
USE
% of perturbation
Llama Guard 4 12B
Llama 2 7B [ 25 ]
RL-ATTACK
82.0 ( ± 3.8)
27.7
0.98
2.0%
BERT (untrained)
87.0
28.3
0.98
1.0%
PAIR [ 17 ]
98.0
56.0
0.60
N/A
GPT-3.5-Turbo
RL-ATTACK
11.5 ( ± 3.3)
16.8
0.98
4.5%
BERT (untrained)
15.5
15.5
0.97
4.7%
PAIR
28.0
29.6
0.44
N/A
TABLE III: Adversarial attack results on the AdvBench dataset . Post-attack accuracy is the percentage of responses still judged safe by the judge. Parenthetical values are standard errors computed from 10 validation folds.
Dataset
Split
Size
Purpose
in-the-wild
train
16,776
Train the attacker
in-the-wild
traindev
50
Checkpoint selection
in-the-wild
evaldev
100
Attack hyperparameter selection
AdvBench
all
500
Final result reported
TABLE IV: Generative Q&A attack data splits.
Fig. 3: Post-attack accuracy on the in-the-wild traindev set. Each point represents a combination of the reward variant, α , and K . No consistent trend is observed.
Fig. 4: Post-attack accuracy over training for six representative hyperparameter combinations under three types of rewards.
Adversarial attacks on large language models probe how crafted inputs, internal changes, and agent interactions bypass safety or disrupt reliable behavior.
East China Normal University Shanghai, China · East China Normal University, Shanghai Innovation Institute Shanghai, China · University of Southampton Southampton, England +2