Organizations: University of Technology Sydney, Sydney, NSW, Australia · Xi’an Jiaotong University, Xi’an, Shaanxi, China · Southeast University, Nanjing, Jiangsu, China · Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu, China
Large language models (LLMs) remain highly vulnerable to jailbreak attacks that induce harmful behaviors and circumvent safety alignment. To defend against such attacks, adversarial training paradigms have been proposed to first simulate failure modes and then train the model to correct them, yielding promising improvements in safety alignment. However, these methods typically construct adversarial samples either by encouraging fixed harmful target completions or by performing targeted activation ablation derived from fixed benign--harmful data pairs. As a result, the generated adversarial samples tend to induce homogeneous harmful behaviors that poorly reflect the diversity of behaviors elicited by real-world jailbreak attacks. This behavior-level narrowness fundamentally limits their robustness. To address this issue, we propose a target-free adversarial training framework that generates adversarial samples in an unsupervised manner. By amplifying and diversifying behavior-level shifts in the model's latent space, our approach produces semantically diverse adversarial samples that induce a wide range of harmful behaviors. This expanded behavioral coverage exposes more diverse failure modes and thereby improves safety alignment. To quantify this effect, we use semantic entropy as an output-level measure of adversarial behavioral diversity. Empirically, our method elicits diverse harmful behaviors in the target model, substantially mitigating behavioral narrowness and improving robustness to jailbreak attacks.
Figures & tables
Figure 1: Overview of the proposed target-free latent adversarial training framework. (A) Prior LAT methods typically optimize a single latent intervention toward one harmful target, which induces behaviorally narrow adversarial samples. (B) Our inner adversary instead learns multiple source-layer intervention directions and maximizes the nuclear norm of the resulting behavior-shift matrix to encourage strong and diverse behavior-level shifts. (C) Applying different learned directions to the same harmful prompt elicits semantically distinct harmful behaviors, increasing output-level diversity as measured by semantic entropy. (D) In the outer alignment stage, the model is trained to maintain consistent refusal behavior on harmful inputs under all learned intervention directions while preserving helpful behavior on benign inputs, yielding a favorable safety–utility trade-off.
Models
Jailbreaking Robustness (SR, ↓ )
Over- refusal
Utility
GCG
TAP
AutoDAN
GPTFuzz
FewShot
Multi- lingual
Code- Chameleon
OR-Bench FPR ↓
AlpacaEval WR. ↑
lm-eval Avg. ↑
Qwen2.5-7B-Instruct
49.94
34.31
38.81
25.68
18.27
15.78
23.34
5.58
50
79.4
Targeted LAT
6.90
17.74
6.92
7.55
5.78
6.42
19.87
21.51
47.47
75.1
ReFAT
6.24
15.05
4.47
6.81
4.83
5.88
18.94
16.72
46.89
74.0
Deep-Refusal
4.98
16.37
5.50
5.34
5.02
5.61
20.16
18.09
48.24
75.3
DAT
4.01
12.65
6.61
6.71
5.57
6.03
19.42
15.35
47.26
75.8
Table 1: Main results. Bold denotes the best training-based defense; original models are included as references.
Figure 2: Optimization dynamics connecting the nuclear-norm objective to behavioral strength and diversity.
Table 2: Semantic interpretation of learned directions. Each column represents one vk ; upper and lower word clouds correspond to its random initialization and optimized state, respectively. The learned directions capture multilingual, role-playing, coding, and direct-compliance behaviors.
Figure 3: Causal ablation on multi-directional adversarial training.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Objective
SGD step size
Step 0
Step 10
Step 100
Nuclear norm
10−5
1.00
4.76
47.56
Frobenius norm
10−5
1.00
1.00
1.00
Nuclear norm
10−2
1.00
64.00
64.00
Frobenius norm
10−2
1.00
1.00
1.00
Appendix
Table 3: Effective rank in the controlled direct-matrix experiment, where C itself is the optimized parameter.
Inner step
Effective rank
Mean SR ↑
Harmful-only entropy ↑
Harmful-output rate ↑
0
55.43
0.11
0.07
0.17
1
22.14
0.21
0.38
0.30
2
39.52
0.56
0.60
0.72
3
44.80
0.48
0.65
0.62
4
42.77
0.53
0.62
0.68
5
39.98
0.54
0.62
0.70
Appendix
Table 4: Ten-step inner-loop trace with K=64 . The harmful-output rate is the fraction with SR>0.3 .
Figure 4: Hyper-parameter sensitivity analysis on Qwen2.5-7B-Instruct over the number of latent directions K , intervention strength λ , and inner harmful batch size. The top row reports inner-adversary quality measured by mean StrongREJECT score and semantic entropy, while the bottom row reports final model performance in terms of jailbreak StrongREJECT score, over-refusal rate, and AlpacaEval win rate.
Hyper-parameter
LLaMA-3-8B Instruct
Mistral-7B v0.2
Qwen2.5-7B Instruct
Model / PEFT
LoRA rank r
16
16
16
LoRA alpha
32
32
32
LoRA target layers
all
all
all
Inner adversary
Source layer s
10
10
10
Appendix
Table 5: Hyper-parameter configuration used in our main experiments.
Method
Adversary Type
Passes per Step
Cost per Adversary
Real Time
Targeted LAT
Single-direction (targeted)
2 (forward + backward)
2×T
0.85s
ReFAT
Single-direction (refusal ablation)
1 (forward only)
1
0.24s
Ours
Multi-direction ( K adversaries)
2 (forward + backward)
K2×T
0.012s
Appendix
Table 6: Comparisions of computational cost for single adversary construction in the inner loop.