Existing large language models typically rely on massive fine-tuning datasets for safety alignment, which results in higher costs for safety alignment and often leads to the problem of over-refusal. We propose Less is Safer (LIS), a compact yet highly effective dataset for safety alignment, constructed through interpretable feature-guided selection and adversarial synthesis. Specifically, we leverage Sparse Autoencoder (SAE) activations to efficiently select prompts that cover diverse and interpretable safety features. Then, we employ a multi-agent adversarial synthesis framework to generate harmless prompts that nevertheless elicit unnecessary refusals. LIS contains only 2.4K samples, yet outperforms much larger datasets, e.g., TRIDENT-Core (26K), TRIDENT-Edge (19K) and WildJailbreak (262K), in overall performance on four safety and five over-refusal benchmarks. Through ablation experiments, we demonstrate that neither safety data nor over-refusal data alone can achieve optimal results. We also used safety features from SAE, annotated by an LLM, for data filtering, but the results were inferior to those obtained using the pipeline proposed in this paper.
Figures & tables
Dataset
Scale
Safety data
Over-refusal data
Key characteristics
Safety-Tuned LLaMAs
2.5K
✓
×
Questions were drawn from Anthropic’s Red Team data, and GPT-3.5 generated responses.
BeaverTails
334K
✓
×
Human labels decouple helpfulness and harmlessness
TRIDENT
19K–26K
✓
×
Broad risk coverage via persona-based generation
STAR-1
1K
✓
×
Policy-grounded reasoning; for reasoning models
CoCoNot ( Brahman et al., 2024 )
11K + 0.9K
✓
✓
A non-specialized safety-aligned dataset; safety data constitutes only a portion of it.
WildJailbreak
262K
✓
✓
In-the-wild jailbreak tactics; very large scale
Table 1: The scale, composition, and characteristics of each training dataset.
Figure 1: Overview of the LIS construction pipeline. Stage 1 : Identifying safety features by comparing SAE activations of the safety data against other data. Stage 2 : Seeds from over-refusal data are processed through a multi-agent adversarial synthesis framework to produce the final over-refusal subset. The safety subset is obtained directly by filtering safety data.
Score
Index
Interpretation
15.87
124376
Copyright/legal implications of media
15.19
97413
Biological weapons and related events
15.06
50879
Pornography-related terminology
− 7.27
77647
Punctuation and structural elements
− 6.97
10944
Programming and syntax elements
− 6.71
13874
Questions and question marks
Table 2: Feature score, index and interpretations; interpretations sourced from Neuronpedia ( Lin, 2023 ) . For more examples, see Appendix G .
Safety (unsafe rate %, ↓ )
Over-Refusal (refusal rate %, ↓ )
Overall
Model
Dataset
hm
s-jb
s-base
sa-jb
sa-base
jd
xs
fr
ev
ph
ok
Rec.
Prec.
F1
LLaMA3.1 -Alpaca
LIS 2.4K
1.27
12.54
7.60
3.24
1.61
0.00
34.80
10.53
51.89
27.73
13.71
95.62
72.27
82.32
LIS/LLM-annot. 2.4K
1.09
13.30
4.88
4.98
1.15
1.29
48.80
22.58
56.01
39.96
28.00
95.55
60.93
74.41
Edge rand. 2.4K
0.91
8.14
1.82
1.36
1.36
0.00
56.40
44.48
72.85
67.65
18.86
97.74
47.95
64.34
Core rand. 2.4K
0.91
9.96
3.30
2.50
1.74
0.00
57.60
34.63
67.01
65.29
14.57
96.93
52.18
67.84
WJB rand. 2.4K
2.00
32.11
19.75
33.14
5.90
13.55
12.40
52.15
55.15
40.73
23.71
82.26
63.17
71.46
Table 3: Size-matched (2.4K) and full-dataset comparison on two model families under strictly unified settings . rand. = random equal-size subset. Recall =1−avg(unsafe rate) , Precision =1−avg(over-refusal rate) ( Jiang et al., 2024 ) . Best result in each column within each model family is in bold .
Strategy
hm
s-jb
s-base
sa-jb
sa-base
jd
Random
1.73
24.34
8.02
19.68
3.49
21.29
Anti-Greedy
2.00
26.36
6.82
21.76
2.27
20.00
K-means
1.64
24.41
7.46
17.76
2.70
19.35
Greedy
5.82
16.91
9.25
11.78
4.41
17.42
Word-avg
0.55
17.31
6.35
10.02
2.23
16.77
Table 4: Comparison of the Word-average Greedy (Word-avg) with other filtering methods: Unsafe rate (%, ↓ ) across selection strategies. Best in bold .
Dataset
Rec.
Prec.
F1
LIS (2.4K)
95.62
72.27
82.32
LIS-Safety (2.4K)
97.91
43.52
60.26
LIS-Over-Refusal (2.4K)
72.60
81.89
76.96
Table 5: Results of training LLaMA3.1-Alpaca on each subset alone
Benchmark
Instruct
Edge
LIS
MMLU (5-shot)
68.43
66.34
66.88
GSM8K (8-shot CoT)
85.67
83.78
83.84
MATH-500 (0-shot CoT)
57.32
54.53
55.22
ARC-Challenge (0-shot)
87.80
87.29
88.40
Table 6: General capabilities of LLaMA3.1-8B-Instruct before fine-tuning (Instruct) and after fine-tuning on TRIDENT-Edge (Edge) or LIS. Accuracy (%, ↑ ).
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Training type
Full fine-tuning
Epochs
2
Learning rate
2e-5
LR scheduler
cosine (default)
Weight decay
0.01
Max sequence length
2,048
Appendix
Table 7: Hyperparameters for instruction tuning on Alpaca.
Hyperparameter
Value
Training type
Full fine-tuning
Epochs
2
Learning rate
1e-5
LR scheduler
cosine (default)
Weight decay
0.01
Max sequence length
2,048
Appendix
Table 8: Hyperparameters for instruction tuning on LIS.
Hyperparameter
Value
Training type
Full fine-tuning
Epochs
2
Learning rate
1e-6
Per-device batch size
1
Effective batch size
8 (8 GPUs)
Max sequence length
2,048
Appendix
Table 9: Hyperparameters for the selection-strategy ablation.
Hyperparameter
Value
Training type
LoRA
LoRA rank
16
LoRA alpha
32
LoRA dropout
0.05
Target modules
all-linear
Epochs
7
Appendix
Table 10: Original TRIDENT recipe (Core / Edge), used only for the cost estimate in Appendix A.4 .
Hyperparameter
Value
Training type
Full fine-tuning
Epochs
2
Learning rate
2e-5
Warmup ratio
0.03
Weight decay
0
Max sequence length
2,048
Appendix
Table 11: Original WildJailbreak recipe (mixed with Tulu-2-Mix), used only for the cost estimate in Appendix A.4 .
Item
Value
Base model
LLaMA3.1-Alpaca
Training data
1,200 vanilla_harmful
samples from WildJailbreak
Trainer
ms-swift, full-parameter SFT
Distributed
8 × GPU, ZeRO-2
Precision
bfloat16
Appendix
Table 12: Fixed training settings in Hyperparameter Grid Search.
Hyperparameter
Values
#
Learning rate
{1,2,5}×10−6,{1,2,5}×10−5
6
Training epochs
{1,2,3}
3
Batch size (per device)
{1,2,4,8,16,32}
6
Total
6×3×6
108
Appendix
Table 13: Search space in SALAD-Bench and S-Eval.
Hyperparameter
Values
#
Learning rate
{1,2,5}×10−6,{1}×10−5
4
Training epochs
{1,2,3}
3
Batch size (per device)
{1,2,4,8,16}
5
Total
4×3×5
60
Appendix
Table 14: Search space in XSTest. All 60 configurations are obtained from the full Cartesian product of the three axes.
Figure 2: The unsafe rate for 108 sets of parameter configurations; the x-axis represents the ID, and the y-axis represents the unsafe rate
Figure 3: The over-refusal rate for 60 sets of parameter configurations; the x-axis represents the ID, and the y-axis represents the over-refusal rate
Figure 4: Distribution of the model’s unsafe rate on SALAD-Bench at epoch 1. The left side is base_set (base), and the right side is attack_enhanced_set (jailbreak)
Figure 5: Distribution of the model’s unsafe rate on SALAD-Bench at epoch 2. The left side is base_set (base), and the right side is attack_enhanced_set (jailbreak)
Figure 6: Distribution of the model’s unsafe rate on SALAD-Bench at epoch 3. The left side is base_set (base), and the right side is attack_enhanced_set (jailbreak)
Figure 7: Distribution of the model’s unsafe rate on S-Eval at epoch 1. The left side is base_set (base), and the right side is attack_enhanced_set (jailbreak)
Figure 8: Distribution of the model’s unsafe rate on S-Eval at epoch 2. The left side is base_set (base), and the right side is attack_enhanced_set (jailbreak)
Figure 9: Distribution of the model’s unsafe rate on S-Eval at epoch 3. The left side is base_set (base), and the right side is attack_enhanced_set (jailbreak)
Figure 10: Distribution of the model’s over-refusal rate on XSTest at epoch 1.
Figure 11: Distribution of the model’s over-refusal rate on XSTest at epoch 2.
Figure 12: Distribution of the model’s over-refusal rate on XSTest at epoch 3.
Figure 13: The F1 score on SALAD-Bench and XSTest for 60 sets of parameter configurations. The model performed consistently well, and the parameters yielding the highest F1 score were a learning rate of 1e-6, a batch size of 1, and 2 epochs.
Figure 14: The model’s performance on the validation set after being trained with varying amounts of safety data.
n
Unsafe Rate ↓
Over-refusal Rate ↓
600
2.75%
11.4%
800
1.85%
11.4%
1000
1.00%
13.0%
1200
0.85%
12.0%
1400
0.80%
12.2%
1600
0.70%
12.4%
Appendix
Table 15: The model’s performance on the validation set after being trained with varying amounts of safety data
nsafe
Unsafe Rate ↓
Over-refusal Rate ↓
nor=600
800
1000
1200
1400
nor=600
800
1000
1200
1400
1000
1.90%
1.95%
1.55%
1.60%
1.50%
3.20%
2.00%
2.20%
1.20%
1.00%
1200
1.45%
1.45%
1.10%
1.10%
1.20%
4.00%
2.80%
2.80%
1.40%
1.80%
1400
1.15%
1.10%
1.10%
1.10%
1.05%
5.00%
2.40%
4.20%
2.00%
1.60%
Appendix
Table 16: Performance under different mixing ratios of safety data ( nsafe ) and over-refusal data ( nor ). Left block: unsafe rate; right block: over-refusal rate. Lower is better in both blocks.
Figure 15: Pareto Analysis for Different Ratio Settings
Dataset
Activated
Alpaca Activated
Safety Features
TRIDENT-Edge
77,291
93,927
54,390
OR-Bench
65,322
93,921
30,861
Appendix
Table 17: Total number of features activated in the dataset versus the number of safety-related features.
Figure 16: Coverage of safety features by greedy and random algorithms. In the early stages of filtering, greedy algorithms can significantly improve the coverage of safety features in the dataset.
ID
Description
Type
Score
124376
references to copyright and legal implications related to media content
Safety
15.876
48254
references to criticisms of organized groups and their practices
Safety
15.307
97413
references to biological weapons and related events
Safety
15.192
50879
terms related to pornography and its various representations
Safety
15.060
110523
references to rape and sexual assault
Safety
14.920
130894
terms related to pedophilia and sexual offenses
Safety
14.772
Appendix
Table 18: Top 50 features with the highest Scores
ID
Description
Type
Score
77647
sentences with varying levels of punctuation and structural elements
Generic
−7.270
10944
programming and syntax elements
Generic
−6.974
13874
questions and question marks
Generic
−6.713
49642
punctuation marks and their formatting context
Generic
−6.605
121859
references to various forms of editing and revising written content
Generic
−6.559
129654
numerical values, equations, and technical specifications related to computing and data processing
Generic
−6.542
Appendix
Table 19: The 50 features with the lowest scores
Setting
Value
Initialization
LlamaScope L24R-32x
Hooked activation
residual stream after layer 24
Base model
LLaMA3.1-8B-Base
Number of features
131,072
Activation (training)
TopK, k=50
Activation (released)
JumpReLU, threshold 1.62
Appendix
Table 20: Training settings of the safety-adapted SAE.
Safety
General
SAE
EV
L0
EV
L0
LlamaScope L24R-32x
0.730
50
0.756
50
Adapted (TopK)
0.847
50
0.759
50
Adapted (JumpReLU)
0.843
46
0.760
54
Appendix
Table 21: Reconstruction quality on held-out text. EV = explained variance of the reconstruction. LlamaScope and the adapted TopK SAE are evaluated with the TopK ( k=50 ) activation used in training.
Index
Score
Explanation
Harmful topics
95966
13.72
“layering” in money laundering
27932
13.70
the word “self-harm”
12668
12.74
stalking and harassment
88854
12.73
illegal online marketplaces
113260
11.91
paying earlier investors with new money
Appendix
Table 22: Examples of safety features of the adapted SAE on TRIDENT-Edge whose index was not in the original safety set. Explanations are shortened.
Safety (unsafe rate %, ↓ )
Over-Refusal (refusal rate %, ↓ )
Overall
Model
SAE
hm
s-jb
s-base
sa-jb
sa-base
jd
xs
fr
ev
ph
ok
Rec.
Prec.
F1
LLaMA3.1-Alpaca
LlamaScope
1.27
12.54
7.60
3.24
1.61
0.00
34.80
10.53
51.89
27.73
13.71
95.62
72.27
82.32
Safety-adapted
0.91
10.66
7.74
3.28
1.55
1.29
36.00
13.65
54.47
38.32
18.57
95.76
67.80
79.39
Qwen3.5-9B-Base
LlamaScope
2.55
26.20
16.32
8.00
2.73
2.58
18.80
8.85
33.51
14.20
4.00
90.27
84.13
87.09
Safety-adapted
3.09
18.49
14.57
2.70
2.39
5.16
16.80
7.67
34.36
14.93
6.00
92.27
84.05
87.97
Appendix
Table 23: LIS selected with the original LlamaScope SAE versus the safety-adapted SAE. Benchmarks, metrics, and training follow Table 3 ; the LlamaScope rows are the LIS rows of Table 3 .
Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to jailbreak attacks that elicit harmful or unsafe outputs. Existing safety alignment approaches, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often require substantial attack-specific supervision and computational resources, while remaining susceptible to shallow safety alignment and over-refusal. To address these challenges, we introduce SSRFT(Supervised Safe-Role Fine-Tuning), the first framework that reformulates safety alignment as the internalization of a predefined safe role. SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset from psychometric questions, limited jailbreak prompts, and a safe-role description. Role-consistent responses are synthesized, validated, and expanded into diverse scenarios, enabling models to internalize safety-oriented values and principles rather than explicit refusal patterns. Experiments across multiple Base and Instruct models show that SSRFT achieves more robust and generalizable safety alignment than standard SFT. SSRFT shows substantially greater robustness to prefilling attacks and better generalization to unseen jailbreak domains, while reducing over-refusal on benign queries and preserving the model's general capabilities. These results establish safe-role internalization as an effective alternative to refusal-centric safety alignment. Warning: This paper contains examples of harmful and toxic language.
Jinghao Pang, Jitai Hao, Qiang Huang +2
Harbin Institute of Technology (Shenzhen) · Leiden University
Safety interpretability advances the study of Large Language Model (LLM) alignment from behavioral constraints driven by data or algorithms towards a deeper understanding of internal mechanisms. However, existing works have focused primarily on safety-related representations, attention heads, or neurons after alignment, while largely overlooking the safety mechanisms in pretrained-only models and their evolution across alignment checkpoints. To address this, we propose SafeEvo, an interpretability framework from the circuit (sparse subgraphs of an LLM) perspective. SafeEvo first applies an optimization-based extraction algorithm to identify weak refusal circuits in pretrained base LLMs that can independently express refusal behavior. Causally ablating these circuits completely eliminates the base model's refusal of harmful inputs. SafeEvo then traces the evolution of refusal circuits across successive alignment checkpoints and finds that their structures change progressively, suggesting that the alignment tax may result from refusal-circuit updates affecting utility-related parameters. To validate this, SafeEvo introduces Safety Circuit Alignment (SCA), which confines safety updates to the refusal circuits. Experiments across three LLMs and two alignment algorithms show that, on average, SCA outperforms vanilla alignment in three aspects: (1) stronger alignment, lowering harmfulness score by 63.21%; (2) less over-refusal, yielding a 58.44% decrease in refusal rates for benign queries; and (3) better utility, retaining 99.58% of the original model capabilities.
Miao Yu, Hao Huang, Lu Yuan +3
The University of Hong Kong (HKU) · Chinese Academy of Sciences (CAS) · Information Engineering University (IEU) +1
Safety alignment of large language models (LLMs) typically depends on high-quality supervision data, such as safe demonstrations or preference pairs. However, in real-world deployment, emerging safety requirements are often specified as natural-language policies, while corresponding supervision data may be costly, delayed, or unavailable. This creates a mismatch between rapidly evolving safety policies and conventional data-driven alignment methods. To address this, we propose PolicyAlign, a simple yet effective framework for directly aligning LLMs with safety policies. Given a safety policy, PolicyAlign first synthesizes policy-violating instructions and then performs on-policy self-distillation to internalize policy-guided behavior. To improve training stability and data efficiency, we further introduce Policy-Sensitive Filtering, which selects instructions where the policy induces the largest behavioral shift. Experiments across multiple models show that PolicyAlign consistently improves safety while maintaining low over-refusal and preserving general capabilities. PolicyAlign also generalizes to medical, legal, and financial safety scenarios, highlighting its potential as a scalable and maintainable approach to policy-based LLM safety alignment. The code is released at https://github.com/Qwen-Applications/PolicyAlign.
Chang Wu, Junfeng Fang, Houcheng Jiang +5
Qwen Large Model Application Team, Alibaba · National University of Singapore · Zhongguancun Academy