Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model
Authors: Pankayaraj Pathmanathan, Furong Huang
Organizations: University of Maryland College Park
While the wide adoption of refusal training in large language models (LLMs) has showcased improvements in model safety, recent works have highlighted shortcomings due to the shallow nature of these alignment methods. To this end, the work on Deliberative alignment proposed distilling reasoning capabilities from stronger reasoning models, thereby instilling deeper safety in LLMs. In this work, we study the impact of deliberative alignment in language models. First, we show that despite being larger in model size and stronger in safety capability, there exists an alignment gap between teacher and student language models, which affects both the safety and general utility of the student model. Furthermore, we show that models aligned through deliberative alignment can retain unsafe behaviors from the base model despite learning the reasoning patterns of larger reasoning models. Building upon this observation, we propose a BoN sampling method that attributes the unsafe behavior back to the base LLMs in the latent space, thereby down-ranking unsafe responses to gain a meaningful improvement in model safety across multiple safety benchmarks with minimal loss in utility. In particular, across 7 teacher models and 6 student models of different classes and sizes, we show an average attack success rate (ASR) reduction of 28.2% in DAN, 31.3% in WildJailbreak and 35.4 % in StrongREJECT benchmarks. We further show that these safety gains prevail post RL training, thus highlighting the uncertainty in safety reasoning and it's explicit attribution to the base model.
Figures & tables
Figure 1: I. Deliberative Alignment : The figure outlines the methodology behind deliberative alignment. For a given labeled training data a stronger reasoning LLM generates a safety reasoning chain of thought (CoT) alongside the refusal. Post filtering, this CoT + Refusal response is then used to instruction tune a non reasoning LLM. As a last stage the instruction tuned LLM is further finetuned via reinforcement learning (eg. GRPO) for further safety alignment. II. Latent Similarity based BoN sampling : This showcases the outline of the our proposed BoN sampling strategy. Based on our empirical observation attributing unsafe behavior to the base model in the latent space we propose a BoN sampling that effectively leverages this observation for a safet response generation.
Figure 2: Capability of teacher models on training data : Figure showcases the safety capability of different teacher reasoning models on the training data. Here use the number of examples for which the teacher models generate a harmful response as an indicator of the capability. Here proportional to the model size the larger models elicit a stronger safety capability
Figure 4: Existence of teach student alignment gap : This figure showcase the existence of an alignment gap between teacher and student model. Despite the increasing model size and capability certain teacher models are less capable at aligning certain student models.
Figure 5: Impact of teacher model size on general utility : This figure presents the utility of aligned student models of various sizes when different teacher models were used. Using smaller reasoning models to align larger students models lead to a higher drop in the general utility of the model. For instance using reasoning traces from DeepSeek-R1-Distill-Qwen-1.5B lead to larger utility drops across larger Qwen-7B-Instruct and Qwen-14B-Instruct models. Here the utility is measure via single zero shot sampling with a temperature of 0.7.
Figure 6: Existence of safer responses in multi sampling : This figure presents the Attack Success Rate (ASR) across three different safety benchmarks. We compare the ASR measured via single sampling against the ASR measured when more than half of the sampled responses are unsafe for deliberately aligned models. The figure demonstrates that safer responses exist within the generations of the deliberately aligned model, suggesting a potential for further safety improvements. For each prompt, we sampled 8 responses. Here DeepSeek-R1-Distill-Qwen-14B model was employed as the teacher model during the alignment process.
Figure 7: Probability density distribution safe and unsafe responses : Figure presents the probability density distribution different metrics over safe and unsafe responses. Notably, KL divergence and Latent Similarity demonstrate a visible separation in their distributions. Specifically, Latent Similarity exhibits more distinct distribution peaks between safe and unsafe responses. Both the KL divergence and Latent Similarity distribution highlights the attribution of harmful responses to the base model. These results correspond to the Qwen-0.5B-Instruct model, which was aligned using reasoning traces from DeepSeek-R1-Distill-Qwen-14B. Evaluations were performed on the DAN dataset.
Metric
Drop in ASR ( % ) ↑
Drop in Utility ( % ) ↓
DAN
WildJailbreak
StrongREJECT
GSM8K
MMLU
Random
-1.5 %
0.3 %
-2.6 %
1.6 %
1.0 %
Perplexity
-26.2 %
-31.8 %
-8.0 %
-5.1 %
-1.7 %
Self Certainty
-46.2 %
-75.5 %
-87.2 %
-6.0 %
5.8 %
KL (ours)
19.1 %
17.0 %
31.9 %
29.8 %
36.4 %
Latent Similarity-
28.2 %
31.3 %
35.4 %
7.6 %
11.4 %
Table 1: Percentage of reduction in ASR and Utility on deliberative alignment (SFT) : Table highlights the average efficacy gain over single sampling of the proposed method in improving reasoning safety across different models and benchmarks post SFT alignment.
Metric
Drop in ASR ( % ) ↑
Drop in Utility ( % ) ↓
DAN
WildJailbreak
StrongREJECT
GSM8K
MMLU
Random
-2.3 %
-25 %
3.5 %
1.9 %
0.7 %
Perplexity
-38.0 %
-53.1 %
-40.1 %
-6.2 %
-6.7 %
Self Certainty
-54.2 %
-93.5 %
-146.4 %
-8.8 %
1.3 %
KL (ours)
-14.2 %
-11.4 %
24.8 %
36.7 %
37.4 %
Latent Similarity-
21.9 %
35.3 %
48.0 %
5.3 %
9.7 %
Table 2: Percentage of reduction in ASR and Utility on deliberative alignment (GRPO) : Table highlights the average efficacy gain over single sampling of the proposed method in improving reasoning safety across different models and benchmarks post RL training.
Metric
Drop in ASR ( % ) ↑
DAN
WildJailbreak
StrongREJECT
Random
0.4 %
6.6 %
-1.0 %
KL (ours)
-1.0 %
0.0 %
-2.0 %
Latent Similarity-
0.8 %
-0.5 %
2.3 %
Layer 12 (ours)
Latent Similarity-
9.2 %
19.9 %
13.8 %
Table 3: Percentage of reduction in ASR and Utility on instruction tuning : Table showcases the marginal average efficacy gain over single sampling of the proposed method on general instruction tuning thus vastly attributing uncertainty to reasoning alignment.
ASR ( % ) ↓
Student Model
Instruction tuning
Deliberative Alignment
Deliberative Alignment
(single sampling)
(ours)
Llama 3.2 1B Instruct
88 %
68 %
66 %
Qwen 2.5 7B Instruct
68 %
18 %
18 %
Table 4: Effectiveness against adaptive attacks : Table showcases the efficacy of the aligned models against jailbreak attacks under the proposed BoN sampling. Results showcase that the proposed method do not reduce the jailbreak immunity gain from deliberative alignment.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Safety Categories
Percentage of training data
Harassment/Hate/Discrimination
23.1 %
Sexual/Adult
6.7 %
Violence/Physical Harm
13.4 %
Self-Harm
3.6 %
Illicit/Criminal Behavior
30.2 %
Misinformation/Disinformation
10.7 %
Appendix
Table 5: Breakdown of the safety categories in the dataset
Dataset
Number of evaluation samples
DAN
1405
Wildjailbreak
2000
StrongReject
313
Appendix
Table 6: Number of data samples in safety evaluation datasets
Dataset
Data subsets
Number of evaluation samples
GSM8K
1319
abstract algebra
100
computer security
100
business ethics
100
high school chemistry
203
college physics
102
Appendix
Table 7: Number of data samples in utility evaluation datasets
Reasoning models
Base model
DeepSeek R1 Distill Qwen 1.5B
Qwen2.5 Math 1.5B
DeepSeek R1 Distill Qwen 7B
Qwen2.5 Math 7B
DeepSeek R1 Distill Llama 8B
Llama 3.1 8B
DeepSeek R1 Distill Qwen 14B
Qwen2.5 14B
DeepSeek-R1 Distill Qwen 32B
Qwen2.5 32B
Qwen QWQ 32B
Qwen2.5 32B
Appendix
Table 8: Reasoning models and the base models from which they were training on.
Figure 12: Model safety with layer : Here we showcase the safety of the model when different layers of models were used for Latent Similarity where
Figure 14: Model safety with layer : Here we showcase the safety of the model when different layers of models were used for Latent Similarity where
Figure 16: Model safety with layer : Here we showcase the safety of the model when different layers of models were used for Latent Similarity where
Figure 18: Model safety with layer : Here we showcase the safety of the model when different layers of models were used for Latent Similarity where
Figure 20: Model safety with layer : Here we showcase the safety of the model when different layers of models were used for Latent Similarity where
Figure 22: Model safety with layer : Here we showcase the safety of the model when different layers of models were used for Latent Similarity where
Figure 24: Model safety with layer : Here we showcase the safety of the model when different layers of models were used for Latent Similarity where
Figure 26: Model safety with layer : Here we showcase the safety of the model when different layers of models were used for Latent Similarity where
Figure 28: Existence of teach student alignment gap : This figure showcase the existence of an alignment gap between teacher and student model. Despite the increasing model size and capability certain teacher models are less capable at aligning certain student models.
Figure 30: Existence of teach student alignment gap : This figure showcase the existence of an alignment gap between teacher and student model. Despite the increasing model size and capability certain teacher models are less capable at aligning certain student models.
Model
Metric
Safety(ASR % ) ↓
DAN
Wildjailbreak
StrongREJECT
First answer
43.1
48.1
55.2
Random sampling
45.1
47.4
54.3
Qwen 2.5 0.5B Instruct
KL (ours)
43.2
48.1
55.2
Latent Similarity (ours) - Layer no = 12
41.4
47.4
53.6
Latent Similarity (ours) - Best layer
38.7
45.5
47.2
Appendix
Table 10: Comparison of BoN decoding on instruct models where base pretrained models were used as a base for comparative metric computation.
Model
Metric
Safety(ASR % ) ↓
Utility(ACC % ) ↑
DAN
Wildjailbreak
StrongREJECT
GSM8K
MMLU
First answer
15.7
14.4
13.4
12.3
27.1
Random sampling
14.1
13.5
12.7
9.6
29.3
Qwen 2.5 0.5B Instruct
Perplexity
35.4
31.3
20.1
17.4
40.47
Self-certainty
45.1
45.1
37.0
13.7
33.9
KL (ours)
3.9
2.5
2.2
9.4
20.7
Appendix
Table 11: Comparison of BoN decoding when the a teacher model of DeepSeek R1 Distill Qwen 1.5B is used in the deliberative alignment (SFT).
Model
Metric
Safety(ASR % ) ↓
Utility(ACC % ) ↑
DAN
Wildjailbreak
StrongREJECT
GSM8K
MMLU
First answer
17.8
16.4
14.6
17.3
31.4
Random sampling
17.6
16.9
19.1
17.2
31.9
Qwen 2.5 0.5B Instruct
Perplexity
20.8
18.0
11.8
28.0
45.3
Self-certainty
30.8
32.0
28.4
26.3
38.5
KL (ours)
6.0
2.7
3.8
15.3
17.3
Appendix
Table 12: Comparison of BoN decoding when the a teacher model of DeepSeek R1 Distill Qwen 7B is used in the deliberative alignment (SFT).
Model
Metric
Safety(ASR % ) ↓
Utility(ACC % ) ↑
DAN
Wildjailbreak
StrongREJECT
GSM8K
MMLU
First answer
13.3
11.6
13.4
14.3
23.7
Random sampling
12.2
12.2
12.4
14.1
23.1
Qwen 2.5 0.5B Instruct
Perplexity
14.2
10.2
3.1
24.1
30.0
Self-certainty
17.5
20.1
15.0
22.4
23.8
KL (ours)
4.4
2.9
2.8
11.1
10.6
Appendix
Table 13: Comparison of BoN decoding when the a teacher model of DeepSeek R1 Distill Llama 8B is used in the deliberative alignment (SFT).
Model
Metric
Safety(ASR % ) ↓
Utility(ACC % ) ↑
DAN
Wildjailbreak
StrongREJECT
GSM8K
MMLU
First answer
16.1
14.0
10.8
12.4
21.7
Random sampling
14.0
14.3
12.1
11.7
23.1
Qwen 2.5 0.5B Instruct
Perplexity
25.9
25.7
16.6
23.2
31.0
Self-certainty
32.4
34.2
37.0
20.3
23.8
KL (ours)
5.1
2.4
1.9
9.7
12.8
Appendix
Table 14: Comparison of BoN decoding when the a teacher model of DeepSeek R1 Distill Qwen 14B is used in the deliberative alignment (SFT).
Model
Metric
Safety(ASR % ) ↓
Utility(ACC % ) ↑
DAN
Wildjailbreak
StrongREJECT
GSM8K
MMLU
First answer
18.5
15.4
12.1
13.7
18.9
Random sampling
16.0
15.7
18.8
13.3
18.5
Qwen 2.5 0.5B Instruct
Perplexity
24.9
20.7
15.6
21.8
29.1
Self-certainty
33.5
32.3
30.3
22.1
22.1
KL (ours)
6.5
4.3
6.0
14.4
12.4
Appendix
Table 15: Comparison of BoN decoding when the a teacher model of DeepSeek R1 Distill Qwen 32B is used in the deliberative alignment (SFT).
Model
Metric
Safety(ASR % ) ↓
Utility(ACC % ) ↑
DAN
Wildjailbreak
StrongREJECT
GSM8K
MMLU
First answer
16.9
13.7
16.9
10.6
23.2
Random sampling
15.9
14.6
17.2
9.7
23.9
Qwen 2.5 0.5B Instruct
Perplexity
21.4
12.6
16.9
17.9
34.1
Self-certainty
22.2
18.15
19.8
16.4
27.8
KL (ours)
6.5
5.2
8.6
9.0
14.3
Appendix
Table 16: Comparison of BoN decoding when the a teacher model of DeepSeek R1 Distill Qwen 70B is used in the deliberative alignment (SFT).
Model
Metric
Safety(ASR % ) ↓
Utility(ACC % ) ↑
DAN
Wildjailbreak
StrongREJECT
GSM8K
MMLU
First answer
31.6
37.6
42.8
4.1
30.5
Random sampling
34.3
36.5
41.5
4.9
31.0
Qwen 2.5 0.5B Instruct
Perplexity
59.6
59.6
61.6
4.0
26.1
Self-certainty
57.6
59.4
62.6
4.6
26.2
KL (ours)
21.4
32.3
46.6
4.3
27.0
Appendix
Table 17: Comparison of BoN decoding when the a teacher model of QWQ 32B is used in the deliberative alignment (SFT).
Model
Metric
Safety(ASR % ) ↓
Utility(ACC % ) ↑
DAN
Wildjailbreak
StrongREJECT
GSM8K
MMLU
First answer
20.7
20.55
13.7
12.3
30.1
Random sampling
21.9
22.8
15.0
11.5
32.4
Qwen 2.5 0.5B Instruct
Perplexity
34.1
25.8
11.5
23.3
34.3
Self-certainty
42.7
45.0
38.6
21.3
31.9
KL (ours)
5.9
5.3
2.2
16.9
26.2
Appendix
Table 18: Comparison of BoN decoding when the a teacher model of DeepSeek R1 Distill Qwen 1.5B is used in the deliberative alignment (GRPO).
Model
Metric
Safety(ASR % ) ↓
Utility(ACC % ) ↑
DAN
Wildjailbreak
StrongREJECT
GSM8K
MMLU
First answer
15.3
14.0
9.2
14.3
31.1
Random sampling
15.8
14.0
8.9
13.1
27.8
Qwen 2.5 0.5B Instruct
Perplexity
10.7
5.7
1.9
32.0
33.7
Self-certainty
15.5
18.3
14.6
29.8
31.5
KL (ours)
4.9
2.3
0.3
20.8
23.0
Appendix
Table 19: Comparison of BoN decoding when the a teacher model of DeepSeek R1 Distill Llama 8B is used in the deliberative alignment (GRPO).
Model
Metric
Safety(ASR % ) ↓
Utility(ACC % ) ↑
DAN
Wildjailbreak
StrongREJECT
GSM8K
MMLU
First answer
22.4
24.0
15.0
13.7
23.1
Random sampling
22.2
24.2
8.9
11.7
23.1
Qwen 2.5 0.5B Instruct
Perplexity
19.0
15.9
4.4
29.4
17.3
Self-certainty
29.5
32.1
19.4
27.5
16.5
KL (ours)
7.1
5.7
3.1
18.3
16.3
Appendix
Table 20: Comparison of BoN decoding when the a teacher model of DeepSeek R1 Distill Qwen 32B is used in the deliberative alignment (GRPO).
How do the methods used to train language models to refuse harmful requests shape how that refusal actually works inside the model? We compare three post-training methods - supervised fine-tuning, reasoning-augmented fine-tuning (training on reasoning chains that justify a safety decision), and preference optimization (ORPO) - across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). We find that training method, not just data, reshapes how refusal is computed internally: reasoning-augmented training consistently produces a distinct kind of refusal computation, visible across all three models, while architecture independently shapes internal structure and how reliably refusal can be steered. Most importantly, no method we study achieves all three properties we would want from safe alignment at once: refusal that isn't concentrated in a few fragile components, safety gains that don't cost general capability, and safety behavior correctable through small, targeted edits. We caution against treating current post-training methods as a solved, reliable defense, especially for security-critical use. Code and models are available in https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.
Large Reasoning Models (LRMs) pose a dual-surface safety challenge: both intermediate reasoning traces and final answers can contain harmful content. Existing alignment methods often operate at the whole-response level, allowing unsafe reasoning to be masked by a safe-looking final answer. We propose Segment-aware Listwise Target DPO (SaLT-DPO), which addresses this gap through three mechanisms: (1) segment-aware listwise alignment that decomposes responses into reasoning and answer segments, independently scores each segment's safety, and aligns length-normalized segment rewards with soft target distributions over multiple candidates; (2) joint safety coherence regularization that applies a weakest-link principle to promote safety consistency across both segments; and (3) utility anchoring on benign prompts to mitigate over-refusal and reasoning degradation. Experiments on three LRMs show that SaLT-DPO consistently reduces unsafe rates for both reasoning and answer segments while mitigating degradation in benign compliance and preserving general reasoning performance. Ablation studies demonstrate the complementary contributions of its components.
JungMin Yun, Junehyoung Kwon, Hayeong Ryu +3
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks but often generate harmful responses to malicious user queries. This paper investigates the underlying cause of these safety risks and shows that the issue lies in the reasoning structure itself. Based on this insight, we claim that effective safety alignment can be achieved by altering the reasoning structure. We propose AltTrain, a simple yet effective post training method that explicitly alters the reasoning structure of LRMs. AltTrain is both practical and generalizable, requiring no complex reinforcement learning (RL) training or reward design, only supervised finetuning (SFT) with a lightweight 1K training examples. Experiments across LRM backbones and model sizes demonstrate strong safety alignment, along with robust generalization across reasoning, QA, summarization, and multilingual setting.