Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment

Mengxuan Hu; Vivek V. Datla; Anoop Kumar; Zihan Guan; Sheng Li; Alfy Samuel; Daben Liu

cs.CLFeb 24, 2026

Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment

Authors: Mengxuan Hu, Vivek V. Datla, Anoop Kumar, Zihan Guan, Sheng Li, Alfy Samuel, Daben Liu

Organizations: University of Virginia · Capital One

Abstract

Recent advances in alignment techniques such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and Direct Preference Optimization (DPO) have improved the safety of large language models (LLMs). However, these LLMs remain vulnerable to jailbreak attacks that disguise harmful intent through indirect or deceptive phrasing. Using causal intervention, we empirically demonstrate that this vulnerability stems from shallow alignment mechanisms that lack deep reasoning, often rejecting harmful prompts without truly understanding why they are harmful. To mitigate this vulnerability, we propose enhancing alignment through reasoning-aware post-training. We construct and release a novel Chain-of-Thought (CoT) fine-tuning dataset that includes both utility-oriented and safety-critical prompts with step-by-step rationales. Fine-tuning on this dataset encourages models to produce principled refusals grounded in reasoning, outperforming standard SFT baselines. Furthermore, inspired by failure patterns in CoT fine-tuning, we introduce Alignment-Weighted DPO, which targets the most problematic parts of an output by assigning different preference weights to the reasoning and final-answer segments. This produces finer-grained, targeted updates than vanilla DPO and improves robustness to diverse jailbreak strategies. Extensive experiments across multiple safety and utility benchmarks show that our method consistently improves alignment robustness while maintaining overall model utility.

Figures & tables

Figure 1: Heatmap of Probing Accuracy for Original and Pruned Llama-2-7b-Chat and Mistral-7B-Instruct-v0.3 on Alignment and Reasoning Tasks.

Figure 2: AW-DPO Pipeline. Step 1: Generate $k$ candidate responses per prompt using the COT-finetuned LLM, and score their harmfulness on (i) reasoning ( $h_{rs}$ ), (ii) response ( $h_{rp}$ ), and (iii) full answer ( $h_{f}$ ) using a judge model. Step 2: Select preference pairs $(x_{\text{chosen}},x_{\text{rejected}})$ where the full harmfulness score difference exceeds threshold $\gamma$ . Step 3: Compute alignment weights and train using $L_{\text{AW-DPO}}$ .

Method Name	Safety						Utility
Method Name	Base $\downarrow$	Writing Styles $\downarrow$	Persuasion Techniques $\downarrow$	Encoding & Encryption $\downarrow$	Multi-languages $\downarrow$	Average $\downarrow$	Average $\uparrow$	Std $\downarrow$
Llama-2-7B Base	78.18%	65.14% ± 6.72	18.68% ± 5.76	0.68% ± 1.05	60.50% ± 10.90	41.32% ± 28.29	17.80%	6.94%
$\hookrightarrow$ +SFT	69.77%	61.04% ± 5.75	13.50% ± 4.62	2.50% ± 2.22	64.09% ± 2.78	39.71% ± 27.55	45.29%	12.24%
$\hookrightarrow$ +Safety SFT	43.86%	31.92% ± 17.75	9.68% ± 2.77	2.67% ± 2.60	50.27% ± 11.52	25.99% ± 21.38	43.77%	12.75%
$\hookrightarrow$ +CoT SFT	63.41%	52.72% ± 12.26	13.41% ± 3.70	0.06% ± 0.10	30.09% ± 19.01	28.45% ± 23.81	44.43%	12.03%
$\hookrightarrow$ + CoT Safety SFT	14.09%	11.26% ± 9.17	7.59% ± 2.66	0.06% ± 0.10	7.82% ± 4.82	7.57% ± 6.92	44.14%	11.40%

Table 1: Safety and utility performance of our methods compared to baselines.

Method Name	Safety						Utility
Method Name	Base $\downarrow$	Writing Styles $\downarrow$	Persuasion Techniques $\downarrow$	Encoding & Encryption $\downarrow$	Multi-languages $\downarrow$	Average $\downarrow$	Average $\uparrow$	Std $\downarrow$
SAFECHAIN ( 23 )	45.23%	40.71% ± 4.32	15.73% ± 3.16	0.23% ± 0.16	34.55% ± 7.33	25.80% ± 16.40	44.88%	9.03%
PP ( 63 )	5.45%	4.67% ± 1.26	4.68% ± 0.23	0.34% ± 0.38	7.45% ± 1.15	4.55% ± 2.50	61.84%	18.26%
STAIR ( 58 )	2.95%	3.34% ± 1.77	4.14% ± 1.84	0.68% ± 0.68	3.68% ± 0.62	3.09% ± 1.83	70.38%	12.44%
STAIR-DPO-3 ( 58 )	1.59%	1.21% ± 0.70	1.45% ± 0.83	0.34% ± 0.47	2.09% ± 0.53	1.33% ± 0.87	71.34%	12.80%
Ours (Instruct)	2.27%	1.14% ± 0.74	0.95% ± 0.56	0.57% ± 0.34	9.05% ± 6.37	2.92% ± 4.66	65.29%	13.83%

Table 2: Safety and utility performance of our methods vs. advanced alignment baselines.

Method Name	Safety						Utility
Method Name	Base $\downarrow$	Writing Styles $\downarrow$	Persuasion Techniques $\downarrow$	Encoding & Encryption $\downarrow$	Multi-languages $\downarrow$	Average $\downarrow$	Average $\uparrow$	Std $\downarrow$
Llama3.2-3B	5.00%	2.16% ± 2.15	1.09% ± 0.53	0.80% ± 0.59	2.45% ± 1.00	1.85% ± 1.62	50.66%	12.69%
Llama3.1-8B	5.23%	2.16% ± 1.13	1.14% ± 0.48	0.51% ± 0.34	1.91% ± 0.59	1.69% ± 1.24	59.41%	14.03%
Mistral-7B-V0.3	3.18%	2.46% ± 1.40	3.00% ± 0.84	0.62% ± 0.74	5.73% ± 3.47	3.05% ± 2.57	55.73%	13.60%

Table 3: Transferability of DPO dataset on other models. Specifically, we use the pre-constructed AW-DPO dataset using LLaMA2-7B and apply it to train other models.

Panel of Figure 3 — Figure 3: Plot (a) shows the distribution within unsafe full responses. Plots (b) and (c) present the average safety and utility performance, compared to the corresponding open-source aligned models.

Panel of Figure 4 — Figure 4: Plot (a) shows the performance improvements of our method on aligned chat models. Plots (b) and (c) show the ablation study comparing the safety and utility performance of standard DPO versus AW-DPO, respectively.

Scaling Factor $\alpha$	Safety						Utility
Scaling Factor $\alpha$	Base $\downarrow$	Writing Styles $\downarrow$	Persuasion Techniques $\downarrow$	Encoding & Encryption $\downarrow$	Multi-languages $\downarrow$	Average $\downarrow$	Average $\uparrow$	Std $\downarrow$
0.05	1.14%	0.45% ± 0.63	0.18% ± 0.17	1.59% ± 2.05	0.68% ± 0.29	0.69% ± 1.09	49.43%	11.47%
0.1	1.14%	0.23% ± 0.23	0.14% ± 0.11	1.48% ± 1.39	0.59% ± 0.37	0.57% ± 0.82	48.09%	11.15%
0.2	1.14%	0.27% ± 0.3	0.09% ± 0.18	1.36% ± 1.37	0.73% ± 0.53	0.58% ± 0.83	48.52%	11.99%
0.5	1.14%	0.34% ± 0.57	0.05% ± 0.09	1.65% ± 1.76	0.59% ± 0.34	0.62% ± 1.01	48.98%	10.87%

Table 4: Ablation study: Sensitivity Analysis of Scaling Factor

\alpha

Learning Rate $lr$	Safety						Utility
Learning Rate $lr$	Base $\downarrow$	Writing Styles $\downarrow$	Persuasion Techniques $\downarrow$	Encoding & Encryption $\downarrow$	Multi-languages $\downarrow$	Average $\downarrow$	Average $\uparrow$	Std $\downarrow$
$5e-8$	14.55%	12.50% ± 14.37	5.27% ± 1.52	1.25% ± 1.35	7.59% ± 2.45	7.57% ± 8.91	51.36%	12.77%
$1e-7$	12.50%	11.48% ± 12.96	4.82% ± 1.84	0.51% ± 0.57	6.64% ± 2.88	6.70% ± 8.19	51.77%	12.37%
$5e-7$	5.23%	2.42% ± 2.35	1.09% ± 0.51	0.63% ± 0.57	2.23% ± 0.94	1.85% ± 1.73	50.68%	12.28%
$1e-6$	1.14%	0.27% ± 0.3	0.09% ± 0.18	1.36% ± 1.37	0.73% ± 0.53	0.58% ± 0.83	48.52%	11.99%
$5e-6$	7.95%	11.49% ± 7.61	0.50% ± 0.30	0.06% ± 0.10	13.18% ± 5.43	6.93% ± 7.60	26.09%	4.57%

Table 5: Ablation study: Sensitivity Analysis of Learning Rate

lr

Figure 5: T-SNE Visualization of Embeddings from Attention Head 5 for Alignment and Reasoning Tasks Across All Layers.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Example of correct reasoning example and incorrect reasoning example.

Model	Reasoning Accuracy $\uparrow$		Safety Rate $\downarrow$
Model	Before deactivating	After deactivating	Before deactivating	After deactivating
LLaMA2-7B	41.42%	23.83% (-42.46%)	100.00%	99.60% (-0.40%)
LLaMA2-13B	48.69%	33.41% (-31.38%)	99.60%	100.00% (+0.40%)
LLaMA3.2-3B	51.30%	32.79% (-36.07%)	97.00%	98.20% (+1.24%)

Appendix

Table 6: Comparison of reasoning accuracy and safety rate before and after pruning. Percent change is shown in parentheses.

Figure 7: Heatmap of probing accuracy for original and pruned Llama-3.2-3B-Instruct and LLaMA2-13B-Chat on alignment and reasoning tasks.

Method Name	Safety						Utility
Method Name	Base $\downarrow$	Writing Styles $\downarrow$	Persuasion Techniques $\downarrow$	Encoding & Encryption $\downarrow$	Multi-languages $\downarrow$	Average $\downarrow$	Average $\uparrow$	Std $\downarrow$
Llama-2-7b-chat	6.36%	6.14% ± 2.84	5.41% ± 2.62	1.48% ± 2.18	8.09% ± 2.87	5.51% ± 3.41	45.54 %	13.44%
AW-DPO	8.41%	4.74% ± 3.70	2.82% ± 1.73	0.00% ± 0.00	4.14% ± 1.96	3.41% ± 3.11	45.23%	12.36%
Llama-3.2-3B-Instruct	7.50%	6.30% ± 1.41	7.91% ± 2.52	0.00% ± 0.00	26.64% ± 18.70	10.38% ± 13.24	61.46%	13.72%
AW-DPO	1.14%	0.27% ± 0.3	0.09% ± 0.18	1.36% ± 1.37	0.73% ± 0.53	0.58% ± 0.83	48.52%	11.99%
Llama-3.1-8B-Instruct	3.41%	3.00% ± 0.66	4.00% ± 1.32	0.91% ± 0.56	21.41% ± 16.30	7.24% ± 11.30	63.27%	15.62%

Appendix

Table 7: Compare with open-source aligned LLMs in terms of safety and utility.

Scenario	Pearson Correlation
Full generation	0.6658
Response only	0.9116
Reasoning only	0.5761

Appendix

Table 8: Pearson correlation between scores produced under original and perturbed judge prompts.

Method Name	Safety						Utility
Method Name	Base $\downarrow$	Writing Styles $\downarrow$	Persuasion Techniques $\downarrow$	Encoding & Encryption $\downarrow$	Multi-languages $\downarrow$	Average $\downarrow$	Average $\uparrow$	Std $\downarrow$
Phi-4-reasoning	75.00%	69.83% $\pm$ 5.93	23.41% $\pm$ 5.81	11.71% $\pm$ 6.91	66.68% $\pm$ 4.09	47.20% $\pm$ 26.06	60.88%	13.17%
Phi-4-reasoning-plus	67.50%	66.99% $\pm$ 5.26	26.91% $\pm$ 4.99	9.32% $\pm$ 11.01	65.05% $\pm$ 1.76	46.02% $\pm$ 24.80	57.71%	11.78%
Qwen3-4B-Thinking-250	8.41%	7.58% $\pm$ 2.00	3.77% $\pm$ 1.09	0.17% $\pm$ 0.19	24.41% $\pm$ 8.79	9.31% $\pm$ 9.91	45.73%	13.59%
Qwen3-30B-A3B-Thinking-250	4.55%	3.64% $\pm$ 1.78	1.45% $\pm$ 0.57	6.09% $\pm$ 7.08	10.50% $\pm$ 3.44	5.26% $\pm$ 4.92	60.39%	15.00%

Appendix

Table 9: Comparison with reasoning models in terms of safety and utility.

Method Name	Safety						Utility
Method Name	Base $\downarrow$	Writing Styles $\downarrow$	Persuasion Techniques $\downarrow$	Encoding & Encryption $\downarrow$	Multi-languages $\downarrow$	Average $\downarrow$	Average $\uparrow$	Std $\downarrow$
Original Performance	1.14%	0.27% ± 0.3	0.09% ± 0.18	1.36% ± 1.37	0.73% ± 0.53	0.58% ± 0.83	48.52%	11.99%
Prefix attack	1.82%	0.53% ± 0.69	0.50% ± 0.56	0.97% ± 0.44	0.91% ± 0.43	0.76% ± 0.62	48.62 %	11.44%

Appendix

Table 10: Evaluation of our model’s robustness against prefix attacks.

Model Name	Safety						Utility
Model Name	Base $\downarrow$	Writing Styles $\downarrow$	Persuasion Techniques $\downarrow$	Encoding & Encryption $\downarrow$	Multi-languages $\downarrow$	Average $\downarrow$	Average $\uparrow$	Std $\downarrow$
Llama-3.1-8B-Instruct	3.41%	3.00% ± 0.66	4.00% ± 1.32	0.91% ± 0.56	21.41% ± 16.30	7.24% ± 11.30	63.27%	15.62%
$\hookrightarrow$ + CoT Safety SFT	6.14%	4.70% ± 3.40	2.82% ± 1.92	0.17% ± 0.30	15.86% ± 9.41	6.12% ± 7.60	59.41%	14.03%
$\hookrightarrow$ + Safety DPO	2.27%	1.14% ± 0.74	0.95% ± 0.56	0.57% ± 0.34	9.05% ± 6.37	2.92% ± 4.66	65.29%	13.83%

Appendix

Table 11: Performance improvements of our method on aligned chat models.

Model Name	Safety						Utility
Model Name	Base $\downarrow$	Writing Styles $\downarrow$	Persuasion Techniques $\downarrow$	Encoding & Encryption $\downarrow$	Multi-languages $\downarrow$	Average $\downarrow$	Average $\uparrow$	Std $\downarrow$
Standard DPO	1.82%	1.21% ± 0.66	0.68% ± 0.38	4.32% ± 6.5	1.73% ± 0.67	1.83% ± 3.18	57.66%	14.34%
AW-weighted DPO	1.82%	0.87% ± 0.56	0.55% ± 0.47	0.11% ± 0.11	1.36% ± 0.61	0.81% ± 0.68	58.27%	14.31%

Appendix

Table 12: Comparison between standard DPO and AW-DPO on LLaMA3.1-8B.

Model	ASR (GPTFuzz) $\downarrow$	ASR (PAP) $\downarrow$	LLM Score (Adaptive) $\downarrow$
Llama-3-3B	0.05	0.25	5.17
Llama-3-3B-AW-DPO	0.05	0.0125	1.75

Appendix

Table 13: Effectiveness of AW-DPO against Additional Attack Methods.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 584 papers

Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment
Oct 4, 2026Jinghao Pang, Jitai Hao, Qiang Huang +2LLM Alignment LLM Safety Alignment
Harbin Institute of Technology (Shenzhen) · Leiden University
Dual-Adversarial Safety Alignment: Cultivating Intrinsic Threat Comprehension in LRMs
Aug 10, 2026Hongli Shen, Shaopeng Fu, Qinbo Zhang +2Adversarial Training Adversarial Attacks on LLMs
1King Abdullah University of Science and Technology · 2Stony Brook University
Adaptive and Explicit safe: Triggering Latent Safety Awareness in Large Reasoning Models
Jun 15, 2026Ke Miao, Jiaxin Li, Hongliang Chen +2LLM Safety Alignment LLM Safety
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou HighTech Zone (Binjiang) Blockchain and Data Security Research Institute, China · Li Auto Inc. +2

Oct 4, 2026J/K move · Enter open · S save

Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment

Jinghao Pang, Jitai Hao, Qiang Huang, Zhaochun Ren+1

Harbin Institute of Technology (Shenzhen) · Leiden University

LLM Alignment LLM Safety Alignment LLM Safety Supervised Fine-Tuning Adversarial Attacks on LLMs

Open paper AskPDF arXiv

Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to jailbreak attacks that elicit harmful or unsafe outputs. Existing safety alignment approaches, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often require substantial attack-specific supervision and computational resources, while remaining susceptible to shallow safety alignment and over-refusal. To address these challenges, we introduce SSRFT(Supervised Safe-Role Fine-Tuning), the first framework that reformulates safety alignment as the internalization of a predefined safe role. SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset from psychometric questions, limited jailbreak prompts, and a safe-role description. Role-consistent responses are synthesized, validated, and expanded into diverse scenarios, enabling models to internalize safety-oriented values and principles rather than explicit refusal patterns. Experiments across multiple Base and Instruct models show that SSRFT achieves more robust and generalizable safety alignment than standard SFT. SSRFT shows substantially greater robustness to prefilling attacks and better generalization to unseen jailbreak domains, while reducing over-refusal on benign queries and preserving the model's general capabilities. These results establish safe-role internalization as an effective alternative to refusal-centric safety alignment. Warning: This paper contains examples of harmful and toxic language.

Abstract

LLM Alignment

Explore similar work

Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment

Dual-Adversarial Safety Alignment: Cultivating Intrinsic Threat Comprehension in LRMs

Adaptive and Explicit safe: Triggering Latent Safety Awareness in Large Reasoning Models