The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training

Rui Zhang; Hongwei Li; Yun Shen; Xinyue Shen; Wenbo Jiang; Guowen Xu; Yang Liu; Michael Backes

cs.CRApr 9, 2026

The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training

Authors: Rui Zhang, Hongwei Li, Yun Shen, Xinyue Shen, Wenbo Jiang, Guowen Xu, Yang Liu, Michael Backes,

Organizations: University of Electronic Science and Technology of China · Flexera · CISPA Helmholtz Center for Information Security · Nanyang Technological University

Abstract

The deployment of large language models (LLMs) raises significant ethical and safety concerns. While LLM alignment techniques are adopted to improve model safety and trustworthiness, adversaries can exploit these techniques to undermine safety for malicious purposes, resulting in misalignment. Misaligned LLMs may be published on open platforms to magnify harm. To address this, additional safety alignment, referred to as realignment, is necessary before deploying untrusted third-party LLMs. This study explores the efficacy of fine-tuning methods in terms of misalignment, realignment, and the effects of their interplay. By evaluating four Supervised Fine-Tuning (SFT) and two Preference Fine-Tuning (PFT) methods across four popular safety-aligned LLMs, we reveal a mechanism asymmetry between attack and defense. While Odds Ratio Preference Optimization (ORPO) is most effective for misalignment, Direct Preference Optimization (DPO) excels in realignment, albeit at the expense of model utility. Additionally, we identify model-specific resistance, residual effects of multi-round adversarial dynamics, and other noteworthy findings. These findings highlight the need for robust safeguards and customized safety alignment strategies to mitigate potential risks in the deployment of LLMs. Our code is available at https://github.com/zhangrui4041/The-Art-of-Mis-alignment.

Figures & tables

Figure 1: Overview of evaluation workflow.

Misalignment Method	Llama3.1	Mistral	GLM4	Gemma2	Avg.
Baseline	76.40	66.39	78.23	77.64	74.66
LoRA	67.71	62.19	69.49	74.61	68.50
QLoRA	68.81	59.39	73.51	77.79	69.87
AdaLoRA	77.45	64.94	77.13	76.29	73.95
IA3	77.52	66.45	76.66	76.79	74.35
DPO	76.23	68.36	78.36	79.83	75.69

Table 1: Model utility after misalignment. We report the average utility score of the four dimensions. See Table 4 for detailed results.

Figure 2: Model unsafety scores following misalignment.

Figure 3: Unsafety scores across 10 categories. We use grey (filled) and red (outlined) polygons to indicate unsafety levels of baseline and misaligned LLMs. A larger occupied area indicates lower model safety.

Figure 4: Model unsafety of different sizes of misalignment dataset.

Panel of Figure 5 — Figure 5: $\Delta S_{\mathrm{utility}}$ and $\Delta S_{\mathrm{unsafety}}$ between the realigned and the misaligned models. We adopt safe-rlhf as the realignment dataset, and Llama3.1 and Gemma2 as the target models. Deeper blue represents a greater decline in unsafety scores or a greater increase in utility scores after realignment, indicating better realignment performance, while deeper red indicates the opposite.

Figure 6: Results of multi-round misalignment and realignment. We use dataset MisQA for every round of misalignment and safe-rlhf for realignment. We use

M_{n}

and

R_{n}

to represent the

n

-th rounds of misalignment and realignment, respectively.

Figure 7: Logit Lens visualization of the internal decoding trajectory on Gemma2. We present the results of LoRA, DPO, and ORPO here, and provide the complete results in Figure 17 .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: The interplay between misalignment and realignment. The cumulative effects of misalignment and realignment remain unexplored.

Dataset	Categories	Category Number	Size
MisQA	Illegal Activity, Hate Speech, Malware Generation, Physical Harm, Economic Harm, Fraud, Pornography, Political Lobbying, Privacy Violence, Legal Opinion, Financial Advice, Health Consultation, Gov Decision	13	390
hh-rlhf	Violent Crimes, Non-Violent Crimes, Sex-Related Crimes, Specialized Advice, Privacy, Intellectual Property, Indiscriminate Weapons, Hate, Suicide & Self-Harm, Sexual Content	10	500
safe-rlhf	Endangering National Security, Insulting Behavior, Discriminatory Behavior, Endangering Public Health, Copyright Issues, Violence, Drugs, Privacy Violation, Economic Crime, Mental Manipulation, Human Trafficking, Physical Harm, Sexual Content, Cybercrime, Disrupting Public Order, Environmental Damage, Psychological Harm, White-Collar Crime, Animal Abuse	19	950

Appendix

Table 2: The details of fine-tuning datasets. MisQA is used for misalignment. hh-rlhf and safe-rlhf are used for realignment.

Category	Sample Number
Illegal Activitiy	288
Hate Speech	484
Malware	162
Physical Harm	190
Fraud	256
Pornography	73

Appendix

Table 3: Test dataset for model unsafety evaluation.

Method	Models	MMLU	GSM8K	BoolQ	PIQA	Avg. (Model)	Avg. (Method)
Baseline	Llama3.1	67.43	75.00	85.20	77.97	76.40	74.66
	Mistral	61.40	50.00	79.69	74.48	66.39
	GLM4	69.10	70.31	89.17	84.33	78.23
	Gemma2	72.71	76.56	88.04	73.23	77.64
LoRA	Llama3.1	62.72	68.75	66.12	73.23	67.71	68.50
	Mistral	54.54	48.44	84.04	61.75	62.19

Appendix

Table 4: Model utility after misalignment, including the details of all the dimensions.

Figure 9: Model unsafety after misalignment using dataset Shadow Alignment ( SA ).

Panel of Figure 10 — Figure 10: Resource efficacy of each method, including (a) time cost and (b) memory cost.

Panel of Figure 11 — Figure 11: $\Delta S_{\mathrm{utility}}$ and $\Delta S_{\mathrm{unsafety}}$ between the realigned and the misaligned models. We adopt hh-rlhf as the realignment dataset, and Llama3.1 and Gemma2 as the target models. Deeper blue represents a greater decline in unsafety scores or a greater increase in utility scores after realignment, indicating better realignment performance, while deeper red indicates the opposite.

Panel of Figure 12 — Figure 12: Resource efficiency of realignment using dataset hh-rlhf and safe-rlhf , including (a) time cost and (b) memory cost.

Panel of Figure 13 — Figure 13: Results of multi-round misalignment and realignment. We use dataset MisQA for every round of misalignment and adopt (a) hh-rlhf and (b) MisQA itself for realignment. We use $M_{n}$ and $R_{n}$ to represent the $n$ -th rounds of misalignment and realignment.

Model	Baseline	LoRA	QLoRA	AdaLoRA	IA3	DPO	ORPO	Full-SFT
Llama3.1	0.3511	0.7358	0.6821	0.7595	0.5200	0.4147	0.7579	0.7374
Mistral	0.5311	0.7811	0.7553	0.7258	0.5600	0.5963	0.7742	0.7916
GLM4	0.2484	0.7537	0.6389	0.4932	0.3384	0.4268	0.6895	0.8011
Gemma2	0.0216	0.0889	0.1095	0.0237	0.0211	0.2258	0.7958	0.5132
Average	0.2881	0.5899	0.5465	0.5006	0.3599	0.4159	0.7544	0.7108

Appendix

Table 5: Comparison of unsafety scores between PEFT methods and Full-Parameter SFT (Full-SFT).

Category	Cosine Similarity
Malware Generation	0.7950
Political Lobbying	0.7438
Hate Speech	0.7363
Privacy Violence	0.7246
Fraud	0.7016
Pornography	0.6851

Appendix

Table 6: Semantic Consistency Analysis of MisQA Categories. Higher cosine similarity indicates greater intra-class semantic homogeneity.

Figure 14: Unsafety score across 10 categories when using dataset Shadow Alignment (SA) as the fine-tuning dataset. We use grey (filled) and red (outlined) polygons to indicate unsafety levels of baseline and misaligned LLMs. A larger occupied area indicates lower model safety.

Figure 15: Unsafety scores across 10 categories of LLMs realigned by safe-rlhf . We use grey (filled), red (outlined), and blue (outlined) polygons to indicate unsafety levels of baseline, misaligned, and realigned LLMs. A larger occupied area indicates lower model safety.

Figure 16: Unsafety score across 10 categories of LLMs realigned by hh-rlhf . We use grey (filled), red (outlined), and blue (outlined) polygons to indicate unsafety levels of baseline, misaligned, and realigned LLMs. A larger occupied area indicates lower model safety.

Panel of Figure 17 — Figure 17: Logit Lens visualization of the internal decoding trajectory on four LLMs.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 598 papers

Understanding the Effects of Safety Unalignment on Large Language Models
Apr 2, 2026John T. HalloranLLM Safety Alignment LLM Security
Leidos
On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment
Jul 29, 2026Yongjian Guo, Wanlun Ma, Lingyu Shen +2LLM Safety Alignment Adversarial Robustness of Language Models
Tsinghua University · Swinburne University of Technology · EPFL
Guardrails in Logit Space: Safety Token Regularization for LLM Alignment
Apr 19, 2026Thong Bach, Truyen TranLLM Fine-Tuning LLM Safety Alignment
Applied Artificial Intelligence Initiative (A2I2), Deakin University

Apr 2, 2026J/K move · Enter open · S save

Understanding the Effects of Safety Unalignment on Large Language Models

John T. Halloran

Leidos

LLM Safety Alignment LLM Security

Open paper AskPDF arXiv

Safety alignment has become a critical step to ensure LLMs refuse harmful requests while providing helpful and harmless responses. However, despite the ubiquity of safety alignment for deployed frontier models, two separate lines of recent work--jailbreak-tuning (JT) and weight orthogonalization (WO)--have shown that safety guardrails may be largely disabled, resulting in LLMs which comply with harmful requests they would normally refuse. In spite of far-reaching safety implications, analysis has largely been limited to refusal rates of each unalignment method in isolation, leaving their relative effects on adversarial LLM capabilities unknown. To fill this gap, we study the impact of unaligning six popular LLMs of various sizes across a large number of malicious and benign tasks, using both JT and WO. Across the evaluated models, we show that while refusal degradation is split between the two methods, WO produces LLMs far more capable of aiding in malicious activity; in contrast to JT, the majority of WO unaligned models are far less prone to hallucinations, better retain their original natural-language performance, and are more effective at state-of-the-art adversarial and cyber attacks. To thus help mitigate the malicious risks of WO unalignment, we conclude by showing that supervised fine-tuning effectively limits the adversarial attack abilities enabled by WO, without drastically affecting hallucination rates or natural language performance.

Abstract

LLM Alignment

Explore similar work

Understanding the Effects of Safety Unalignment on Large Language Models

On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment

Guardrails in Logit Space: Safety Token Regularization for LLM Alignment