SLDR: Defending Against Malicious Fine-tuning via Selective Layers Recovery and Dynamic Routing

Hui Zhang; Yachao Yuan; Jiayun Wang; Yuanzhuo Li; Hongtao Wang; Yali Yuan

cs.CROct 7, 2026

SLDR: Defending Against Malicious Fine-tuning via Selective Layers Recovery and Dynamic Routing

Authors: Hui Zhang, Yachao Yuan, Jiayun Wang, Yuanzhuo Li, Hongtao Wang, Yali Yuan

Organizations: North China Electric Power University · Southeast University · Soochow University · Engineering Research Center of Intelligent Computing for Complex Energy Systems, Ministry of Education

Abstract

Fine-tuning-as-a-service enables users to adapt aligned large language models (LLMs) to specialized tasks, but malicious fine-tuning can erode refusal behavior while preserving task performance on legitimate inputs. We revisit recent layer-wise safety diagnostics and find that safety sensitivity is signed: scaling different layers can strengthen refusal, weaken it, or have little effect. Motivated by this observation, we propose SLDR, a post-fine-tuning defense based on Selective Layers Recovery and Dynamic Routing. SLDR trains a LoRA recovery adapter only on the layers with the maximum and minimum sensitivity scores in the signed spectrum, and uses representation-based dynamic routing inference to activate the adapter only for malicious queries. Across four model architectures, five downstream tasks, and four harmful benchmarks, SLDR substantially reduces harmful outputs while preserving downstream utility. On Llama3.1/SST2, SLDR reduces the average harmful score from 11.54 to 0.08 while maintaining downstream accuracy, and the harmful score remains near zero under poisoning ratios up to 0.9. The code is available at https://github.com/Stardust457/SLDR.

Figures & tables

Figure 1: Signed layer-sensitivity scores for Llama3.1-8B-Instruct. Positive scores (green) indicate layers whose scaling increases refusal, while negative scores (red) indicate layers whose scaling decreases refusal.

Figure 2: Overview of SLDR. (a) Signed layer selection; (b) two-stage fine-tuning with targeted safety recovery; (c) representation-based dynamic routing for inference-time safety-utility balance.

	Harmful Score $\downarrow$						Finetune Accuracy $\uparrow$
Method	clean	$\rho=0.05$	$\rho=0.1$	$\rho=0.15$	$\rho=0.2$	Average	clean	$\rho=0.05$	$\rho=0.1$	$\rho=0.15$	$\rho=0.2$	Average
SFT	1.60	5.50	11.40	18.30	20.90	11.54	92.43	92.20	92.43	92.32	92.55	92.39
Lisa	1.50	1.60	1.90	1.80	1.70	1.70	73.74	74.08	74.54	74.31	74.20	74.17
Antidote	1.50	1.40	1.60	1.50	1.50	1.50	84.86	84.75	84.75	85.09	84.86	84.86
Panacea	1.20	3.60	6.40	7.00	7.80	5.20	87.84	90.02	89.79	89.56	89.22	89.29
STAR-DSS	1.60	1.70	1.20	1.40	1.70	1.52	84.75	85.67	85.78	85.55	85.67	85.48

Table 1: Comparison with SOTA baselines. Using the Llama3.1 and the SST2 dataset.

Table 4

	Harmful Score $\downarrow$
Method	clean	$\rho=0.05$	$\rho=0.1$	$\rho=0.15$	$\rho=0.2$	Average
BDS	1.40	1.46	1.47	1.51	1.49	1.47
One-shot FT	1.77	1.82	1.78	1.84	1.86	1.81
Ours	1.08	1.08	1.06	1.09	1.14	1.09

Table 4: Robustness Across Different LLM Judge. Using the Llama3.1 and the SST2 dataset.

	Harmful Score $\downarrow$								Finetune Accuracy $\uparrow$
Method	$\rho=0.3$	$\rho=0.4$	$\rho=0.5$	$\rho=0.6$	$\rho=0.7$	$\rho=0.8$	$\rho=0.9$	$\rho=1.0$	$\rho=0.3$	$\rho=0.4$	$\rho=0.5$	$\rho=0.6$	$\rho=0.7$	$\rho=0.8$	$\rho=0.9$	$\rho=1.0$
SFT	29.30	30.40	34.30	34.20	36.00	36.70	35.70	36.70	92.32	92.09	91.97	91.63	91.74	91.17	90.37	–
BDS	2.90	2.70	2.60	2.90	3.30	3.10	3.50	3.60	91.74	91.40	91.97	91.74	91.74	91.40	91.28	–
One-shot FT	5.00	5.90	6.50	6.30	7.30	7.10	7.40	8.00	89.79	89.56	89.68	89.22	87.96	87.96	87.61	–
Ours	0.20	0.30	0.20	0.30	0.20	0.40	0.20	0.20	92.32	92.09	91.97	91.51	91.51	90.94	90.14	–

Table 5: Robustness across different poisoning ratios. Using the Llama3.1 and the SST2 dataset.

Table 7

Table 8

Table 10

Statistic	Top-5 layers (ranked; score)
SLDR	13 (+200), 12 ( $-$ 185), 10 (+180), 8 ( $-$ 170), 15 (+135)
Panacea	11 (58.13), 9 (49.03), 8 (46.75), 12 (45.06), 17 (44.15)
Targeted Vaccine	0 (0.992), 18 (0.490), 19 (0.484), 1 (0.482), 16 (0.482)
Surgery	0 (+0.154), 2 ( $-$ 0.178), 1 ( $-$ 0.182), 3 ( $-$ 0.371), 4 ( $-$ 0.623)

Table 12: Comparison of layer-wise statistics on Llama3.1-8B-Instruct. Scores are shown for the Top-5 layers. Since the statistics have different scales, scores should not be compared across methods.

Statistic	Top-5 overlap	Top-10 overlap
Panacea	2/5	4/10
Targeted Vaccine	0/5	4/10
Surgery	0/5	2/10

Table 13: Overlap between the Top-

K

layers identified by SLDR and other layer-wise statistics.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

\mathbf{W}^{(l)+}\leftarrow(1+\alpha)\,\mathbf{W}^{(l)},\quad\forall\,\mathbf{W}^{(l)}\in\left\{W_{Q}^{(l)},W_{K}^{(l)},W_{V}^{(l)},W_{O}^{(l)},W_{\mathrm{up}}^{(l)},W_{\mathrm{gate}}^{(l)},W_{\mathrm{down}}^{(l)}\right\}

Appendix

Algorithm 1 Layer Selection

h_{l_{\text{max}}}(x)=\mathcal{M}_{\text{rec}}^{l_{\text{max}}}(x)[-1,:]\in\mathbb{R}^{d}

Appendix

Algorithm 2 Representation-based Dynamic Routing Inference

Figure 5: Refusal count results for layer-wise scaling of Llama3.1-8B-Instruct. (a) shows the results for the first 16 layers, and (b) shows the results for the last 16 layers. The x-axis denotes the scaling parameter

(1\pm\alpha)

applied to all weight matrices in layer

l

, where

\alpha\in\{0.1,0.2\}

. The value 1.0 corresponds to the unscaled baseline.

Figure 6: Refusal count results for layer-wise scaling of Llama3-8B-Instruct. (a) shows the results for the first 16 layers, and (b) shows the results for the last 16 layers. The x-axis denotes the scaling parameter

(1\pm\alpha)

applied to all weight matrices in layer

l

, where

\alpha\in\{0.1,0.2\}

. The value 1.0 corresponds to the unscaled baseline.

Figure 7: Refusal count results for layer-wise scaling of Qwen2.5-7B-Instruct. (a) shows the results for the first 14 layers, and (b) shows the results for the last 14 layers. The x-axis denotes the scaling parameter

(1\pm\alpha)

applied to all weight matrices in layer

l

, where

\alpha\in\{0.1,0.2\}

. The value 1.0 corresponds to the unscaled baseline.

Figure 8: Refusal count results for layer-wise scaling of Mistral-7B-Instruct-v0.2. (a) shows the results for the first 16 layers, and (b) shows the results for the last 16 layers. The x-axis denotes the scaling parameter

(1\pm\alpha)

applied to all weight matrices in layer

l

, where

\alpha\in\{0.1,0.2\}

. The value 1.0 corresponds to the unscaled baseline.

Figure 9: Layer-wise sensitivity scores of Llama3-8B-Instruct. Positive scores (green) indicate layers that enhance safety refusal, while negative scores (red) indicate layers that undermine it.

Figure 10: Layer-wise sensitivity scores of Qwen2.5-7B-Instruct. Positive scores (green) indicate layers that enhance safety refusal, while negative scores (red) indicate layers that undermine it.

Figure 11: Layer-wise sensitivity scores of Mistral-7B-Instruct-v0.2. Positive scores (green) indicate layers that enhance safety refusal, while negative scores (red) indicate layers that undermine it.

	Harmful Score $\downarrow$					Finetune Accuracy $\uparrow$
Method	Seed 0	Seed 1	Seed 2	Seed 42	Mean $\pm$ Std	Seed 0	Seed 1	Seed 2	Seed 42	Mean $\pm$ Std
BDS	1.80	2.40	1.70	1.60	$1.88\pm 0.36$	90.71	91.86	91.28	91.63	$91.37\pm 0.50$
One-shot FT	4.50	3.90	4.40	3.90	$4.18\pm 0.32$	90.83	90.60	90.71	90.48	$90.66\pm 0.15$
Ours	0.10	0.00	0.20	0.00	$0.08\pm 0.10$	93.00	93.12	93.12	92.43	$92.92\pm 0.33$

Appendix

Table 14: Multi-seed stability analysis on Llama3.1 with the SST2 dataset.

Method	Harmful Score $\downarrow$	Finetune Accuracy $\uparrow$
BDS	47.70	88.42
One-shot FT	51.10	93.46
Ours	1.10	92.55

Appendix

Table 15: Performance under full-parameter downstream fine-tuning. Using the Llama3.1 and the SST2 dataset.

Component	Cost
Layer probing across 32 layers	0.3663 hours
Standard SFT inference	0.3121 seconds per prompt
SLDR inference with routing	0.3293 seconds per prompt
Additional online overhead	0.0172 seconds per prompt

Appendix

Table 16: Computational overhead of SLDR on Llama3.1. Inference time is averaged over 1,000 BeaverTails prompts with a generation length of 64 tokens.

Method	Zulu HS $\downarrow$	IJP HS $\downarrow$
Original aligned model	66.82	12.94
SFT (no defense)	66.82	18.24
EnchTable	72.71	13.53
Ours	61.53	15.53

Appendix

Table 17: Comparison under jailbreak distribution shifts on Llama3.1 with the SST2 dataset. Lower Harmful Score (HS) indicates better safety.

Method	BeaverTails HS $\downarrow$	SST2 FA $\uparrow$
Original aligned model	1.50	70.64
SFT (no defense)	11.40	92.43
EnchTable	1.50	88.65
Ours	0.00	92.43

Appendix

Table 18: Safety and downstream utility comparison on Llama3.1 with the SST2 dataset. HS is evaluated on BeaverTails.

Recovery Epochs	HS $\downarrow$	FA $\uparrow$
5	0.10	92.20
10	0.00	92.20
20 (default)	0.00	92.43
30	0.30	92.20
40	0.60	92.20
50	0.60	92.20

Appendix

Table 19: Impact of the number of recovery epochs on SLDR. Experiments are conducted on Llama3.1 with the SST2 dataset under

\rho=0.1

and

|\mathcal{D}_{\text{train}}|=1000

Metric / Threshold	$-0.10$	$-0.08$	$-0.06$	$-0.04$	$-0.02$	$0.00$	$0.02$	$0.04$	$0.06$	$0.08$	$0.10$
TPR $\uparrow$	100.00	100.00	100.00	100.00	100.00	100.00	99.25	94.75	89.25	76.25	59.25
FPR $\downarrow$	100.00	63.00	15.25	2.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Benign over refusal rate $\downarrow$	14.50	14.50	7.50	1.75	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Appendix

Table 20: Router performance under different routing thresholds

\tau

Explore similar work

HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models
Jul 13, 2026Aznaur Aliev, Carlos Hinojosa, Abdelrahman Eldesokey +3LLM Fine-Tuning LLM Safety Alignment
King Abdullah University of Science and Technology, Saudi Arabia
Jailbreak to Protect: Buffering and Reinforcing via Temporary Jailbreaking for Safe Fine-Tuning in Large Language Models
May 23, 2026Seokil Ham, Jaehyuk Jang, Wonjun Lee +1LLM Fine-Tuning LLM Safety Alignment
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.
When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning
Sep 1, 2026Yitong Guo, Xiaoyi Chen, Siyuan Zhang +2LLM Fine-Tuning LLM Safety Alignment
Indiana University Bloomington · Tsinghua University · Nanyang Technological University

Jul 13, 2026, cs.LGJ/K move · Enter open · S save

HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models

Aznaur Aliev, Carlos Hinojosa, Abdelrahman Eldesokey, Bang An+2

King Abdullah University of Science and Technology, Saudi Arabia

LLM Fine-Tuning LLM Safety Alignment LLM Safety Inference-Time Intervention

Open paper AskPDF arXiv

Safety alignment in large language models can be fragile under fine-tuning, as even benign task adaptation may increase harmful compliance. Existing defenses mainly follow two directions: they either intervene during or after fine-tuning through retraining or weight modification, which can be costly and may hurt task performance, or they use model-agnostic safety classifiers, which may miss failures specific to a given fine-tuned checkpoint. These limitations motivate a post hoc, model-specific, and non-invasive approach to safety restoration. To meet these requirements, we propose HyperSafe, a framework that restores safety behavior by generating a model-specific Safe Side Network (SSN) for each fine-tuned checkpoint. HyperSafe uses layer-wise activation fingerprints to capture how fine-tuning changes the model's inner representations. With a small set of given calibration prompts, the hypernetwork maps these fingerprints to the parameters of the \ssn{} in a single forward pass. The generated \ssn{} runs alongside the frozen fine-tuned model and performs prompt-level safety classification: harmful prompts are routed to refusal, while safe prompts are answered by the original fine-tuned model. Thus, HyperSafe requires no gradient updates, no safety data at deployment time, and no modification to the deployed model weights. We evaluate HyperSafe on two model families, Qwen2-7B and LLaMA-3-8B, across multiple safety benchmarks. HyperSafe reduces harmful response rates from 19-31% to below 1% on every held-out checkpoint, while keeping downstream task accuracy within 1% of the fine-tuned baseline on average. Code is available at https://github.com/nokronim/project-safety-remedy.

Abstract

Explore similar work

HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models

Jailbreak to Protect: Buffering and Reinforcing via Temporary Jailbreaking for Safe Fine-Tuning in Large Language Models

When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning