Refusal Localizes, the Damage Relocates: Safety Layers Under Few-Sample Fine-Tuning

Jungseob Lee; Dongyub Jude Lee; Sugyeong Eo; Seongtae Hong; Seungyoon Lee; Heuiseok Lim

cs.CLSep 29, 2026

Refusal Localizes, the Damage Relocates: Safety Layers Under Few-Sample Fine-Tuning

Authors: Jungseob Lee, Dongyub Jude Lee, Sugyeong Eo, Seongtae Hong, Seungyoon Lee, Heuiseok Lim

Organizations: Korea University · Zoom Communications · Yonsei University Mirae Campus

Abstract

Fine-tuning adapts aligned large language models (LLMs) to downstream tasks, but a few dozen harmful examples can remove their refusal of harmful requests. Prior work localizes safety-related behavior to specific layers, directions, and tokens, suggesting targets for protection. We test whether successful localization and recovery support defenses that survive changes in the attack. Across six checkpoints from four model families, harmful and benign prompts remain linearly separable after attack, and patching full clean hidden states into the compromised model restores refusal at a reproducible transition depth. Building on a prior layer-freezing defense, we freeze every layer up to this depth and repeat the attack. At a hundred harmful examples, refusal remains near zero on all six checkpoints, with recovery transitions above the frozen boundary. In a second study, removing the update's top two singular directions restores refusal after short attention-only fine-tunes on four checkpoints. On Llama-3.1-8B, ordinary training changes weaken this repair and an attacker who spreads the update defeats it. A spectral detector calibrated on benign Llama fine-tunes misses most repair failures on that checkpoint. Localized freezing can nevertheless help preserve refusal when a few harmful examples enter training data unintentionally. These results show that an attacker can bypass a region identified by recovery and defeat a repair that works across multiple checkpoints, motivating five checks for defenses against adaptive fine-tuning. Code is available at https://github.com/js-lee-AI/refusal-relocates.

Figures & tables

Figure 1: (a) Lockstep activation patching overwrites the compromised model’s layer output with the clean state at prefill and each decoding step. (b) In the matched Llama attack, freezing layers 0–17 shifts the half-ceiling transition from 15 to 27–28.

Figure 2: Pooled accuracy of the frozen probe under three thresholds on

200

harmful and

100

benign prompts per cell, four checkpoints at four depths, bars over three seeds. On the right, lockstep recovery from injecting the clean state of this prompt, another harmful prompt, or a benign one.

Figure 3: Recovery from patching one clean layer into the compromised checkpoint.

			Coherent refusal $\uparrow$				Llama-Guard unsafe $\downarrow$
Model	$L$	Frozen	Clean	Attack	Freeze	Matched	Clean	Attack	Freeze	Matched
LoRA, each checkpoint at its own measured depth
Llama-3.1-8B	32	$[0,17]$	.92–.96	.00	.00–.01	.00	.04–.05	.94–.98	.89–.92	.96–.97
Tulu-3-8B-DPO	32	$[0,15]$	.99–1.00	.01–.06	.00–.03	.01–.04	.00	.89–.94	.53–.74	.89–.92
OLMo-2-7B	32	$[0,16]$	.99–1.00	.00–.02	.00–.02	.06–.07	.00	.84–.94	.49–.72	.84–.88
OLMo-2-13B	40	$[0,20]$	.99–1.00	.00–.02	.00–.01	.00–.02	.00	.89–.95	.23–.63	.89–.96

Table 1: Coherent refusal and Llama-Guard unsafe rates on AdvBench after an attack that cannot write to the frozen layers. Dose

100

, three seeds per checkpoint, ranges over seeds. Matched freezes the same number of layers elsewhere. The blocks differ in what is frozen.

Figure 4: Coherent refusal on AdvBench, bars spanning the seeds that pass the gate. (a) The evaluated low-dose comparisons. (b) Llama-3.1-8B with layers

[0,17]

frozen against dose, and (c) against the harmful fraction of a mixed adapter.

Figure 5: (a, b) Recovery rescaled from the attacked floor to the clean ceiling. Gray cells are frozen; outlines mark the first measured layer with at least half-range recovery in all seeds. (c) Thick and thin bars leave two or one writable layers.

	Coherent refusal $\uparrow$				Llama-Guard unsafe $\downarrow$
Model	Clean	All	Top 4	Bottom 4	Clean	All	Top 4	Bottom 4
Llama-3.1-8B	.92–.96	.00	.00–.01	.00–.03	.04–.05	.93–.98	.70–.75	.90–.94
Tulu-3-8B-DPO	.99–1.00	.01–.06	.00–.01	.00–.04	.00	.89–.94	.03–.28	.77–.95
OLMo-2-7B	.99–1.00	.00–.02	.10–.16	.05–.14	.00	.84–.94	.04–.12	.50–.84
OLMo-2-13B	.99–1.00	.00–.02	.57–.69	.01–.03	.00	.89–.95	.01–.03	.78–.94
Qwen2.5-14B	.97–.99	.00–.01	.01–.03	.16–.30	.00	.92–.98	.00–.02	.60–.74

Table 2: Coherent refusal and Llama-Guard unsafe on AdvBench when four layers are left writable at one end. All is the unrestricted attack, Top four freezes

[0,L{-}5]

and Bottom four leaves layers

0

3

. Dose

100

, three seeds, ranges over seeds.

Figure 6: Coherent refusal after singular-direction removal: four checkpoints, 3 seeds, 5 doses (top); top-two removal by epoch on Llama-3.1-8B (bottom).

	Standard	Spread	Projected
	5 ep	$\lambda{=}1$	$k{=}2$
$\\|\Delta W\\|_{F}$	.607–.623	.228–.229	.222–.226
Part. ratio	4.47–4.73	15.6	5.99–6.00
Top-2 energy	.597–.607	.149	.407–.408
Refusal, no repair	.00	.00–.02	.02–.06
LG unsafe, no repair	.96–.98	.96–.98	.88–.90

Table 3: Three dose-

100

attacks on Llama-3.1-8B, evaluated on the same

50

AdvBench prompts. Ranges cover three training seeds, except two landed seeds for the projected attack. Norm and spectral statistics average adapted modules.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	AdvBench	HarmBench	StrongREJECT	XSTest-safe over-ref.
Original	0.955	0.882	0.938	0.035
Compromised	0.010	0.003	0.003	0.000
Repaired (top-2)	0.940	0.813	0.818	0.008
Random-2 removed	0.008	0.003	0.001	0.000

Appendix

Table 4: Coherent refusal on three harmful benchmarks and over-refusal on XSTest-safe. Means cover

12

runs with fixed evaluation prompts and three training seeds at each setting: Llama-3.1-8B at doses

25

and

100

, and Tulu-3-8B-DPO and OLMo-2-7B at dose

50

. Top-2 removes the leading two singular directions; random-2 removes two random directions.

Model	Layer	Fixed	Recal.	Oracle	Model	Layer	Fixed	Recal.	Oracle
Llama-3.1-8B	9	0.90–0.95	0.81–0.89	0.93–0.96	OLMo-2-7B	9	0.69–0.94	0.96–0.97	0.97–0.98
	14	0.35–0.39	0.79–0.87	0.94–0.97		14	0.36–0.42	0.88–0.91	0.92–0.95
	19	0.38–0.54	0.69–0.77	0.88–0.90		19	0.34	0.72–0.75	0.87–0.90
	24	0.56–0.74	0.72–0.74	0.85–0.90		24	0.33–0.34	0.63–0.70	0.84–0.85
Tulu-3-8B-DPO	9	0.33–0.85	0.75–0.93	0.91–0.96	Qwen2.5-14B	14	0.34–0.73	0.84–0.94	0.91–0.95
	14	0.33	0.69–0.85	0.90–0.92		21	0.33–0.80	0.81–0.92	0.89–0.94

Appendix

Table 5: Pooled accuracy of the frozen probe at dose

100

200

harmful and

100

benign prompts per cell, three seeds per checkpoint, ranges over seeds. Fixed uses the clean threshold, recalibrated uses the

95

th percentile of calibration-half benign scores, and oracle maximizes pooled accuracy using evaluation labels.

Checkpoint	Layer	Base	Aligned	Gap	Checkpoint	Layer	Base	Aligned	Gap
Llama-3.1-8B	9	0.915	0.958	$+0.043$	OLMo-2-13B	12	0.931	0.961	$+0.030$
	14	0.944	0.972	$+0.029$		18	0.941	0.969	$+0.028$
	19	0.936	0.971	$+0.035$		24	0.943	0.972	$+0.029$
	24	0.937	0.965	$+0.027$		30	0.943	0.969	$+0.026$
OLMo-2-7B	9	0.907	0.946	$+0.038$
	14	0.932	0.970	$+0.038$

Appendix

Table 6: Fresh probe AUROC inside never-aligned base checkpoints and their aligned siblings, on the identical contrast and identical input tokens, at four relative depths.

Model	Dose	Clean	Attack	Freeze	Matched
Llama-3.1-8B	5	.92–.96	.00–.05	.91–.96	.16–.24
OLMo-2-7B	25	.99–1.00	.10–.18	.95–.98	.30–.50
Tulu-3-8B-DPO	25	.99–1.00	.01–.05	.21–.52	.01–.05
Qwen2.5-14B	25	.97–.99	.00–.02	.33–.60	.26–.35
Yi-1.5-9B	10	.93–.94	.05–.13	.85–.93	.80–.87
OLMo-2-13B	5, 10	No successful unrestricted attack at either dose

Appendix

Table 7: The evaluated low-dose freezes, three seeds, ranges over the seeds that pass the gate, coherent refusal on AdvBench. Matched is a freeze of the same size at the other end of the network.

Model	$n{=}5$	$n{=}10$	$n{=}25$	$n{=}50$	$n{=}100$
Llama-3.1-8B (32L)	.375 $\pm$ .000	.438 $\pm$ .054	.562 $\pm$ .094	.531 $\pm$ .054	.531 $\pm$ .108
Tulu-3-8B-DPO (32L)	—	—	.406 $\pm$ .054	.469 $\pm$ .000	.469 $\pm$ .000
OLMo-2-7B (32L)	—	—	.344 $\pm$ .054	.500 $\pm$ .054	.500 $\pm$ .054
OLMo-2-13B (40L)	.375 [1]	.442 $\pm$ .058	.375 $\pm$ .000	.525 $\pm$ .043	.525 $\pm$ .043
Qwen2.5-14B (48L)	—	.375 $\pm$ .000 [2]	.681 $\pm$ .060	.646 $\pm$ .000	.646 $\pm$ .000
Yi-1.5-9B (48L)	.681 $\pm$ .060	.806 $\pm$ .048	.750 $\pm$ .000	.806 $\pm$ .048	.806 $\pm$ .048

Appendix

Table 8: Transition depth

\ell^{*}

as a fraction of network depth, mean over surviving seeds with the seed standard deviation. Each cell is three seeds unless a smaller count is in brackets, and a dash is a dose at which the attack does not land.

Frozen	L12	L15	L18	L21	L24	L27	L28	L29	L30	$\ell^{*}$ by seed
none	.00–.05	.77–.91	.90–.93	.92–.94	.93–.96	.92–.96	.92–.96	.92–.96	.92–.96	15, 15, 15
$[0,5]$	.00–.01	.72–.93	.91–.94	.92–.94	.93–.95	.92–.96	.92–.96	.92–.96	.92–.96	15, 15, 15
$[0,11]$	.00	.05–.67	.77–.90	.91–.94	.93–.96	.92–.96	.91–.96	.91–.96	.92–.96	18, 15, 18
$[0,17]$	.00–.01	.00–.01	.00–.04	.01–.04	.03–.08	.08–.67	.72–.94	.86–.94	.91–.95	28, 27, 27
$[0,23]$	.00–.01	.00–.01	.00–.01	.00–.01	.01–.04	.00–.03	.03–.16	.15–.49	.85–.94	29, 30, ${\geq}29$
$[0,27]$	.00–.01	.00–.01	.00–.01	.00–.01	.00–.01	.00–.01	.00–.02	.01–.02	.05–.50	30, ${\geq}30$ , ${>}30$

Appendix

Table 9: Coherent refusal under patching at one layer, Llama-3.1-8B at dose

100

, three seeds at every rung, ranges over seeds,

100

evaluation prompts. Gray cells lie at or below the frozen boundary. The last column gives

\ell^{*}

for seeds

42

43

and

44

Model	$b$	Unrestr. $\ell^{*}$	$b{+}1$	$b{+}2$	$b{+}3$	$b{+}4$	$b{+}5$	$\ell^{*}$ by seed
Llama-3.1-8B	17	15, 15, 15	.00–.11	.00–.05	.00–.03	.00–.03	.03–.11	${>}27$ , 27, 27
Tulu-3-8B-DPO	15	15, 15, 15	.04–.42	.62–.86	.88–.94	.89–.94	.96–.99	17, 17, 17
OLMo-2-7B	16	15, 15, 18	.10–.26	.66–.77	.73–.84	.74–.84	.75–.85	18, 18, 18
OLMo-2-13B	20	19, 19, 19	.00–.33	.33–.50	.62–.65	.65–.78	.78–.90	23, 22, 23
Qwen2.5-14B	31	31, 31, 31	.00	.00–.25	.15–.47	.28–.62	.28–.68	35, 36, 40
Yi-1.5-9B	39	36, 40, 40	.00–.11	.00–.11	.08–.29	.59–.76	.62–.76	43, 43, 43

Appendix

Table 10: Recovery rescaled to the floor-to-ceiling range at the five layers above each frozen boundary

b

, dose

100

, ranges over seeds. Bold marks the first layer with at least half-range recovery in every seed. The

\ell^{*}

columns report measured half-ceiling crossings or bounds, following Section 4 .

Model, boundary $b$	Below	Donated at, with the layer in brackets
Llama-3.1-8B, 17	.93–.95 [12]	.00 [28]	.00 [30]	.00 [31]
benign donor	1.00 [12]	.78–.85 [28]	.72–.82 [30]	.70–.80 [31]
Tulu-3-8B-DPO, 15	1.00 [10]	.88–.90 [16]	.28–.47 [17]	.12–.25 [18]
OLMo-2-7B, 16	1.00 [10]	.97–1.00 [17]	.53–.65 [18]	.10–.35 [19]

Appendix

Table 11: Coherent refusal after donor-state patching into the clean model on

40

AdvBench prompts, ranges over seeds; layers are in brackets. For restricted donors, the unpatched reference is

0.93

–

0.95

on Llama and

1.00

on Tulu and OLMo-2-7B, and Below is a no-op patch inside the frozen prefix.

			Coherent refusal $\uparrow$				Llama-Guard unsafe $\downarrow$
Model	Frozen	Free	Clean	Attack	Freeze	Matched	Clean	Attack	Freeze	Matched
Llama-3.1-8B	$[0,29]$	2	.92–.96	.00	.00–.01	.02–.16	.04–.05	.93–.98	.63–.71	.79–.94
	$[0,30]$	1	.92–.96	.00	.00–.03	.01–.03	.04–.05	.93–.98	.45–.62	.88–.95
Tulu-3-8B-DPO	$[0,29]$	2	.99–1.00	.00–.05	.01–.04	.00–.03	.00	.90–.95	.08–.20	.85–.94
	$[0,30]$	1	.99–1.00	.00–.05	.00–.04	.00–.11	.00	.90–.95	.51–.61	.70–.85
Qwen2.5-14B	$[0,45]$	2	.97–.99	.00	.06–.13	.14–.51	.00	.92–.97	.00–.03	.41–.72

Appendix

Table 12: Coherent refusal and Llama-Guard unsafe on AdvBench with the attacker confined to the last one or two layers. Free is the number of writable layers and Matched freezes the same number at the other end. Dose

100

, three seeds, ranges over seeds.

Attacker	Refusal $\uparrow$	LG unsafe $\downarrow$	Over-refusal	Train loss	$\\|\Delta W\\|_{F}$ ratio
Clean	0.92–0.96	0.04–0.05	0.04	–	–
Full FT	0.00	0.96–0.98	0.00	0.13–0.17	1.00
Full FT, $[0,17]$ frozen	0.00–0.01	0.67–0.85	0.00	0.14–0.16	1.09–1.25
Full FT, $[14,31]$ frozen	0.00–0.01	0.96–0.98	0.00	0.39–0.42	1.14–1.28

Appendix

Table 13: Full fine-tuning under three freeze conditions, Llama-3.1-8B at dose

100

, ranges over seeds. Refusal and over-refusal are coherent refusal rates on AdvBench and on XSTest-safe.

\|\Delta W\|_{F}

is the mean over trainable weight matrices, relative to the unrestricted arm.

$f$ and arm	Refusal $\uparrow$	LG unsafe $\downarrow$	Over-refusal	Task NLL $\downarrow$
Clean	0.92–0.96	0.04–0.05	0.06	1.942
$f{=}0.05$ , unrestricted	0.00–0.10	0.78–0.94	0.00	1.441–1.532
$f{=}0.05$ , $[0,17]$ frozen	0.74–0.92	0.08–0.16	0.00	1.524–1.562
$f{=}0.05$ , matched	0.02–0.06	0.80–0.90	0.00	1.350–1.369
$f{=}0.15$ , unrestricted	0.00–0.02	0.82–0.98	0.00	1.457–1.494
$f{=}0.15$ , $[0,17]$ frozen	0.08–0.14	0.42–0.66	0.00	1.535–1.540

Appendix

Table 14: The freeze against a mixed adapter of

100

training examples of which a fraction

f

is harmful, Llama-3.1-8B, three seeds, ranges over seeds. Task is mean negative log-likelihood of held-out reference responses, lower being better.

Run	Layer	Full	Rank-1	Complement	Ratio
Llama-3.1-8B, all-7, 5 ep	$18/32$	$0.90$	$0.35$	$0.28$	$0.39$
	$26/32$	$0.95$	$0.03$	$0.68$	$0.03$
Llama-3.1-8B, attn, 3 ep	$22/32$	$0.93$	$0.00$	$0.10$	$0.00$
	$26/32$	$0.95$	$0.00$	$0.07$	$0.00$
Yi-1.5-9B	$40/48$	$0.85$	$0.17$	$0.03$	$0.21$
OLMo-2-13B	$17/40$	$0.93$	$0.82$	$0.05$	$0.89$

Appendix

Table 15: Coherent refusal after a full hidden-state transplant, its rank-one summary or the complementary component, at the listed layers. Ratio is rank-1 divided by full.

Model	Remove top-1	Remove top-2	Remove top-4	Random-2
Llama-3.1-8B	0.04–0.98	0.76–1.00	0.94–0.98	0.00–0.14
Tulu-3-8B-DPO	0.94–1.00	1.00	1.00	0.00–0.06
OLMo-2-7B	0.98–1.00	1.00	1.00	0.00–0.26
Qwen2.5-14B	0.88–1.00	1.00	1.00	0.00–0.26

Appendix

Table 16: Coherent refusal after removing the top singular directions of the update, under attention-only adaptation at three epochs. Ranges cover three seeds and five doses, with

50

seed-specific AdvBench prompts, and only cells where the attack landed. Random-2 removes two randomly chosen directions.

Epochs	1	2	3	5	8
Mean $\\|\Delta W\\|_{F}$	0.202	0.323	0.465	0.614	0.697
Participation ratio	6.19	5.26	4.70	4.61	4.58
Top-2 energy fraction	0.504	0.548	0.590	0.601	0.605
Refusal after removing top-2	0.85	0.79	0.71	0.41	0.34

Appendix

Table 17: Update statistics and repair by training length on Llama-3.1-8B at dose

100

. All seven projections are adapted; means cover three training seeds evaluated on the same

50

AdvBench prompts. Participation ratio is out of rank

16

Arm	Residual	Std, all-7, 5 ep	Std, attn, 3 ep	Spread, $\lambda{=}1$	Projected, $k{=}2$
No repair ( $\tau{=}0$ )	1.00	0.00	0.00	0.00	0.02–0.06
Projection, $\tau{=}0.5$	0.28–0.79	0.00–0.04	0.00	0.02–0.06	0.82–0.92
Projection, $\tau{=}0.7$	0.17–0.31	0.52–0.80	0.06–0.12	0.94–0.98	0.94–0.96
Projection, $\tau{=}1.0$	0.08–0.17	0.88–0.96	0.80–0.94	0.94–0.98	0.92–0.94
Scale-matched control	0.08–0.17	0.48–0.78	0.28–0.70	0.94–0.98	0.92–0.94
Random subspace	0.42–0.46	0.02–0.36	0.02–0.22	0.44–0.78	0.94–0.96

Appendix

Table 18: Safe LoRA-style projection on Llama-3.1-8B, spectral projector, three seeds except two for the projected attacker, ranges over seeds, coherent refusal on AdvBench. Residual is the fraction of the update’s norm the arm leaves.

$\lambda$	Detector score	$\\|\Delta W\\|_{F}$	Refusal after removal
$1.0$	0.972–0.974	0.228–0.229	0.00
$0.5$	0.974–0.976	0.231–0.233	0.00
$0.25$	0.977	0.247–0.249	0.00–0.03
$0.1$	0.983	0.306–0.308	0.00
$0.03$	0.971–0.972	0.419–0.422	0.03
$0.01$	0.926	0.483–0.486	0.02–0.05

Appendix

Table 19: Sweeping the concentration penalty on the spread attacker, Llama-3.1-8B at dose

100

, rank

16

, three seeds at the two ends of the sweep and two at the others. The detector score is participation ratio over rank.

$f$	Seed	$k^{*}$	Part. ratio	Top-2 energy	$f$	Seed	$k^{*}$	Part. ratio	Top-2 energy
$0.00$	42	1	$3.971$	$0.645$	$0.15$	44	2	$3.865$	$0.648$
$0.00$	43	1	$3.838$	$0.653$	$0.35$	42	4	$3.730$	$0.661$
$0.00$	44	1	$3.626$	$0.669$	$0.35$	43	2	$3.586$	$0.673$
$0.05$	42	1	$3.866$	$0.652$	$0.35$	44	2	$3.925$	$0.652$
$0.05$	43	1	$3.543$	$0.669$	$1.00$	42	2	$3.526$	$0.685$
$0.05$	44	2	$3.716$	$0.660$	$1.00$	43	2	$3.513$	$0.687$

Appendix

Table 20: Smallest repairing rank

k^{*}

, reaching at least

80\%

of clean refusal with positive negative-NLL task gain over clean, against the update’s spectrum. Llama-3.1-8B, attention-only, three epochs; each row is one fine-tune of

100

examples at harmful fraction

f

Figure 7: Transition depth

\ell^{*}

(solid) and

d_{75}

, the depth below which

75\%

\|\Delta W\|_{F}^{2}

sits (dashed), against attack dose. One panel per checkpoint, on a shared relative-depth axis.

Clean state taken from	L10	L14	L18	L23	L27
This prompt	0.00	0.15–0.75	0.90–0.95	0.95–0.97	0.93–0.95
Another harmful prompt	0.00	0.15–0.47	0.47–0.55	0.47–0.57	0.90–0.95
A benign prompt	0.00–0.03	0.05–0.10	0.00–0.07	0.00–0.03	0.00

Appendix

Table 21: Recovery under a matched source and two foreign ones, dose

100

, three seeds, ranges over seeds.

LLM Safety

Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.

See all 440 papers

SLDR: Defending Against Malicious Fine-tuning via Selective Layers Recovery and Dynamic Routing
Oct 7, 2026Hui Zhang, Yachao Yuan, Jiayun Wang +3LLM Security LLM Safety
North China Electric Power University · Southeast University · Soochow University +1
RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs
May 3, 2026Sadia Asif, Mohammad Mohammadi AmiriFine-Tuning LLM Fine-Tuning
Department of Computer Science Rensselaer Polytechnic Institute Troy, NY, USA
HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models
Jul 13, 2026Aznaur Aliev, Carlos Hinojosa, Abdelrahman Eldesokey +3LLM Fine-Tuning LLM Safety Alignment
King Abdullah University of Science and Technology, Saudi Arabia

Oct 7, 2026J/K move · Enter open · S save

SLDR: Defending Against Malicious Fine-tuning via Selective Layers Recovery and Dynamic Routing

Hui Zhang, Yachao Yuan, Jiayun Wang, Yuanzhuo Li+2

North China Electric Power University · Southeast University · Soochow University+1

LLM Security LLM Safety LLM Safety Alignment

Open paper AskPDF arXiv

Fine-tuning-as-a-service enables users to adapt aligned large language models (LLMs) to specialized tasks, but malicious fine-tuning can erode refusal behavior while preserving task performance on legitimate inputs. We revisit recent layer-wise safety diagnostics and find that safety sensitivity is signed: scaling different layers can strengthen refusal, weaken it, or have little effect. Motivated by this observation, we propose SLDR, a post-fine-tuning defense based on Selective Layers Recovery and Dynamic Routing. SLDR trains a LoRA recovery adapter only on the layers with the maximum and minimum sensitivity scores in the signed spectrum, and uses representation-based dynamic routing inference to activate the adapter only for malicious queries. Across four model architectures, five downstream tasks, and four harmful benchmarks, SLDR substantially reduces harmful outputs while preserving downstream utility. On Llama3.1/SST2, SLDR reduces the average harmful score from 11.54 to 0.08 while maintaining downstream accuracy, and the harmful score remains near zero under poisoning ratios up to 0.9. The code is available at https://github.com/Stardust457/SLDR.

Abstract

LLM Safety

Explore similar work

SLDR: Defending Against Malicious Fine-tuning via Selective Layers Recovery and Dynamic Routing

RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs

HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models