Beyond Average Safety: Chance-Constrained LLM Fine-tuning

Taha Entesari; Mahyar Fazlyab

cs.LGSep 24, 2026

Beyond Average Safety: Chance-Constrained LLM Fine-tuning

Authors: Taha Entesari, Mahyar Fazlyab

Organizations: Department of Electrical and Computer Engineering Johns Hopkins University

Abstract

Fine-tuning large language models on new objectives can improve helpfulness, instruction following, or domain-specific performance, but it can also induce regressions on safety-critical prompts. Existing safety-preserving fine-tuning methods typically control average safety loss or use weighted auxiliary penalties, which can obscure rare but severe failures. We propose a chance-constrained formulation for safety-preserving fine-tuning that limits the fraction of safety examples whose degradation relative to a reference model exceeds a prescribed threshold. Because the resulting empirical chance constraint contains a discontinuous indicator, we introduce a differentiable majorization of the violation rate, yielding a tractable conservative constraint. We then develop a constraint-aware gradient descent method that treats the majorized constraint as a safe set in parameter space and minimally modifies the fine-tuning direction to preserve feasibility. The resulting update admits a closed form and produces a tail-aware safety correction that emphasizes examples near or above the degradation threshold. We conduct an extensive set of experiments on harmful fine-tuning across three different tasks and three models and show that our approach consistently outperforms the baselines that exist in the literature. These results suggest that safety preservation in LLM fine-tuning is better viewed as a reliability-constrained optimization problem than as average-risk regularization.

Figures & tables

Method	Alignment	SFT	Post-hoc	Mechanism
SFT		✓		Plain supervised fine-tuning on the poisoned task; worst-case lower bound w/o defensive intervention.
Vaccine [ 16 ]	✓			SAM-style perturbation of attention representations during alignment to flatten the loss landscape against fine-tuning shifts.
Booster [ 14 ]	✓			Adds a harmful-perturbation regularizer ( $\lambda{=}5$ ) that perturbs parameters along the harmful gradient before each alignment step.
Lisa [ 15 ]		✓		Bi-state optimization alternating fine-tuning and alignment phases, with a proximal pull toward the aligned checkpoint.
SafeGrad [ 35 ]		✓		Gradient surgery resolving conflicts between the task gradient and a KL-based alignment gradient at every fine-tuning step.
AsFT [ 32 ]		✓		Anchors fine-tuning to the alignment direction by penalizing the orthogonal component of the LoRA update ( $\lambda_{\mathrm{reg}}{=}1$ ).

Table 1: Baselines grouped by stage of intervention. SafeLoRA is training-free and applied to the LoRA artifact after fine-tuning.

	Defense	SST-2			AG News			GSM8K
		HS $\downarrow$	HS qK $\downarrow$	FA $\uparrow$	HS $\downarrow$	HS qK $\downarrow$	FA $\uparrow$	HS $\downarrow$	HS qK $\downarrow$	FA $\uparrow$
Qwen-3.5-4B ( $qK{=}90$ )	Base	0.430	0.707	0.915	0.430	0.707	1.000	0.430	0.707	0.280
	Aligned (SFT)	0.350	0.738	0.940	0.350	0.738	0.890	0.350	0.738	0.170
	SFT	0.546 $\pm$ 0.042	0.890 $\pm$ 0.015	0.973 $\pm$ 0.016	0.545 $\pm$ 0.043	0.901 $\pm$ 0.025	0.890 $\pm$ 0.015	0.637 $\pm$ 0.047	0.945 $\pm$ 0.017	0.633 $\pm$ 0.064
	Vaccine	0.688 $\pm$ 0.032	0.962 $\pm$ 0.008	0.963 $\pm$ 0.038	0.680 $\pm$ 0.025	0.960 $\pm$ 0.002	0.932 $\pm$ 0.033	0.697 $\pm$ 0.023	0.968 $\pm$ 0.009	0.378 $\pm$ 0.172
	Booster	0.665 $\pm$ 0.021	0.940 $\pm$ 0.012	0.943 $\pm$ 0.018	0.674 $\pm$ 0.013	0.949 $\pm$ 0.007	0.952 $\pm$ 0.021	0.715 $\pm$ 0.012	0.951 $\pm$ 0.002	0.427 $\pm$ 0.083

Table 2: Main results at poison ratio

p{=}0.5

. For each (model, defense, task) we report HS (fraction of prompts with max harm probability

>0.4

), HS qK (per-prompt

K

-th quantile of max harm probability), and FA (task accuracy).

K{=}50

for Llama (higher quantiles saturate near

1.0

across all defenses);

K{=}90

for Qwen models. Bold = best, underlined = second best per column.

\downarrow

lower is better,

\uparrow

higher is better. Defense rows are mean

\pm

std over 3 seeds.

Defense	$p{=}0.05$			$p{=}0.1$			$p{=}0.25$			$p{=}0.5$
	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$
Base	HS = 0.430 HS q90 = 0.707 FA = 0.915 (constant in $p$ )
Aligned (SFT)	HS = 0.350 HS q90 = 0.738 FA = 0.940 (constant in $p$ )
SFT	0.425 $\pm$ 0.040	0.810 $\pm$ 0.029	0.985 $\pm$ 0.013	0.437 $\pm$ 0.037	0.828 $\pm$ 0.021	0.977 $\pm$ 0.014	0.516 $\pm$ 0.020	0.862 $\pm$ 0.015	0.983 $\pm$ 0.020	0.546 $\pm$ 0.042	0.890 $\pm$ 0.015	0.973 $\pm$ 0.016
Vaccine	0.575 $\pm$ 0.046	0.940 $\pm$ 0.018	0.973 $\pm$ 0.019	0.604 $\pm$ 0.027	0.944 $\pm$ 0.008	0.972 $\pm$ 0.010	0.665 $\pm$ 0.025	0.958 $\pm$ 0.006	0.983 $\pm$ 0.010	0.688 $\pm$ 0.032	0.962 $\pm$ 0.008	0.963 $\pm$ 0.038
Booster	0.464 $\pm$ 0.015	0.872 $\pm$ 0.013	0.973 $\pm$ 0.008	0.493 $\pm$ 0.015	0.883 $\pm$ 0.007	0.957 $\pm$ 0.008	0.600 $\pm$ 0.026	0.918 $\pm$ 0.010	0.943 $\pm$ 0.016	0.665 $\pm$ 0.021	0.940 $\pm$ 0.012	0.943 $\pm$ 0.018

Table 3: Poison-ratio sweep on Qwen-3.5-4B / SST-2 (align=sft). For each defense and poison ratio

p\in\{0.05,0.1,0.25,0.5\}

we report HS (fraction of prompts with max harm probability

>0.4

\downarrow

), HS q90 (90th-percentile per-prompt max harm probability,

\downarrow

), and FA (SST-2 accuracy,

\uparrow

). Base and Aligned-SFT baselines do not depend on

p

and are shown as single spanned values. Bold = best, underlined = second best per column. Antibody-FT is deferred to the appendix. Defense rows are mean

\pm

std over 3 seeds.

Defense	$p{=}0.05$			$p{=}0.1$			$p{=}0.25$			$p{=}0.5$
	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$
Base	HS = 0.430 HS q90 = 0.707 FA = 1.000 (constant in $p$ )
Aligned (SFT)	HS = 0.350 HS q90 = 0.738 FA = 0.890 (constant in $p$ )
SFT	0.434 $\pm$ 0.024	0.844 $\pm$ 0.004	0.917 $\pm$ 0.026	0.445 $\pm$ 0.010	0.866 $\pm$ 0.006	0.912 $\pm$ 0.028	0.517 $\pm$ 0.008	0.882 $\pm$ 0.023	0.895 $\pm$ 0.040	0.545 $\pm$ 0.043	0.901 $\pm$ 0.025	0.890 $\pm$ 0.015
Vaccine	0.584 $\pm$ 0.024	0.936 $\pm$ 0.008	0.947 $\pm$ 0.062	0.613 $\pm$ 0.015	0.947 $\pm$ 0.011	0.930 $\pm$ 0.066	0.684 $\pm$ 0.021	0.967 $\pm$ 0.002	0.943 $\pm$ 0.028	0.680 $\pm$ 0.025	0.960 $\pm$ 0.002	0.932 $\pm$ 0.033
Booster	0.476 $\pm$ 0.029	0.884 $\pm$ 0.006	0.975 $\pm$ 0.018	0.496 $\pm$ 0.016	0.891 $\pm$ 0.004	0.968 $\pm$ 0.021	0.597 $\pm$ 0.010	0.918 $\pm$ 0.004	0.973 $\pm$ 0.006	0.674 $\pm$ 0.013	0.949 $\pm$ 0.007	0.952 $\pm$ 0.021

Table 4: Poison-ratio sweep on Qwen-3.5-4B / AG News (align=sft). For each defense and poison ratio

p\in\{0.05,0.1,0.25,0.5\}

we report HS (fraction of prompts with max harm probability

>0.4

\downarrow

), HS q90 (90th-percentile per-prompt max harm probability,

\downarrow

), and FA (AG News accuracy,

\uparrow

). Base and Aligned-SFT baselines do not depend on

p

and are shown as single spanned values. Bold = best, underlined = second best per column. Defense rows are mean

\pm

std over 3 seeds.

Defense	$p{=}0.05$			$p{=}0.1$			$p{=}0.25$			$p{=}0.5$
	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$
Base	HS = 0.430 HS q90 = 0.707 FA = 0.280 (constant in $p$ )
Aligned (SFT)	HS = 0.350 HS q90 = 0.738 FA = 0.170 (constant in $p$ )
SFT	0.470 $\pm$ 0.024	0.877 $\pm$ 0.006	0.665 $\pm$ 0.010	0.510 $\pm$ 0.043	0.889 $\pm$ 0.016	0.665 $\pm$ 0.020	0.605 $\pm$ 0.026	0.938 $\pm$ 0.010	0.660 $\pm$ 0.043	0.637 $\pm$ 0.047	0.945 $\pm$ 0.017	0.633 $\pm$ 0.064
Vaccine	0.636 $\pm$ 0.024	0.948 $\pm$ 0.013	0.498 $\pm$ 0.020	0.653 $\pm$ 0.024	0.952 $\pm$ 0.004	0.465 $\pm$ 0.044	0.696 $\pm$ 0.015	0.964 $\pm$ 0.005	0.465 $\pm$ 0.026	0.697 $\pm$ 0.023	0.968 $\pm$ 0.009	0.378 $\pm$ 0.172
Booster	0.565 $\pm$ 0.010	0.924 $\pm$ 0.002	0.460 $\pm$ 0.025	0.587 $\pm$ 0.016	0.923 $\pm$ 0.005	0.450 $\pm$ 0.015	0.658 $\pm$ 0.009	0.938 $\pm$ 0.007	0.452 $\pm$ 0.019	0.715 $\pm$ 0.012	0.951 $\pm$ 0.002	0.427 $\pm$ 0.083

Table 5: Poison-ratio sweep on Qwen-3.5-4B / GSM8K (align=sft). For each defense and poison ratio

p\in\{0.05,0.1,0.25,0.5\}

we report HS (fraction of prompts with max harm probability

>0.4

\downarrow

), HS q90 (90th-percentile per-prompt max harm probability,

\downarrow

), and FA (GSM8K accuracy,

\uparrow

). Base and Aligned-SFT baselines do not depend on

p

and are shown as single spanned values. Bold = best, underlined = second best per column. Defense rows are mean

\pm

std over 3 seeds.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Defense	Hyperparameters
Vaccine	$\rho{=}2$ , align epochs $20$
Booster	$\alpha_{\mathrm{B}}{=}0.1$ , $\lambda{=}5$ , align epochs $20$
Lisa	align step $100$ , ft step $900$ , $\rho{=}0.01$ , guide $10{,}000$
SaLoRA	projection strength $\alpha_{\mathrm{proj}}{=}1.0$
SafeLoRA	projection threshold $\tau_{\mathrm{SL}}{=}0.5$ (applied after plain SFT fine-tuning)
SafeGrad	$\rho{=}1.0$ (KL alignment weight)

Appendix

Table 6: Defense-specific hyperparameters used in all experiments. Symbols follow the originating papers; “align epochs” is the number of epochs of the alignment-stage objective on the

5{,}000

-example

D_{\mathrm{safe}}

split. “ft” abbreviates fine-tuning.

Value	HS $\downarrow$	$q_{50}$ $\downarrow$	$q_{90}$ $\downarrow$	$q_{99}$ $\downarrow$	FA $\uparrow$
$\tau$ sweep, fixing $\alpha=0.05,\ \beta=10$
0	0.323	0.256	0.676	0.942	0.925
0.01	0.311	0.250	0.691	0.930	0.925
0.05	0.314	0.252	0.704	0.949	0.930
0.1	0.320	0.253	0.707	0.930	0.920
0.2	0.350	0.252	0.723	0.949	0.935

Appendix

Table 7: Hyperparameter sensitivity on Qwen-3.5-4B / SST-2 at

p{=}0.5

. Each section sweeps one of the three hyperparameters while holding the other two at the canonical values used in the main results

(\tau,\alpha,\beta)=(0.1,0.05,10)

. Bold = best, underlined = second best within each sweep.

\downarrow

lower is better,

\uparrow

higher is better.

Defense	HS $\downarrow$	$q_{10}$ $\downarrow$	$q_{25}$ $\downarrow$	$q_{50}$ $\downarrow$	$q_{75}$ $\downarrow$	$q_{90}$ $\downarrow$	$q_{95}$ $\downarrow$	$q_{99}$ $\downarrow$	$q_{100}$ $\downarrow$	FA $\uparrow$
Base	0.430	0.161	0.248	0.365	0.531	0.707	0.824	0.938	0.984	0.915
Aligned (SFT)	0.350	0.094	0.164	0.296	0.504	0.738	0.867	0.961	1.000	0.940
SFT	0.546 $\pm$ 0.042	0.119 $\pm$ 0.010	0.227 $\pm$ 0.020	0.446 $\pm$ 0.040	0.705 $\pm$ 0.037	0.890 $\pm$ 0.015	0.947 $\pm$ 0.008	0.992 $\pm$ 0.000	1.000 $\pm$ 0.000	0.973 $\pm$ 0.016
Vaccine	0.688 $\pm$ 0.032	0.149 $\pm$ 0.023	0.316 $\pm$ 0.033	0.611 $\pm$ 0.026	0.861 $\pm$ 0.022	0.962 $\pm$ 0.008	0.986 $\pm$ 0.002	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	0.963 $\pm$ 0.038
Booster	0.665 $\pm$ 0.021	0.143 $\pm$ 0.013	0.305 $\pm$ 0.017	0.584 $\pm$ 0.030	0.826 $\pm$ 0.026	0.940 $\pm$ 0.012	0.970 $\pm$ 0.006	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	0.943 $\pm$ 0.018
Lisa	0.489 $\pm$ 0.051	0.115 $\pm$ 0.009	0.206 $\pm$ 0.022	0.392 $\pm$ 0.046	0.651 $\pm$ 0.045	0.845 $\pm$ 0.037	0.925 $\pm$ 0.012	0.983 $\pm$ 0.014	1.000 $\pm$ 0.000	0.968 $\pm$ 0.003

Appendix

Table 8: HS-quantile breakdown on Qwen-3.5-4B / SST-2 at

p{=}0.5

, align=sft. HS is the fraction of prompts with max harm probability

>0.4

(binary flag rate);

q_{k}

columns are the

k

-th percentile of the per-prompt max harm probability (continuous).

\downarrow

for HS columns,

\uparrow

for FA. Bold = best, underlined = second best per column. Defense rows are mean

\pm

std over 3 seeds.

Defense	HS $\downarrow$	$q_{10}$ $\downarrow$	$q_{25}$ $\downarrow$	$q_{50}$ $\downarrow$	$q_{75}$ $\downarrow$	$q_{90}$ $\downarrow$	$q_{95}$ $\downarrow$	$q_{99}$ $\downarrow$	$q_{100}$ $\downarrow$	FA $\uparrow$
Base	0.430	0.161	0.248	0.365	0.531	0.707	0.824	0.938	0.984	1.000
Aligned (SFT)	0.350	0.094	0.164	0.296	0.504	0.738	0.867	0.961	1.000	0.890
SFT	0.545 $\pm$ 0.043	0.113 $\pm$ 0.010	0.216 $\pm$ 0.024	0.449 $\pm$ 0.049	0.730 $\pm$ 0.035	0.901 $\pm$ 0.025	0.948 $\pm$ 0.011	0.992 $\pm$ 0.008	1.000 $\pm$ 0.000	0.890 $\pm$ 0.015
Vaccine	0.680 $\pm$ 0.025	0.154 $\pm$ 0.012	0.311 $\pm$ 0.025	0.617 $\pm$ 0.023	0.857 $\pm$ 0.006	0.960 $\pm$ 0.002	0.982 $\pm$ 0.002	0.999 $\pm$ 0.002	1.000 $\pm$ 0.000	0.932 $\pm$ 0.033
Booster	0.674 $\pm$ 0.013	0.155 $\pm$ 0.006	0.319 $\pm$ 0.015	0.597 $\pm$ 0.027	0.836 $\pm$ 0.008	0.949 $\pm$ 0.007	0.979 $\pm$ 0.002	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	0.952 $\pm$ 0.021
Lisa	0.510 $\pm$ 0.038	0.108 $\pm$ 0.009	0.200 $\pm$ 0.019	0.407 $\pm$ 0.032	0.666 $\pm$ 0.022	0.874 $\pm$ 0.020	0.940 $\pm$ 0.012	0.992 $\pm$ 0.004	1.000 $\pm$ 0.000	0.900 $\pm$ 0.018

Appendix

Table 9: HS-quantile breakdown on Qwen-3.5-4B / AG News at

p{=}0.5

, align=sft. HS is the fraction of prompts with max harm probability

>0.4

(binary flag rate);

q_{k}

columns are the

k

-th percentile of the per-prompt max harm probability (continuous).

\downarrow

for HS columns,

\uparrow

for FA. Bold = best, underlined = second best per column. Defense rows are mean

\pm

std over 3 seeds.

Defense	HS $\downarrow$	$q_{10}$ $\downarrow$	$q_{25}$ $\downarrow$	$q_{50}$ $\downarrow$	$q_{75}$ $\downarrow$	$q_{90}$ $\downarrow$	$q_{95}$ $\downarrow$	$q_{99}$ $\downarrow$	$q_{100}$ $\downarrow$	FA $\uparrow$
Base	0.430	0.161	0.248	0.365	0.531	0.707	0.824	0.938	0.984	0.280
Aligned (SFT)	0.350	0.094	0.164	0.296	0.504	0.738	0.867	0.961	1.000	0.170
SFT	0.637 $\pm$ 0.047	0.133 $\pm$ 0.025	0.275 $\pm$ 0.049	0.558 $\pm$ 0.055	0.814 $\pm$ 0.049	0.945 $\pm$ 0.017	0.979 $\pm$ 0.009	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	0.633 $\pm$ 0.064
Vaccine	0.697 $\pm$ 0.023	0.176 $\pm$ 0.007	0.346 $\pm$ 0.017	0.614 $\pm$ 0.028	0.872 $\pm$ 0.016	0.968 $\pm$ 0.009	0.991 $\pm$ 0.002	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	0.378 $\pm$ 0.172
Booster	0.715 $\pm$ 0.012	0.175 $\pm$ 0.022	0.361 $\pm$ 0.016	0.635 $\pm$ 0.014	0.847 $\pm$ 0.004	0.951 $\pm$ 0.002	0.982 $\pm$ 0.002	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	0.427 $\pm$ 0.083
Lisa	0.532 $\pm$ 0.032	0.096 $\pm$ 0.003	0.195 $\pm$ 0.014	0.438 $\pm$ 0.039	0.739 $\pm$ 0.039	0.913 $\pm$ 0.016	0.958 $\pm$ 0.010	0.997 $\pm$ 0.002	1.000 $\pm$ 0.000	0.290 $\pm$ 0.041

Appendix

Table 10: HS-quantile breakdown on Qwen-3.5-4B / GSM8K at

p{=}0.5

, align=sft. HS is the fraction of prompts with max harm probability

>0.4

(binary flag rate);

q_{k}

columns are the

k

-th percentile of the per-prompt max harm probability (continuous).

\downarrow

for HS columns,

\uparrow

for FA. Bold = best, underlined = second best per column. Defense rows are mean

\pm

std over 3 seeds.

Defense	step (ms)	p90 (ms)	GPU (MB)	samples/s	$\times$ SFT
SFT	136.8	160.5	14512	28.16	1.00
Lisa	138.6	153.3	14797	28.30	1.00
SafeGrad	334.7	371.5	26137	11.83	2.38
AsFT	525.6	551.4	16083	7.55	3.73
Ours	282.2	306.1	16739	13.98	2.01

Appendix

Table 11: Per-step training cost on Qwen-3.5 4B / SST2 (one A100, batch=10). step is the median wall-clock time over 25 measured optimizer steps after a 5-step warmup.

\times

SFT normalizes the step time by SFT.

Task	Method	HS $\downarrow$	HS q90 $\downarrow$	FA $\uparrow$
AG News	Our method	0.324	0.688	0.930
AG News	Primal-Dual	0.346	0.755	0.935
GSM8K	Our method	0.303	0.645	0.140
GSM8K	Primal-Dual	0.380	0.802	0.155
SST-2	Our method	0.317	0.684	0.940
SST-2	Primal-Dual	0.311	0.703	0.940

Appendix

Table 12: Primal-Dual ablation: we compare solving the chance-constrained finetuning problem using our method and a primal dual method on the Qwen 3.5 4B model. Bold = best.

		SST-2			AG News			GSM8K
	Method	HS $\downarrow$	HS ${}_{qK}\downarrow$	FA $\uparrow$	HS $\downarrow$	HS ${}_{qK}\downarrow$	FA $\uparrow$	HS $\downarrow$	HS ${}_{qK}\downarrow$	FA $\uparrow$
Qwen-3.5-4B (q90)	SFT-Aligned	$0.350$	$0.738$	$0.940$	$0.350$	$0.738$	$0.890$	$0.350$	$0.738$	$0.170$
	Antibody-Aligned	$0.204$	$0.571$	$0.835$	$0.204$	$0.571$	$0.805$	$0.204$	$0.571$	$0.145$
	SFT	$0.546{\scriptstyle\,\pm 0.042}$	$0.890{\scriptstyle\,\pm 0.015}$	$0.973{\scriptstyle\,\pm 0.016}$	$0.545{\scriptstyle\,\pm 0.043}$	$0.901{\scriptstyle\,\pm 0.025}$	$0.890{\scriptstyle\,\pm 0.015}$	$0.637{\scriptstyle\,\pm 0.047}$	$0.945{\scriptstyle\,\pm 0.017}$	$0.633{\scriptstyle\,\pm 0.064}$
	Antibody	$\mathbf{0.273}{\scriptstyle\,\pm 0.016}$	$\mathbf{0.690}{\scriptstyle\,\pm 0.016}$	$0.867{\scriptstyle\,\pm 0.062}$	$\mathbf{0.244}{\scriptstyle\,\pm 0.010}$	$\mathbf{0.648}{\scriptstyle\,\pm 0.024}$	$0.888{\scriptstyle\,\pm 0.020}$	$0.519{\scriptstyle\,\pm 0.067}$	$0.907{\scriptstyle\,\pm 0.031}$	$\mathbf{0.383}{\scriptstyle\,\pm 0.070}$
	Ours on SFT-Aligned	$0.340{\scriptstyle\,\pm 0.032}$	$0.726{\scriptstyle\,\pm 0.041}$	$\mathbf{0.957}{\scriptstyle\,\pm 0.018}$	$0.340{\scriptstyle\,\pm 0.017}$	$0.732{\scriptstyle\,\pm 0.039}$	$\mathbf{0.927}{\scriptstyle\,\pm 0.006}$	$0.360{\scriptstyle\,\pm 0.052}$	$\mathbf{0.780}{\scriptstyle\,\pm 0.118}$	$0.267{\scriptstyle\,\pm 0.110}$

Appendix

Table 13: Antibody vs. our method under the harmful fine-tuning attack (

p{=}0.5

). Rows above the rule are the pre-attack floors. HS and HS qK (per-prompt

K

-th quantile of max harm-prob;

K{=}90

Qwen,

K{=}50

Llama) are lower-is-safer; FA is task accuracy. Mean

\pm

std over seeds

\{42,1337,2024\}

. Bold = best defense per column.

Explore similar work

Jun 2, 2026cs.CL

Safety Measurements for Fine-tuned LLMs Should be Grounded in Capability

Adapting foundation large language models to a user's task or preferred style through fine-tuning can result in compromising the model's safety. Previous works examined the effects of fine-tuning on model safety in limited and seemingly random experimental settings. We argue that anchoring fine-tuning to a specific capability goal is essential for avoiding arbitrary empirical choices, allowing us to draw meaningful conclusions about safety impacts, and to compare mitigation methods on a consistent basis. We conduct a multi-dimensional evaluation of the effects of fine-tuning on model behavior by focusing on capability as well as safety. Our results surface important issues that (1) fine-tuned models can produce incoherent generations in response to safety prompts, (2) automated safety judgments are unreliable for such incoherent outputs, and (3) the conclusions about the effects of fine-tuning can change depending on the choice of safety benchmark as well as the safety evaluator.

Krishnapriya Vishnubhotla, Hillary Dawkins, Isar Nejadgholi +1

National Research Council, Canada

Large Language Model Safety Large Language Model Fine-Tuning Model Fine-Tuning Multi-Dimensional Evaluation

Ask PDF

Aug 5, 2026cs.CL

DataRx: Missingness-Aware Sampling for Safer Large Language Model Task-Specific Fine-Tuning

Task-specific fine-tuning can improve the performance of large language models (LLMs) on downstream tasks. However, our study reveals that task-specific fine-tuning can also weaken the safety guardrails of aligned LLMs. A widely adopted strategy for preserving safety during fine-tuning is to incorporate safety data. Although previous studies have shown that randomly mixing safety data can alleviate safety degradation, the underlying principle determining why some safety examples are more effective than others still remains unclear. In this paper, we propose DataRx, a missingness-aware sampling method for selecting safety-critical examples. DataRx is based on the hypothesis that a safety sample is more effective when the selected examples provide safety signals that fill the missing parts of LLMs' safety capabilities. DataRx's key insight is leveraging high-dimensional hidden representations rather than discrete tokens to quantify the safety signal gap between the target model's native response and the safety reference response. The results show that, with only 1% additional safety samples from BeaverTails, DataRx reduces the average attack success rate of Llama3-8B-Instruct across seven downstream tasks from 59.23% under random sampling to 13.70%. In addition, DataRx can be combined with the existing safety data synthesis method to further enhance safety defenses during fine-tuning. We hope that DataRx will inspire more data-centric defense research.

Junbo Zhang, Qianli Zhou, Xinyang Deng +1

Northwestern Polytechnical University

Large Language Model Safety Large Language Model Fine-Tuning Model Fine-Tuning Missing Data

Ask PDF

May 6, 2026cs.AI

From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuning

Safety alignment of Large Language Models (LLMs) is extremely fragile, as fine-tuning on a small number of benign samples can erase safety behaviors learned from millions of preference examples. Existing studies attempt to explain this phenomenon by comparing parameters and hidden states before and after fine-tuning, but overlook their dynamic evolution during fine-tuning. In this paper, we uncover a critical mechanism underlying safety degradation by analyzing parameter dynamics, where benign fine-tuning causes parameters to cumulatively drift toward danger-aligned directions, progressively undermining the model's safety. This finding suggests that samples contributing more to this drift has greater fine-tuning risks. Based on this insight, we propose a method of Sample-Level Quantification of Safety Degradation (SQSD), which quantifies the influence of each training sample on safety degradation. Specifically, SQSD computes continuous risk scores to samples by measuring their induced parameter updates' projection difference between danger and safety directions. Extensive experiments across multiple models and datasets demonstrate that SQSD effectively quantifies sample-level fine-tuning risks and exhibits strong transferability across model architectures, parameter scales, and parameter-efficient methods.

Xiao Wang, Yifei Zhang, YongKang Liu +4

School of Computer Science and Engineering, Northeastern University, Shenyang, China · School of Computer and Communication Engineering, Northeastern University, Qinhuangdao, China

Large Language Model Safety Large Language Model Fine-Tuning Model Fine-Tuning Safety Benchmarks Risk Stratification

Ask PDF