Preserving Safety During Sequential Fine-Tuning: A Factorial and Geometric Analysis of Replay

George Ruellan; Diana Benavides-Prado

cs.LGOct 27, 2026

Preserving Safety During Sequential Fine-Tuning: A Factorial and Geometric Analysis of Replay

Authors: George Ruellan, Diana Benavides-Prado

Organizations: School of Electronic Engineering and Computer Science, Queen Mary University of London, London, United Kingdom

Abstract

Benign fine-tuning can erode safety alignment in large language models, and sequential adaptation substantially compounds this risk. Prior work mitigates this erosion by replaying harmful-prompt refusals and benign-compliance examples, while other work instead matches base-model logits via distillation. However, the individual contributions of supervised exposure and distillation to safety behavior and refusal geometry during sequential adaptation remain unclear. We factor these two strategies apart and study their behavioral and geometric effects during sequential LoRA fine-tuning of Mistral-7B and Llama-3-8B across diverse task sequences (code generation, German-language summarization, and science question-answering). Behaviorally, supervised replay exposure yields the largest reduction in HarmBench attack success rate (ASR) on Mistral, averaging 51.1 points below base, but at a substantial over-refusal cost, averaging 53.4 points above base on OR-Bench-hard. Logit distillation, without any supervised replay loss, largely prevents safety erosion at a fraction of this cost: an average 5.4-point ASR reduction against only a 5.1-point rise in over-refusal. Llama shows the same asymmetry between the two strategies. Combining exposure and distillation moderates exposure's over-refusal while leaving downstream task scores largely unaffected. Geometrically, the two models diverge: on Mistral, distillation preserves the base refusal direction more closely than exposure does, whereas on Llama, refusal behavior after exposure largely survives ablation of the extracted refusal direction. This dissociation between models indicates that preserving the base refusal direction is not a reliable proxy for behavioral safety.

Figures & tables

Ordering	Task 1	Task 2	Task 3
O1	code	sum	qa
O2	qa	sum	code
O3	sum	code	qa
O4	code	qa	sum
O5	sum	qa	code
O6	qa	code	sum

TABLE I: Task sequences for ordering labels O1–O6. Code denotes Py150, sum denotes 20Minuten and qa denotes ScienceQA.

Fig. 1: HarmBench DirectRequest ASR trajectories under Plain ( $n{=}240$ ) for Mistral (left) and Llama (right), with orderings colored by first task and dashed $M_{0}$ baselines.

Model	Ordering	$M_{0}$	$M_{1}$	$M_{2}$	$M_{3}$	$\Delta$
Mistral-7B	O1	52.1	59.2	72.5	76.2	$+24.2$
	O2	52.1	65.4	72.9	57.9	$+5.8$
	O3	52.1	68.3	66.2	74.6	$+22.5$
	O4	52.1	59.2	75.0	73.8	$+21.7$
	O5	52.1	68.3	75.8	67.1	$+15.0$
	O6	52.1	65.4	62.5	62.5	$+10.4$

TABLE II: HarmBench ASR (%) under Plain across six task orderings and checkpoints

M_{0}

–

M_{3}

;

\Delta=M_{3}-M_{0}

Model	$M_{0}$	$M_{1}$	$M_{2}$	$M_{3}$	$\Delta$
Mistral-7B	52.1	72.1	72.1	69.6	$+17.5$
Llama-3-8B	5.8	34.2	37.9	28.7	$+22.9$

TABLE III: HarmBench ASR (%,

n{=}240

) over three Py150 shards at seed 42.

Fig. 2: HarmBench ASR versus OR-Bench-hard ORR over O1–O3, seed 42 (lower is better on both axes). Faint points show orderings, bold points their means and stars the base models.

Model	Method	HarmBench ASR	XSTest ORR	ORB-hard ORR
Mistral-7B	$M_{0}$	52.1	4.8	13.0
	Plain	69.6 $\pm$ 10.1	2.8 $\pm$ 0.7	3.9 $\pm$ 1.8
	Matched Plain	71.2 $\pm$ 10.5	2.5 $\pm$ 1.2	4.9 $\pm$ 1.9
	ER	1.0 $\pm$ 0.9	28.0 $\pm$ 2.8	66.4 $\pm$ 2.8
	DER	46.7 $\pm$ 3.8	7.9 $\pm$ 0.8	18.1 $\pm$ 2.3
	DER++	7.1 $\pm$ 1.1	17.3 $\pm$ 0.5	60.5 $\pm$ 3.5

TABLE IV: Final-

M_{3}

ASR and ORR (%), mean

\pm

sample s.d. over O1–O3 at seed 42.

Fig. 3: Final HarmBench ASR under Plain and DER++ across O1–O6 for Mistral (left) and Llama (right) ( $n{=}240$ , excluding copyright behaviors); dashed lines show $M_{0}$ .

Fig. 4: Downstream task performance at O3 $M_{3}$ (left) and mean forgetting over available orderings (right), with Mistral in the top row and Llama in the bottom row.

Model	Method	sum	code	qa	Forgetting
		(ROUGE-L)	(edit sim.)	(acc)
Mistral-7B	$M_{0}$	18.7	9.8	67.2	—
	Plain	21.4 $\pm$ 0.9	54.1 $\pm$ 1.2	87.0 $\pm$ 4.2	2.3 $\pm$ 1.9
	ER	22.7 $\pm$ 0.5	53.2 $\pm$ 0.6	89.1 $\pm$ 1.8	1.2 $\pm$ 0.7
	DER	22.1 $\pm$ 0.2	53.9 $\pm$ 0.6	90.8 $\pm$ 2.6	1.1 $\pm$ 1.3
	DER++	22.4 $\pm$ 0.6	53.6 $\pm$ 0.3	89.6 $\pm$ 1.3	1.0 $\pm$ 0.6

TABLE V: Final-checkpoint

M_{3}

task performance and forgetting, mean

\pm

s.d. over available orderings with complete evaluation chains. Lowest mean forgetting per model in bold.

Model	Ordering	HarmBench ASR	XSTest	ORB-hard
Mistral-7B	O1	6.2	16.8	58.2
	O2	8.3	17.6	58.8
	O3	6.7	17.6	64.6
	O4	6.2	17.6	56.0
	O5	8.3	17.6	63.8
	O6	8.8	22.4	67.8

TABLE VI: DER++ final-

M_{3}

HarmBench ASR and XSTest/ORB-hard ORR (%) by ordering, with mean

\pm

s.d. over O1–O6.

Model	Metric	$M_{0}$	Plain	DER	ER	DER++
Mistral-7B	HarmBench ASR	52.1	68.7 $\pm$ 7.4	46.7 $\pm$ 3.8	1.0 $\pm$ 0.9	7.4 $\pm$ 1.2
	AdvBench ASR	35.8	66.3 $\pm$ 5.6	30.3 $\pm$ 1.8	0.0 $\pm$ 0.0	0.7 $\pm$ 0.4
	JBB ASR	41.0	73.5 $\pm$ 8.4	39.0 $\pm$ 4.4	0.3 $\pm$ 0.6	1.0 $\pm$ 0.6
	ORB-toxic ASR	43.1	75.8 $\pm$ 2.8	37.4 $\pm$ 2.4	2.0 $\pm$ 0.5	8.3 $\pm$ 2.5
	XSTest ORR	4.8	2.7 $\pm$ 0.6	7.9 $\pm$ 0.8	28.0 $\pm$ 2.8	18.3 $\pm$ 2.1
	ORB-hard ORR	13.0	4.0 $\pm$ 1.6	18.1 $\pm$ 2.3	66.4 $\pm$ 2.8	61.5 $\pm$ 4.5

TABLE VII: Final-

M_{3}

ASR and ORR (%) at seed 42, mean

\pm

s.d. over O1–O6 for Plain/DER++ and O1–O3 for ER/DER.

Metric (Mistral-7B)	O2 70:30	O2 50:50	O3 70:30	O3 50:50
HarmBench ASR	8.3	13.8	6.7	7.9
XSTest ORR	17.6	16.0	17.6	16.8
ORB-hard ORR	58.8	48.0	64.6	50.0

TABLE VIII: Final-checkpoint

M_{3}

ASR and ORR (%) for the 70:30 and 50:50 buffers on Mistral O2/O3.

Model	Method	$\cos$	$\lVert\textbf{r}\rVert$	Refusal	$\Delta$ Refusal
Mistral-7B	$M_{0}$	1.000	2.89	0.72	0.42
	Plain	0.737	2.08	0.15	$0.15\rightarrow-0.16\rightarrow-0.29$
	ER	0.718	2.44	0.99	$0.16\rightarrow 0.24\rightarrow 0.19$
	DER	0.976	2.92	0.70	$0.33\rightarrow 0.28\rightarrow 0.32$
	DER++	0.960	2.85	0.94	$0.28\rightarrow 0.31\rightarrow 0.30$
Llama-3-8B	$M_{0}$	1.000	2.77	0.97	0.69

TABLE IX: Refusal geometry and ablation effects, means over O1–O3 at

M_{3}

, except the

\Delta

Refusal chain (

M_{1}{\rightarrow}M_{2}{\rightarrow}M_{3}

). Negative

\Delta

Refusal indicates increased refusal after ablation; no cells are bolded as these metrics have no preferred direction.

Fig. 5: Final-checkpoint cosine similarity to the base refusal direction versus HarmBench ASR (faint: orderings, bold: means, stars: $M_{0}$ ).

Fig. 6: Refusal geometry across O1–O3 for Mistral (left) and Llama (right), with magnitude normalized to each trajectory’s base and each arrow representing one task step.

Fig. 7: $M_{3}$ cosine similarity to the base refusal direction versus relative magnitude over O1–O3 (faint: orderings; bold: model–method means).

Model	Method	HarmBench ASR	XSTest ORR	ORB-hard ORR
Mistral-7B	$M_{0}$	52.1	4.8	13.0
	Plain	75.3 $\pm$ 1.3	2.9 $\pm$ 0.5	5.9 $\pm$ 1.3
	ER	1.1 $\pm$ 1.0	27.5 $\pm$ 2.0	67.7 $\pm$ 3.8
	DER	45.4 $\pm$ 1.8	7.1 $\pm$ 0.6	18.5 $\pm$ 2.0
	DER++	8.3 $\pm$ 2.1	17.9 $\pm$ 1.2	57.7 $\pm$ 3.8
Llama-3-8B	$M_{0}$	5.8	4.8	72.4

TABLE X: O1 final-

M_{3}

ASR and ORR (%), mean

\pm

s.d. over seeds 42–44. Lowest mean per measure and model in bold.

Model	Method	$\Delta$ Refusal
Mistral-7B	$M_{0}$	0.42
	Plain	$-$ 0.26 $\pm$ 0.36
	ER	0.20 $\pm$ 0.03
	DER	0.28 $\pm$ 0.03
	DER++	0.27 $\pm$ 0.04
Llama-3-8B	$M_{0}$	0.69

TABLE XI: O1 final-

M_{3}

\Delta

Refusal, mean

\pm

s.d. over seeds 42–44.

Method	HarmBench ASR	XSTest ORR	ORB-hard ORR
$M_{0}$	2.1	16.0	79.0
Plain	15.4	18.0	60.6
ER	1.2	26.8	91.6
DER	2.1	17.2	82.4
DER++	2.5	16.8	78.6

TABLE XII: Gemma O1 factorial ASR and ORR at

M_{3}

(%).

Setting	Harmful / total	ASR (%)
$M_{0}$	125/240	52.1
$\tau=0.1$	186/240	77.5
$\tau=0.3$	175/240	72.9
$\tau=0.5$	186/240	77.5
$\tau=1.0$	121/240	50.4

TABLE XIII: SafeLoRA threshold sensitivity on Mistral O1 at seed 42.

	$\beta=0$	$\beta=1$
$\alpha=0$	Plain	ER
	Lower base cosine	Lower base cosine
	ASR above base	ASR below base
$\alpha=1$	DER	DER++
	Higher base cosine	Higher base cosine
	ASR near base	ASR below base

TABLE XIV: Descriptive

(\alpha,\beta)

factorial summary;

\alpha

: distillation,

\beta

: replay exposure, cosine: similarity to the base refusal direction, ASR: HarmBench.

Measurement	Conditions	Checkpoints
Primary HarmBench	Plain/DER++ O1–O6; ER/DER O1–O3; seed 42	$M_{0}$ – $M_{3}$
Primary held-out ASR and ORR	Same method/ordering sets	$M_{0}$ , $M_{3}$
Matched Plain control	Primary models, O1–O3, seed 42	$M_{0}$ , $M_{3}$ ASR/ORR
Primary geometry/ablation	Four methods, O1–O3, seed 42	$M_{0}$ – $M_{3}$
Primary seed replication	Four methods, O1, seeds 42–44	ASR/geometry chains; held-out ASR/ORR endpoints
Gemma check	Plain/DER++ O1–O6; four methods at O1; seed 42	ASR chains and ORR endpoints; geometry O1–O3 for Plain/DER++ and O1 for ER/DER

TABLE XV: Reported evaluation coverage by measurement, training condition and model checkpoint.

Safety-Preserving Fine-Tuning of Language Models

See all 44 papers

RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs
May 3, 2026Sadia Asif, Mohammad Mohammadi AmiriFine-Tuning LLM Fine-Tuning
Department of Computer Science Rensselaer Polytechnic Institute Troy, NY, USA
When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning
Sep 1, 2026Yitong Guo, Xiaoyi Chen, Siyuan Zhang +2LLM Fine-Tuning LLM Safety Alignment
Indiana University Bloomington · Tsinghua University · Nanyang Technological University
Safety Offset Learning: When Benign Fine-Tuning Misaligns LLMs
Oct 19, 2026Jesson Wang, Zhanhao Hu, Minghao Lin +3LLM Fine-Tuning LLM Safety Alignment
University of Southern California · University of California, Berkeley · University of Washington

May 3, 2026J/K move · Enter open · S save

RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs

Sadia Asif, Mohammad Mohammadi Amiri

Department of Computer Science Rensselaer Polytechnic Institute Troy, NY, USA

Fine-Tuning LLM Fine-Tuning LLM Safety Alignment Adversarial Robustness of Language Models Safety-Preserving Fine-Tuning of Language Models

Open paper AskPDF arXiv

Fine-tuning safety-aligned language models for downstream tasks often leads to substantial degradation of refusal behavior, making models vulnerable to adversarial misuse. While prior work has shown that safety-relevant features are encoded in structured representations within the model's activation space, how these representations change during fine-tuning and why alignment degrades remains poorly understood. In this work, we investigate the representation-level mechanisms underlying alignment degradation. Our analysis shows that standard fine-tuning induces systematic drift in safety-relevant representations, distorts their geometric structure, and introduces interference between task optimization and safety features. These effects collectively lead to increased harmful compliance. Motivated by these findings, we introduce REFUSALGUARD, a representation-level fine-tuning framework that preserves safety-relevant structure during model adaptation. Our approach constrains updates in hidden representation space, ensuring that safety-mediating components remain stable while allowing task-specific learning in complementary directions. We evaluate REFUSALGUARD across multiple model families, including LLaMA, Gemma, and Qwen, on adversarial safety benchmarks such as AdvBench, DirectHarm4, and JailbreakBench, as well as downstream utility tasks. Our approach achieves attack success rates comparable to base safety-aligned models while maintaining competitive task performance, significantly outperforming baselines.

Abstract

Safety-Preserving Fine-Tuning of Language Models

Explore similar work

RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs

When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning

Safety Offset Learning: When Benign Fine-Tuning Misaligns LLMs