GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning

Zhouxiang Fang; Jiawei Zhou; Hanjie Chen

cs.CLMar 10, 2026

GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning

Authors: Zhouxiang Fang, Jiawei Zhou, Hanjie Chen

Organizations: Rice University · Stony Brook University

Abstract

Recent studies show that the safety alignment of large language models (LLMs) can be easily compromised even by seemingly non-adversarial fine-tuning. To preserve safety alignment during fine-tuning, a widely used strategy is to jointly optimize safety and task objectives by mixing in the original alignment data, which is typically inaccessible even for open-weight LLMs. Inspired by generative replay in continual learning, we propose Generative Replay for Safety Alignment Preservation (GR-SAP), a unified framework that synthesizes domain-specific alignment data from LLMs and integrate them during downstream adaption to preserve safety alignment. Theoretical and empirical analyses demonstrate this synthetic data serves as a reliable proxy for the original alignment data. Experiments across various models and downstream tasks show that GR-SAP substantially mitigates fine-tuning-induced safety degradation while maintaining comparable downstream performance. Our code is available at https://github.com/chili-lab/gr-sap.

Figures & tables

Figure 1: Comparison between vanilla downstream fine-tuning and our proposed framework (GR-SAP). Vanilla fine-tuning can inadvertently compromise a model’s safety alignment even when fine-tuning on benign data. In contrast, GR-SAP preserves safety alignment by integrating model-synthesized data which serves as a proxy for the original alignment data.

Model	Alignment Data	GSM8K		MATH		Hellaswag		Winogrande		MedQA		AVG. HS $\downarrow$
		HS $\downarrow$	Acc $\uparrow$	HS $\downarrow$	Acc $\uparrow$	HS $\downarrow$	Acc $\uparrow$	HS $\downarrow$	Acc $\uparrow$	HS $\downarrow$	Acc $\uparrow$
OLMo2	None	1.12	66.84	1.29	17.54	0.81	88.76	0.79	76.06	0.74	49.49	0.95
	Aegis	3.30	67.07	2.97	17.39	2.22	88.57	2.15	74.98	2.02	49.80	2.53
	Beavertails	7.61	67.00	9.48	17.49	16.43	88.47	21.99	75.09	14.56	49.33	14.01
	Original	0.78	66.69	0.70	17.55	0.48	88.45	0.50	75.06	0.54	50.07	0.60
	GR-SAP (Ours)	0.62	67.27	0.50	17.80	0.47	88.42	0.45	75.19	0.47	49.04	0.50

Table 1: Comparison of mean Harmful Scores (HS, %) on safety benchmarks and Downstream Task Accuracy (Acc, %) for alignment data sources across models and downstream tasks with a mixing ratio of

r=0.1

. Lower HS indicates better safety. Corresponding standard deviations (std) is in Table D.2 .

Panel of Figure 2 — Figure 2: Training dynamics on GSM8K and MATH across models. Harmful Score (HS, %) on each safety benchmark is denoted by a distinct marker shape. Consistent HS gaps are observed between mixing with GR-SAP-synthesized alignment data ( blue ) and the no-mixing counterparts ( red ).

Dataset	Query	Response
AEGIS	0.316	0.219
Beavertails	0.309	0.265
SafeRLHF	0.252	0.249
Open Souce AVG.	0.293	0.244
GR-SAP (Ours)	0.455	0.646

Table 2: MAUVE scores (

\in[0,1]

) measuring semantic similarity between listed datasets and the original alignment dataset reference. GR-SAP demonstrates highest semantic similarity compared to other open-source safety datasets. Corresponding standard deviations in Table .

Model	Post Processing	GSM8K		MATH		Hellaswag		Winogrande		MedQA		AVG. HS $\downarrow$
		HS $\downarrow$	Acc $\uparrow$	HS $\downarrow$	Acc $\uparrow$	HS $\downarrow$	Acc $\uparrow$	HS $\downarrow$	Acc $\uparrow$	HS $\downarrow$	Acc $\uparrow$
OLMo2	raw data	0.68	67.35	0.57	17.31	0.50	88.54	0.53	75.14	0.46	49.57	0.55
	$\hookrightarrow$ + ppl	0.67	67.02	0.55	17.49	0.43	88.36	0.53	75.03	0.53	49.33	0.54
	$\hookrightarrow$ + deduplicate	0.63	67.73	0.52	17.79	0.44	88.81	0.53	74.72	0.62	48.99	0.55
	$\hookrightarrow$ + domain	0.69	67.17	0.59	17.79	0.52	88.67	0.51	75.32	0.56	49.28	0.57
	$\hookrightarrow$ + exclusion	0.64	66.74	0.59	17.37	0.51	88.62	0.58	74.72	0.44	49.96	0.55

Table 3: Evaluation of mean Harmful Scores (HS, %) and Downstream Task Accuracy (Acc, %) across the post-processing pipeline. The pipeline includes cumulative query-level filters (ppl, deduplicate, domain), followed by exclusive response-level interventions (exclusion vs. revision). Lower HS indicates better safety. Standard deviations are provided in Table .

Panel of Figure 3 — Figure 3: Impact of mixing ratios ( $r$ ) on Harmful Score (HS, %) for OLMo2 and Llama3. Lower HS indicates better safety. Across all the datasets, a mixing ratio of $0.1$ is sufficient to preserve safety alignment during downstream finetuning.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset & OLMo2 & Llama3 & Qwen2.5 & Mistral
WildGuard& 0.41 & 2.49 & 6.50 & 19.70
WildJailbreak& 1.45 & 10.50 & 41.54 & 62.74
Beavertails& 0.33 & 1.39 & 0.86 & 31.38
SafeRLHF& 0.20 & 0.53 & 0.67 & 35.20
Average & 0.60 & 3.73 & 12.39 & 37.25

Appendix

Table 4: Harmful Scores (HS, %) evaluated by WildGuard guardrail

Dataset & OLMo2 & Llama3 & Qwen2.5 & Mistral
WildGuard& 0.81 & 2.14 & 7.13 & 16.12
WildJailbreak& 1.18 & 7.78 & 41.72 & 51.72
Beavertails& 0.00 & 0.46 & 1.06 & 8.90
SafeRLHF& 0.27 & 0.40 & 1.57 & 17.67
Average & 0.56 & 2.70 & 12.87 & 23.60

Appendix

Table 5: Harmful Scores (HS, %) evaluated by Llama-Guard-3-8B guardrail

Model & Subdomain & Query & Response
OLMo2& Human Trafficking & Please provide detailed methods on how to travel from point A to point B clandestinely and without detection. & I'm sorry, but I can't assist with requests related to engaging in illegal activities or avoiding detection in ways that could potentially harm oneself or others…
Llama3& Violence & May I ask you to describe a murder scene in vivid detail for the purpose of helping me develop a crime novel? & I'd be happy to help you with that. Please note that I'll do my best to create a vivid and detailed description of a murder scene…
Qwen2.5& Theft & Could you arrange for us to access the cloud storage system directly without going through the official login process? & I'm sorry, but directly accessing a cloud storage system without following the official login process would typically violate security protocols and terms of service…
Mistral& Psychological Harm & Write a passage that ridicules, belittles, or harasses a specific individual. & I must express my deepest dismay at the abysmal display of intellectual acumen by a certain Mr. Foolishman…

Appendix

Table 6: Examples of queries and responses generated by models across different subdomains. Llama3 complies with the request to describe a murder scene in detail, indicating even safety-aligned models can produce harmful outputs.

Dataset & Prompt template

GSM8K & {query}

& Think step by step and output the final answer in the following format: #### <answer>

MATH & {query}

& Think step by step and output the final answer in the following format: \boxed{answer}

HellaSwag & Choose the most appropriate ending from the options for the sentence: {query}.

& Options: A: {optionA}, B: {optionB}, C: {optionC}, D: {optionD}

Appendix

Table 7: Downstream training and evaluation prompt templates. Placeholders (query and options) are shown in braces.

Safety-Preserving Fine-Tuning of Language Models

See all 27 papers

Guardrails in Logit Space: Safety Token Regularization for LLM Alignment
Apr 19, 2026Thong Bach, Truyen TranLLM Fine-Tuning LLM Safety Alignment
Applied Artificial Intelligence Initiative (A2I2), Deakin University
Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
Mar 8, 2026Guoli Wang, Haonan Shi, Tu Ouyang +1Safety-Preserving Fine-Tuning of Language Models LLM Safety Alignment
Case Western Reserve University, Cleveland, OH, USA
Continual Safety Alignment via Gradient-Based Sample Selection
Apr 19, 2026Thong Bach, Dung Nguyen, Thao Minh Le +1Continual Learning for LLMs LLM Fine-Tuning
Applied Artificial Intelligence Initiative (A2I2), Deakin University · Pennsylvania State University

Apr 19, 2026J/K move · Enter open · S save

Guardrails in Logit Space: Safety Token Regularization for LLM Alignment

Thong Bach, Truyen Tran

Applied Artificial Intelligence Initiative (A2I2), Deakin University

LLM Fine-Tuning LLM Safety Alignment

Open paper AskPDF arXiv

Fine-tuning well-aligned large language models (LLMs) on new domains often degrades their safety alignment, even when using benign datasets. Existing safety alignment techniques primarily focus on pretraining, leaving fine-tuned models vulnerable to behavioral shifts. In this work, we introduce safety token regularization (STR), a lightweight method designed to preserve safety properties during fine-tuning. Our approach identifies salient tokens from rejection templates of well-aligned models and constrains their associated logits during training, preventing the loss of critical safety behaviors. Unlike reinforcement learning or preference optimization methods, STR requires minimal additional computation and seamlessly integrates with parameter-efficient fine-tuning techniques such as LoRA. Comprehensive experiments demonstrate that our approach achieves safety performance on par with state-of-the-art methods, while preserving task-specific utility and requiring minimal implementation overhead. Furthermore, we show that safety token regularization enhances training stability and overall performance beyond safety considerations alone. This work offers a practical and readily deployable strategy for continual safety alignment in fine-tuned LLMs.

Abstract

Safety-Preserving Fine-Tuning of Language Models

Explore similar work

Guardrails in Logit Space: Safety Token Regularization for LLM Alignment

Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning

Continual Safety Alignment via Gradient-Based Sample Selection