Fine-tuning Large Language Models (LLMs) on instruction data can quietly erode their safety, even when no training example looks harmful. Because this safety-degrading data looks benign, standard content filters struggle to catch it. We tackle this problem by developing SimpleLoRIF, a scalable gradient embedding method that identifies fine-tuning examples whose gradients align with unsafe behavior. By simplifying a low-rank attribution method and restricting backpropagation to the final four transformer blocks, SimpleLoRIF can embed datasets on the order of a million examples, boosting gradient attribution from a small-scale analysis tool into a practical filter for full post-training datasets. We first validate it on emergent misalignment, where we find that gradients can connect insecure-code training data to the broadly harmful behaviors that appear after fine-tuning, with AUC as high as 0.865 in the gradient space of misaligned models. We then use SimpleLoRIF to build SafeTulu, a version of Tulu 3 with the 20% of examples most gradient-aligned with unsafe responses removed. Training on SafeTulu reduces GCG attack success rate by up to 22% across Qwen, Llama, and Olmo, outperforming both LLM-as-a-judge and prior gradient-based filtering, while largely preserving or even improving capability. Finally, we analyzed the removed data and identify possibly under-studied sources of safety degradation, such as tool-execution failures and narrative arithmetic. We release SafeTulu along with a Hugging Face Space that lets researchers build their own customized versions of SafeTulu to fine-tune on, such as by changing the removal fraction or filtering only certain subsets.
Figures & tables
Figure 1: Detection AUC against embedding wall-clock time for each optimization strategy. Restricting backpropagation to the last four transformer blocks gives the best balance of embedding quality and speed.
Figure 2: Probability that a randomly selected insecure-code example receives a higher gradient-similarity score than a randomly selected secure-code example (ROC-AUC), before and after insecure-code fine-tuning. This measures score ordering, not the frequency of harmful responses.
Figure 3: SafeTulu construction: score Tulu 3 examples against unsafe SafeRLHF responses, remove the 20% with the highest gradient similarity, and train and evaluate on the retained 80%.
Figure 4: Evaluation results after baseline training, training on randomly filtered data, training on the LLM-as-a-judge filtered dataset, training on data filtered with the method of He et al., and training on SafeTulu. All values are percentages, reported as mean ± standard error. Lower is better on GCG & ActorBreaker attack success rate and ORBench over-refusal rate, while higher accuracy is better on all capability metrics. Across all models, training on SafeTulu decreases ASR while preserving comparable capability.
Model
Training Data
GCG ↓
ActorBreaker ↓
MMLU Pro
IFEval
Qwen
Baseline Training
47.67 ± 0.67
43.0 ± 4.4
19.8 ± 2.5
37.2 ± 7.4
Random Removal
52.00 ± 2.08
41.7 ± 2.7
21.3 ± 1.2
45.2 ± 0.7
LLM-as-a-Judge
47.33 ± 1.20
39.7 ± 0.9
21.1 ± 0.9
45.8 ± 1.3
He et al.
46.33 ± 2.60
42.0 ± 2.6
22.4 ± 0.5
45.3 ± 0.6
SafeTulu (Ours)
37.00 ± 1.53
36.3 ± 1.2
22.4 ± 0.7
46.1 ± 1.4
Llama
Baseline Training
38.00 ± 1.53
42.7 ± 0.7
15.3 ± 0.1
43.8 ± 0.7
Table 1: Evaluation results after baseline training, training on randomly filtered data, training on the LLM-as-a-judge filtered dataset, training on data filtered with the method of He et al., and training on SafeTulu, for Qwen 3 1.7B, Llama 3.2 1B, Olmo 2 1B, and the larger Llama 3.1 8B. All values are percentages, reported as mean ± standard error. Lower is better on GCG & ActorBreaker attack success rate and ORBench over-refusal rate, while higher accuracy is better on all capability metrics. Across all models, including at the 8B scale, training on SafeTulu decreases ASR while preserving comparable capability.
Subset
Gradient Removal Rate
Judge Removal Rate
personahub
9.2%
15.5%
codealpaca
17.3%
15.5%
wildchat
18.7%
28.2%
aya
11.7%
19.2%
flan
22.1%
17.0%
wildjailbreak
39.4%
32.8%
Table 2: Fraction of each Tulu 3 subset removed by each method under a 20% budget, with math subsets highlighted. While the LLM judge removes math at a roughly flat rate, gradient filtering varies its rates, indicating it is finding a signal beyond surface level content.
Subset
Feature
RR
numinamath
Python traceback/exception
1.71 ×
personahub
“Write a Python function” code task
1.74 ×
personahub
Calculus/differential-equation modeling
0.51 ×
codealpaca
High-level architecture discussion
1.75 ×
codealpaca
Runnable code in response
0.89 ×
open_math_2_gsm8k
Long repeating decimal
0.36 ×
Table 3: Relative rate (RR) of each feature among removed vs kept documents. We observe trends such as removing tool-execution failures and simple coding tasks at a higher rate.
Subset
Removed
Kept
numinamath
… from scipy.optimize import minimize → ModuleNotFoundError: No module named ’scipy’ “It seems that the scipy library is not available…”
“Given… an=sin3nπ , find the value of a3 .” → import sympy as sp … a_3 = a_n.subs(n, 3)
personahub
“Write a python function to convert a list of historical event years into a list of centuries…” → def convert_years_to_centuries(years): …
“An entrepreneur is analyzing…determine the constants in the revenue functions and predict when total revenue will reach 50million”\rightarrow$ “ Step 1: Determine…”
codealpaca
“…building a resilient serverless architecture using…AWS Lambda and API Gateway…Could you provide a detailed, sequential guide…” → “…These include planning, establishing the environment, continuous integration…”
“Create a function that takes two input strings, and returns a dictionary containing the count of unique characters…” → def count_chars(string1, string2): … print(char_counts)
open_math_2_gsm8k
“Emily planted a cherry blossom tree…when she was 6…takes 9 years to bloom…currently 10…” → “… 9−4=5 years are left… 10+5=15 … 15 ”
“…The smallest building is 1/3 the size of the second-largest building…” → “… 1/3×135=0.3333333333333333×135=45 feet tall.”
personas_math
“Lawmaker Lisa…plans to allocate resources to 5 different community centers. Each center will receive an equal number…”
“A television producer…divide the show into three parts…” → “Let’s define variables for the time allocated to each part…”
wildchat
“hi” → “Hello! How can I assist you today?”
“Title: Dyanra: Misteryo, Kababalaghan. Tauhan: Mike, Elmer, Jones, Erica…” → “…Sa isang tahimik at mapayapang baryo, sinimulan ng magkakapatid na Mike, Elmer…”
Table 4: Examples of removed and kept documents from each subset. Responses are truncated to their distinguishing parts.
Subset
% Removed Data is Shorter By
table_gpt
40%
personahub
33%
aya
30%
no_robots_converted
29%
wildjailbreak
21%
flan
20%
Table 5: The percentage that the data removed by gradient filtering is shorter than the kept documents for every major subset. The percentage is only large for half (6/11) of the major subsets, indicating that gradient filtering is not simply optimizing for shorter length examples.
Removal Fraction
LLM-as-a-Judge ASR ↓
Gradient ASR ↓
0%
38.00 ± 1.53
38.00 ± 1.53
5%
36.33 ± 1.86
35.00 ± 1.53
10%
35.00 ± 1.53
33.67 ± 1.20
20%
39.00 ± 1.58
30.33 ± 0.33
40%
43.00 ± 2.52
39.00 ± 0.58
Table 6: GCG attack success rate (%) vs. the fraction of training data removed, for training Llama 3.2 1B after LLM-as-a-judge filtering and gradient-based filtering. Values are mean ± standard error; lower is better.
Embedding Model
Target Responses
Model State
Matched AUC
Transfer AUC
Mistral-Small-24B-Instruct-2501
Qwen harmful responses
Fine-tuned
0.865
0.832
Qwen2.5-Coder-32B-Instruct
Mistral harmful responses
Fine-tuned
0.604
0.247
Mistral-Small-24B-Instruct-2501
Qwen harmful responses
Base
0.431
0.502
Qwen2.5-Coder-32B-Instruct
Mistral harmful responses
Base
0.552
0.535
Table 7: Cross-model transfer of the emergent-misalignment gradient signal. Matched AUC uses harmful target responses produced by the same model family as the embedding model, while Transfer AUC replaces them with responses from the other fine-tuned model family. The embedding model used is important: models’ gradient representations do not necessarily transfer to different models’ responses.
Model
AUC
Qwen2.5-Coder-0.5B-Instruct
0.507
Qwen2.5-Coder-3B-Instruct
0.549
Qwen2.5-Coder-7B-Instruct
0.573
Qwen2.5-Coder-14B-Instruct
0.590
Qwen2.5-Coder-32B-Instruct
0.552
Table 8: Gradient-embedding AUC across non-fine-tuned Qwen2.5-Coder model sizes. The gradient signal increases up to 14B, suggesting that smaller models may have a sharper gradient signal than larger models.
Large language models (LLMs) suffer from degraded safety capabilities even when fine-tuned with benign datasets. However, existing methods for identifying safety-degrading samples in benign datasets suffer from high computational costs and significant noise issues. In this paper, we propose DataShield to efficiently and effectively identify potential safety-degrading samples. Our key intuition is based on the observation that benign fine-tuning increases the overall response compliance of LLMs. DataShield's key technical insight is to quantify each sample's contribution to the model's compliance behavior as its safety degradation score. DataShield consists of three core components: (1) Compliance Vector Extraction, which captures the LLM's compliance behavior tendency; (2) a novel Compliance-Aware Score (CAS), which automatically identifies the optimal safety-critical layer; and (3) Safety-degrading Sample Filtering, which quantifies the projection shift of training data along the compliance direction. Extensive experimental evaluation on Llama3-8B, Llama3.1-8B, and Qwen2.5-7B using the Alpaca and Dolly benign datasets validates our method's effectiveness in identifying high-risk and low-risk data subsets. We also observe that open-ended question answering is more likely to trigger safety degradation, and corresponding responses tend to be longer. We hope this work can provide new insights into data-centric defense methods. The source code is available at: https://github.com/ZJunBo/DataShield.
Large Language Models (LLMs) pose a significant risk of safety misalignment after finetuning, as models can be compromised by both explicitly and implicitly harmful data. Even some seemingly benign data can inadvertently steer a model towards misaligned behaviors. To address this, we introduce GradShield, a principled filtering method that safeguards LLMs during finetuning by identifying and removing harmful data points before they corrupt the model's alignment. It removes potentially harmful data by computing a Finetuning Implicit Harmfulness Score (FIHS) for each data point and employs an adaptive thresholding algorithm. We apply GradShield to multiple utility fine-tuning tasks across varying levels of harmful data and evaluate the safety and utility performance of the resulting LLMs using various metrics. The results show that GradShield outperforms all baseline methods, consistently maintaining an Attack Success Rate (ASR) below 6% while preserving utility performance.
Zhanhao Hu, Xiao Huang, Patrick Mendoza +4
University of California, Berkeley · 2HUMAIN · 3King Abdulaziz City for Science and Technology +1
Methods for addressing safety drift in fine-tuned Large Language Models (LLMs) are scattered across incompatible implementations, lifecycle stages, and evaluation protocols, making them difficult to adopt and compare. We introduce SafeTune, a source-available library that unifies four intervention paradigms: post-hoc weight recovery, safety-constrained fine-tuning, gradient-based unlearning, and inference-time steering, alongside shared interpretability, evaluation, and deployment utilities. SafeTune provides a consistent configuration-driven workflow while preserving the distinct inputs and intervention points each paradigm requires. Its modular registry supports new methods, benchmarks, judges, models, and fine-tuning domains without redesigning the surrounding pipeline. We demonstrate SafeTune through controlled comparisons and finance and medical deployment case studies, showing how it characterizes safety drift, evaluates feasible interventions on common refusal-behavior and capability evaluations, and supports calibrated or layered mitigation.