Large language models are trained to have safety guardrails, but these can be compromised while fine-tuning for new capabilities, such as reasoning. One way to mitigate this degradation is to modify the fine-tuning objective to preserve safety behaviour, but doing so effectively and efficiently remains challenging. We revisit Elastic Weight Consolidation (EWC), which penalises parameter updates based on sensitivity estimates from the diagonal of the Fisher information matrix (FIM). While EWC provides some protection, the diagonal approximation ignores interactions between parameters. We address this limitation by exploiting recent findings that safety-relevant behaviour is located in a low-rank subspace within models. Our method, Subspace-Projected Elastic Weight Consolidation (SP-EWC), projects gradients onto this subspace to capture non-diagonal curvature efficiently. We investigate three different ways of identifying the safety subspace: through safety fine-tuning updates (LoRA), safety-related activations (ActSVD), and contrastive representations of safety and utility data (cPCA). We find that this choice substantially affects performance, with SP-EWC using cPCA comparing favourably against existing methods by better maintaining safety guardrails without introducing excessive over-refusal.
Figures & tables
Figure 1: The prohibitive size of the full Fisher information matrix forces EWC to use a diagonal approximation, capturing only parameter-wise sensitivity. We instead project gradients onto the safety subspace, capturing interactions between safety-critical directions.
Figure 2: The performance of Llama-3.1-8B-it and various interventions on MATH-500, OlympiadBench, its average on jailbreak resistance benchmarks, and its average on over-refusal benchmarks. Notably, SafeInstr over-optimises for jailbreak resistance, introducing over-refusal as a new pathology. All other baselines struggle to maintain safety guardrails, while SP-EWC strikes a favourable trade-off. The full results table detailing individual safety benchmark results can be found in Table 2 .
Figure 3: The performance of OLMo-2-7B-it and various interventions on maths and safety benchmarks. Notably, SafeInstr over-optimises for jailbreak resistance, introducing over-refusal as a new pathology. SP-EWC better balances this while still enabling in-domain gains. The full results table detailing individual safety benchmark results can be found in Table 2 .
Figure 4: We plot cPCA for a single linear layer to illustrate its effects. We collect activations from the MLP down projection of layer 16 of Llama-3.1-8B-it, and project them onto the top two contrastive principal components. Activations come from the first 20 response tokens of 2000 safety (target) and 2000 utility (background) examples. As α rises, the principal components exhibit comparatively more variance in safety than utility activations.
Figure 5: Comparison of the performance change in percentage points when locating the safety subspace for SP-EWC via LoRA, ActSVD or cPCA.
Variant
M-500
Olymp
JR
OA
Llama-3.1-8B-it
44.9
16.1
92.7
88.2
Diagonal only
51.8
20.8
77.3
96.4
Diagonal + subspace
51.0
19.8
92.0
87.0
Subspace only
50.6
21.8
89.9
87.2
OLMo-2-7B-it
35.5
12.8
87.8
95.2
Diagonal only
39.5
14.1
88.0
93.6
Table 1: Ablation separating the contributions of the diagonal EWC penalty and the projected Fisher penalty in SP-EWC (cPCA).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Math
Jailbreak resistance
Over-refusal avoidance
Method
M-500
Olymp
Trust
WGT
WJB:H
DAN
Avg.
XSTest
WJB:B
Avg.
Llama-3.1-8B-it
Before training
44.9
16.1
91.0
97.0
83.8
99.0
92.7
94.8
81.6
88.2
Low-LR SFT
55.6
23.2
95.0
69.7
8.3
15.7
47.2
100.0
100.0
100.0
SafeInstr
52.6
23.0
95.0
100.0
98.7
94.0
96.9
64.0
18.0
41.0
SafeGrad
54.8
21.8
81.2
84.0
39.8
74.7
69.9
90.4
99.2
94.8
Appendix
Table 2: Per-benchmark results for Llama-3.1-8B-it and OLMo-2-7B-it trained on maths with each intervention. All scores are in %, and higher is better across all benchmarks.
Method
Time (h)
Overhead vs. SFT
Low-LR SFT
12.5
–
EWC
14.3
+15%
SP-EWC (cPCA)
14.2
+14%
SP-EWC (ActSVD)
14.7
+18%
SP-EWC (LoRA)
14.8
+19%
SafeGrad
21.0
+69%
Appendix
Table 3: Wall-clock training time of OLMo-2-7B-it on 4xH200, including pre-computation before training.
Figure 6: The performance of Llama-3.1-8B-it on maths and safety benchmarks when varying the λ hyper-parameter which modulates the strength of the EWC penalty.
Figure 7: The performance of Llama-3.1-8B-it on maths and safety benchmarks when using SafeGrad, or when using DER under varying amounts of safety data.
Model
JR
OA
Gemma-2-9B-it
57.8
90.8
Llama-3.1-8B-it
92.7
88.2
Ministral-8B-it
29.7
97.8
OLMo-2-7B-it
87.8
95.2
Qwen-2.5-7B-it
47.9
98.2
Appendix
Table 4: Jailbreak resistance (JR) and over-refusal avoidance (OA) of candidate instruct models before any training, averaged over the same benchmarks as in our main results. All scores in %, higher is better.
Learning rate
M-500
Olymp
JR
OA
Llama-3.1-8B-it
Before training
44.9
16.1
92.7
88.2
1×10−6
55.6
23.2
47.2
100.0
5×10−6
55.0
23.3
23.8
99.2
1×10−5
54.1
24.4
17.3
99.6
5×10−5
42.1
16.5
24.7
90.8
Appendix
Table 5: Performance of Llama-3.1-8B-it and OLMo-2-7B-it on maths and safety benchmarks after SFT with different learning rates. All scores in %, higher is better.
Safety data
M-500
Olymp
JR
OA
Llama-3.1-8B-it
Before training
44.9
16.1
92.7
88.2
Low-LR SFT
55.6
23.2
47.2
100.0
1%
56.1
21.4
98.7
34.2
2%
51.4
21.3
95.8
41.2
3%
52.6
23.0
96.9
41.0
Appendix
Table 6: Performance of Llama-3.1-8B-it and OLMo-2-7B-it on maths and safety benchmarks after mixing varying percentages of safety data into the training data (SafeInstr). We reproduce the finding by Bianchi et al. (2024) that this maintains alignment very well while causing over-refusal. All scores in %, higher is better.
Most Large Language Models exhibit a fundamental tension between two sequential tasks, such as logical reasoning and safety alignment. The high-variance internal states required for sophisticated Chain-of-Thought (CoT) deduction can geometrically interfere with latent representations encoding safety constraints. We identify this phenomenon as Sequential Subspace Interference, showing that standard fine-tuning on logical tasks such as multi-step mathematics and code generation can result in a 23.3% interference penalty on alignment benchmarks, substantially weakening the model's safety priors. This Reasoning Drift is not adequately captured by current adaptation methods because gradients for logical tasks are rarely orthogonal to safety objectives. To address this issue, we propose AURA (Adaptive Unique Residual Allocation), a spectral regularization framework that enforces Spectral Independence between reasoning and safety. By explicitly estimating the null space of the alignment manifold and constraining reasoning updates to its orthogonal complement, AURA enables models to improve logical reasoning without compromising safety. Empirically, AURA recovers 23.0% of the lost performance while preserving greater than 0.98 cosine fidelity to the safe state, demonstrating that reasoning and alignment can be effectively decoupled through geometric regularization.
Instruction-following language models are trained to be helpful and safe, yet their safety behavior can deteriorate under benign fine-tuning and worsen under adversarial updates. Existing defenses often offer limited protection or force a trade-off between safety and utility. We introduce a training framework that adapts regularization in response to safety risk, enabling models to remain aligned throughout fine-tuning. To estimate safety risk at training time, we explore two distinct approaches: a judge-based Safety Critic that assigns high-level harm scores to training batches, and an activation-based risk predictor built with a lightweight classifier trained on intermediate model activations to estimate harmful intent. Each approach provides a risk signal that is used to constrain updates deemed higher risk to remain close to a safe reference policy, while lower-risk updates proceed with standard training. We empirically verify that harmful intent signals are predictable from pre-generation activations and that judge scores provide effective high-recall safety guidance. Across multiple model families and attack scenarios, adaptive regularization with either risk estimation approach consistently lowers attack success rate compared to standard fine-tuning, preserves downstream performance, and adds no inference-time cost. This work demonstrates a principled mechanism for maintaining safety without sacrificing utility.
Jyotin Goel, Souvik Maji, Pratik Mazumder
Indian Institute of Technology Jodhpur Jodhpur, Rajasthan, India
Low-rank adaptation has become a standard method for parameter-efficient fine-tuning of large language models, but even small amounts of unsafe or adversarial fine-tuning data can substantially weaken the safety behavior of aligned models. Existing safety-preserving LoRA methods often rely on hard interventions such as projection, pruning, thresholding, or additional training objectives. While these methods can suppress unsafe update directions, they may also remove task-relevant information or require extra tuning. We introduce CSULoRA, a post-hoc method for correcting trained LoRA adapters through closest safe update estimation. CSULoRA estimates a safety-aligned subspace from the weight displacement between a safety-aligned model and its corresponding base checkpoint. It then decomposes each LoRA update into fully aligned, partially aligned, and off-subspace components. Instead of discarding components outside the estimated safety subspace, CSULoRA solves a closed-form penalized minimum-change problem that preserves the fully aligned component while smoothly attenuating potentially unsafe directions according to their relative energy. In adversarial fine-tuning experiments, CSULoRA substantially reduces attack success rate while preserving most of the utility gains obtained from standard LoRA fine-tuning.