Large language models are trained to have safety guardrails, but these can be compromised while fine-tuning for new capabilities, such as reasoning. One way to mitigate this degradation is to modify the fine-tuning objective to preserve safety behaviour, but doing so effectively and efficiently remains challenging. We revisit Elastic Weight Consolidation (EWC), which penalises parameter updates based on sensitivity estimates from the diagonal of the Fisher information matrix (FIM). While EWC provides some protection, the diagonal approximation ignores interactions between parameters. We address this limitation by exploiting recent findings that safety-relevant behaviour is located in a low-rank subspace within models. Our method, Subspace-Projected Elastic Weight Consolidation (SP-EWC), projects gradients onto this subspace to capture non-diagonal curvature efficiently. We investigate three different ways of identifying the safety subspace: through safety fine-tuning updates (LoRA), safety-related activations (ActSVD), and contrastive representations of safety and utility data (cPCA). We find that this choice substantially affects performance, with SP-EWC using cPCA comparing favourably against existing methods by better maintaining safety guardrails without introducing excessive over-refusal.
Figures & tables
Figure 1: The prohibitive size of the full Fisher information matrix forces EWC to use a diagonal approximation, capturing only parameter-wise sensitivity. We instead project gradients onto the safety subspace, capturing interactions between safety-critical directions.
Figure 2: The performance of Llama-3.1-8B-it and various interventions on MATH-500, OlympiadBench, its average on jailbreak resistance benchmarks, and its average on over-refusal benchmarks. Notably, SafeInstr over-optimises for jailbreak resistance, introducing over-refusal as a new pathology. All other baselines struggle to maintain safety guardrails, while SP-EWC strikes a favourable trade-off. The full results table detailing individual safety benchmark results can be found in Table 2 .
Figure 3: The performance of OLMo-2-7B-it and various interventions on maths and safety benchmarks. Notably, SafeInstr over-optimises for jailbreak resistance, introducing over-refusal as a new pathology. SP-EWC better balances this while still enabling in-domain gains. The full results table detailing individual safety benchmark results can be found in Table 2 .
Figure 4: We plot cPCA for a single linear layer to illustrate its effects. We collect activations from the MLP down projection of layer 16 of Llama-3.1-8B-it, and project them onto the top two contrastive principal components. Activations come from the first 20 response tokens of 2000 safety (target) and 2000 utility (background) examples. As α rises, the principal components exhibit comparatively more variance in safety than utility activations.
Figure 5: Comparison of the performance change in percentage points when locating the safety subspace for SP-EWC via LoRA, ActSVD or cPCA.
Variant
M-500
Olymp
JR
OA
Llama-3.1-8B-it
44.9
16.1
92.7
88.2
Diagonal only
51.8
20.8
77.3
96.4
Diagonal + subspace
51.0
19.8
92.0
87.0
Subspace only
50.6
21.8
89.9
87.2
OLMo-2-7B-it
35.5
12.8
87.8
95.2
Diagonal only
39.5
14.1
88.0
93.6
Table 1: Ablation separating the contributions of the diagonal EWC penalty and the projected Fisher penalty in SP-EWC (cPCA).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Math
Jailbreak resistance
Over-refusal avoidance
Method
M-500
Olymp
Trust
WGT
WJB:H
DAN
Avg.
XSTest
WJB:B
Avg.
Llama-3.1-8B-it
Before training
44.9
16.1
91.0
97.0
83.8
99.0
92.7
94.8
81.6
88.2
Low-LR SFT
55.6
23.2
95.0
69.7
8.3
15.7
47.2
100.0
100.0
100.0
SafeInstr
52.6
23.0
95.0
100.0
98.7
94.0
96.9
64.0
18.0
41.0
SafeGrad
54.8
21.8
81.2
84.0
39.8
74.7
69.9
90.4
99.2
94.8
Appendix
Table 2: Per-benchmark results for Llama-3.1-8B-it and OLMo-2-7B-it trained on maths with each intervention. All scores are in %, and higher is better across all benchmarks.
Method
Time (h)
Overhead vs. SFT
Low-LR SFT
12.5
–
EWC
14.3
+15%
SP-EWC (cPCA)
14.2
+14%
SP-EWC (ActSVD)
14.7
+18%
SP-EWC (LoRA)
14.8
+19%
SafeGrad
21.0
+69%
Appendix
Table 3: Wall-clock training time of OLMo-2-7B-it on 4xH200, including pre-computation before training.
Figure 6: The performance of Llama-3.1-8B-it on maths and safety benchmarks when varying the λ hyper-parameter which modulates the strength of the EWC penalty.
Figure 7: The performance of Llama-3.1-8B-it on maths and safety benchmarks when using SafeGrad, or when using DER under varying amounts of safety data.
Model
JR
OA
Gemma-2-9B-it
57.8
90.8
Llama-3.1-8B-it
92.7
88.2
Ministral-8B-it
29.7
97.8
OLMo-2-7B-it
87.8
95.2
Qwen-2.5-7B-it
47.9
98.2
Appendix
Table 4: Jailbreak resistance (JR) and over-refusal avoidance (OA) of candidate instruct models before any training, averaged over the same benchmarks as in our main results. All scores in %, higher is better.
Learning rate
M-500
Olymp
JR
OA
Llama-3.1-8B-it
Before training
44.9
16.1
92.7
88.2
1×10−6
55.6
23.2
47.2
100.0
5×10−6
55.0
23.3
23.8
99.2
1×10−5
54.1
24.4
17.3
99.6
5×10−5
42.1
16.5
24.7
90.8
Appendix
Table 5: Performance of Llama-3.1-8B-it and OLMo-2-7B-it on maths and safety benchmarks after SFT with different learning rates. All scores in %, higher is better.
Safety data
M-500
Olymp
JR
OA
Llama-3.1-8B-it
Before training
44.9
16.1
92.7
88.2
Low-LR SFT
55.6
23.2
47.2
100.0
1%
56.1
21.4
98.7
34.2
2%
51.4
21.3
95.8
41.2
3%
52.6
23.0
96.9
41.0
Appendix
Table 6: Performance of Llama-3.1-8B-it and OLMo-2-7B-it on maths and safety benchmarks after mixing varying percentages of safety data into the training data (SafeInstr). We reproduce the finding by Bianchi et al. (2024) that this maintains alignment very well while causing over-refusal. All scores in %, higher is better.