cs.LGNov 15, 2026

Protecting Safety during Reasoning Training with Low-Rank Fisher Regularisation

Authors: Neel Rajani, Ivan Titov

Organizations: ILCC, University of Edinburgh · ILLC, University of Amsterdam

Abstract

Large language models are trained to have safety guardrails, but these can be compromised while fine-tuning for new capabilities, such as reasoning. One way to mitigate this degradation is to modify the fine-tuning objective to preserve safety behaviour, but doing so effectively and efficiently remains challenging. We revisit Elastic Weight Consolidation (EWC), which penalises parameter updates based on sensitivity estimates from the diagonal of the Fisher information matrix (FIM). While EWC provides some protection, the diagonal approximation ignores interactions between parameters. We address this limitation by exploiting recent findings that safety-relevant behaviour is located in a low-rank subspace within models. Our method, Subspace-Projected Elastic Weight Consolidation (SP-EWC), projects gradients onto this subspace to capture non-diagonal curvature efficiently. We investigate three different ways of identifying the safety subspace: through safety fine-tuning updates (LoRA), safety-related activations (ActSVD), and contrastive representations of safety and utility data (cPCA). We find that this choice substantially affects performance, with SP-EWC using cPCA comparing favourably against existing methods by better maintaining safety guardrails without introducing excessive over-refusal.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuning

    Feb 19, 2026Jyotin Goel, Souvik Maji, Pratik MazumderSafety-Preserving Fine-Tuning of Language ModelsLLM Safety

  2. CSULoRA: Closest Safe Update Low-Rank Adaptation

    May 28, 2026Oleksandr Marchenko Breneur, Adelaide Danilov, Aria Nourbakhsh +1Safety-Preserving Fine-Tuning of Language ModelsAdapter Tuning