cs.LGSep 7, 2026

The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists

Authors: Srikanth Malla, Chiho Choi, Joon Hee Choi

Organizations: Samsung Semiconductor US

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Post-hoc safety training (RLHF, DPO) is the dominant way to align large language models, yet jailbreaks (Zou et al., 2023), fine-tuning attacks (Qi et al., 2024), and activation-space edits (Arditi et al., 2024) keep recovering the behaviors it was meant to remove. We give this fragility one geometric explanation and follow it into pretraining. We measure the safety update Δ=Wsafe−WbaseΔ= W_{safe} - W_{base} against the curvature of the model's capabilities (the empirical Fisher of a capability loss). Across five model families, post-hoc safety lands in a suppression regime: ΔΔ is nearly orthogonal to the capability directions, and its small in-subspace part concentrates on a few high-curvature ones. The update is thin but sharp, a refusal gate laid over intact capabilities rather than erasure of them. A kernel-immobility lemma explains why such an update can only mask a capability, not remove it, so a little benign fine-tuning restores it: 100 benign examples cut the AdvBench refusal of Qwen-2.5-7B-Instruct and Llama-3-8B-Instruct by 35 to 38 pp. Following the account into pretraining, a pretraining-checkpoint sweep of OLMo-2-1B (Team OLMo et al., 2024) shows the features that refusal attaches to emerging in a sharp transition between 1B and 63B pretraining tokens. We then use the account constructively: models trained from scratch with safety co-training spread continuously across pretraining reach 87 to 98% AdvBench refusal that the same attack erodes by only 2 to 14 pp at every scale from 410M to 6.9B, against 35 to 38 pp for post-hoc installs, at a small cost on short-answer capability probes; a windowed schedule of equal total safety weight installs no refusal. Persistence of the safety signal across pretraining, not its timing, is what buys attack robustness.

Figures & tables

Figure 1: Post-hoc safety lands orders of magnitude short of erasure and loses 3535 to 3838 pp of refusal to a benign fine-tuning attack; co-training loses 22 to 1414 pp. All points measured. (a) Where safety updates land in the (ρ,qnorm)(\rho,{q_{\text{norm}}}) plane, one point per transformer block and capability probe. Post-hoc DPO/RLHF across five model families (blue, light to dark by size) is thin but sharp : two to three orders of magnitude below the erasure boundary ρpart\rho_{\text{part}} , yet at its peak blocks above each family’s own random floor k/dk/d (dashed; the five span 6.2×6.2\times ). Erasure is reachable but post-hoc safety never reaches it: an adversarial-SFT control built to saturate the capability subspace (squares) crosses ρpart\rho_{\text{part}} . Applying post-hoc DPO to an under-trained base (circles) lands elsewhere, which Sec. 6 traces to a developmental transition. This panel is post-hoc installs only: a co-trained model never received a patch, so no such Δ\Delta exists (App. M ); its claim is panel (b). (b) Why that geometry matters: AdvBench refusal rate before →\rightarrow after the same 100-example benign-SFT attack, regex-scored throughout so rows are comparable. Post-hoc installs lose 3535 to 3838 pp (Llama-3 to 11%11\% ); co-trained refusal loses 22 to 1414 pp, ending at 8484 to 91%91\% at every scale to 6.96.9 B. Regex tracks the refusal gate , which is what the geometry predicts; App. L repeats both attack tables under a harmful-output classifier.
Figure 2: What the install added is flat through the stack until it turns on inside a narrow band of blocks. Per-layer L2 distance between base and safe residual streams over 50 harmful and 50 topic-matched benign prompts. The two curves track each other through most of the stack; their difference (black) is the harmful-specific part of what the install changed. The shaded safety-install band marks the blocks where qnorm{q_{\text{norm}}} peaks (App. E , Fig. 5 ). On Qwen and Mistral the difference turns on inside it; on Llama the difference peaks mid-stack (block 1212 ) while qnorm{q_{\text{norm}}} peaks at the top, so there the two measurements pick out different blocks.
Instruct modelAdvBenchHarmBenchStrongReject
Qwen-2.5-7B93.0→58.293.0\to 58.267.5→45.567.5\to 45.552.3→56.652.3\to 56.6
Llama-3-8B49.2→10.949.2\to 10.946.5→6.546.5\to 6.546.5→7.846.5\to 7.8
Mistral-7B-v0.31.2→0.41.2\to 0.41.5→0.01.5\to 0.03.5→0.83.5\to 0.8
Table 1: The attack takes 3535 to 3838 pp of AdvBench refusal from both models that had any, leaving Llama-3 at 11%11\% . Refusal rate (%) before →\to after the same 100100 -example benign-SFT attack on the three shipped instruct models. Mistral ships with almost no refusal, so its row has little to strip. n=256/200/256n=256/200/256 ; the one cell that rises is within noise (criterion in App. A ). App. L repeats the table under a harmful-output classifier.
Figure 3: At 1B pretraining tokens the base carries nothing for a refusal circuit to engage; by 63B the substrate is there. OLMo-2-1B substrate sweep, the final-stage Instruct weights held fixed while the base is swept across 9 pretraining revisions. (a) Max selective divergence (how much further the Instruct weights move the hidden state from the base’s on harmful than on benign prompts) against pretraining tokens: at noise, then up two orders of magnitude, then a plateau. (b) The same measurement at each residual-stream position, one curve per revision, darker with more pretraining. Among post-substrate revisions the peak position settles forward through the residual stream ( ℓ⋆=6\ell^{\star}=6 at 105B, 99 at 420B to 2T, 1111 at 4T), having sat after the final block ( ℓ⋆=16\ell^{\star}=16 ) at the transition’s early edge.
Condition (safety window)410M (pre →\to post)1B (pre →\to post)
cotrain-continuous [0,1][0,1]84.4→84.4\mathbf{84.4\to 84.4}87.5→90.6\mathbf{87.5\to 90.6}
cotrain-post-substrate [0.30,1][0.30,1]n/a84.4→84.4\mathbf{84.4\to 84.4}
cotrain-windowed [0.10,0.30][0.10,0.30]0.0→1.60.0\to 1.63.1→4.73.1\to 4.7
base (LM only)0.0→0.00.0\to 0.00.0→0.00.0\to 0.0
Table 2: Both schedules that persist to the end of pretraining survive the attack; the windowed schedule installs nothing. From-scratch co-training refusal (regex, %) on 64 AdvBench prompts, pre →\to post benign-SFT attack. Conditions differ in the fractional pretraining window in which the safety loss is active, at λ=0.10\lambda=0.10 per step except cotrain-windowed ’s 0.50.5 , which matches cotrain-continuous ’s total. The 11 B 87.5→90.687.5\to 90.6 change is two prompts, within noise at n=64n=64 (App. A ); Table 3 is a separate run of cotrain-continuous on 256256 prompts.
ScaleAdvBench (in-dist.)HarmBench (out)StrongReject (out)
410M98.4→84.098.4\to 84.058.0→34.558.0\to 34.58.2→7.48.2\to 7.4
1B87.1→84.887.1\to 84.856.0→32.056.0\to 32.07.8→8.27.8\to 8.2
2.8B96.1→90.296.1\to 90.245.5→31.545.5\to 31.58.6→8.28.6\to 8.2
6.9B96.9→90.6\mathbf{96.9\to 90.6}32.5→39.032.5\to 39.09.0→10.99.0\to 10.9
Table 3: Post-attack AdvBench refusal holds at 8484 to 91%91\% at every scale to 6.96.9 B. From-scratch cotrain-continuous refusal (%), pre →\to post benign-SFT attack, by scale and prompt distribution. The recipe’s refusal templates are AdvBench-like, so only AdvBench is in-distribution. The LM-only base refuses at most 1.6%1.6\% by regex at any scale and benchmark ( 24.6%24.6\% by the classifier, Table 26 ). n=256/200/256n=256/200/256 ; changes under about 66 pp are within noise (App. A ). Classifier columns: App. L .
Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Modelρobs\rho_{\text{obs}}ρobs/(k/d)\rho_{\text{obs}}/(k/d)zz (SDs above null)
Qwen-2.5-7B4.1×10−54.1\times 10^{-5}67×67\times528528
Mistral-7B-v0.33.2×10−53.2\times 10^{-5}44×44\times343343
Llama-3-8B1.5×10−51.5\times 10^{-5}21×21\times159159
Appendix
Table 4: Random-subspace significance of the Fisher alignment. ρobs\rho_{\text{obs}} is the observed overlap (max over layers); the null is a random kk -subspace at the model’s MLP parameter dimensionality dd ; z=(ρobs−k/d)/stdnullz=(\rho_{\text{obs}}-k/d)/\text{std}_{\text{null}} . All p≈0p\approx 0 (two-sided). The alignment is 2121 to 6767 times the floor and hundreds of null standard deviations above it, so it is small but not a high-dimensional coincidence.
  mean ρ\rhoqnorm{q_{\text{norm}}} -peak layer
ModelProbe0-shot3-shot0-shot3-shot
Qwen-2.5-7Breason1.40×10−51.40\times 10^{-5}1.40×10−51.40\times 10^{-5}L0L0
 arith1.21×10−51.21\times 10^{-5}1.01×10−51.01\times 10^{-5}L26L0
 recall1.20×10−51.20\times 10^{-5}9.37×10−69.37\times 10^{-6}L18L27
 instruct1.07×10−51.07\times 10^{-5}1.05×10−51.05\times 10^{-5}L27L17
Llama-3-8Breason5.07×10−65.07\times 10^{-6}5.07×10−65.07\times 10^{-6}L30L30
Appendix
Table 5: Geometry robustness check under raw 3-shot probe prompts. Mean ρ\rho averaged across all transformer blocks; qnorm{q_{\text{norm}}} -peak reports the layer where qnorm(c){q_{\text{norm}}}^{(c)} attains its maximum. All three families remain in the suppression regime ( ρ≪ρpart=10−2\rho\ll\rho_{\text{part}}=10^{-2} ) under 3-shot; peak- qnorm{q_{\text{norm}}} layer is stable on the probes driving the safety-install-band claim.
Regimeρ\rhoqnorm{q_{\text{norm}}}Observed-in; sub-patterns within
Suppression≪ρpart\ll\rho_{\text{part}} (block means 10−610^{-6} to 1.1×10−41.1\times 10^{-4} )≫1\gg 1 (thin and sharp) or ∼1\sim 1 (diffuse)Baseline (pure DPO/RLHF on mature base, ρ∼10−6\rho\sim 10^{-6} to 10−510^{-5} ): Pythia-1.4B+DPO (§ 4 ); Qwen/Llama/Mistral-Instruct (§ 4 ); post-substrate OLMo Weak/Strong (§ 6 ). Sub-pattern (a), under-trained base : Lemma assumption fails, qnorm\textscarith∈[600,863]{q_{\text{norm}}}^{\textsc{arith}}\in[600,863] (pre-substrate OLMo step 300300 , § 6 ). Sub-pattern (b), SFT-stage recipe : SFT’s gradient lies in FcapF_{\text{cap}} ’s top- kk , raising ρ\rho 66 to 19×19\times over the Weak recipe to ∼10−4\sim 10^{-4} (SFT+DPO at every post-substrate revision, § 6 ).
Erasure≳ρpart\gtrsim\rho_{\text{part}}anyunobserved under DPO/RLHF; reachable by adversarial SFT on (prompt, wrong-answer) pairs (max ρ≈10−2\rho\approx 10^{-2} on target probe; App. G ); target of unlearning (RMU and related) ( Li et al., 2024 )
Appendix
Table 6: Regime dichotomy with two sub-patterns inside suppression. Every DPO/RLHF recipe we tested (Pythia-1.4B, Qwen-2.5-7B, Llama-3-8B, Mistral-7B-v0.3, OLMo-2-1B under Weak / Strong / SFT+DPO) lands in row 1; row 2 is unobserved under post-hoc alignment and is the target of unlearning ( Li et al., 2024 ) .
Probemean ρℓ\rho_{\ell}mean qnormℓ{q_{\text{norm}}}_{\ell}ΔLcap\Delta\mathcal{L}_{\text{cap}}Regime
reason (MMLU fl)1.7×10−51.7\times 10^{-5}7.307.30−0.015-0.015suppression
arith (GSM8K)2.4×10−52.4\times 10^{-5}21.3921.39−0.027-0.027suppression
recall (TriviaQA)2.2×10−52.2\times 10^{-5}10.7310.73−0.010-0.010suppression
instruct (FLAN)2.0×10−52.0\times 10^{-5}6.506.50−0.007-0.007suppression
Appendix
Table 7: Pythia-1.4B post-DPO: per-probe ρ\rho (mean over 24 transformer blocks), normalized Fisher-quadratic form qnorm{q_{\text{norm}}} , and capability-loss shift ΔLcap\Delta\mathcal{L}_{\text{cap}} . All rows are in the suppression regime; capability loss decreases slightly on every probe.
Figure 4: Per-layer ρℓ\rho_{\ell} (log-scale) for each capability probe on Pythia-1.4B after DPO. Layers 00 , 33 , and 2323 (residual-stream boundaries and the final block) dominate; middle layers ( ∼16\sim 16 to 2222 ) sit near the random floor. Pattern is consistent with per-layer localization reported for the refusal direction ( Arditi et al., 2024 ) .
Figure 5: Layer-localized signals across three model families (columns: Qwen-2.5-7B-Instruct, Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3), referenced from Sec. 4 . The top row is Fig. 2 in the main text, repeated here so the two rows can be read against each other. Top row: per-layer L2 divergence between base and instruct residual streams on harmful vs. benign topic-matched prompts ( npairs=50n_{\text{pairs}}{=}50 each), plus selective divergence (harm −- benign). Bottom row: per-layer qnorm(c){q_{\text{norm}}}^{(c)} for the four capability probes (log-scale). Shaded bands bracket each model’s qnorm\textscreason{q_{\text{norm}}}^{\textsc{reason}} / qnorm\textscarith{q_{\text{norm}}}^{\textsc{arith}} peaks (Qwen ℓ=17\ell{=}17 to 2626 ; Llama ℓ=28\ell{=}28 to 3131 ; Mistral ℓ=3\ell{=}3 to 1313 ). Selective-divergence magnitude scales with pre-attack refusal rate: +0.166+0.166 (Qwen, 93%93\% ), +0.053+0.053 (Llama, 49%49\% ), +0.032+0.032 (Mistral, 1.2%1.2\% ).
Probemean ρ\rhomax ρ\rhomean qnorm{q_{\text{norm}}}max qnorm{q_{\text{norm}}} (layer)ΔLcap\Delta\mathcal{L}_{\text{cap}}Regime
reason1.4×10−51.4\times 10^{-5}4.1×10−54.1\times 10^{-5}76.276.2220.6 (ℓ=19)220.6\ (\ell{=}19)+3.45+3.45suppression
arith1.2×10−51.2\times 10^{-5}2.9×10−52.9\times 10^{-5}47.147.1198.4 (ℓ=26)198.4\ (\ell{=}26)+3.15+3.15suppression
recall1.2×10−51.2\times 10^{-5}2.2×10−52.2\times 10^{-5}29.829.894.7 (ℓ=18)94.7\ (\ell{=}18)+1.07+1.07suppression
instruct1.1×10−51.1\times 10^{-5}2.1×10−52.1\times 10^{-5}18.618.631.3 (ℓ=27)31.3\ (\ell{=}27)+0.67+0.67suppression
Appendix
Table 8: Qwen-2.5-7B →\to Qwen-2.5-7B-Instruct: per-probe overlap ρ\rho (mean / max over 2828 blocks), normalized Fisher-quadratic form qnorm{q_{\text{norm}}} , raw-0-shot ΔLcap\Delta\mathcal{L}_{\text{cap}} averaged over the 256256 gradient examples. Max qnorm{q_{\text{norm}}} is over blocks 11 to 2727 : block 00 , next to the embeddings, reaches 525.6525.6 on reason through raw-gradient magnitude alone and is excluded from every peak- qnorm{q_{\text{norm}}} figure in the paper except Table 5 , which reports the unexcluded argmax (App. B ). Source for the prose summary in § 4 .
SignalQwen-2.5-7B-InstructLlama-3-8B-InstructMistral-7B-Instruct-v0.3
mean ρ\rho1.2×10−51.2\times 10^{-5}5.6×10−65.6\times 10^{-6}1.2×10−51.2\times 10^{-5}
peak ρ/(k/d)\rho/(k/d)67×67\times21×21\times44×44\times
peak qnorm\textscreason{q_{\text{norm}}}^{\textsc{reason}} (layer)220.6 (ℓ=19)220.6\ (\ell{=}19)48.9 (ℓ=30)48.9\ (\ell{=}30)158.7 (ℓ=12)158.7\ (\ell{=}12)
peak qnorm\textscarith{q_{\text{norm}}}^{\textsc{arith}} (layer)198.4 (ℓ=26)198.4\ (\ell{=}26)47.7 (ℓ=30)47.7\ (\ell{=}30)92.1 (ℓ=4)92.1\ (\ell{=}4)
peak selective divergence+0.166 (ℓ=24)+0.166\ (\ell{=}24)+0.053 (ℓ=12)+0.053\ (\ell{=}12)+0.032 (ℓ=13)+0.032\ (\ell{=}13)
max raw-0-shot ΔLcap\Delta\mathcal{L}_{\text{cap}}+3.45+3.45+0.88+0.88+0.59+0.59
Appendix
Table 9: Cross-model summary across three instruct-tuned 77 to 88 B models. All replicate the suppression regime; the absolute qnorm{q_{\text{norm}}} band is family-specific; the behavioral attack number is a joint measurement of (refusal installed) ×\times (robustness of that install). Peak qnorm{q_{\text{norm}}} excludes Qwen’s block 00 (Table 8 ). Geometry only; the behavioral effect of the attack is Table 1 in the main text, measured in a single run under both detectors.
 AdvBenchHarmBenchStrongReject
Base modelregexclassifierregexclassifierregexclassifier
Qwen-2.5-7B71.1\mathbf{71.1}69.5\mathbf{69.5}31.531.534.034.028.528.530.930.9
Llama-3-8B0.00.07.07.01.01.010.510.51.61.612.512.5
Mistral-7B-v0.30.00.016.416.40.00.014.014.00.80.819.919.9
Appendix
Table 10: Refusal (%) of the three shipped base models, before any install, under both detectors. Same prompts, decoding and scoring as Table 1 , with no chat template applied to either half of a pair. n=256/200/256n=256/200/256 .
Proberaw (0-shot)raw (3-shot)chat (0-shot)chat (3-shot)
reason+3.35+3.35+0.87+0.87+21.01+21.01+11.07+11.07
arith+3.06+3.06+1.10+1.10+15.19+15.19+9.14+9.14
recall+0.90+0.90+0.54+0.54+5.39+5.39+0.84+0.84
instruct+0.46+0.46+0.37+0.37+3.16+3.16+2.99+2.99
Appendix
Table 11: Chat-template ablation on Qwen-2.5-7B. Four prompt formats on the same four capability probes, n=64n=64 examples per probe, three demonstrations in the 3-shot columns. The raw 0-shot column uses a different sample from Table 8 , which averages over the 256256 gradient examples, so the two differ by up to 0.210.21 nats ( instruct ). 3-shot collapses the gap by 4040 to 74%74\% on reason , arith and recall , and by 20%20\% on instruct , whose raw gap is already small, by showing the Instruct model the raw-completion format via in-context examples. Chat formatting inflates the gap because we score NLL of a bare gold token that Instruct no longer naturally emits after its chat template; 3-shot partially recovers by demonstrating the target format in context.
target probeprobe under evalmean ρ\rhomax ρ\rhomax qnorm{q_{\text{norm}}}ΔLcap\Delta\mathcal{L}_{\text{cap}}target?
reasonreason3.33×10−33.33\times 10^{-3}1.04×10−2\mathbf{1.04\times 10^{-2}}3394533945−0.96-0.96✓\checkmark
reasonarith1.16×10−31.16\times 10^{-3}4.66×10−34.66\times 10^{-3}1577315773−0.62-0.62 
reasonrecall4.97×10−44.97\times 10^{-4}1.49×10−31.49\times 10^{-3}18951895−0.35-0.35 
reasoninstruct4.12×10−44.12\times 10^{-4}1.33×10−31.33\times 10^{-3}328328+0.15+0.15 
aritharith1.16×10−31.16\times 10^{-3}3.01×10−33.01\times 10^{-3}1506415064−2.08-2.08✓\checkmark
arithreason3.47×10−43.47\times 10^{-4}8.95×10−48.95\times 10^{-4}29562956−0.72-0.72 
Appendix
Table 12: Adversarial-SFT erasure demonstration on Pythia-1.4B step143000 . One SFT run per target probe; geometry measured against the base on all four probes. Max ρ\rho on the target probe reaches the ρpart=10−2\rho_{\text{part}}=10^{-2} boundary for target= reason and comes within about 3×3\times of it for target= arith ; qnorm{q_{\text{norm}}} on target is 1717 to 39×39\times the largest value seen under DPO/RLHF ( 863863 , pre-substrate OLMo). Off-target probes see 33 to 8×8\times lower mean ρ\rho than the target, establishing target specificity. ΔLcap\Delta\mathcal{L}_{\text{cap}} is NLL-based; on capability probes with a stable wrong-token format ( reason , arith ), adversarial SFT lowers NLL because format-learning dominates at this intervention scale (see caveat below). ρrand=3.75×10−6\rho_{\text{rand}}=3.75\times 10^{-6} throughout.
StepTokensmean ρ\rhomax selective_divℓ\text{selective\_div}_{\ell}ℓ⋆\ell^{\star}
000B1.4×10−61.4\times 10^{-6}+4×10−4+4\times 10^{-4}n/a
3003001B1.6×10−61.6\times 10^{-6}+4×10−4+4\times 10^{-4}n/a
1×1041\times 10^{4}21B1.8×10−61.8\times 10^{-6}+2.7×10−2\mathbf{+2.7\times 10^{-2}}16
3×1043\times 10^{4}63B1.3×10−61.3\times 10^{-6}+7.3×10−2\mathbf{+7.3\times 10^{-2}}16
5×1045\times 10^{4}105B1.0×10−61.0\times 10^{-6}+7.7×10−2+7.7\times 10^{-2}6
2×1052\times 10^{5}420B9.3×10−79.3\times 10^{-7}+7.8×10−2+7.8\times 10^{-2}9
Appendix
Table 13: Substrate sweep on OLMo-2-1B (full data behind Fig. 3 ): WbaseW_{\text{base}} swept across nine pretraining revisions spanning 0 to 4T tokens, Wsafe=OLMo-2-1B-InstructW_{\text{safe}}=\text{OLMo-2-1B-Instruct} fixed. The random floor for this model is k/d≈2.6×10−6k/d\approx 2.6\times 10^{-6} . Selective divergence goes from noise at 1B to plateau by 63B pretraining tokens, with the first signal at 21B. ℓ⋆\ell^{\star} = the residual-stream position (stream after ℓ⋆\ell^{\star} blocks) with the largest selective divergence; reported as n/a at pre-substrate checkpoints where the curve is flat at noise ( ∼10−4\mathord{\sim}10^{-4} ) and argmax would label a random fluctuation rather than a locus.
Revision∥Δ∥\|\Delta\|mean ∥∇Lcap∥\|\nabla\mathcal{L}_{\text{cap}}\|mean cos⁡(∇Lcap,Δ)\cos(\nabla\mathcal{L}_{\text{cap}},\Delta)mean linear/ ∥Δ∥2\|\Delta\|^{2}
step300-tokens1B12.6912.6920.4620.46−6×10−4-6\times 10^{-4}−7×10−4-7\times 10^{-4}
step10000-tokens21B9.659.6513.6913.69−5×10−4-5\times 10^{-4}−6×10−4-6\times 10^{-4}
step20000-tokens42B9.119.1110.0910.09−1×10−4-1\times 10^{-4}−3×10−5-3\times 10^{-5}
step50000-tokens105B8.878.877.977.97−0\phantom{-}0 to 44 d.p.+3×10−4+3\times 10^{-4}
step1907359-tokens4001B8.128.1229.5629.56−4×10−4-4\times 10^{-4}−6×10−4-6\times 10^{-4}
Appendix
Table 14: Alignment of the safety update with the capability gradient at OLMo developmental-axis revisions. Mean is over the four capability probes; Δ\Delta is an SFT ++ DPO update with the recipe of App. J , from a second, independent run set placed at 21B and 42B to resolve the transition, which shares only the 1B, 105B and 4001B revisions with the runs of Table 18 . Across the entire axis ∣cos⁡(∇Lcap,Δ)∣<0.002|\cos(\nabla\mathcal{L}_{\text{cap}},\Delta)|<0.002 , including the over-trained 40014001 B-token endpoint where ∥∇Lcap∥\|\nabla\mathcal{L}_{\text{cap}}\| peaks at 29.629.6 . Translated into absolute nats, the linear Taylor term (∇Lcap)⊤Δ=cos⁡ ⁣⋅ ⁣∥∇Lcap∥ ⁣⋅ ⁣∥Δ∥(\nabla\mathcal{L}_{\text{cap}})^{\top}\Delta=\cos\!\cdot\!\|\nabla\mathcal{L}_{\text{cap}}\|\!\cdot\!\|\Delta\| is at most ∼ ⁣0.11\sim\!0.11 nats at the pre-substrate step300 revision and ≤0.06\leq 0.06 nats at post-substrate mature revisions: a fraction of 10−310^{-3} of the maximum-aligned upper bound ∥∇Lcap∥ ⁣⋅ ⁣∥Δ∥∈[71,260]\|\nabla\mathcal{L}_{\text{cap}}\|\!\cdot\!\|\Delta\|\in[71,260] , confirming the kernel-induced gradient orthogonality empirically.
Update∥Δ∥\|\Delta\|reasonrecallarithinstruct
refusal-SFT only6.996.99−1.1×10−4-1.1\times 10^{-4}+1.4×10−5+1.4\times 10^{-5}+1.7×10−4+1.7\times 10^{-4}−6.0×10−5-6.0\times 10^{-5}
DPO only0.01550.0155−1.6×10−4-1.6\times 10^{-4}+9.5×10−5+9.5\times 10^{-5}+6.1×10−5+6.1\times 10^{-5}−4.8×10−5-4.8\times 10^{-5}
Appendix
Table 15: Safety-only controls on Qwen-2.5-1.5B: cos⁡(∇Lcap,Δ)\cos(\nabla\mathcal{L}_{\text{cap}},\Delta) per capability probe for a refusal-SFT-only and a DPO-only update, N=128N=128 per-example gradients. Neither update contains general instruction-tuning, so the orthogonality cannot come from it.
RevisionTokensmean ∥∇Lcap∥\|\nabla\mathcal{L}_{\text{cap}}\|reasonarithrecallinstruct
step51211 B119.5\mathbf{119.5}146.7146.7209.9209.998.798.722.522.5
step100022 B96.096.0100.5100.5140.3140.3128.5128.514.914.9
step300066 B38.838.851.551.572.272.222.322.39.19.1
step100002121 B32.632.650.450.451.551.521.021.07.57.5
step300006363 B28.8\mathbf{28.8}31.231.256.156.119.819.88.28.2
step60000126126 B32.732.741.441.459.359.322.422.47.87.8
Appendix
Table 16: ∥∇Lcap∥\|\nabla\mathcal{L}_{\text{cap}}\| across Pythia-1.4B pretraining revisions, mean over the four capability probes. The U-shape from OLMo replicates: minimum at step30000 ( ∼ ⁣63\sim\!63 B tokens, about twice the compute-optimal ratio), large at both extremes.
Revisionmax⁡selective_divℓ\max\text{selective\_div}_{\ell}argmax layer (per seed)
step20000-tokens42B0.043±0.0020.043\pm 0.00216,16,1616,16,16
step40000-tokens84B0.100±0.0080.100\pm 0.0089,9,99,9,9
step60000-tokens126B0.073±0.0070.073\pm 0.0076,6,66,6,6
step70000-tokens147B0.122±0.0090.122\pm 0.0096,6,66,6,6
step80000-tokens168B0.068±0.0100.068\pm 0.0107,7,67,7,6
step90000-tokens189B0.063±0.0050.063\pm 0.0058,8,88,8,8
Appendix
Table 17: Substrate divergence at 33 shuffled-seed runs per revision (mean ±\pm std). Argmax layers reported per-seed, in run order.
  WeakStrongSFT+DPO
StepTokensρ\rhoqnorm\textscarith{q_{\text{norm}}}^{\textsc{arith}}ρ\rhoqnorm\textscarith{q_{\text{norm}}}^{\textsc{arith}}ρ\rhoqnorm\textscarith{q_{\text{norm}}}^{\textsc{arith}}pre-refpost-ref
3×1023\times 10^{2}1B8.1×10−58.1\times 10^{-5}6206209.5×10−59.5\times 10^{-5}8638638.3×10−58.3\times 10^{-5}959521.9%21.9\%0%0\%
3×1043\times 10^{4}63B4.6×10−64.6\times 10^{-6}23236.8×10−66.8\times 10^{-6}34345.8×10−55.8\times 10^{-5}6036030%0\%0%0\%
5×1045\times 10^{4}105B4.3×10−64.3\times 10^{-6}13136.6×10−66.6\times 10^{-6}13138.0×10−58.0\times 10^{-5}2832834.7%4.7\%3.1%3.1\%
5×1055\times 10^{5}1049B7.2×10−67.2\times 10^{-6}33331.2×10−51.2\times 10^{-5}40401.1×10−41.1\times 10^{-4}7707700%0\%0%0\%
1.9×1061.9\times 10^{6}4001B9.8×10−69.8\times 10^{-6}22221.8×10−51.8\times 10^{-5}48485.9×10−55.9\times 10^{-5}52952942.2%\mathbf{42.2\%}15.6%\mathbf{15.6\%}
Appendix
Table 18: Per-revision DPO on OLMo-2-1B under three recipes of increasing strength. qnorm\textscarith{q_{\text{norm}}}^{\textsc{arith}} is the per-recipe layer-max on the arith probe. All rows are in the suppression regime by absolute ρ\rho . Weak/Strong show the under-trained-base sub-pattern: pre-substrate qnorm\textscarith{q_{\text{norm}}}^{\textsc{arith}} ( 620/863620/863 at 11 B) is more than an order of magnitude above any of their post-substrate values ( 1313 to 4848 ). SFT+DPO shows the SFT-stage-recipe sub-pattern instead: at every post-substrate revision ρ\rho stays elevated ( 66 to 19×19\times the Weak recipe’s) and qnorm\textscarith{q_{\text{norm}}}^{\textsc{arith}} stays high ( 283283 to 770770 ), reflecting the SFT-stage NLL gradient projecting onto FcapF_{\text{cap}} ’s top- kk ; at 11 B the recipes coincide in ρ\rho . ρ\rho is the mean over blocks; single blocks reach 7.4×10−47.4\times 10^{-4} (SFT+DPO, 10491049 B). Table 14 uses a second, independent SFT+DPO run set. Behavioral refusal appears only under SFT+DPO; Weak/Strong produce no measurable refusal at any revision (behavioral columns omitted). This is a weight-space geometry table, not the fragility result: the headline fragility claim is on Qwen/Llama (§ 4 ), and the developmental behavioral points that exist show the attack succeeding (SFT+DPO: 21.9%→0%21.9\%\to 0\% at 11 B, 42.2%→15.6%42.2\%\to 15.6\% at 40014001 B; the 105105 B point, 4.7%→3.1%4.7\%\to 3.1\% , is within noise), so the table should not be read as “no successful attack.” Source for the prose summary in § 6 .
    capability NLL
ConditionPre-refPost-refShiftreasonarithrecallinstruct
base0.0%0.0\%0.0%0.0\%00 pp5.345.3410.0110.016.426.425.555.55
cotrain-windowed0.0%0.0\%1.6%1.6\%+1.6+1.6 pp4.784.789.119.116.486.485.615.61
cotrain-continuous84.4%\mathbf{84.4\%}84.4%\mathbf{84.4\%}0.0\mathbf{0.0} pp4.954.958.578.576.396.395.605.60
post-hoc-dpo0.0%0.0\%0.0%0.0\%00 pp5.455.4510.0010.006.426.425.555.55
Appendix
Table 19: From-scratch Pythia-410M, 10B C4 tokens, four conditions (referenced from Sec. 7 ). Refusal rate on 64 AdvBench prompts before and after the benign-SFT attack. This is the original run; the four-condition comparison is valid within it, but for cotrain-continuous the separate from-scratch run scored on 256 prompts gives 98.4→84.098.4\to 84.0 rather than the 00 pp shift here, so the erosion is modest rather than absent. Capability is per-probe NLL, where lower is better : cotrain-continuous improves on the LM-only base on three probes and is within 0.060.06 nats of it on instruct . It is the only condition that installs refusal. This run predates the data-loading fix (App. A ). Capability of the models behind Table 3 is in Table 20 , where arith comes out 1.291.29 nats worse at 410410 M rather than 1.441.44 better, a measure of that probe’s run-to-run spread.
 410M1B2.8B6.9B
LM-only base, NLL
reason4.724.724.554.554.524.524.554.55
arith8.598.597.977.978.348.348.468.46
recall6.106.105.415.415.075.074.954.95
instruct6.086.085.925.925.825.825.635.63
co-training cost, continuous −- base (nats), with 95%95\% interval
Appendix
Table 20: Capability of the co-trained models at every scale: the eight from-scratch checkpoints behind Table 3 , scored on the four capability probes (NLL of the gold answer in nats, natural-log units; lower is better). A cost of cc nats means the co-trained model gives the correct answer e−ce^{-c} times the base’s probability, equivalently ece^{c} times the perplexity: +0.2+0.2 is about 18%18\% less probability, or 22%22\% higher perplexity. Each co-trained model is compared with its LM-only base on the same examples ( 126126 for reason , the whole formal_logic test split; 256256 for the others), with a 95%95\% paired-bootstrap interval over examples; bold marks an interval that excludes zero. Each condition is one training run, so the interval covers which examples were asked, not run-to-run variance.
Conditionmean ρ\rhomean qnorm{q_{\text{norm}}}post-attack refusal
cotrain-continuous (uniform)2.77×10−52.77\times 10^{-5}2.312.3184.4%\mathbf{84.4\%}
cotrain-windowed2.75×10−52.75\times 10^{-5}2.272.271.6%1.6\%
post-hoc-dpo3.01×10−53.01\times 10^{-5}4.014.010.0%0.0\%
Appendix
Table 21: Geometry of the safety direction at 410410 M scale, measured against base . Mean over 9696 (layer, probe) cells. All three conditions sit in the suppression band; behaviorally, only cotrain-continuous resists the benign-SFT attack. Refusal is from the original 6464 -prompt run; the separate scale run of cotrain-continuous reads 84.0%84.0\% at n=256n=256 , and the other two conditions have no scale run.
Scalereasonrecallarithinstruct∥∇Lsafe∥\|\nabla\mathcal{L}_{\text{safe}}\|
410M0.93→0.830.93\to 0.831.62→1.791.62\to 1.790.88→0.960.88\to 0.963.56→12.983.56\to 12.9810.0→3.4×10−410.0\to 3.4{\times}10^{-4}
1B1.53→1.041.53\to 1.042.48→1.982.48\to 1.981.86→1.981.86\to 1.985.81→7.725.81\to 7.724.7→1.1×10−44.7\to 1.1{\times}10^{-4}
2.8B1.35→0.741.35\to 0.741.98→0.431.98\to 0.431.75→0.601.75\to 0.603.56→23.243.56\to 23.242.0→1.5×10−52.0\to 1.5{\times}10^{-5}
6.9B2.97→2.192.97\to 2.193.81→5.203.81\to 5.203.79→3.033.79\to 3.037.05→14.487.05\to 14.481.1→1.0×10−51.1\to 1.0{\times}10^{-5}
Appendix
Table 22: ρsafe\rho_{\text{safe}} ( ×10−3\times 10^{-3} ), LM-only base →\to cotrain-continuous at the same scale, per capability probe: the safety gradient, taken within one model, projected onto that model’s own FcapF_{\text{cap}} . Reported for completeness; the two arms are not comparable. The base’s safety gradient points toward installing refusal, while the co-trained model, having already fit the refusal templates, has a safety gradient 10410^{4} to 10510^{5} times smaller in norm (last column), whose direction carries little information. Absolute values are not comparable to Fig. 1 a’s ρ\rho or to ρpart\rho_{\text{part}} , which are calibrated for a displacement rather than a gradient.
Learning rate200200 steps400400 steps800800 steps
2×10−52\times 10^{-5}90.690.690.690.693.893.8
5×10−55\times 10^{-5}92.292.285.985.993.893.8
1×10−41\times 10^{-4}78.178.175.0\mathbf{75.0}84.484.4
identity-shifting attack100.0100.0
Appendix
Table 23: Stronger attacks on the 11 B cotrain-continuous model: post-attack refusal (%, regex) on 6464 AdvBench prompts, pre-attack 87.5%87.5\% . Benign-SFT on 100100 Alpaca examples at three learning rates and three step counts; the default attack of the main text is 2×10−52\times 10^{-5} for 200200 steps. The identity-shifting attack runs at the default setting.
ConditionPre-attack refusalPost-attack refusalRefusal shift
baseline0.0%0.0\%0.0%0.0\%0.00.0 pp
cotrain-early0.0%0.0\%1.6%1.6\%+1.6+1.6 pp
cotrain-late84.4%\mathbf{84.4\%}15.6%\mathbf{15.6\%}−68.8\mathbf{-68.8} pp
post-hoc-dpo0.0%0.0\%0.0%0.0\%0.00.0 pp
Appendix
Table 24: 160160 M continued-training pilot for completeness. cotrain-early ends with no refusal after the LM-only steps that follow its window; cotrain-late installs refusal that the attack partially erodes; the two DPO-style conditions install no measurable refusal at 160160 M (the 5K-pair DPO recipe does not engage). Refusal on 6464 AdvBench prompts; this pilot has no 256256 -prompt rerun and no main-text counterpart.
 AdvBenchHarmBenchStrongReject
Instruct modelregexclassifierregexclassifierregexclassifier
Qwen-2.5-7B93.0→58.293.0\to 58.293.0→85.293.0\to 85.267.5→45.567.5\to 45.574.0→59.574.0\to 59.552.3→56.652.3\to 56.672.3→73.872.3\to 73.8
Llama-3-8B49.2→10.949.2\to 10.939.5→27.739.5\to 27.746.5→6.546.5\to 6.549.0→28.049.0\to 28.046.5→7.846.5\to 7.853.9→32.453.9\to 32.4
Mistral-7B-v0.31.2→0.41.2\to 0.412.1→6.212.1\to 6.21.5→0.01.5\to 0.07.0→8.57.0\to 8.53.5→0.83.5\to 0.823.0→14.123.0\to 14.1
Appendix
Table 25: Table 1 under both detectors: refusal (%) before →\to after the same 100100 -example benign-SFT attack on the three shipped instruct models. The regex columns are the main-text table. n=256/200/256n=256/200/256 ; the three cells that rise are within noise (App. A ).
 AdvBench (in-dist.)HarmBench (out)StrongReject (out)
Scaleregexclassifierregexclassifierregexclassifier
410M98.4→84.098.4\to 84.091.8→64.891.8\to 64.858.0→34.558.0\to 34.557.0→33.557.0\to 33.58.2→7.48.2\to 7.425.8→21.525.8\to 21.5
1B87.1→84.887.1\to 84.883.2→63.783.2\to 63.756.0→32.056.0\to 32.054.0→25.054.0\to 25.07.8→8.27.8\to 8.223.0→20.323.0\to 20.3
2.8B96.1→90.296.1\to 90.291.0→77.391.0\to 77.345.5→31.545.5\to 31.549.0→37.549.0\to 37.58.6→8.28.6\to 8.231.2→19.531.2\to 19.5
6.9B96.9→90.6\mathbf{96.9\to 90.6}97.3→86.7\mathbf{97.3\to 86.7}32.5→39.032.5\to 39.033.0→41.533.0\to 41.59.0→10.99.0\to 10.929.7→19.529.7\to 19.5
LM-only base
Appendix
Table 26: Table 3 under both detectors: from-scratch cotrain-continuous refusal (%), pre →\to post benign-SFT attack, by scale and prompt distribution. The regex columns are the main-text table. n=256/200/256n=256/200/256 ; changes under about 66 pp are within noise (App. A ). The lower block is the LM-only base at each scale.
Condition (safety window)regexclassifier
cotrain-continuous [0,1][0,1]87.5→90.687.5\to 90.682.8→65.682.8\to 65.6
cotrain-post-substrate [0.30,1][0.30,1]84.4→84.484.4\to 84.485.9→76.685.9\to 76.6
cotrain-windowed [0.10,0.30][0.10,0.30]3.1→4.73.1\to 4.76.2→7.86.2\to 7.8
base (LM only)0.0→0.00.0\to 0.04.7→7.84.7\to 7.8
Appendix
Table 27: The 11 B schedule runs of Table 2 under both detectors: refusal (%) before →\to after the benign-SFT attack, 6464 AdvBench prompts. The regex columns are the main-text table’s 11 B column.
More in this topic

LLM Safety

Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.

See all 421 papers

Explore similar work

CardsList

  1. Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness

    Sep 3, 2026Hoang Cuong Nguyen, Mark Dras, Usman NaseemMechanistic InterpretabilityCircuits in Language Models

    Macquarie University, Sydney, Australia

  2. When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning

    Sep 1, 2026Yitong Guo, Xiaoyi Chen, Siyuan Zhang +2LLM Fine-TuningLLM Safety Alignment

    Indiana University Bloomington · Tsinghua University · Nanyang Technological University

  3. Mood Matters: How Syntactic Sensitivity Undermines Safety Alignment

    Aug 5, 2026Alina Klerings, Jannik Brinkmann, Heiner Stuckenschmidt +1Adversarial Attacks on LLMsLLM Safety Alignment

    University of Mannheim · Technical University Clausthal