cs.LGApr 3, 2026
SaveShifting the Gradient: Understanding How Defensive Training Methods Protect Language Model Integrity
Organizations: Stanford University & MATS · MATS · Goodfire
Abstract
Defensive training methods such as positive preventative steering (PPS) and inoculation prompting (IP) offer surprising results through seemingly similar processes: both add trait-inducing objects to large language models (LLMs) during training, and both defend the LLM against acquiring the trait. The surprising success of these methods comes with the question: how do they work? Are PPS and IP doing the same thing? We provide behavioral and mechanistic comparisons of these two methods using "evilness" as a case-study trait. Our central finding is that PPS and IP achieve their defensive benefits through distinct mechanisms. Behaviorally, we show that neither PPS nor IP operates through a purely associative mechanism; and PPS can both defend against trait acquisition and actively reduce pre-existing expression, whereas IP is ineffective in models that were previously finetuned to express the trait. This behavioral divergence is reflected mechanistically: PPS shifts the activation gradient towards an attenuating direction along the PPS vector axis. When the PPS vector is aligned with a trait-expressing axis, it can reverse the gradient pressure, reducing rather than increasing activation along that axis. In contrast, IP continues to resist a precise mechanistic account. Direct cosine similarity analyses reveal that IP has a characteristically different gradient signature than PPS, and qualitative analyses reveal IP's gradient to be more diffuse. Furthermore, IP reduces the next-token prediction loss on trait-expressing data where PPS need not, consistent with the notion that IP "explains away" the trait-expression in the training data. Taken together, our analyses reveal distinct mechanisms by which each method operates and highlight open questions about IP's mechanistic picture.
Figures & tables
Figure 1: Panels (a) and (b) are visual depictions of the gradient of the loss with respect to the activations, , when training on “evil”-coded data with and without PPS. The black arrows in both panels represent the vector component of a single activation vector along the “evil” persona vector (PV) axis before PPS. The dashed purple arrow represents the addition of the PPS vector (in this case, along the “evil” PV axis). The red and green solid arrows represent the direction of the gradient along the “evil” PV axis without and with PPS in the forward pass. PPS induces suppressive gradient pressure along the PPS vector direction which would otherwise be amplified. Panel (c) shows the mean tokenwise cosine similarity between produced from IP and produced from a form of PPS over all model layers designed to be equivalent to IP (Section 4.4 ). The baseline comparison shows compared to (gradient from the model without IP or PPS). We see that IP is not equivalent to this unique form of PPS—nor other forms (Section 4 ).
Figure 2: PPS and IP have different behavioral effects. Panel (a) shows the raw “evil” trait-expression for models trained without defense (e.g., without PPS or IP). The x-axis shows the “evilness” intensity of the training data. The y-axis shows the training data used to pre-finetune starting models to have varying baseline trait-expression. The None column—equivalent to the None row—shows these starting models’ trait-expression before post-finetuning on the datasets shown on the x-axis. Panel (b) shows the difference in resulting trait-expression between the defenseless post-finetuned models shown in panel (a) and models post-finetuned with PPS and IP respectively. More negative trait expression means more successful defense in the top row and more active trait-expression reduction in the Medium and Heavy rows. Both PPS and IP show defensive effects, and neither meaningfully increase trait-expression, but only PPS shows clear expression reduction. Panel (c) shows the trait expression (y-axis) for PPS and IP trainings on “evil” data using various trait-specific PPS vectors and prompts indicated on the x-axis. Color refers to the defensive method, where gray is defenseless. Base Model simply refers to the starting model. We see that trait-alignment is important for defense. Panels (d), (e), and (f) reveal characteristically different trait-expression of the defenseless, PPS, and IP training methods over the course of training.
Figure 3: PPS causes suppressive gradient pressure along the PPS vector direction. Panels show the effect of defensive objects on the gradients’ cosine similarity with “evil” PV and random vector directions at layer 20, computed on “evil” data. Each point corresponds to a single data sequence averaged over token positions. The x-axis shows the cosine similarity when PPS or IP is included in the forward pass; the y-axis shows the same with the defensive object excluded. Points below the identity (dashed) indicate that the defensive object shifts the gradient away from amplification, towards attenuation along the compared vector direction. Results shown here are taken before training; see Figure 11 for results over training. (a) A random PPS vector (light) in the forward pass induces a greater suppressive gradient shift along the same random direction than PPS using the “evil” PV (dark). (b) PPS using the “evil” PV flips the gradients’ sign along the “evil” PV, whereas PPS using the random vector has little effect. IP shifts the gradient closer to zero along the “evil” PV. (c) Neither “neutral” prompts (light) nor “evil” prompts (dark) show a meaningful gradient shift along a random vector direction. (d) “Evil” IP prompts shift the otherwise amplifying gradient closer to zero along the “evil” PV. Neutral prompts, in contrast, do not.
| Dataset | Defense | Ablation Vector | Default | Attenuate | Amplify | Neutralize |
|---|---|---|---|---|---|---|
| Evil | None | Evil | ||||
| Evil | None | Rand | ||||
| Normal | None | Evil |
Table 1: “Evilness” trait expression (mean SEM) under gradient manipulations applied to all layers during 1-epoch of fine-tuning on evil/normal data. Attenuate forces the gradient component along the Ablation Vector direction to be positive, Amplify forces it negative, and Neutralize removes it entirely. PPS and IP use “evil” PPS vector and inoculation prompts regardless of the ablation vector. See details in Section 4.2 .
Figure 4: IP reduces the loss and PPS vectors can too, but PPS vectors can increase the loss at effective defensive intensities. Panels (a) and (b) show scatter plots where each point shows the next-token prediction loss on a single trait-expressing data sequence averaged over token positions, where PPS or IP is included in the forward pass on the x-axis and excluded on the y-axis. Points that are above the dashed line indicate that defense decreases the loss; points below the line indicate that defense increases the loss. We see in panel (a) that “evil” PPS does not decrease the loss on “evil” data relative to a random vector, whereas “evil” IP does reduce the loss relative to neutral IP in panel (b). Panel (c) shows the loss on the left y-axis and the gradient cosine similarity with the “evil” PV on the right y-axis as a function of steering intensity on the x-axis. The blue lines show results with IP included in the forward pass, whereas orange excludes IP. Baseline Qwen2.5-7B-Instruct is represented at zero intensity without prompts; IP is represented at zero intensity with prompts. We see that prompts reduce the loss and induce a neutralization of the gradient cosine along the “evil” axis. PPS also reduces the loss at low intensities, but increases the loss at larger, more effective intensities. Panel (d) shows the corresponding trait expression (y-axis) of PPS trained models resulting from steering intensities indicated on the x-axis. Taken with Panel (c), we see that PPS effectiveness increases with steering intensity regardless of the loss.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Gradient cosine similarity predicts cross-trait PPS effectiveness. Panels (a)–(d) show the cosine similarity between the gradient and different vectors at each model layer, before the PPS finetuning. Color denotes the steering intensity of the PPS vector applied in the forward pass, where black shows the defenseless baseline and dashed green denotes the default steering intensity of 1.5. The top panels (a, b) use “evil” data; the bottom panels (c, d) use “sycophantic” data. The left panels (a, c) apply the “evil” PPS vector in the forward pass; the right panels (b, d) apply the “sycophantic” PPS vector. The “evil” PPS vector induces strong positive (attenuating) cosines with both the “evil” and “sycophantic” PVs, whereas the “sycophantic” PPS vector only induces a positive cosine in the “sycophantic” PV direction, not the “evil” PV direction. This asymmetry predicts the behavioral results: the evil vector defends against both traits, while the sycophantic vector only defends against sycophancy. Panels (e) and (f) show the residual stream activations projected onto the evil PV (e) and sycophantic PV (f) after defensive finetuning under each cross-trait condition. Dashed lines show the raw projections with the steering vector included in the forward pass, applied at layer 20. Activation reductions along each trait direction track the behavioral trait-expression scores (shown in the adjacent tables), confirming that the pre-training gradient cosine analysis is predictive of post-training outcomes.
Figure 6: Raw activation projections onto “evil” PPS axis show that PPS-trained models reduce “evil” activity whereas IP models do not.
Figure 7: Results of Figure 2 hold even with system prompt change oddity. This figure is similar to Figure 2 except that the trait-expression was evaluated using a different system prompt than the one used during training. Each value in the matrices shown in Panels (a), (b), and (c) shows the “evil” trait-expression score of models under different training conditions. The x-axis shows the evilness intensity of the training data and the y-axis shows the starting model condition (Qwen2.5-7B-Instruct pre-finetuned without defense on the same datasets shown on the x-axis). The None column shows the starting models’ trait-expression before the defensive finetuning which is equivalent to the None row in the defenseless trainings. Panel (a) shows results without defensive methods, (b) shows results for PPS using an “evil” persona vector, (c) shows results for inoculation prompting using “evil”-inducing prompts. Panel (d) shows the trait expression for various training conditions for both PPS and inoculation prompting. The x-axis refers to the defensive object used for the training. Color refers to the defensive method, where gray is defenseless (Base Model simply refers to the starting model: Qwen2.5-7B-Instruct). Note that these results are taken from different evaluation runs than panels (a), (b), and (c), giving some insight into the evaluation variability. Panels (e), (f), and (g) reveal characteristically different trait-expression of the defenseless, PPS, and inoculation prompting training methods respectively over the course of training, starting from Qwen2.5-7B-Instruct, trained on heavily evil data.
Figure 8: More IP prompts leads to worse defense. The y-axis shows “evil” trait-expression for models trained with IP using different numbers of sampled prompts during training. The x-axis shows the number of prompts in the set from which prompts are sampled from for each training sample, where prompts are freshly sampled for each training batch. Fewer prompts leads to lower trait-acquisition for the same training settings.
Figure 9: The cosine similarity between the gradient and the evil persona vector across layers with PPS applied in the forward pass at different layers (denoted by hue). The steering intensity was 1.5.
Figure 10: The cosine similarity between the gradient and the evil persona vector across layers with and without the presence of inoculation prompting.
Figure 11: Gradient cosine analysis along the “evil” PV axis over the course of training. The y-axis shows the cosine similarity between the gradient induced by Defenseless (base) Qwen2.5-7B-Instruct, PPS, and IP along the “evil” PV over the course of each method’s training in the respective panels.
Figure 12: Each panel shows the tokenwise cosine similarity between the gradient and the activation delta in black across layers. Each panel also shows the cosine similarity between the gradient and the delta with a persona vector direction in blue and red respectively. See Appx D for more detail.
Figure 13: Each panel shows the cosine similarity between the and the evil persona vector at a specified layer where the delta was computed from a gradient descent step without PPS on the x-axis and with PPS on the y-axis (defense was not included for the calculation, only for determining the updated weights ). See Appx D for more detail.
Figure 14: Each panel shows the cosine similarity between the and the evil persona vector at a specified layer averaged over the sequence dimension where the delta was computed from a gradient descent step without inoculation prompting on the x-axis and with prompting on the y-axis (defense was not included for the calculation, only for determining the updated weights ). Suffix matching indicates that only the tokens following the inoculation prompts were included in the visualized values. See Appx D for more detail.
| Defense | Ablation Vector | Default | Attenuate | Amplify | Neutralize |
|---|---|---|---|---|---|
| None | Evil | ||||
| PPS | Evil | ||||
| IP | Evil | ||||
| None | Rand | ||||
| PPS | Rand | ||||
| IP | Rand |
Table 2: “Evilness” trait expression (mean SEM) under gradient manipulations applied to all layers during 1-epoch of fine-tuning on evil data. Attenuate forces the gradient component along the Ablation Vector direction to be positive, Amplify forces it negative, and Neutralize removes it entirely. PPS and IP use “evil” PPS vector and inoculation prompts regardless of the ablation vector. See details in Section 4.2 .
| Defense | Ablation Vector | Default | Atten. | Ampl. |
|---|---|---|---|---|
| None | Evil | |||
| PPS | Evil | |||
| IP | Evil |
Table 3: Peak “evilness” trait expression (mean SEM) under gradient manipulations applied to all layers fine-tuning on “normal” (non-evil) data. The reported values are the peak evil-expression over training with at least a coherence of 20 (coherence drops to zero over training). Attenuate forces the gradient component along the Ablation Vector direction to be positive, Amplify forces it negative, and Default does not manipulate the gradient. PPS and IP use “evil” PPS vector and inoculation prompts. See details in Section 4.2 .
Figure 15: Loss curves corresponding to all 3 training epochs when manually manipulating the gradients. We show trait-expression results for models taken at the vertical line (epoch 1). Otherwise, coherence goes to 0.
Figure 16: The cosine similarity between the gradients induced by IP compared to the gradients induced by activation difference PPS—using the difference in activations with minus without inoculation prompts, preserving layers—as a function of steering intensity (applied equally at all layers; Section 4.4 ). Both panels show the same information sliced differently. The left panel shows steering intensities on the x-axis, where hue denotes the cosine similarity for select layers. The right panel shows the same cosine similarities across layers, using hue to denote the steering intensity.
Figure 17: Cosine similarity (y-axis) measurements across model layers (x-axis) between the “evil” PVs and the prompt-induced mean activation difference vectors from Section 4.4 . See Appx E.6 for details on how the vectors are constructed.
Figure 18: Uncentered PCA decomposition of the defended gradient difference reveals heavily concentrated, trait-aligned structure for PPS and more diffuse, less aligned structure for IP. Panels (a) and (b) show cosine similarity matrices between reference trait vectors (x-axis) and the the top PCs (y-axis) extracted from PCA performed on the difference between the gradient with and without the defensive object in the forward pass. We perform the PCA on the uncentered data to preserve potential common differences among all samples. Each PC is oriented so that it has a positive cosine similarity with the “evil” trait vector. Panel (c) shows the fraction of variance explained by each of the PCs shown in panels (a) and (b). We see that the explained variance for PPS is heavily concentrated on the first PC, whereas the explained variance for IP is more diffuse.
Figure 19: Trait expression and coherence scores as evaluated by GPT 4.1 for finetunings with different LoRA ranks (and LoRA alphas) and for different inoculation prompt placements. These results were collected with the default Qwen system prompt during evaluation, whereas the defensive trainings used a neutral system prompt.
Figure 20: Trait expression and coherence scores as evaluated by GPT 4.1 for finetunings with different LoRA ranks (and LoRA alphas) in PPS trainings. These results were collected with the default Qwen system prompt during evaluation, whereas the defensive trainings used a neutral system prompt. This is why there is a trait-expression discrepancy between the models shown here, and the models shown in the main text.
Figure 21: Evil trait expression and coherence across a range of PPS steering intensities at layer 20 with rank 64.
Figure 22: Evil trait expression and coherence across a range of model layers, using steering intensity 1.5 and rank 64.
More in this topic
Adversarial Robustness of Language Models
Adversarial robustness of language models studies how to keep their outputs and decisions reliable under manipulative prompts, input changes, and malicious fine-tuning.
See all 192 papers