cs.AIOct 12, 2026

The Sharp Directions Are Against You: Curvature Analysis of Activation Steering

Authors: Seyedarmin Azizi, Arya Fayyazi, Parsa Razmara, Erfan Baghaei Potraghloo, Souvik Kundu, Massoud Pedram

Organizations: University of Southern California, Los Angeles, USA · Intel AI, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Activation steering adds a vector to a language model's internal representations at inference time. It is a lightweight alternative to fine-tuning for behavioral control. The standard construction, contrastive activation addition (CAA), is fragile, as on many inputs it shifts behavior in the wrong direction. To explain this fragility, we first study the geometry of the steering vector. We introduce steering Hessian, the matrix of second order derivative of the model's behavioral loss with respect to the steering vector. It captures the output behavior response sensitivity of the model under small perturbations of the steering vector along each direction. Its top eigenvectors are sharp directions: tiny perturbations along them can cause large, erratic, input-dependent behavioral swings. Bottom eigenvectors are flat directions: behavior is robust along them. Removing the sharp component of a steering vector approximately preserves its direction and magnitude (cosine > 0.974, norm reduction < 3% for typical settings) while reducing the vector's directional curvature by up to 12x (median 4x across 36 conditions). We evaluate on three model families (Qwen2.5-7B, Llama-3.1-8B, Mistral-7B), with four behavioral tasks, and 36 token-level conditions. Steering along sharp directions reverses the intended behavior in every condition tested, with anti-steer rates of 53-100%. Sharp directions account for only 1-5% of the CAA vector's squared norm. However, they drive a disproportionate share of the gap between the steered and unsteered next-token distributions during autoregressive generation. We term this one-line correction as sharp removal; it outperforms standard CAA in open-ended generation across all 25 conditions tested. A jailbreaking experiment further shows that sharp directions act as noise rather than signal: they disrupt coherent behavioral control regardless of steering polarity.

Figures & tables

Figure 1: Overview of the steering Hessian framework. (a) The behavioral loss landscape is anisotropic in activation space. The CAA vector (gray) has a small component along the sharp eigenvectors (red dashed). Removing this component gives vcleaned\mathbf{v}_{\text{cleaned}} (green), which avoids the high-curvature region. (b) Real eigenspectrum from Llama-3.1-8B, sycophancy, layer 16: eigenvalues span three orders of magnitude ( κ=3727\kappa=3727 ). The top- kk sharp eigenvectors (red, Sk\mathcal{S}_{k} ) are the components removed from CAA. (c) Behavioral shift Δ\Delta across the α\alpha grid (Qwen2.5-7B, sentiment, layer 21): vflat\mathbf{v}_{\text{flat}} has a 0% anti-steer rate at every α\alpha , confirming the geometric structure. vsharp\mathbf{v}_{\text{sharp}} reverses behavior. (d) The full pipeline. Sharp removal (step 4, highlighted) is a one-line operation after Hessian computation.
  Condition number κ\kappaAnti-steer rate of vsharp\mathbf{v}_{\text{sharp}}
ModelTaskEarlyMidLateEarlyMidLate
QwenSentiment63 (L7)50 (L14)70 (L21)0.690.740.77
Sycophancy181 (L10)71 (L14)250 (L16)0.530.840.96
Toxicity51 (L10)148 (L16)51 (L22)0.690.721.00
Refusal673 (L10)182 (L16)143 (L20)0.990.970.99
LlamaSentiment94 (L8)58 (L16)1279 (L24)0.880.790.76
Table 1: Left: Condition number κ=∣λmax⁡∣/∣λmin⁡∣\kappa=|\lambda_{\max}|/|\lambda_{\min}| of the steering Hessian. Every value exceeds 50. Right: Anti-steer rate of vsharp\mathbf{v}_{\text{sharp}} , averaged over all α\alpha . Every entry exceeds 0.50; bold marks values ≥0.95\geq 0.95 .
Figure 2: Per-position KL divergence between the steered and unsteered next-token distributions ( α=3.0\alpha{=}3.0 , N=30N{=}30 prompts, 5-point moving average). (a) On Qwen, CAA’s KL roughly doubles over the sequence while sharp removal grows more slowly; the gap widens at later positions. (b) On Llama, CAA and vsharp\mathbf{v}_{\text{sharp}} produce nearly identical trajectories ( ∼\sim 19 nats); sharp removal reduces divergence by 3–4 ×\times . On Mistral L14 (not shown), CAA’s KL grows 3.8×3.8\times over 100 tokens, vs 2.5×2.5\times for sharp removal.
Figure 3: Generation-level evaluation across three tasks. Sharp removal (green) outperforms CAA (red) in all 25 conditions. Dashed lines are per-model baselines. (a) Sentiment: sharp removal reaches near-ceiling positive sentiment across all layers. (b) Sycophancy: on a task where CAA is inverted at the token level, sharp removal consistently reduces honesty degradation. (c) Refusal: CAA actively weakens safety alignment (Qwen drops from 78% to 36–44%; Llama from 62% to 24–52%); sharp removal restores and in most cases exceeds baseline refusal rates (Qwen: 88–98%; Llama: 74–76%). Labels: Q=Qwen, L=Llama, M=Mistral; the number is the layer index.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

ModelTaskEarlyMidLate
QwenSentiment2.6 (L7)2.5 (L14)1.7 (L21)
Sycophancy4.1 (L10)2.5 (L14)4.0 (L16)
Toxicity3.0 (L10)3.0 (L16)2.5 (L22)
Refusal8.9 (L10)5.9 (L16)3.7 (L20)
LlamaSentiment3.2 (L8)2.3 (L16)2.8 (L24)
Sycophancy7.0 (L10)6.0 (L14)5.4 (L16)
Appendix
Table 2: Curvature reduction factor κgap=∣λπ(1)∣/∣λπ(16)∣\kappa_{\text{gap}}=|\lambda_{\pi(1)}|/|\lambda_{\pi(16)}| at k=15k{=}15 . Eigenvalues are sorted by absolute value. Larger values indicate a stronger spectral gap above the sharp subspace and therefore a larger curvature reduction from sharp removal. Across all 36 conditions, κgap\kappa_{\text{gap}} ranges from 1.7 to 12.3 (median 4.0). Values do not depend on α\alpha since Hsteer∝α2\mathbf{H}_{\text{steer}}\propto\alpha^{2} .
TaskQwenLlamaMistralDepth pattern
SentimentL21 (0%)L16 (50%)L15 (47%)Mid-to-late
SycophancyL14 (50%)L14 (25%)L19 (59%)Mid
ToxicityL10 (36%)L20 (63%)L21 (66%)Varies
RefusalL20 (71%)L10 (31%)L8 (25%)Varies
Appendix
Table 3: Best layer for vflat\mathbf{v}_{\text{flat}} per task, reported as (layer index, anti-steer rate at that layer).
Figure 4: Qualitative refusal examples (Mistral-7B-Instruct-v0.3, Layer 12, α=5.0\alpha{=}5.0 ). Mistral-7B has limited safety training (baseline refusal rate: 12%). Dark red : the unsteered model complies with harmful requests, producing actionable content including step-by-step hacking instructions, a romance scam script, a phishing email template, and a fraud guide. This is consistent with the competing-objectives failure mode [ Wei et al., 2023 ] , where helpfulness training overrides safety. Light red : CAA-steered responses add warnings but still partially engage with the harmful request. Green : sharp removal introduces explicit refusal or redirects to legitimate alternatives, increasing the overall refusal rate from 12% to 62%. This demonstrates that Hessian-informed steering can strengthen safety alignment even when the base model’s own alignment is limited [ Qi et al., 2025 ] . Responses truncated for space.
ModelConditionComply (%)Partial (%)Refuse (%)
Qwen (L10)Baseline2494
−vCAA-\mathbf{v}_{\text{CAA}}561224
−vsharp-\mathbf{v}_{\text{sharp}}24058
−vcleaned-\mathbf{v}_{\text{cleaned}}661420
Llama (L15)Baseline2692
−vCAA-\mathbf{v}_{\text{CAA}}23167
Appendix
Table 4: Jailbreak effectiveness across two models ( α=3.0\alpha{=}3.0 , GPT-4o-mini judge, N=50N{=}50 Qwen, N=100N{=}100 Llama). −vsharp-\mathbf{v}_{\text{sharp}} is the least effective jailbreak on both models: on Llama it produces 100% refusal (safer than baseline). −vcleaned-\mathbf{v}_{\text{cleaned}} is the most effective. Qualitative examples in Appendix M .
Figure 5: Qualitative jailbreaking examples: Qwen2.5-7B-Instruct (Layer 10, α=3.0\alpha{=}3.0 , negative-direction steering). Green : baseline cleanly refuses. Orange : −vsharp-\mathbf{v}_{\text{sharp}} produces incoherent responses that waver between refusal and engagement. Red : −vcleaned-\mathbf{v}_{\text{cleaned}} produces coherent compliance. Responses truncated for space.
Figure 6: Qualitative jailbreaking examples: Llama-3.1-8B-Instruct (Layer 15, α=3.0\alpha{=}3.0 , negative-direction steering). The −vsharp-\mathbf{v}_{\text{sharp}} responses exhibit a distinctive repetition loop failure mode: the model collapses into repeating its refusal phrase verbatim, producing degenerate text. This is a hallmark of high-curvature noise disrupting the autoregressive generation process—the sharp component destabilizes the hidden state so severely that the model cannot sustain coherent output, defaulting to its most probable token sequence (the refusal phrase) on repeat. In contrast, −vcleaned-\mathbf{v}_{\text{cleaned}} produces coherent responses that partially or fully engage with the request. This confirms Finding 5 across model families.
Figure 7: Sensitivity to sharp subspace dimension kk . (a) On Qwen refusal (high κ\kappa ), k=1k{=}1 more than doubles CAA’s refusal rate; the curve plateaus by k=5k{=}5 . (b) On Llama sycophancy (weaker signal), sharp removal peaks at k=5k{=}5 and degrades at k=25k{=}25 from over-removal. Together, these motivate k=10k{=}10 – 1515 as the default.
More in this topic

Language Model Steering

Language model steering controls a model’s behavior at inference time by intervening in its representations, components, or decoding process.

See all 281 papers

Explore similar work

CardsList
Sep 7, 2026

Steering Interference Reflects the Model's Defaults, Not the Behavior Directions

Activation steering promises modular control of language model behavior: a behavior such as politeness corresponds to a direction in a model's activations, and adding that direction while it generates should switch the behavior on and leave everything else alone. It does not. We ask what decides which other behaviors move, and by how much, and find that it is the model rather than the behavior being steered. A steer relaxes the model toward a small set of behaviors it already favors, chiefly refusal, sycophancy, and poeticism, and that set is much the same whatever is steered. Three results across 24 behaviors and ten instruction-tuned models support this, every effect read off the generated text by a language-model judge rather than off a probe. That readout matters: all 24 behaviors are linearly decodable, but only 20 change what the model writes. First, a direction carrying no behavioral content, matched to a real steer only in the size of the vector it adds, moves the same behaviors in the same order as real steers do, while producing none of the behaviors that need a specific direction. Second, most interference runs one way, so it cannot be an overlap between two directions: steering profanity makes the model toxic, while steering toxicity leaves profanity untouched. Third, with a behavior held out entirely, geometry measured on the others explains almost none of the interference it takes part in. The account holds on all ten models, the pull toward defaults strongest below 10B parameters and weakening in each family's largest. Reading a steer as a perturbation whose endpoint the model fixes implies that disentangling behavior directions cannot by itself make steering modular.

Srikanth Malla, Chiho Choi, Joon Hee Choi

Samsung Semiconductor US

Language Model SteeringCausal Interventions in Language ModelsActivation Steering
AskPDF
Jun 7, 2026

Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior

Activation steering provides a lightweight inference-time mechanism for controlling large language models (LLMs) by modifying their internal activation vectors toward desired behaviors. Most existing methods compute a fixed steering direction in the original activation space, typically from pairs of contrastive examples using mean differences, linear probes, or arbitrary separability criteria. While effective to a certain extent, these methods treat behavioral control as a global, linear, additive offset: the same direction is applied across inputs, and behaviors are linearly separable. This can be restrictive when behavioral features vary nonlinearly across the activation space or lie on curved and anisotropic manifolds, where the optimal intervention may be input-dependent. To address this limitation, we propose INNSteer, a nonlinear activation steering framework based on invertible latent transformations. Rather than searching for a better steering vector in the original representation space, INNSteer learns a lightweight invertible neural network φφ that maps an LLM's activations into a latent space where behavioral classes are more amenable to linear control. At inference time, activations are mapped through φφ, steered in the latent space, and mapped back through the exact inverse transformation φ−1φ^{-1}. This makes a simple latent-space translation become a nonlinear, input-dependent intervention in the original activation space. Across experiment settings on multiple LLM families, scales, behavioral traits, and safety benchmarks, INNSteer consistently improves model control over linear, transport-based, and nonlinear steering baselines while largely preserving generation fluency.

Tuc Nguyen, Thai Le

Indiana University Bloomington, USA

Invertible Neural NetworksLanguage Model SteeringActivation Steering
AskPDF
Sep 12, 2026

Signatures of Steerability in Activation Space of Language Models

Steering language models using a set of contrastive representations has been a canonical and computationally efficient method for controlling model behavior. Despite this success in controlling certain model behaviors, the effectiveness of activation steering varies markedly across concepts; the generalization properties of steering vectors are often considered a function of the dataset used to construct them. We make this dataset-dependence claim more rigorous and show that simple separation metrics strongly correlate with the downstream steerability of language models across diverse settings, even after controlling for layers and dataset effects. Beyond prediction, we provide evidence from a synthetic superposition experiment that separation metrics are strongly correlated with alignment between the empirical and true feature direction. Our results suggest that simple separability statistics can serve as practical diagnostics for when steering vectors are likely to work.

Prajjwal Bhattarai, Tuka Alhanai

New York University Abu Dhabi, Abu Dhabi, UAE · New York University, Tandon School of Engineering, Brooklyn, NY, USA

Language Model SteeringActivation SteeringRepresentation Geometry in Language Models
AskPDF