Understanding Unreliability of Steering Vectors in Language Models: Geometric Predictors and the Limits of Linear Approximations

Joschka Braun

Property	Examples
Sentiment	How are you doing? Not too bad actually.
Toxicity	He called me an idiot , but he was just joking.
Bias	They hired her because she’s a woman .
Tense	He will go to the meeting, I go to every week.
Truthfulness	The world is flat .
Lexical Ambiguity	Words such as ring , lie , spring , leaves , and bank exhibit polysemy, possessing multiple distinct meanings contingent upon their contextual usage.

Table 1: Examples where static word embeddings lack crucial context information to adequately represent a text property. The local context could disambiguate the text properties and make contextual embeddings superior to static embeddings in such cases.

Figure 1: The hypothetical representation space illustrates how features like gender and royalty can be represented as linear directions. The representation of “king” for instance, can be decomposed into its components along the “male” and “royal” direction within this subspace. This linear structure enables vector arithmetic operations such as analogies (e.g., “king” - “man” + “woman”

\approx

“queen”) through vector addition and scalar multiplication. Such representation vector arithmetics, using linearly encoded features, were first demonstrated by Mikolov et al. (2013a) in their work on Word2Vec

Figure 2: Linear scaling of sentiment and object size along their respective feature directions.

Figure 3: Because of feature entanglement in (a), changing the gender also changes the professional position. For disentangled features, gender and professional positions can be varied independently.

Figure 4: Illustrating CAA steering vector computation in a favorable scenario: A 2D projection at layer

l

shows distinct positive and negative activation clusters. This allows the steering vector

\mathbf{s}^{l}

to accurately approximate individual activation differences. The means of positive and negative activations are

\boldsymbol{\mu}^{l,+}=\frac{1}{|\mathcal{D}_{\text{train}}|}\sum\mathbf{a}^{l}(x_{i},y^{+}_{i})

and

\boldsymbol{\mu}^{l,-}=\frac{1}{|\mathcal{D}_{\text{train}}|}\sum\mathbf{a}^{l}(x_{i},y^{-}_{i})

, respectively. The steering vector can be computed as the difference between these means

\mathbf{s}^{l}=\boldsymbol{\mu}^{l,+}-\boldsymbol{\mu}^{l,-}

, which is equivalent to the previous definition as the mean of paired activation differences.

Figure 5: Mean cosine similarity between the activation differences and the resulting steering vector provides a concise metric to quantify how well, on average, the steering vector aligns directionally with each individual activation differences. The directional agreement metric captures whether the activation differences point into the same direction and are well directionally approximated by a common steering vector. While the computation of the steering vector itself is insensitive to how activations are paired, the mean cosine similarity metric remains sensitive to those pairings.

Figure 6: In (a) and (b), the positive and negative activations are shown in a 2D subspace at the top. The bottom shows the same activations projected onto the difference-of-means line. The d’ values are the discriminability index, explained in Section 4.9.1 , which measures separability along the difference-of-means line (high values mean high separability).

Figure 7: Convergence of steering vector cosine similarity for increasing sample size. The plot shows the mean cosine similarity between steering vectors computed from subsets of activation differences and a reference steering vector derived from 500 samples. Each line represents one of 36 datasets, with error bars indicating standard deviation over 25 trials. While cosine similarity generally increases with sample size, convergence rates vary notably between datasets. Most datasets achieve a cosine similarity greater than 0.9 for 100 samples and greater than 0.95 for 150 samples, suggesting that approximately 150 samples are sufficient for most datasets to construct a stable steering vector. As stated in 4.3 , I use 250 training samples per steering vector by default.

Figure 8: I group the 36 datasets by how effective the resulting steering vector is (“steerability rank”). The steerability rank is determined on a set of 500 prompts, that are different from the 250 prompt pairs used to train the dataset-specific steering vectors. The most steerable group (ranks 1-6) exhibit high directional agreement between the individual activation differences and the steering vectors, whereas directions in the least steerable group (ranks 31-36) are more dispersed or even orthogonal. I measure directional agreement as the cosine similarity between the 250 activation differences and their resulting steering vector. Conceptually, high directional agreement suggests a coherent linear representation of the behavior.

Figure 9: The same experiment results as Figure 8 , but with a focus on correlation between the measure of directional agreement and the three metrics for steerability. The mean cosine similarity between individual activation differences and the resulting steering vector correlates statistically significantly, as measured by the Spearman’s correlation, with the steerability rank, mean per-sample steerability and the fraction of anti-steerable samples. This suggests that directional agreement is a predictor for steerability.

Figure 10: Steering vector convergence rates correlate with dataset steerability. The plot displays the mean cosine similarity between steering vectors derived from subsets of activation differences and a reference steering vector, computed from 500 samples. Each of the 36 lines represents a unique dataset, color-coded by its steerability rank (green: most steerable, low rank; red: least steerable, high rank), with error bars indicating standard deviation over 25 trials. The visualization generally shows that more steerable datasets (green/yellow lines) tend to exhibit faster convergence and maintain higher cosine similarities even at smaller sample sizes, supporting the idea that higher internal directional agreement in activations contributes to more rapid stabilization of the steering vector.

Figure 11: Subplot (a) shows the baseline distributions, which are broadly symmetric and unimodal, with higher means and variances for more reliable (low steerability rank) datasets. Subplot (b) demonstrates that normalization by the steering vector norm effectively distinguishes datasets by steerability, linking higher normalized values to directional disagreement. Subplot (c) illustrates that normalization by the dataset-specific mean activation difference norm collapses all dataset distributions to mean of 1 with similar shape and variance, regardless of steerability rank.

Figure 12: For datasets where the behavior is steerable, activations are clearly separated along the difference-of-activation-means line (top). Less steerable datasets have overlapping positive and negative activations (bottom). The CAA steering vector is added to the negative activations to shift them along the difference-of-means line, such that their mean matches the positive mean. Negative and positive activation distributions have similar shape, each is shown for 500 samples.

Figure 13: Correlation between the discriminability along the difference-of-means line with the three measures of steering success across all 36 MWE datasets. As the representative examples in Figure 12 showcase, the discriminability is significantly correlated as measured by the Spearman’s correlation, with the steerability rank, mean per-sample steerability and the fraction of anti-steerable samples. This suggests that separability along the steering direction is a predictor for steerability.

Figure 14: Correlation between discriminability metric d’ with other separability metrics. d’ correlates significantly with AUROC, Kolmogorov-Smirnov Statistic and Overlap Coefficient.

Figure 15: For a highly steerable dataset (top), positive and negative activations form distinct, well-separated distributions. For a less steerable dataset (bottom), the distributions are closer but still fully separated, resulting in a lower discriminability index (d’). Unlike for the difference-of-means projection, there is no overlap for less steerable datasets, yet the separability remains quantitatively lower. Applying the CAA steering vector shifts the negative activations to completely overlap with the positive ones along this component.

Figure 16: Separability of activations projected onto the direction learned by a logistic regression classifier. Similar to other projections, highly steerable datasets (top) exhibit clear separation between positive and negative activation distributions. Less steerable datasets (bottom) show reduced separation and a lower discriminability index (d’), though with minimal distributional overlap. Adding the CAA steering vector to the negative activations causes them to align with the positive activation distribution along the classifier’s direction.

Figure 17: Correlation of the discriminability index (d’) across three different projection methods. The plots show the Spearman’s rank correlation between the d’ values calculated for all 36 datasets when activations are projected onto the difference-of-means line, the first LDA component, and the direction learned by a logistic regression classifier. Separability measured along the difference-of-means line and the logistic regression direction are significantly correlated (p<0.05), as are the LDA and logistic regression directions. However, the correlation between the difference-of-means and LDA projections is not statistically significant.

Figure 18: Steering vectors trained with different prompt types all increase the mean logit-difference relative to no steering and perform similarly across datasets. Yet, for all prompt types, steering effect size is unreliable, with a significant fraction of the test samples shifted in the opposite direction (“anti-steerable”). I use 250 training samples and 500 evaluation samples for each combination of prompt type and dataset.

Figure 19: Steering vectors trained with different prompt types all increase the mean logit-difference relative to no steering and perform similarly across datasets. Yet, for all prompt types, steering effect size is unreliable, with 29% - 43% of all samples shifted in the opposite direction. Both steering effect size and faction of such anti-steerable samples vary substantially between datasets, as shown by the six most steerable datasets (top row) outperforming those in the middle row (average) and the bottom row (six least steerable datasets). For the six least steerable datasets the mean logit difference compared to no steering is negative for some prompt types and the fraction of anti-steerable samples around half of all samples. Conversely, the six most steerable datasets have mean effect sizes between 2.12 and 3.15 and only between 22.5% and 9.5% anti-steerable samples. I use 250 steering vector training samples and 500 evaluation samples for each combination of prompt type and dataset.

Figure 20: Steering vectors trained on the same datasets but with different prompt types have cosine similarities ranging from 0.07 to 0.86. Steering vectors trained with similar prompt types have higher cosine similarity than for different prompt types. Cosine similarities between steering vectors from prefilled prompts range from 0.25 to 0.86. Cosine similarities between steering vectors from non-prefilled prompts range from 0.32 and 0.44. One straightforward reason for why prefilled and non-prefilled activation differences are not similar is because generating an answer token (A/B, Yes/No) requires different computations/representations than generating the token after the answer token. Very similar prompts (prefilled 5-shot, prefilled instruction and prefilled instruction 5-shot) have comparatively high cosine similarities (0.61 to 0.86). The ranking counts for prompt types show that no single prompt type is systematically better than the others, if compared by their dataset wise mean logit-difference.

Figure 21: Steering efficacy is correlated across (a) datasets and (b) individual samples. Overall, the correlation of steerability across datasets is high, for many prompt types with correlation > 0.8. As in Figure 20(a) , the prompts that combine multiple behavior elicitation methods have the highest similarity to each other. Figure 21(b) shows that correlation across individual samples is much more mixed, with the 5-shot prompt even having negative Spearman rank correlation coefficient with many other prompt types.

Figure 22: Prompt types that leverage in-context learning by prepending an instruction or few-shot examples lead to higher directional agreement between the individual training activation differences. This results in a much faster convergence of steering vectors trained on subsets to the full-data reference steering vector. The convergence speed is highest for prompt types that don’t prefill answer tokens, like “instruction” and “instruction and 5-shot”. Their mean directional agreement between steering vectors trained on only 10 paired samples is > 0.95 for most datasets. Prefilling the answer token on the other hand generally decreases convergence speed when comparing the results for the non-prefilled prompt types in Figure 22(b) to same prompt types but with prefilled answer tokens in Figure 22(c) .

Symbol	Explanation
$[\mathbb{N}]:=\{1,\ldots,\mathbb{N}\}$ set of integers	1, 2, …, $\mathbb{N}-1,\mathbb{N}$
$i,j\in\mathbb{N}$	generic integer indices
$V\subseteq[\mathbb{N}_{V}]$	vocabulary
$\mathbb{N}_{V}\in\mathbb{N}$	vocabulary size
$V^{*}=\bigcup_{\tau=0}^{\infty}V^{\tau}$ set	set of token sequences; elements include, e.g., sentences or documents
$\tau_{\text{max}}\in\mathbb{N}$	maximum sequence length

rank	prefilled	instruction	5-shot	prefilled instruction	prefilled 5-shot	instruction 5-shot	prefilled instruction 5-shot
1	$0.90\pm 0.09$	$0.99\pm 0.00$	$0.99\pm 0.01$	$0.92\pm 0.08$	$0.93\pm 0.06$	$0.99\pm 0.00$	$0.93\pm 0.07$
2	$0.91\pm 0.07$	$0.98\pm 0.01$	$0.99\pm 0.00$	$0.91\pm 0.07$	$0.92\pm 0.06$	$0.99\pm 0.00$	$0.92\pm 0.05$
3	$0.89\pm 0.08$	$0.98\pm 0.01$	$0.98\pm 0.02$	$0.91\pm 0.06$	$0.92\pm 0.06$	$0.98\pm 0.01$	$0.91\pm 0.07$
4	$0.91\pm 0.06$	$0.99\pm 0.00$	$0.98\pm 0.01$	$0.91\pm 0.07$	$0.91\pm 0.06$	$0.99\pm 0.01$	$0.91\pm 0.07$
5	$0.90\pm 0.09$	$0.98\pm 0.01$	$0.98\pm 0.02$	$0.89\pm 0.08$	$0.89\pm 0.09$	$0.99\pm 0.00$	$0.91\pm 0.08$
6	$0.84\pm 0.18$	$0.99\pm 0.01$	$0.97\pm 0.03$	$0.90\pm 0.11$	$0.91\pm 0.10$	$0.96\pm 0.05$	$0.93\pm 0.07$

Understanding Unreliability of Steering Vectors in Language Models: Geometric Predictors and the Limits of Linear Approximations

Abstract

Figures & tables

Appendix

Language Model Steering

Signatures of Steerability in Activation Space of Language Models

What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal

Disentangling Steering Vectors

Abstract

Language Model Steering

Explore similar work

Signatures of Steerability in Activation Space of Language Models

What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal

Disentangling Steering Vectors