Neyman Orthogonal Activation Steering

David Arbour; Erfan Jahanparast; Serina Chang; Avi Feller

cs.LGOct 23, 2026

Neyman Orthogonal Activation Steering

Authors: David Arbour, Erfan Jahanparast, Serina Chang, Avi Feller

Organizations: Adobe Research · MIT · University of California, Berkeley

Abstract

Activation steering is a popular approach for shifting LLM output toward a desired behavior, and increasingly for generating realistic synthetic data. The standard recipe estimates a steering direction by contrasting LLM activations on prompts with and without a target concept, such as truthfulness. In this paper, we separate the estimand (the steering direction these methods target) from the estimator used to recover it, two notions the literature often conflates. We frame nearly all existing methods as plug-in estimators of the average gradient of an outcome regression, with the binary contrastive label fixed as the outcome. Using this framing, we can immediately incorporate flexible modeling approaches and extend existing methods to allow for continuous outcomes such as log-probabilities or oracle ratings. Flexible ML models that fit the outcome well, however, might not estimate the gradient well, which motivates estimators that target the gradient directly. Drawing on the debiased machine learning literature, we develop a complementary estimator built on the Stein score, the Riesz representer of the gradient functional. Combining the two routes gives an orthogonalized estimator with classical double-robustness properties. We illustrate the framework on activation steering applied to a number of benchmark datasets.

Figures & tables

Method or construction	Outcome model or assumption	Direction
CAA / difference of means	Linear, isotropic working covariance	$\Delta$
OLS	Linear regression of $L$	$\Sigma_{X}^{-1}\Delta$
LDA	Gaussian classes, shared covariance	$\Sigma_{W}^{-1}\Delta$
PCA-projected difference of means	Isotropic working slope, projected onto PCs	$U_{k}U_{k}^{\top}\Delta$
ITI mean-difference direction	Per-head linear, selected attention heads	$\Delta_{(\ell,h)}$
Logistic probe / ITI probe direction	Logistic, global or per-head	$\hat{\beta}^{\mathrm{logit}}$

Table 1: Regression constructions and their connections to steering methods.

\Delta

is the difference of class means,

\mathbb{E}[X\mid L=1]-\mathbb{E}[X\mid L=0]

;

\Sigma_{X}=\operatorname{Cov}(X)

is the marginal activation covariance;

\Sigma_{W}

is the within-class covariance; and the columns of

U_{k}

are the leading eigenvectors of

\Sigma_{X}

. Directions are shown up to positive multiplicative constants.

Model	Target	Unsteered	CAA	Neyman	Unsteered	CAA	Neyman	TV	KL
		TV $\downarrow$			KL $\downarrow$			Gain (%)
Llama-3.1-8B	Dem	0.271	0.265	0.263	0.254	0.229	0.216	0.6	5.7
	Rep	0.301	0.276	0.259	0.303	0.237	0.218	6.0	8.0
Qwen3-4B-Base	Dem	0.241	0.232	0.231	0.209	0.192	0.176	0.6	8.1
	Rep	0.315	0.270	0.269	0.356	0.266	0.257	0.3	3.5
Qwen3-8B	Dem	0.354	0.308	0.284	0.725	0.442	0.323	7.8	26.9

Table 2: OpinionQA partisan steering, test error averaged over five seeds; lower is better. “Unsteered” is the model on the neutral question. “Gain” is the relative reduction from CAA to the Neyman estimator in each setting. The aggregate gain is the unweighted mean of the twelve setting-specific relative reductions; the 95% interval resamples the five seeds, which share splits across settings.

Model	Unst.	Probe	Stein	CAA	Ney.	Gain
Llama-3.1-8B	.492	.463	.465	.459	.444	3.2
Q3-4B-B	.495	.423	.334	.323	.281	12.8
Q3-8B	.475	.394	.225	.298	.195	34.6
Q3-8B-B	.452	.424	.350	.393	.283	28.0
Q3.5-9B	.432	.367	.312	.294	.271	7.9
Q3.5-9B-B	.461	.415	.336	.372	.312	16.0

Table 3: Truthfulness with CAA as the outcome model: test TV averaged over nine datasets and two seeds, then the all-model averages of TV and KL; lower is better. Q: Qwen; B: base; “Unst.”: unsteered. “Probe” is the label mass-mean direction; “Stein” is the score-only estimator with the isotropic score on demeaned activations. “Gain” is the relative reduction in mean error from CAA to the Neyman estimator, weighting dataset–seed units equally. All-model 95% intervals, resampling the nine datasets, are

[11,24]

for TV and

[17,30]

for KL.

	Linear			KRR
Model	Plug	Ney.	Gain	Plug	Ney.	Gain
Llama-3.1-8B	.471	.448	4.9	.469	.444	5.2
Q3-4B-B	.430	.322	25.1	.432	.326	24.6
Q3-8B	.367	.213	42.0	.383	.208	45.8
Q3-8B-B	.404	.324	19.8	.390	.313	19.5
Q3.5-9B	.378	.288	23.8	.366	.286	21.7

Table 4: Truthfulness with linear ridge and KRR outcome models fit to

Y

: plug-in (“Plug”) and Neyman (“Ney.”) estimators built on each, same averaging as Table 3 ; lower is better. “Gain” is the relative reduction in mean error from the plug-in to the Neyman estimator. All-model 95% intervals, resampling the nine datasets: linear

[15,34]

(TV) and

[19,42]

(KL); KRR

[16,34]

and

[17,40]

Genre	Prompt	CAA	Neyman	Gain
Text (332)	0.523	0.665	0.740	$+.075$ [.044,.106]
Code (122)	0.335	0.559	0.559	—
Math (46)	0.409	0.499	0.499	—
All (500)	0.466	0.624	0.673	$+.050$ [.029,.070]

Table 5: AxBench Concept500, Gemma-2-2B-it, layer 10. Overall steering (0–2; higher is better) at the validation-selected strength, by concept genre (number of concepts in parentheses). “Prompt” instructs the model to include the concept, with no intervention; for code and math, validation selects CAA, so the two columns coincide by construction. The “All” row averages concepts equally. Intervals are 95% paired bootstrap intervals over concepts.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Linear			KRR			RFM
Model	Plug	Ney.	Gain	Plug	Ney.	Gain	Plug	Ney.	Gain
Llama-3.1-8B	.268	.260	2.8	.268	.259	3.1	.268	.260	3.0
Qwen3-4B-Base	.257	.255	0.2	.256	.252	1.6	.256	.252	1.7
Qwen3-8B	.331	.281	14.6	.338	.276	18.2	.338	.275	18.5
Qwen3-8B-Base	.246	.237	3.6	.247	.235	4.7	.246	.235	4.3
Qwen3.5-9B	.295	.261	11.3	.291	.269	6.9	.291	.269	7.2

Appendix

Table 6: OpinionQA with linear ridge, KRR and RFM outcome models: plug-in (“Plug”) and Neyman (“Ney.”) estimators built on each, test TV averaged over both targets and five seeds, then all-model TV and KL; lower is better. “Gain” is the mean of the setting-level relative reductions, as in Table 2 . All-model 95% intervals, resampling seeds: linear

[5.4,7.0]

(TV) and

[12.8,18.2]

(KL); KRR

[4.7,7.8]

and

[11.5,17.4]

; RFM

[4.9,8.1]

and

[10.6,17.7]

	TV		KL
Dataset	CAA	Neyman	CAA	Neyman
Cities	0.221	0.121	0.295	0.154
Sp-En Trans.	0.350	0.264	0.550	0.346
Neg Cities	0.365	0.297	0.506	0.377
Neg Sp-En Trans.	0.406	0.370	0.591	0.510
Cities Conj.	0.280	0.195	0.379	0.254

Appendix

Table 7: Truthfulness by dataset, mean test error over six models and two seeds; lower is better.

Language Model Steering

Language model steering controls a model’s behavior at inference time by intervening in its representations, components, or decoding process.

See all 283 papers

Orthogonal and Parametrized Multi-Attribute Steering of LLMs
Oct 27, 2026Remy Ogasawara, Cullen Anderson, Narmeen Fatimah Oozeer +4Language Model Steering Activation Steering
Kahlert School of Computing University of Utah Salt Lake City, UT 84112 · Department of Computer Science University of Massachusetts Amherst Amherst, MA 01003 · Martian AI San Francisco, CA 94117, USA +1
Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methods
Oct 6, 2026Haotian Yang, Huikang Jiang, Yucheng Wu +4LLM Evaluation Language Model Steering
State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · Columbia University +2
Steer Like the LLM: Activation Steering that Mimics Prompting
May 5, 2026Geert Heyman, Frederik VandeputteLanguage Model Steering LLM Prompting
Nokia Bell Labs, Belgium.

Oct 27, 2026J/K move · Enter open · S save

Orthogonal and Parametrized Multi-Attribute Steering of LLMs

Remy Ogasawara, Cullen Anderson, Narmeen Fatimah Oozeer, Foad Namjoo+3

Kahlert School of Computing University of Utah Salt Lake City, UT 84112 · Department of Computer Science University of Massachusetts Amherst Amherst, MA 01003 · Martian AI San Francisco, CA 94117, USA+1

Language Model Steering Activation Steering

Open paper AskPDF arXiv

Activation steering offers an effective and efficient way to modify an LLM's behavior at inference time. However, it remains unclear how to apply steering such that the signal is consistently strong and the steering strength has a universally interpretable meaning. We address this first by identifying the most steerable interval for each steering vector and rescaling the steering range accordingly. Building on this, we introduce a more interpretable method for steering, called parameterized steering which instead of pushing activations in a direction, it sets them at the desired trait strength. Next, with multi-trait steering, we find that naively combining vectors causes them to interfere with one another. In response, we develop a method to orthogonalize the vectors; it decouples these traits, allowing a model to exhibit one behavior without inducing the others, thus enabling multiple traits to be steered simultaneously.

Abstract

Language Model Steering

Explore similar work

Orthogonal and Parametrized Multi-Attribute Steering of LLMs

Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methods

Steer Like the LLM: Activation Steering that Mimics Prompting