LocUS: Head Selection and Subspace Projection for Targeted Activation Steering

Irene Tallini; Lorenzo Basile; Valentino Maiorca; Francesco Locatello; Alberto Cazzaniga

cs.CLSep 25, 2026

LocUS: Head Selection and Subspace Projection for Targeted Activation Steering

Authors: Irene Tallini, Lorenzo Basile, Valentino Maiorca, Francesco Locatello, Alberto Cazzaniga

Organizations: Area Science Park, Trieste, Italy · Institute of Science and Technology Austria · Université Côte d’Azur, Inria, LJAD, Maasai Project Team, Nice, France

Abstract

Activation steering is a powerful training-free paradigm for controlling large language models at inference time. However, standard approaches estimate a per-layer steering direction from contrastive data and apply it on the layer's entire representation space, which may couple the intervention to off-target properties present in the contrastive data and degrade unrelated capabilities. To mitigate this issue, we introduce LocUS (Localized Unembedding Steering), a method which grounds activation steering to the model's own output vocabulary subspace. By identifying a property-specific linear subspace within the unembedding matrix, LocUS enforces a geometric constraint that restricts the steering transformation to a specific subspace and at the same time localizes its application to a sparse subset of attention heads. Extensive evaluations across three model families on toxicity mitigation, sentiment redirection and sycophancy suppression show that LocUS matches or outperforms state-of-the-art baselines while intervening on under 6% of parameters and better preserving general capability.

Figures & tables

Figure 1: The LocUS pipeline. A property dictionary

\mathcal{P}

(here: stupid , hate , idiot ) selects columns of unembedding matrix

\bm{W}_{U}

to define the atoms matrix

\bm{U}_{\mathcal{P}}

. (2) Each head’s output is reconstructed through aa subset of atoms in

\bm{U}_{\mathcal{P}}

. Heads with low reconstruction error are retained. The steering vector

\bm{s}

is restricted to the per-head selected atoms and added to the retained heads at inference.

Mistral-7B	Llama-8B
Prompt: POSSIBLE SPOILERS	Prompt: Early in the movie, Cagney
Steered continuation: AHEAD. It’s been a long time since I watched the first episode of this series. The stupidity and inanities that were present there made me want to throw up. But now, after watching this one, it	Steered continuation: ’s bad guy is so mean that he even kills a dog. It was such an unnecessary scene and it didn’t add anything to the story except for cheap shock value. I don’t think I’ve ever seen any of his movies before unless you count

Table 1: Qualitative examples of LocUS steering positive

\rightarrow

negative on the IMDb sentiment task.

Figure 2: Per-head EVR scores and selected atoms. Above: EVR matrix across all (layer, head) pairs for IMDb on deepseek-7b; red squares mark the heads selected by the $\mu+2\sigma$ rule. Below: for the top-5 heads by EVR, the SOMP-selected property atoms in order of selection.

Mistral-7B-I
	TET			IMDb			Sycophancy
Method	tox $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	POS $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	SYC $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$
Unsteered	0.237 (—)	0.620	7.2	0.630 (—)	0.624	6.8	66.5 (—)	0.620	7.2
DoM	0.060 ( $-75\%$ )	0.614	8.2	0.254 ( $-60\%$ )	0.620	6.9	61.5 ( $-8\%$ )	0.619	7.4
ITI	0.102 ( $-57\%$ )	0.616	8.1	0.444 ( $-30\%$ )	0.621	6.8	58.5 ( $-12\%$ )	0.612	7.6
SMH	0.075 ( $-68\%$ )	0.623	7.6	0.552 ( $-12\%$ )	0.621	6.8	68.0 ( $+2\%$ )	0.619	7.2

Table 2: LocUS vs. DoM, ITI ( Li et al., 2023 ) and SMH ( Izawa et al., 2026 ) . DoM is the layer-wise difference-of-means update at every layer. Best target and MMLU value per task in bold , second best underlined .

	TET					IMDb					Sycophancy
Method	$k$	$\alpha_{1}$ / $\beta$	tox $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	$k$	$\alpha_{1}$ / $\beta$	POS $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	$k$	$\alpha_{1}$ / $\beta$	SYC $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$
Mistral-7B-I
Unsteered	–	0	0.237 (—)	0.620	7.2	–	0	0.630 (—)	0.624	6.8	–	0	66.5 (—)	0.620	7.2
LocUS -h	42	2 / 29.4	0.003 ( $-99\%$ )	0.612	13.0	41	4 / 49.5	0.012 ( $-98\%$ )	0.622	11.2	42	4 / 53.8	59.3 ( $-11\%$ )	0.615	8.3
LocUS -s	1024	2.5 / 1.10	0.010 ( $-96\%$ )	0.614	10.1	1024	1.5 / 1.15	0.008 ( $-99\%$ )	0.622	7.9	1024	4 / 4.46	61.8 ( $-7\%$ )	0.621	7.4
LocUS -hs	1024	3 / 5.15	0.036 ( $-85\%$ )	0.618	8.5	1024	2 / 3.45	0.156 ( $-75\%$ )	0.616	7.0	1024	4 / 7.66	58.6 ( $-12\%$ )	0.620	7.4

Table 3: Ablation of the LocUS components. LocUS -h keeps only the EVR head selection (

\mu+2\sigma

k

heads, no subspace projection); LocUS -s projects the update on the property subspace at the residual stream of every layer (no head selection); LocUS -hs projects at the raw head output of every head (no selection); LocUS combines head selection with head-level projection.Best target and MMLU value per task in bold , second best underlined .

Mistral-7B-I
Method	$k$	$\alpha_{1}$ / $\beta$	tox $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$
Unsteered	–	0	0.237 (—)	0.618	7.2
DoM	–	4 / 4	0.0943 ( $-60\%$ )	0.618	8.7
LocUS	19	0.5 / 27.2	0.0358 ( $-85\%$ )	0.619	8.3
Llama-8B-I
Unsteered	–	0	0.132 (—)	0.683	9.4

Table 4: Steering fitted on an external corpus (Jigsaw), evaluated on TET . Best target and MMLU value per task in bold .

Method	$k$	$\alpha_{1}$ / $\beta$	tox $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	$k$	$\alpha_{1}$ / $\beta$	POS $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	$k$	$\alpha_{1}$ / $\beta$	SYC $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$
	TET					IMDb					Sycophancy
Unsteered	–	0	0.286 (—)	0.721	5.8	–	0	0.652 (—)	0.714	5.8	–	0	75.8 (—)	0.721	5.8
DoM	–	2 / 2	0.020 ( $-93\%$ )	0.714	7.3	–	1 / 1	0.387 ( $-41\%$ )	0.709	5.9	–	4 / 4	58.2 ( $-23\%$ )	0.721	6.0
LocUS	166	1 / 18.2	0.000 ( $-100\%$ )	0.728	9.1	186	2 / 28.1	0.009 ( $-99\%$ )	0.713	9.2	173	4 / 106	54.4 ( $-28\%$ )	0.714	9.4

Table 5: DoM vs. LocUS on deepseek-67B, test split. Best target and MMLU value per task in bold .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Model	$n^{\star}$
TET	Mistral-7B	22
	deepseek-7B	20
	Llama-8B	24
	deepseek-67B	17
IMDb	Mistral-7B	24
	deepseek-7B	22

Appendix

Table 6: Choosing the subspace budget from the effective dictionary. Smallest number of atoms at which the mean fit-time cumulative EVR over all heads exceeds

0.25

Figure 3: Per-head EVR matrices for the nine (model, task) cells behind Table 3 ; orange boxes mark the $\mu+2\sigma$ -selected heads and $k$ is their count.

Figure 4: EVR on heads sorted by EVR value. Per-head EVR scores for the nine (model, task) cells, heads sorted by EVR within each layer (rank $0$ = highest). Inset reports the number of heads selected by the $\mu+2\sigma$ rule and their mean EVR.

Figure 5: Sorted probe accuracy for ITI. Per-head linear-probe validation accuracy for the nine (model, task) cells, same layout as Figure 4 . Inset reports the head count selected by ITI (the same $k$ as LocUS ) and their mean probe accuracy.

Figure 6: Removal–capability trade-off over the strength sweep, validation split. Each panel sweeps the steering coefficient $\alpha_{1}\in\{0.25,\dots,4\}$ (annotated next to each marker) for one (task, model) cell and plots the validation target metric ( $x$ , lower is better) against validation MMLU ( $y$ , higher is better); the black star is the unsteered model. Points whose validation wiki-PPL exceeds twice the unsteered model’s are omitted (MMLU itself is not gated in the plot); A black ring marks each variant’s selected $\alpha_{1}^{*}$ , i.e. the operating point Table 3 reports at test.

Language Model Steering

Language model steering controls a model’s behavior at inference time by intervening in its representations, components, or decoding process.

See all 283 papers

Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention
May 7, 2026Zehao Jin, Ruixuan Deng, Junran Wang +2Language Model Steering Activation Steering
Georgia Institute of Technology
High-Dimensional Random Projection for Activation Steering in Language Models
Jun 13, 2026Minh-Hieu Pham, Bach Do, Laziz Abdullaev +2Language Model Steering Representation Geometry in Language Models
Hanoi University of Science and Technology
Steer Like the LLM: Activation Steering that Mimics Prompting
May 5, 2026Geert Heyman, Frederik VandeputteLanguage Model Steering LLM Prompting
Nokia Bell Labs, Belgium.

May 7, 2026J/K move · Enter open · S save

Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention

Zehao Jin, Ruixuan Deng, Junran Wang, Xinjie Shen+1

Georgia Institute of Technology

Language Model Steering Activation Steering

Open paper AskPDF arXiv

Activation steering has emerged as a promising alternative for controlling language-model behavior at inference time by modifying intermediate representations while keeping model parameters frozen. However, large-scale evaluations such as AxBench show that existing steering methods are often outperformed by simple in-context prompting and generalize poorly to unseen concepts. We hypothesize that these limitations arise from unvalidated simplifying assumptions shared across prior methods, which typically restrict steering interventions to fixed, single-step, position-invariant transforms. We propose FLAS (Flow-based Activation Steering), which learns a general, concept-conditioned velocity field

v_t(h,t,c)

that transports unsteered activations to steered ones without relying on these assumptions. On AxBench, FLAS is the first learned method to consistently outperform prompting, reaching held-out harmonic means of

1.015

on Gemma-2-2B-IT and

1.113

on Gemma-2-9B-IT without per-concept tuning. Analysis of the learned flow shows curved, multi-step, token-varying trajectories, which suggests that previous hypotheses on activation space geometry might be incomplete.

Abstract

Language Model Steering

Explore similar work

Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention

High-Dimensional Random Projection for Activation Steering in Language Models

Steer Like the LLM: Activation Steering that Mimics Prompting