LocUS: Head Selection and Subspace Projection for Targeted Activation Steering

Irene Tallini; Lorenzo Basile; Valentino Maiorca; Francesco Locatello; Alberto Cazzaniga

cs.CLSep 25, 2026

LocUS: Head Selection and Subspace Projection for Targeted Activation Steering

Authors: Irene Tallini, Lorenzo Basile, Valentino Maiorca, Francesco Locatello, Alberto Cazzaniga

Organizations: Area Science Park, Trieste, Italy · Institute of Science and Technology Austria · Université Côte d’Azur, Inria, LJAD, Maasai Project Team, Nice, France

Abstract

Activation steering is a powerful training-free paradigm for controlling large language models at inference time. However, standard approaches estimate a per-layer steering direction from contrastive data and apply it on the layer's entire representation space, which may couple the intervention to off-target properties present in the contrastive data and degrade unrelated capabilities. To mitigate this issue, we introduce LocUS (Localized Unembedding Steering), a method which grounds activation steering to the model's own output vocabulary subspace. By identifying a property-specific linear subspace within the unembedding matrix, LocUS enforces a geometric constraint that restricts the steering transformation to a specific subspace and at the same time localizes its application to a sparse subset of attention heads. Extensive evaluations across three model families on toxicity mitigation, sentiment redirection and sycophancy suppression show that LocUS matches or outperforms state-of-the-art baselines while intervening on under 6% of parameters and better preserving general capability.

Figures & tables

Figure 1: The LocUS pipeline. A property dictionary

\mathcal{P}

(here: stupid , hate , idiot ) selects columns of unembedding matrix

\bm{W}_{U}

to define the atoms matrix

\bm{U}_{\mathcal{P}}

. (2) Each head’s output is reconstructed through aa subset of atoms in

\bm{U}_{\mathcal{P}}

. Heads with low reconstruction error are retained. The steering vector

\bm{s}

is restricted to the per-head selected atoms and added to the retained heads at inference.

Mistral-7B	Llama-8B
Prompt: POSSIBLE SPOILERS	Prompt: Early in the movie, Cagney
Steered continuation: AHEAD. It’s been a long time since I watched the first episode of this series. The stupidity and inanities that were present there made me want to throw up. But now, after watching this one, it	Steered continuation: ’s bad guy is so mean that he even kills a dog. It was such an unnecessary scene and it didn’t add anything to the story except for cheap shock value. I don’t think I’ve ever seen any of his movies before unless you count

Table 1: Qualitative examples of LocUS steering positive

\rightarrow

negative on the IMDb sentiment task.

Figure 2: Per-head EVR scores and selected atoms. Above: EVR matrix across all (layer, head) pairs for IMDb on deepseek-7b; red squares mark the heads selected by the $\mu+2\sigma$ rule. Below: for the top-5 heads by EVR, the SOMP-selected property atoms in order of selection.

Mistral-7B-I
	TET			IMDb			Sycophancy
Method	tox $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	POS $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	SYC $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$
Unsteered	0.237 (—)	0.620	7.2	0.630 (—)	0.624	6.8	66.5 (—)	0.620	7.2
DoM	0.060 ( $-75\%$ )	0.614	8.2	0.254 ( $-60\%$ )	0.620	6.9	61.5 ( $-8\%$ )	0.619	7.4
ITI	0.102 ( $-57\%$ )	0.616	8.1	0.444 ( $-30\%$ )	0.621	6.8	58.5 ( $-12\%$ )	0.612	7.6
SMH	0.075 ( $-68\%$ )	0.623	7.6	0.552 ( $-12\%$ )	0.621	6.8	68.0 ( $+2\%$ )	0.619	7.2

Table 2: LocUS vs. DoM, ITI ( Li et al., 2023 ) and SMH ( Izawa et al., 2026 ) . DoM is the layer-wise difference-of-means update at every layer. Best target and MMLU value per task in bold , second best underlined .

	TET					IMDb					Sycophancy
Method	$k$	$\alpha_{1}$ / $\beta$	tox $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	$k$	$\alpha_{1}$ / $\beta$	POS $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	$k$	$\alpha_{1}$ / $\beta$	SYC $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$
Mistral-7B-I
Unsteered	–	0	0.237 (—)	0.620	7.2	–	0	0.630 (—)	0.624	6.8	–	0	66.5 (—)	0.620	7.2
LocUS -h	42	2 / 29.4	0.003 ( $-99\%$ )	0.612	13.0	41	4 / 49.5	0.012 ( $-98\%$ )	0.622	11.2	42	4 / 53.8	59.3 ( $-11\%$ )	0.615	8.3
LocUS -s	1024	2.5 / 1.10	0.010 ( $-96\%$ )	0.614	10.1	1024	1.5 / 1.15	0.008 ( $-99\%$ )	0.622	7.9	1024	4 / 4.46	61.8 ( $-7\%$ )	0.621	7.4
LocUS -hs	1024	3 / 5.15	0.036 ( $-85\%$ )	0.618	8.5	1024	2 / 3.45	0.156 ( $-75\%$ )	0.616	7.0	1024	4 / 7.66	58.6 ( $-12\%$ )	0.620	7.4

Table 3: Ablation of the LocUS components. LocUS -h keeps only the EVR head selection (

\mu+2\sigma

k

heads, no subspace projection); LocUS -s projects the update on the property subspace at the residual stream of every layer (no head selection); LocUS -hs projects at the raw head output of every head (no selection); LocUS combines head selection with head-level projection.Best target and MMLU value per task in bold , second best underlined .

Mistral-7B-I
Method	$k$	$\alpha_{1}$ / $\beta$	tox $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$
Unsteered	–	0	0.237 (—)	0.618	7.2
DoM	–	4 / 4	0.0943 ( $-60\%$ )	0.618	8.7
LocUS	19	0.5 / 27.2	0.0358 ( $-85\%$ )	0.619	8.3
Llama-8B-I
Unsteered	–	0	0.132 (—)	0.683	9.4

Table 4: Steering fitted on an external corpus (Jigsaw), evaluated on TET . Best target and MMLU value per task in bold .

Method	$k$	$\alpha_{1}$ / $\beta$	tox $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	$k$	$\alpha_{1}$ / $\beta$	POS $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$	$k$	$\alpha_{1}$ / $\beta$	SYC $\downarrow$ (vs U)	MMLU $\uparrow$	PPL $\downarrow$
	TET					IMDb					Sycophancy
Unsteered	–	0	0.286 (—)	0.721	5.8	–	0	0.652 (—)	0.714	5.8	–	0	75.8 (—)	0.721	5.8
DoM	–	2 / 2	0.020 ( $-93\%$ )	0.714	7.3	–	1 / 1	0.387 ( $-41\%$ )	0.709	5.9	–	4 / 4	58.2 ( $-23\%$ )	0.721	6.0
LocUS	166	1 / 18.2	0.000 ( $-100\%$ )	0.728	9.1	186	2 / 28.1	0.009 ( $-99\%$ )	0.713	9.2	173	4 / 106	54.4 ( $-28\%$ )	0.714	9.4

Table 5: DoM vs. LocUS on deepseek-67B, test split. Best target and MMLU value per task in bold .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Model	$n^{\star}$
TET	Mistral-7B	22
	deepseek-7B	20
	Llama-8B	24
	deepseek-67B	17
IMDb	Mistral-7B	24
	deepseek-7B	22

Appendix

Table 6: Choosing the subspace budget from the effective dictionary. Smallest number of atoms at which the mean fit-time cumulative EVR over all heads exceeds

0.25

Figure 3: Per-head EVR matrices for the nine (model, task) cells behind Table 3 ; orange boxes mark the $\mu+2\sigma$ -selected heads and $k$ is their count.

Figure 4: EVR on heads sorted by EVR value. Per-head EVR scores for the nine (model, task) cells, heads sorted by EVR within each layer (rank $0$ = highest). Inset reports the number of heads selected by the $\mu+2\sigma$ rule and their mean EVR.

Figure 5: Sorted probe accuracy for ITI. Per-head linear-probe validation accuracy for the nine (model, task) cells, same layout as Figure 4 . Inset reports the head count selected by ITI (the same $k$ as LocUS ) and their mean probe accuracy.

Figure 6: Removal–capability trade-off over the strength sweep, validation split. Each panel sweeps the steering coefficient $\alpha_{1}\in\{0.25,\dots,4\}$ (annotated next to each marker) for one (task, model) cell and plots the validation target metric ( $x$ , lower is better) against validation MMLU ( $y$ , higher is better); the black star is the unsteered model. Points whose validation wiki-PPL exceeds twice the unsteered model’s are omitted (MMLU itself is not gated in the plot); A black ring marks each variant’s selected $\alpha_{1}^{*}$ , i.e. the operating point Table 3 reports at test.

Language Model Steering

Language model steering controls a model’s behavior at inference time by intervening in its representations, components, or decoding process.

See all 283 papers

May 7, 2026

Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention

Activation steering has emerged as a promising alternative for controlling language-model behavior at inference time by modifying intermediate representations while keeping model parameters frozen. However, large-scale evaluations such as AxBench show that existing steering methods are often outperformed by simple in-context prompting and generalize poorly to unseen concepts. We hypothesize that these limitations arise from unvalidated simplifying assumptions shared across prior methods, which typically restrict steering interventions to fixed, single-step, position-invariant transforms. We propose FLAS (Flow-based Activation Steering), which learns a general, concept-conditioned velocity field

v_t(h,t,c)

that transports unsteered activations to steered ones without relying on these assumptions. On AxBench, FLAS is the first learned method to consistently outperform prompting, reaching held-out harmonic means of

1.015

on Gemma-2-2B-IT and

1.113

on Gemma-2-9B-IT without per-concept tuning. Analysis of the learned flow shows curved, multi-step, token-varying trajectories, which suggests that previous hypotheses on activation space geometry might be incomplete.

Zehao Jin, Ruixuan Deng, Junran Wang +2

Georgia Institute of Technology

Language Model Steering Activation Steering

Ask PDF

Jun 13, 2026

High-Dimensional Random Projection for Activation Steering in Language Models

Activation steering has emerged as a key methodology for controlling the behavior of large language models (LLMs). Existing difference-in-means based methods, however, are fundamentally limited: they capture only mean differences between class activations and fail to recover discriminative signals that naturally exist in the nonlinear feature subspace under the superposition hypothesis. Motivated by that, we propose High-Dimensional Random-projection for Activation Steering (HiDRA), a training-free approach that integrates seamlessly with existing activation steering methods. By performing activation addition in the projected high-dimensional space, HiDRA can provably capture a better discriminative structure beyond the reach of linear methods. Experiments across diverse LLM families and benchmarks demonstrate that HiDRA consistently outperforms baseline counterparts, achieving stronger behavioral control without significant computational overhead.

Minh-Hieu Pham, Bach Do, Laziz Abdullaev +2

Hanoi University of Science and Technology

Language Model Steering Representation Geometry in Language Models Activation Steering

Ask PDF

May 5, 2026

Steer Like the LLM: Activation Steering that Mimics Prompting

Large language models can be steered at inference time through prompting or activation interventions, but activation steering methods often underperform compared to prompt-based approaches. We propose a framework that formulates prompt steering as a form of activation steering and investigates whether distilling successful prompt steering behavior into simpler, interpretable models can close this gap. Our analysis reveals that popular activation steering methods are not faithful to the mechanics of prompt steering, which applies strong interventions on some tokens while barely affecting others. Based on these insights, we introduce Prompt Steering Replacement (PSR) models that estimate token-specific steering coefficients from the activations themselves and are trained to imitate prompt-based interventions. Experiments on three steering benchmarks across multiple language models show that PSR models outperform existing activation steering methods, especially when controlling for high-coherence completions, and also compare favorably to prompting on AxBench and persona steering.

Geert Heyman, Frederik Vandeputte

Nokia Bell Labs, Belgium.

Language Model Steering LLM Prompting Causal Interventions in Language Models Activation Steering

Ask PDF

Abstract

Language Model Steering

Explore similar work

Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention

High-Dimensional Random Projection for Activation Steering in Language Models

Steer Like the LLM: Activation Steering that Mimics Prompting