Spectral Attention Steering for Prompt Highlighting

Weixian Waylon Li; Yuchen Niu; Yongxin Yang; Keshuang Li; Tiejun Ma; Shay B. Cohen

cs.CLMar 1, 2026

Spectral Attention Steering for Prompt Highlighting

Authors: Weixian Waylon Li, Yuchen Niu, Yongxin Yang, Keshuang Li, Tiejun Ma, Shay B. Cohen

Organizations: University of Edinburgh, UK · RayNeo, China · Queen Mary University of London, UK · Huawei Research Ltd., UK

Abstract

Attention steering is an important technique for controlling model focus, enabling capabilities such as prompt highlighting, where the model prioritises user-specified text. However, existing attention steering methods require explicit storage of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compatibility with optimised attention.

Figures & tables

Panel of Figure 1 — Figure 1: Visualisation of pairwise key embedding shifts across different (layer, head) in Qwen3-1.7B-Base via PCA. Positive vs. negative representations are plotted for 26 shared token spans. Grey arrows trace individual shifts; the dark blue arrow shows the average displacement.

Figure 2: An overview of SEKA and AdaSEKA .

{\bm{x}}

: context;

{\bm{h}}

: key embedding;

{\bm{\Omega}}

: cross-covariance;

{\bm{U}}

: left singular vectors;

{\bm{S}}

: singular values;

g

: gain coefficient. SEKA applies fixed gains, while AdaSEKA uses the query to compute dynamic steering weights.

Panel of Figure 3 — Figure 3: Heatmaps of the average per-token $\ell_{2}$ distance between positive and negative key embeddings across all KV heads and layers for four Qwen3 model sizes. Higher values ( green ) indicate greater separation between positive and negative key representations.

Task	Prompts	Metrics
Counterfact	Previously, [old fact] . Currently, [new fact] . [question] .	Efficacy score (ES), Paraphrase score (PS)
Bias in Bios	[person’s occupation] . [career history, may not directly related to prediction] . [person] has the occupation of a/an	Accuracy (Acc.)
Pronouns changing	[biographical contexts] . Substitute ‘she’ and ‘he’ with ‘they’ and generate the occupation of [person] after changing pronouns.	Pronoun-weighted Lexical overlap Score (P. Score), All-changed P. Score

Table 1: Summary of standard benchmarks for attention steering. Tokens in bold indicate where attention steering is applied.

Model	Metric	Baselines				Our Methods
Model	Metric	Original	**-marked	PASTA	SPA	SEKA	w/o learn	w/o learn&filt	AdaSEKA
Qwen3-4B	CounterFact (ES)	45.00	57.70	97.16	65.24	99.02	94.96	86.12	98.90
	CounterFact (PS)	45.64	52.12	96.03	57.71	98.61	92.38	86.20	98.72
	Bias in Bios (Acc.)	79.84	82.94	89.58	68.00	91.02	86.62	71.76	91.86
	Pronoun (P. Score)	93.14	95.76	95.82	80.27	95.18	90.42	41.98	94.54
	Pronoun (A. P. Score)	90.52	93.88	94.64	78.19	93.26	88.66	36.95	92.08

Table 2: Performance on standard benchmarks. Bold

=

best. Underline

=

second best. We include two ablation studies for SEKA : “w/o learn” uses random projections instead of spectrally learned ones, and “w/o learn&filt” further removes the head filtering mechanism.

Figure 4: Exact match scores on the lost-in-the-middle task for Qwen3 models of three different sizes, comparing the original model, PASTA/ SEKA applied to the middle region (5 th to 25 th passages), and PASTA/ SEKA applied to all passages.

Figure 5: Exact match scores when applying SEKA to the middle region with different threshold

\delta_{\text{min}}

Method	Avg. Time (s)	Peak Mem. (GB, B=10)	Peak Mem. (GB, B=1)
Original	0.55	27.63	16.72
PASTA	1.58 (+1.03)	50.75 (+23.12)	-
SPA	5.87 (+5.32)	-	17.71 (+0.99)
SEKA	0.58 (+0.03)	27.66 (+0.03)	16.75 (+0.03)
AdaSEKA	0.82 (+0.27)	43.22 (+15.59)	18.23 (+1.51)

Table 3: Inference overhead on Qwen3-8B-Base. Time is per-sample; memory is average peak usage.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Prompt
Neutral	Context: The portfolio manager allocates capital across equities and bonds.
Positive	Question: What does the portfolio manager allocate across equities and bonds? Context: The portfolio manager allocates capital across equities and bonds.
Negative	Question: What does the climate model simulate? Context: The portfolio manager allocates capital across equities and bonds.
Neutral	Context: The climate model simulates sea-level rise under different scenarios.
Positive	Question: What does the climate model simulate? Context: The climate model simulates sea-level rise under different scenarios.
Negative	Question: What does the portfolio manager allocate across equities and bonds? Context: The climate model simulates sea-level rise under different scenarios.

Appendix

Table 4: Constructed prompt triplets for both answer spans. Each group provides a neutral, positive, and negative variant based on question-context alignment.

Context 1 ( $C_{1}$ )	The portfolio manager allocates capital across equities and bonds.
Context 2 ( $C_{2}$ )	The climate model simulates sea-level rise under different scenarios.
Question 1 ( $Q_{1}$ )	What does the portfolio manager allocate across equities and bonds?
Answer 1 ( $A_{1}$ )	capital
Question 2 ( $Q_{2}$ )	What does the climate model simulate?
Answer 2 ( $A_{2}$ )	sea-level rise

Appendix

Table 5: Synthetic data instance.

Panel of Figure 6 — Figure 6: Prompt template used to generate synthetic contrastive examples.

Dataset	Variant	Prompt
CounterFact	Neutral	Context: Previously, The mother tongue of Danielle Darrieux is French. Currently, The mother tongue of Danielle Darrieux is English .
	Positive	Question: Danielle Darrieux, a native ? Context: Previously, The mother tongue of Danielle Darrieux is French. Currently, The mother tongue of Danielle Darrieux is English .
BiasBios	Neutral	Context: Nora is an assistant professor of international relations at Bahcesehir University in Istanbul. She is also a Ronald D. Asmus Policy Entrepreneur Fellow with the German Marshall Fund and is a Visiting Fellow at the Centre for International Studies (CIS) at the University of Oxford. This commentary first appeared at Sada, an online journal published by the Carnegie Endowment for International Peace.
	Positive	Question: Nora has the occupation of a/an ? Context: Nora is an assistant professor of international relations at Bahcesehir University in Istanbul. She is also a Ronald D. Asmus Policy Entrepreneur Fellow with the German Marshall Fund and is a Visiting Fellow at the Centre for International Studies (CIS) at the University of Oxford. This commentary first appeared at Sada, an online journal published by the Carnegie Endowment for International Peace.
HotpotQA	Neutral	Context: Radio City (Indian radio station) Radio City is India’s first private FM radio station and was started on 3 July 2001. $\dots$ Arthur’s Magazine Arthur’s Magazine (1844–1846) was an American literary periodical published in Philadelphia in the 19th century. Edited by T.S. Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others. In May 1846 it was merged into “Godey’s Lady’s Book”. $\dots$ First for Women First for Women is a womanś magazine published by Bauer Media Group in the USA. The magazine was started in 1989. It is based in Englewood Cliffs, New Jersey. $\dots$ The company started first as a denim line, later evolving into a men’s and women’s clothing line.
	Positive	Question: Which magazine was started first Arthur’s Magazine or First for Women? Context: Radio City (Indian radio station) Radio City is India’s first private FM radio station and was started on 3 July 2001. $\dots$ Arthur’s Magazine Arthur’s Magazine (1844–1846) was an American literary periodical published in Philadelphia in the 19th century. Edited by T.S. Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others. In May 1846 it was merged into “Godey’s Lady’s Book”. $\dots$ First for Women First for Women is a woman’s magazine published by Bauer Media Group in the USA. The magazine was started in 1989. It is based in Englewood Cliffs, New Jersey. $\dots$ The company started first as a denim line, later evolving into a men’s and women’s clothing line.

Appendix

Table 6: Constructed prompt pairs for multi-expert projection learning. Each dataset provides neutral and positive variants based on question-context alignment.

		SEKA				AdaSEKA
Model	Task	$\gamma$	$\delta_{\textrm{min}}$	$g^{+}$	$g^{-}$	$\delta_{\textrm{min}}$	$g$
Qwen3-4B-Base	CounterFact	0.960	0.13	1.56	0.00	0.1	3.0
	Bias in Bios	0.998	0.12	1.00	0.80	0.1	0.5
	Pronoun Changing	0.880	0.22	0.16	0.00	0.5	0.6
Qwen3-8B-Base	CounterFact	0.850	0.12	2.40	0.00	0.1	3.0
	Bias in Bios	0.998	0.12	0.60	0.30	0.1	0.5

Appendix

Table 7: Hyperparameters for SEKA and AdaSEKA methods. SEKA uses the variance threshold (

\gamma

), KV-head selection threshold (

\delta_{\textrm{min}}

), positive gain (

g^{+}

), and negative gain (

g^{-}

). AdaSEKA uses the KV-head selection threshold (

\delta_{\textrm{min}}

) and steering gain (

g

Panel of Figure 7 — Figure 7: Sensitivity of SEKA to hyper-parameters across three benchmark tasks. Each curve varies a single hyper-parameter while keeping others fixed at their optimal settings on the validation set.

Panel of Figure 8 — Figure 8: Layer-wise mean attention (all heads) in Qwen3-4B-Base at selected layers for the CounterFact data sample, shown before and after SEKA is applied.

Model	PASTA Configuration	CounterFact		Bias in Bios	Pronoun Changing
Model	PASTA Configuration	ES	PS	Acc.	P. Score	A. P. Score
Qwen3-4B-Base	SEKA KV-heads	83.62	80.43	89.58	55.40	54.08
	Transformed attention heads	82.60	83.02	79.34	95.82	94.64
	Original head-search	97.16	96.03	87.84	47.20	47.20
Qwen3-8B-Base	SEKA KV-heads	82.08	71.72	86.32	62.91	61.44
	Transformed attention heads	78.20	79.69	77.30	98.86	98.72

Appendix

Table 8: Complete PASTA results with different configurations: (1) using SEKA ’s KV-head configuration (runtime 1–2 minutes), (2) using attention head configuration transformed from SEKA ’s KV-heads (1–2 minutes), and (3) using PASTA’s original head-search routine (

\approx

2 hours).

Prompt

Previously David Sainsbury, Baron Sainsbury of Turville used to work in London. Currently David Sainsbury, Baron Sainsbury of Turville used to work in Berlin. After 15 years of work, he published Pillboxes in 1985. David Sainsbury, Baron Sainsbury of Turville took up work in

Appendix

Table 9: CounterFact Example 1: Overriding a known location.

Prompt

Previously Jean Baptiste Pompallier holds the position of bishop. Currently Jean Baptiste Pompallier holds the position of cardinal. Jean Baptiste Pompallier has the position of

Appendix

Table 10: CounterFact Example 2: Overriding a known job title.

Prompt

Tania is an Illinois poet who has published one chapbook… and two full-length collections . Her most recent book… is divided into ten sections… Its predecessor Simple Weight is also concerned with the Biblical narrative… Barbara Crooker said of that collection, ”The poems have weight emotional, spiritual, political but are anything but simple.”.Tania has the occupation of a/an

Appendix

Table 11: Bias in Bios Example 1: Correcting a prediction.

Prompt

Michelle, PhD is an Adjunct Assistant Professor in the Emory Department of Psychiatry and Behavioral Sciences . She received her Ph.D. in 2003 from Emory University. Her areas of expertise include intergenerational processes of risk and resilience… She supervises postdoctoral residents… Michelle has the occupation of a/an

Appendix

Table 12: Bias in Bios Example 2: Improving confidence in the correct answer.

Prompt

Jason is an assistant professor of English… For the aforementioned text, substitute ‘she’ and ‘he’ with ‘they’ and generate the occupation of Jason after changing pronouns.

Appendix

Table 13: Pronouns Changing Example 1.

Prompt

Anna, JD, MA, LPC, is a depth psychotherapist in Houston, Texas… She holds degrees in philosophy, law, and clinical psychology… For the aforementioned text, substitute ‘she’ and ‘he’ with ‘they’ and generate the occupation of Anna after changing pronouns.

Appendix

Table 14: Pronouns Changing Example 2.

Figure 9: The performance of SEKA with varying numbers of synthetic samples used for learning projections across different models and tasks.

Figure 10: Exact match scores on the lost-in-the-middle task when applying SEKA to the middle region with different

\delta_{\text{min}}

thresholds for Qwen3-4B and Qwen3-14B.

Language Model Steering

Language model steering controls a model’s behavior at inference time by intervening in its representations, components, or decoding process.

See all 278 papers

PRISM- $Δ$ : Differential Subspace Steering for Prompt Highlighting in Large Language Models
Mar 11, 2026Yuyao Ge, Shenghua Liu, Yiwei Wang +5Language Model Steering LLM Prompting
♣Institute of Computing Technology, Chinese Academy of Sciences · △University of California, Merced
Don't Lose Focus: Activation Steering via Key-Orthogonal Projections
May 7, 2026Haoyan Luo, Mateo Espinosa Zarlenga, Mateja JamnikLanguage Model Steering Self-Attention
University of Cambridge · University of Oxford
Steer Like the LLM: Activation Steering that Mimics Prompting
May 5, 2026Geert Heyman, Frederik VandeputteLanguage Model Steering LLM Prompting
Nokia Bell Labs, Belgium.

Mar 11, 2026J/K move · Enter open · S save

PRISM- $Δ$ : Differential Subspace Steering for Prompt Highlighting in Large Language Models

Yuyao Ge, Shenghua Liu, Yiwei Wang, Baolong Bi+4

♣Institute of Computing Technology, Chinese Academy of Sciences · △University of California, Merced

Language Model Steering LLM Prompting Activation Steering

Open paper AskPDF arXiv

Prompt highlighting steers a large language model to prioritize user-specified text spans during generation. A key challenge of existing Key-editing approaches is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM-

Δ

(Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, our methods match or exceed the best existing method on 19 of 20 configurations, with relative gains up to +13.0%, while halving the fluency cost of steering. Our methods also scale to long-context retrieval, outperforming the best existing method by up to +5.1% relative gain. PRISM-

Δ

is compatible with FlashAttention and adds negligible memory overhead.

Abstract

Language Model Steering

Explore similar work

PRISM-ΔΔΔ: Differential Subspace Steering for Prompt Highlighting in Large Language Models

Don't Lose Focus: Activation Steering via Key-Orthogonal Projections

Steer Like the LLM: Activation Steering that Mimics Prompting

PRISM- $Δ$ : Differential Subspace Steering for Prompt Highlighting in Large Language Models