Attention steering is an important technique for controlling model focus, enabling capabilities such as prompt highlighting, where the model prioritises user-specified text. However, existing attention steering methods require explicit storage of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compatibility with optimised attention.
Figures & tables
Figure 1: Visualisation of pairwise key embedding shifts across different (layer, head) in Qwen3-1.7B-Base via PCA. Positive vs. negative representations are plotted for 26 shared token spans. Grey arrows trace individual shifts; the dark blue arrow shows the average displacement.
Figure 2: An overview of SEKA and AdaSEKA . x : context; h : key embedding; Ω : cross-covariance; U : left singular vectors; S : singular values; g : gain coefficient. SEKA applies fixed gains, while AdaSEKA uses the query to compute dynamic steering weights.
Figure 3: Heatmaps of the average per-token ℓ2 distance between positive and negative key embeddings across all KV heads and layers for four Qwen3 model sizes. Higher values ( green ) indicate greater separation between positive and negative key representations.
[person’s occupation] . [career history, may not directly related to prediction] . [person] has the occupation of a/an
Accuracy (Acc.)
Pronouns changing
[biographical contexts] . Substitute ‘she’ and ‘he’ with ‘they’ and generate the occupation of [person] after changing pronouns.
Pronoun-weighted Lexical overlap Score (P. Score), All-changed P. Score
Table 1: Summary of standard benchmarks for attention steering. Tokens in bold indicate where attention steering is applied.
Model
Metric
Baselines
Our Methods
Original
**-marked
PASTA
SPA
SEKA
w/o learn
w/o learn&filt
AdaSEKA
Qwen3-4B
CounterFact (ES)
45.00
57.70
97.16
65.24
99.02
94.96
86.12
98.90
CounterFact (PS)
45.64
52.12
96.03
57.71
98.61
92.38
86.20
98.72
Bias in Bios (Acc.)
79.84
82.94
89.58
68.00
91.02
86.62
71.76
91.86
Pronoun (P. Score)
93.14
95.76
95.82
80.27
95.18
90.42
41.98
94.54
Pronoun (A. P. Score)
90.52
93.88
94.64
78.19
93.26
88.66
36.95
92.08
Table 2: Performance on standard benchmarks. Bold = best. Underline = second best. We include two ablation studies for SEKA : “w/o learn” uses random projections instead of spectrally learned ones, and “w/o learn&filt” further removes the head filtering mechanism.
Figure 4: Exact match scores on the lost-in-the-middle task for Qwen3 models of three different sizes, comparing the original model, PASTA/ SEKA applied to the middle region (5 th to 25 th passages), and PASTA/ SEKA applied to all passages.
Figure 5: Exact match scores when applying SEKA to the middle region with different threshold δmin .
Method
Avg. Time (s)
Peak Mem. (GB, B=10)
Peak Mem. (GB, B=1)
Original
0.55
27.63
16.72
PASTA
1.58 (+1.03)
50.75 (+23.12)
-
SPA
5.87 (+5.32)
-
17.71 (+0.99)
SEKA
0.58 (+0.03)
27.66 (+0.03)
16.75 (+0.03)
AdaSEKA
0.82 (+0.27)
43.22 (+15.59)
18.23 (+1.51)
Table 3: Inference overhead on Qwen3-8B-Base. Time is per-sample; memory is average peak usage.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Group
Prompt
Neutral
Context: The portfolio manager allocates capital across equities and bonds.
Positive
Question: What does the portfolio manager allocate across equities and bonds? Context: The portfolio manager allocates capital across equities and bonds.
Negative
Question: What does the climate model simulate? Context: The portfolio manager allocates capital across equities and bonds.
Neutral
Context: The climate model simulates sea-level rise under different scenarios.
Positive
Question: What does the climate model simulate? Context: The climate model simulates sea-level rise under different scenarios.
Negative
Question: What does the portfolio manager allocate across equities and bonds? Context: The climate model simulates sea-level rise under different scenarios.
Appendix
Table 4: Constructed prompt triplets for both answer spans. Each group provides a neutral, positive, and negative variant based on question-context alignment.
Context 1 ( C1 )
The portfolio manager allocates capital across equities and bonds.
Context 2 ( C2 )
The climate model simulates sea-level rise under different scenarios.
Question 1 ( Q1 )
What does the portfolio manager allocate across equities and bonds?
Answer 1 ( A1 )
capital
Question 2 ( Q2 )
What does the climate model simulate?
Answer 2 ( A2 )
sea-level rise
Appendix
Table 5: Synthetic data instance.
Figure 6: Prompt template used to generate synthetic contrastive examples.
Dataset
Variant
Prompt
CounterFact
Neutral
Context: Previously, The mother tongue of Danielle Darrieux is French. Currently, The mother tongue of Danielle Darrieux is English .
Positive
Question: Danielle Darrieux, a native ? Context: Previously, The mother tongue of Danielle Darrieux is French. Currently, The mother tongue of Danielle Darrieux is English .
BiasBios
Neutral
Context: Nora is an assistant professor of international relations at Bahcesehir University in Istanbul. She is also a Ronald D. Asmus Policy Entrepreneur Fellow with the German Marshall Fund and is a Visiting Fellow at the Centre for International Studies (CIS) at the University of Oxford. This commentary first appeared at Sada, an online journal published by the Carnegie Endowment for International Peace.
Positive
Question: Nora has the occupation of a/an ? Context: Nora is an assistant professor of international relations at Bahcesehir University in Istanbul. She is also a Ronald D. Asmus Policy Entrepreneur Fellow with the German Marshall Fund and is a Visiting Fellow at the Centre for International Studies (CIS) at the University of Oxford. This commentary first appeared at Sada, an online journal published by the Carnegie Endowment for International Peace.
HotpotQA
Neutral
Context: Radio City (Indian radio station) Radio City is India’s first private FM radio station and was started on 3 July 2001. … Arthur’s Magazine Arthur’s Magazine (1844–1846) was an American literary periodical published in Philadelphia in the 19th century. Edited by T.S. Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others. In May 1846 it was merged into “Godey’s Lady’s Book”. … First for Women First for Women is a womanś magazine published by Bauer Media Group in the USA. The magazine was started in 1989. It is based in Englewood Cliffs, New Jersey. … The company started first as a denim line, later evolving into a men’s and women’s clothing line.
Positive
Question: Which magazine was started first Arthur’s Magazine or First for Women? Context: Radio City (Indian radio station) Radio City is India’s first private FM radio station and was started on 3 July 2001. … Arthur’s Magazine Arthur’s Magazine (1844–1846) was an American literary periodical published in Philadelphia in the 19th century. Edited by T.S. Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others. In May 1846 it was merged into “Godey’s Lady’s Book”. … First for Women First for Women is a woman’s magazine published by Bauer Media Group in the USA. The magazine was started in 1989. It is based in Englewood Cliffs, New Jersey. … The company started first as a denim line, later evolving into a men’s and women’s clothing line.
Appendix
Table 6: Constructed prompt pairs for multi-expert projection learning. Each dataset provides neutral and positive variants based on question-context alignment.
SEKA
AdaSEKA
Model
Task
γ
δmin
g+
g−
δmin
g
Qwen3-4B-Base
CounterFact
0.960
0.13
1.56
0.00
0.1
3.0
Bias in Bios
0.998
0.12
1.00
0.80
0.1
0.5
Pronoun Changing
0.880
0.22
0.16
0.00
0.5
0.6
Qwen3-8B-Base
CounterFact
0.850
0.12
2.40
0.00
0.1
3.0
Bias in Bios
0.998
0.12
0.60
0.30
0.1
0.5
Appendix
Table 7: Hyperparameters for SEKA and AdaSEKA methods. SEKA uses the variance threshold ( γ ), KV-head selection threshold ( δmin ), positive gain ( g+ ), and negative gain ( g− ). AdaSEKA uses the KV-head selection threshold ( δmin ) and steering gain ( g ).
Figure 7: Sensitivity of SEKA to hyper-parameters across three benchmark tasks. Each curve varies a single hyper-parameter while keeping others fixed at their optimal settings on the validation set.
Figure 8: Layer-wise mean attention (all heads) in Qwen3-4B-Base at selected layers for the CounterFact data sample, shown before and after SEKA is applied.
Model
PASTA Configuration
CounterFact
Bias in Bios
Pronoun Changing
ES
PS
Acc.
P. Score
A. P. Score
Qwen3-4B-Base
SEKA KV-heads
83.62
80.43
89.58
55.40
54.08
Transformed attention heads
82.60
83.02
79.34
95.82
94.64
Original head-search
97.16
96.03
87.84
47.20
47.20
Qwen3-8B-Base
SEKA KV-heads
82.08
71.72
86.32
62.91
61.44
Transformed attention heads
78.20
79.69
77.30
98.86
98.72
Appendix
Table 8: Complete PASTA results with different configurations: (1) using SEKA ’s KV-head configuration (runtime 1–2 minutes), (2) using attention head configuration transformed from SEKA ’s KV-heads (1–2 minutes), and (3) using PASTA’s original head-search routine ( ≈ 2 hours).
Prompt
Previously David Sainsbury, Baron Sainsbury of Turville used to work in London. Currently David Sainsbury, Baron Sainsbury of Turville used to work in Berlin. After 15 years of work, he published Pillboxes in 1985. David Sainsbury, Baron Sainsbury of Turville took up work in
Appendix
Table 9: CounterFact Example 1: Overriding a known location.
Prompt
Previously Jean Baptiste Pompallier holds the position of bishop. Currently Jean Baptiste Pompallier holds the position of cardinal. Jean Baptiste Pompallier has the position of
Appendix
Table 10: CounterFact Example 2: Overriding a known job title.
Prompt
Tania is an Illinois poet who has published one chapbook… and two full-length collections . Her most recent book… is divided into ten sections… Its predecessor Simple Weight is also concerned with the Biblical narrative… Barbara Crooker said of that collection, ”The poems have weight emotional, spiritual, political but are anything but simple.”.Tania has the occupation of a/an
Appendix
Table 11: Bias in Bios Example 1: Correcting a prediction.
Prompt
Michelle, PhD is an Adjunct Assistant Professor in the Emory Department of Psychiatry and Behavioral Sciences . She received her Ph.D. in 2003 from Emory University. Her areas of expertise include intergenerational processes of risk and resilience… She supervises postdoctoral residents… Michelle has the occupation of a/an
Appendix
Table 12: Bias in Bios Example 2: Improving confidence in the correct answer.
Prompt
Jason is an assistant professor of English… For the aforementioned text, substitute ‘she’ and ‘he’ with ‘they’ and generate the occupation of Jason after changing pronouns.
Appendix
Table 13: Pronouns Changing Example 1.
Prompt
Anna, JD, MA, LPC, is a depth psychotherapist in Houston, Texas… She holds degrees in philosophy, law, and clinical psychology… For the aforementioned text, substitute ‘she’ and ‘he’ with ‘they’ and generate the occupation of Anna after changing pronouns.
Appendix
Table 14: Pronouns Changing Example 2.
Figure 9: The performance of SEKA with varying numbers of synthetic samples used for learning projections across different models and tasks.
Figure 10: Exact match scores on the lost-in-the-middle task when applying SEKA to the middle region with different δmin thresholds for Qwen3-4B and Qwen3-14B.
Prompt highlighting steers a large language model to prioritize user-specified text spans during generation. A key challenge of existing Key-editing approaches is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM-Δ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, our methods match or exceed the best existing method on 19 of 20 configurations, with relative gains up to +13.0%, while halving the fluency cost of steering. Our methods also scale to long-context retrieval, outperforming the best existing method by up to +5.1% relative gain. PRISM-Δ is compatible with FlashAttention and adds negligible memory overhead.
Yuyao Ge, Shenghua Liu, Yiwei Wang +5
♣Institute of Computing Technology, Chinese Academy of Sciences · △University of California, Merced
Activation steering controls LLM behaviour towards target behaviour by intervening in internal representations, yet it often degrades reasoning and retrieval performance. We argue that a primary cause of this trade-off is attention rerouting: steering vectors alter query-key matching, shifting attention away from contextually important tokens toward less informative ones. To address this, we propose Steering via Key-Orthogonal Projections (SKOP), a steering method that constrains harmful attention rerouting without eliminating steering efficacy. SKOP achieves this by preserving attention patterns on a small set of focus tokens the model relies on for reasoning and retrieval, while allowing redistribution among less critical tail tokens. Across multiple steering benchmarks, we show that SKOP achieves the best joint steering-utility trade-off, reducing utility degradation by 5-7x while retaining over 95% of vanilla steering efficacy. Our results further suggest that, in long-context retrieval settings where vanilla steering approaches are ineffective, SKOP can maintain robust performance by avoiding attention rerouting.
Large language models can be steered at inference time through prompting or activation interventions, but activation steering methods often underperform compared to prompt-based approaches. We propose a framework that formulates prompt steering as a form of activation steering and investigates whether distilling successful prompt steering behavior into simpler, interpretable models can close this gap. Our analysis reveals that popular activation steering methods are not faithful to the mechanics of prompt steering, which applies strong interventions on some tokens while barely affecting others. Based on these insights, we introduce Prompt Steering Replacement (PSR) models that estimate token-specific steering coefficients from the activations themselves and are trained to imitate prompt-based interventions. Experiments on three steering benchmarks across multiple language models show that PSR models outperform existing activation steering methods, especially when controlling for high-coherence completions, and also compare favorably to prompting on AxBench and persona steering.