PRISM-$Δ$: Differential Subspace Steering for Prompt Highlighting in Large Language Models

Yuyao Ge; Shenghua Liu; Yiwei Wang; Baolong Bi; Lingrui Mei; Jiayu Yao; Jiafeng Guo; Xueqi Cheng

cs.CLMar 11, 2026

PRISM- $Δ$ : Differential Subspace Steering for Prompt Highlighting in Large Language Models

Authors: Yuyao Ge, Shenghua Liu, Yiwei Wang, Baolong Bi, Lingrui Mei, Jiayu Yao, Jiafeng Guo, Xueqi Cheng

Organizations: ♣Institute of Computing Technology, Chinese Academy of Sciences · △University of California, Merced

Abstract

Prompt highlighting steers a large language model to prioritize user-specified text spans during generation. A key challenge of existing Key-editing approaches is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM- $Δ$ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, our methods match or exceed the best existing method on 19 of 20 configurations, with relative gains up to +13.0%, while halving the fluency cost of steering. Our methods also scale to long-context retrieval, outperforming the best existing method by up to +5.1% relative gain. PRISM- $Δ$ is compatible with FlashAttention and adds negligible memory overhead.

Figures & tables

Figure 1: Overview of Prism- $\Delta$ . SVD decomposes $\Omega_{\Delta}$ into per-head projections ( $P_{K}$ , $P_{V}$ ) and importance weights ( $w_{\ell,h}$ ), steering both Key and Value channels at inference.

Method	Steering target	Projection	Head selection	FlashAttn
Prefix-Tuning	Key + Value (prefix)	Gradient	None	✓
PASTA	Attn. matrix $\alpha$	None	Profiling	✗
SPA	Logit distribution	None	None	✓
Spotlight	Attn. logits (bias)	Analytic	None	✗
REAL	Head activation	Codebook	Scoring top- $S$	✓
SEKA	Key	Indep. SVD	Hard threshold	✓

Table 1: Comparison of prompt highlighting methods.

Panel of Figure 2 — Figure 2: Dual-channel discriminative signals in Qwen3-4B -Base. (a) Each point is one attention head; Key and Value shifts are weakly correlated ( $r{=}0.342$ ), confirming that the two channels carry complementary information. (b) Key signal peaks in middle layers, while Value signal peaks in late layers, suggesting functional specialization across depth.

Panel of Figure 3 — Figure 3: Projection matrix structure for layer 21, head 4 of Qwen3-4B with $d{=}128$ . Independent projections $P^{+}$ and $P^{-}$ exhibit overlapping subspaces, while the differential projection $P_{\Delta}$ directly targets the discriminative subspace.

BiasBios
Model	Metric	Baselines					Ours
Model	Metric	Vanilla	**-marked	PASTA	SPA	SEKA	Prism- $\Delta$	Prism- $\Delta$ V
Qwen3-4B	Accuracy	79.80	82.94 ▲3.14	89.58 ▲9.78	68.00 ▼11.80	90.92 ▲11.12	92.38 ▲12.58	92.36 ▲12.56
Qwen3-8B		76.22	80.60 ▲4.38	86.32 ▲10.10	37.02 ▼39.20	88.74 ▲12.52	89.62 ▲13.40	89.12 ▲12.90
Qwen3-14B		85.10	90.94 ▲5.84	88.46 ▲3.36	57.86 ▼27.24	90.28 ▲5.18	91.68 ▲6.58	91.20 ▲6.10
Gemma3-4B		89.90	91.00 ▲1.10	82.58 ▼7.32	48.02 ▼41.88	92.42 ▲2.52	92.90 ▲3.00	91.64 ▲1.74

Table 2: Main results across three benchmarks (%). Bold : best; underline : second best. Subscripts show absolute change vs. Vanilla.

Configuration	Projection	Weighting	Channels	Qwen3-4B		Qwen3-8B
Configuration	Projection	Weighting	Channels	Acc	$\Delta$	Acc	$\Delta$
Prism- $\Delta$ V	Diff. $\Omega_{\Delta}$	Softplus	K+V	92.36	+15.7	89.12	+16.9
Prism- $\Delta$ ( $-$ V)	Diff. $\Omega_{\Delta}$	Softplus	K only	92.38	+15.8	89.62	+17.6
Prism- $\Delta$ -V ( $-$ K)	Diff. $\Omega_{\Delta}$	Softplus	V only	82.44	+3.3	79.16	+3.9
$-$ Differential	Independent	Softplus	K only	91.52	+14.7	88.90	+16.6
$-$ Softplus	Differential	Uniform $w{=}1$	K only	91.42	+14.6	88.22	+15.7

Table 3: Ablation study on BiasBios (Top-1 Accuracy %).

\Delta

: relative improvement (%) over Vanilla.

Model	Vanilla	SEKA	Prism- $\Delta$	Prism- $\Delta$ V
Qwen3-4B	45.47	49.14 ▲3.67	51.43 ▲5.96	50.57 ▲5.10
Qwen3-8B	46.69	50.29 ▲3.60	52.71 ▲6.02	51.86 ▲5.17
Qwen3-14B	57.13	60.86 ▲3.73	62.57 ▲5.44	62.00 ▲4.87
Gemma3-4B	44.36	47.71 ▲3.35	49.29 ▲4.93	50.14 ▲5.78
Gemma3-12B	51.64	55.57 ▲3.93	54.43 ▲2.79	55.57 ▲3.93

Table 4: Lost-in-the-Middle: Average Exact Match (%) across 7 gold positions. Steering targets the middle region (passages 4–25). Setup details in Appendix F .

Figure 4: Direction consistency analysis.

\Omega^{+}

directions show high cross-head similarity (dominated by shared structural directions), while

\Omega_{\Delta}

directions are nearly independent (close to random baseline), confirming that differential projection extracts head-specific discriminative directions.

Method	Avg. Latency (s)	Relative	Peak Mem (GB)	FlashAttn
Vanilla	1.180	1.0 $\times$	26.39	✓
SEKA	1.194 (+0.01)	1.0 $\times$	26.43 (+0.04)	✓
Prism- $\Delta$	1.481 (+0.30)	1.26 $\times$	26.41 (+0.02)	✓
Prism- $\Delta$ V	1.502 (+0.32)	1.27 $\times$	26.43 (+0.04)	✓

Table 5: Inference efficiency on Qwen3-8B .

Configuration	Top-1 Acc (%)	Fluency ( $\uparrow$ )	Consistency ( $\uparrow$ )	Fluency Cost ( $\downarrow$ )
Vanilla	79.80	4.638	0.116	0.000
SEKA	90.92 ▲11.12	3.681	0.112	0.957
Prism- $\Delta$	92.38 ▲12.58	4.134	0.127	0.504
Prism- $\Delta$ -V	82.44 ▲2.64	4.666	0.121	$-$ 0.028
Prism- $\Delta$ V ( $g_{V}{=}0.10$ )	92.36 ▲12.56	4.116	0.125	0.522

Table 6: Dual-channel decomposition on BiasBios with Qwen3-4B . Best per metric in bold . Fluency Cost = Vanilla Fluency

-

Method Fluency.

Figure 5:

\delta_{\min}

sensitivity on BiasBios, Qwen3-4B .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition	Description
Model architecture
$L$	Number of layers	Total Transformer layers in the model
$n_{h}$	Number of heads	KV heads per layer
$d$	Head dimension	Dimensionality of each head’s Key/Value space
$(\ell,h)$	Layer-head index	Identifies a specific attention head
$\mathbf{x}_{j}$	Token embedding	Input representation of token $j$

Appendix

Table 7: Notation used throughout the paper.

Benchmark	Model	SEKA	Prism- $\Delta$	Prism- $\Delta$ V	AdaSEKA	$\Delta$
BiasBios	Qwen3-4B	90.92	92.38	92.36	91.86	+0.52
	Qwen3-8B	88.74	89.62	89.12	88.50	+1.12
	Qwen3-14B	90.28	91.68	91.20	91.22	+0.46
	Gemma3-4B	92.42	92.90	91.64	92.92	$-$ 0.02
	Gemma3-12B	93.04	92.22	91.94	91.14	+1.90
CounterFact	Qwen3-4B	99.02	99.14	98.08	98.90	+0.24

Appendix

Table 8: Full comparison including AdaSEKA (%). Bold : best among lightweight methods; underline : best overall (including AdaSEKA).

\Delta

is the best lightweight method minus AdaSEKA.

Model	Benchmark	$\gamma$	$\delta_{\min}$	$g_{K}$
Qwen3-4B	BiasBios	0.998	0.08	0.40
Qwen3-8B		0.998	0.08	0.40
Qwen3-14B		0.998	0.08	0.40
Gemma3-4B		0.850	0.08	0.50
Gemma3-12B		0.980	0.00	0.40
Qwen3-4B	CounterFact	0.998	0.13	1.90

Appendix

Table 9: Prism-

\Delta

hyperparameters.

Model	Benchmark	$\gamma$	$\delta_{\min}$	$g_{K}$	$g_{V}$
Qwen3-4B	BiasBios	0.998	0.08	0.40	0.10
Qwen3-8B		0.998	0.08	0.40	0.10
Qwen3-14B		0.998	0.08	0.40	0.10
Gemma3-4B		0.800	0.12	0.30	0.10
Gemma3-12B		0.994	0.00	0.40	0.10
Qwen3-4B	CounterFact	0.998	0.13	1.90	0.02

Appendix

Table 10: Prism-

\Delta

V hyperparameters.

Model	Benchmark	Metric	Vanilla	SEKA	Prism- $\Delta$	Prism- $\Delta$ V
Qwen3-4B-Instruct	BiasBios	Accuracy	82.14	91.56	93.10	92.84
Qwen3-4B-Instruct	CounterFact	Efficacy	58.42	99.18	99.32	99.16
Qwen3-4B-Instruct	Pronoun Change	P. Score	96.52	97.24	97.86	97.94
Qwen3-8B	BiasBios	Accuracy	83.46	91.28	92.54	92.58
Qwen3-8B	CounterFact	Efficacy	54.76	99.22	99.38	99.12
Qwen3-8B	Pronoun Change	P. Score	98.86	99.12	99.54	99.62

Appendix

Table 11: Instruction-tuned results, averaged over five projection subsets. All values are percentages.

Benchmark	Selector	Recall	Vanilla	Prism- $\Delta$	Gain
BiasBios	BM25	0.58	79.80	86.42	+6.62
	Contriever	0.68	79.80	89.14	+9.34
	BGE-M3	0.76	79.80	90.86	+11.06
	Oracle span	1.00	79.80	92.38	+12.58
Lost-in-the-Middle	BM25	0.48	45.47	47.86	+2.39
	Contriever	0.62	45.47	49.42	+3.95

Appendix

Table 12: End-to-end accuracy with automatic span selection on Qwen3-4B , averaged over three seeds.

	BiasBios / Qwen3-4B		Pronoun Change / Gemma3-4B
Method	Factuality	Coherence	Factuality	Coherence
Vanilla	3.12	4.28	2.45	4.15
SEKA	3.68	3.15	3.82	3.22
Prism- $\Delta$	3.82	3.78	4.25	3.68
Prism- $\Delta$ V	3.81	4.05	4.32	3.85

Appendix

Table 13: LLM-as-Judge evaluation on a 1–5 scale. On the pooled 50-sample validation set, judge and human scores have Pearson

r=0.87

Model	Task	Metric	Vanilla	SEKA	Prism- $\Delta$	Prism- $\Delta$ -V	Prism- $\Delta$ V
Qwen3-4B	BiasBios	Accuracy	79.800	90.920	92.380	82.440	92.360
		Fluency	4.638	3.681	4.134	4.666	4.116
		Fluency cost	0.000	0.957	0.504	$-$ 0.028	0.522
Gemma3-4B	BiasBios	Accuracy	89.900	92.420	92.900	90.120	91.640
		Fluency	4.850	3.920	4.410	4.902	4.398
		Fluency cost	0.000	0.930	0.440	$-$ 0.052	0.452

Appendix

Table 14: Extended dual-channel decomposition. Fluency cost is Vanilla fluency minus method fluency.

Model	Target	Vanilla	Task-specific	Transfer	Degradation
Qwen3-4B	CounterFact	45.00	99.14	94.86	$-$ 4.28
Qwen3-4B	BiasBios	79.80	92.38	87.42	$-$ 4.96
Gemma3-4B	CounterFact	55.04	98.10	91.24	$-$ 6.86
Gemma3-4B	BiasBios	89.90	92.90	91.08	$-$ 1.82

Appendix

Table 15: Cross-domain projection transfer. The target-domain gain is retained without rebuilding the projection. Degradation is measured from the task-specific projection.

Model	BiasBios $\rightarrow$ CounterFact	CounterFact $\rightarrow$ BiasBios	Average
Qwen3-4B	0.64	0.60	0.62
Gemma3-4B	0.64	0.78	0.71

Appendix

Table 16: Frobenius similarity between source- and target-domain projection subspaces.

Noise ratio	Accuracy	$\boldsymbol{\Delta}$ $\downarrow$
0%	92.38	0.00
5%	91.95	$-$ 0.43
10%	91.42	$-$ 0.96
15%	90.68	$-$ 1.70
20%	89.75	$-$ 2.63

Appendix

Table 17: Robustness to irrelevant tokens substituted into highlighted spans on Qwen3-4B BiasBios, averaged over three seeds.

\Delta

is the accuracy change relative to the noise-free setting.

Model	Layers	Heads	K mean	V mean	K/V ratio
Qwen3-4B	36	8	0.137	0.135	1.02
Qwen3-8B	36	8	0.129	0.174	0.74
Qwen3-14B	40	8	0.138	0.237	0.58
Gemma3-4B	34	4	0.451	0.338	1.33
Gemma3-12B	48	8	0.509	0.420	1.21

Appendix

Table 18: Key and Value discriminative signal strength across models.

Figure 6: Layer-wise Key and Value discriminative signal on BiasBios for five models. In these measurements, Qwen3 models show increasing Value dominance in late layers, while Gemma3 models remain Key-dominant.

Seed	Qwen3-4B	Qwen3-8B
1	92.22	89.08
2	92.12	89.34
3	92.08	88.98
4	92.14	89.30
5	92.12	89.34
Mean $\pm$ Std	92.14 $\pm$ 0.05	89.21 $\pm$ 0.15

Appendix

Table 19: Projection sensitivity across overlapping random subsets (BiasBios, Top-1 Accuracy %).

Configuration	Projection	Weighting	Efficacy (%)
Prism- $\Delta$	Differential	Softplus	99.14
Indep. + softplus	Independent	Softplus	98.84
Diff. + uniform	Differential	Uniform	98.62

Appendix

Table 20: Ablation on CounterFact with Qwen3-4B (Efficacy %, test set).

Synthetic samples	Top-1 Accuracy (%)
50	91.92
100 (default)	92.38
200	92.40

Appendix

Table 21: Effect of synthetic data quantity on Prism-

\Delta

for Qwen3-4B on BiasBios.

	$\Omega^{+}$ (independent)	$\Omega_{\Delta}$ (differential)	Random baseline
Cross-head similarity (mean)	0.254	0.068	0.071
Cross-head similarity (early layers)	0.353	0.053	0.071

Appendix

Table 22: Cross-head cosine similarity of top singular vectors on Qwen3-4B .

Figure 7: Head weight heatmaps. Left: Prism- $\Delta$ softplus weights vary continuously across heads. Right: SEKA hard thresholding deactivates 108 heads, including 90% of early-layer heads.

Figure 8: Per-sample target log-probability comparison on BiasBios with Qwen3-4B . Prism-

\Delta

rescues 154 samples that SEKA misses and loses 81, for a net gain of 73.

Language Model Steering

Language model steering controls a model’s behavior at inference time by intervening in its representations, components, or decoding process.

See all 278 papers

Steer Like the LLM: Activation Steering that Mimics Prompting
May 5, 2026Geert Heyman, Frederik VandeputteLanguage Model Steering LLM Prompting
Nokia Bell Labs, Belgium.
Spectral Attention Steering for Prompt Highlighting
Mar 1, 2026Weixian Waylon Li, Yuchen Niu, Yongxin Yang +3Language Model Steering Efficient Attention
University of Edinburgh, UK · RayNeo, China · Queen Mary University of London, UK +1
Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?
May 28, 2026Pawel Batorski, Abtin Pourhadi, Jerzy Sarosiek +2Prompt Sensitivity LLM Prompting
1Heinrich Heine University Düsseldorf · 2Jagiellonian University · 3IDEAS Research Institute

May 5, 2026J/K move · Enter open · S save

Steer Like the LLM: Activation Steering that Mimics Prompting

Geert Heyman, Frederik Vandeputte

Nokia Bell Labs, Belgium.

Language Model Steering LLM Prompting Causal Interventions in Language Models Activation Steering

Open paper AskPDF arXiv

Large language models can be steered at inference time through prompting or activation interventions, but activation steering methods often underperform compared to prompt-based approaches. We propose a framework that formulates prompt steering as a form of activation steering and investigates whether distilling successful prompt steering behavior into simpler, interpretable models can close this gap. Our analysis reveals that popular activation steering methods are not faithful to the mechanics of prompt steering, which applies strong interventions on some tokens while barely affecting others. Based on these insights, we introduce Prompt Steering Replacement (PSR) models that estimate token-specific steering coefficients from the activations themselves and are trained to imitate prompt-based interventions. Experiments on three steering benchmarks across multiple language models show that PSR models outperform existing activation steering methods, especially when controlling for high-coherence completions, and also compare favorably to prompting on AxBench and persona steering.

Abstract

Language Model Steering

Explore similar work

Steer Like the LLM: Activation Steering that Mimics Prompting

Spectral Attention Steering for Prompt Highlighting

Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?