Steering vectors (SVs) are residual-stream directions that shift the behavior of LLMs when added to hidden states at inference time. While prior work has mainly used SVs as post-hoc controls, we ask whether the same directions can also guide parameter updates during post-training. We study this question across supervised learning, including SFT and distillation, and reinforcement learning with verifiable rewards (RLVR). Our framework considers two intervention regimes: a hidden-state hook during teacher-forced supervised training, and an advantage-repair mechanism on wrong rollouts in GRPO. For each regime, we analyze three design axes, which are the steering direction, its placement, and the operation used to combine it with training. In supervised settings, SV injection improves over plain SFT and distillation, with gains scaling with the off-policy gap and reaching up to a 12% improvement when distilling from thinking-mode teachers. These gains hold across three random seeds. In RL settings, applying a correctness-contrast direction as advantage repair on wrong rollouts, including otherwise zero-gradient all-wrong groups, recovers a learning signal that transfers to held-out math benchmarks and acts as a composable auxiliary signal that adds to existing zero-variance remedies such as entropy-aware advantage shaping. We further identify three failure boundaries. In detail, code generation can break the direction, cross-family layer transfer can break the placement, and inference-time application on an SV-trained checkpoint can break the intervention regime. Together, these results establish steering vectors as a conditional but effective training signal for LLM post-training. Our code is available here.
Figures & tables
Figure 1: Overall framework. Panel (A) precomputes the off-policy displacement vector, which serves as the injection signal for the Supervised Regime in (B). In parallel, the Reinforcement Learning Regime in (C) applies the steering mechanism independently to rescue zero-variance rollout groups.
Model
Dataset
Plain
Mean-SV
Per-sample Slerp
Qwen2.5-1.5B
GSM8K
59.14
62.85
64.14
MATH
24.60
27.00
27.20
MBPP
47.00
45.80
45.60
StrategyQA
62.74
64.48
63.61
Qwen2.5-7B
GSM8K
76.88
80.14
80.21
MATH
43.20
44.00
47.40
Table 1: Comparison of test accuracy (%) after SFT across four models and datasets. The best result per row is shown in bold . SV-augmented SFT improves over plain SFT in general.
Figure 2: Analysis of our SV-augmented SFT on Qwen2.5-1.5B with GSM8K.
Method
α
Accuracy
Δ
Plain LoRA (no SV)
–
59.14
0
Random per-sample Gaussian (control)
0.50
54.82
−4.32
PCA SV
0.50
59.29
+0.15
Multi-layer (L=10/15/20)
0.50/0.50/0.50
60.85
+1.71
Multi-layer (L=10/15/20)
0.10/0.50/0.10
62.62
+3.48
Per-sample additive
0.50
62.93
+3.79
Table 2: Four representative SV injection variants for SV SFT on Qwen2.5-1.5B with GSM8K. The full set of variants we tested is reported in Appendix Appendix C . The best result is shown in bold . Per-sample Slerp outperforms every alternative SV construction we tested.
Student
Dataset
Qwen3-4B-Instruct
Qwen3-8B
Gemma-4-26B (no-thinking)
Gemma-4-26B (thinking)
Plain
Mean-SV
Slerp
Plain
Mean-SV
Slerp
Plain
Mean-SV
Slerp
Plain
Mean-SV
Slerp
Qwen2.5-1.5B
GSM8K
74.83
74.83
74.68
65.81
67.55
67.40
70.58
71.11
70.43
49.58
60.42
60.50
MATH
41.60
41.40
42.20
21.20
31.20
33.60
36.60
37.60
37.40
11.80
15.00
14.00
MBPP
49.00
48.80
50.00
46.00
45.60
45.40
48.80
48.00
48.80
27.00
31.80
37.40
StrategyQA
56.91
58.08
55.31
55.31
59.83
60.84
61.86
60.99
58.52
20.82
19.07
21.11
Qwen3-4B
GSM8K
87.87
88.02
87.95
76.42
75.51
78.62
86.88
87.34
86.73
62.70
52.77
67.63
Table 3: Comparison of test accuracy (%) across three students, four teachers, and four datasets. The best result is shown in bold . SV-augmented distillation improves over plain distillation on math, with the largest gains on thinking-mode teachers.
Figure 3: Analysis on distillation settings. SV-augmented distillation closes the off-policy gap, and the SV direction is reshaped more by switching teacher mode than by switching teacher family.
1.5B GSM8K
7B GSM8K
1.5B MATH
Qwen3-4B MATH
7B MATH
1.5B MATH (Distill)
1.5B MBPP
Plain
58.81±0.37
77.66±1.73
25.80±0.53
42.47±2.20
42.47±1.22
18.13±2.30
46.13±0.31
Per-sample Slerp
62.75±0.50
80.74±0.49
27.00±0.20
47.07±1.70
46.20±1.40
33.47±1.14
46.47±1.01
Δ
+3.94
+3.08
+1.20
+4.60
+3.73
+15.34
+0.34
Table 4: Accuracy (%, mean ± std over three seeds) of Plain and Per-sample Slerp. “Distill” uses the Qwen3-8B thinking teacher.
Table 8
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model, dataset
Plain
SV during training
SV at inference
Qwen2.5-1.5B, GSM8K
59.14
64.14 ( +5.00 )
44.43 ( −14.71 )
Qwen2.5-1.5B, MATH
24.60
27.20 ( +2.60 )
19.60 ( −5.00 )
Qwen2.5-7B, GSM8K
76.88
80.21 ( +3.33 )
71.87 ( −5.01 )
Qwen3-4B, GSM8K
80.06
84.61 ( +4.55 )
77.03 ( −3.03 )
Qwen2.5-1.5B, distill from Qwen3-8B, GSM8K
65.81
67.55 ( +1.74 )
63.53 ( −2.28 )
Qwen2.5-1.5B, distill from Qwen3-8B, MATH
21.20
31.20 ( +10.00 )
10.80 ( −10.40 )
Appendix
Table 7: The same SV used during training versus at inference. “SV at inference” injects the training direction ( α=0.5 ) into the Plain-SFT checkpoint at test time. Training with the SV helps in every cell; injecting it at inference hurts in every cell.
Method
α
Accuracy
Δ
Plain LoRA (no SV)
–
59.14
0
Negated Mean-SV additive (sign-flip control)
−0.50
54.58
−4.56
Random per-sample Gaussian (control)
0.50
54.82
−4.32
α decay schedule (linear 0.5→0 )
0.5→0
58.07
−1.07
Gated Slerp (drop cos<0.7 )
0.10
58.45
−0.69
Slerp first- K=64 (extended scope)
0.10
58.61
−0.53
Appendix
Table 8: Full sweep of SV injection variants on Qwen2.5-1.5B with GSM8K at ℓ⋆=15 . Methods are sorted by accuracy.
Figure 4: Full off-policy mitigation sweep on all 10 cells (eight distillation cells of the headline grid plus the two SFT cells), in two complementary scales. Eight of ten cells are positive at frac(ΔNLLsum>0)≥73% ; the two near-zero/negative cells in panel (a) are exactly the Qwen2.5-1.5B × non-thinking-teacher distillation cells that also show no SV-distill accuracy gain in Table 3 . The linear-probability metric in panel (b) is non-negative on every cell because the per-token probability mass moved onto label tokens is positive even on the cells where token-level NLL gains and losses cancel.
Figure 5: Three statistics of the SV direction at ℓ⋆=15 broken down by teacher mode and averaged over the 12 (student, dataset) cells of each teacher, namely the direction norm in panel (a), the per-sample alignment in panel (b), and Cohen’s d for the GT-versus-Pred projection in panel (c). Both thinking teachers dominate the two non-thinking teachers on every axis, while the two non-thinking teachers (different families) are statistically indistinguishable on all three quantities.
Figure 6: Accuracy under a normalized Mean-SV α -sweep on the Qwen3-thinking-teacher MATH cells, where the steering direction is unit-normalized so that α is interpretable as the literal injection magnitude rather than a Slerp rotation fraction. Plain SFT is shown as a dashed grey line and the Per-sample Slerp α=0.1 result from Table 3 as a dotted orange line. Both panels show that a tuned α∈[10,30] on the normalized direction outperforms Per-sample Slerp at α=0.1 .
Cell (student × teacher × dataset)
∥v∥
mean cos
Cohen’s d
SV Δ (pp)
7B × Qwen3-thinking × MBPP
22.39
0.801
7.10
−1.80
Qwen3-4B × Qwen3-thinking × MBPP
12.93
0.814
6.77
0.00
1.5B × Qwen3-thinking × MBPP
18.33
0.835
5.85
−0.60
Qwen3-4B × Qwen3-thinking × GSM8K
5.51
0.783
5.84
−2.20
7B × Qwen3-thinking × GSM8K
10.57
0.778
5.79
+3.94
7B × Qwen3-thinking × MATH
11.23
0.749
4.76
+19.80
Appendix
Table 9: The six (student, teacher, dataset) cells with the largest Cohen’s d in the distillation grid, alongside the direction norm ∥v∥ , the per-sample alignment cos , and the Per-sample Slerp gain over Plain. A high Cohen’s d on the training distribution does not imply a positive gain at test time, as illustrated by the three top-ranked MBPP cells whose gain is at most zero. The cell with the largest gain in this subset is shown in bold .
Student
Random direction
SV-Rescue (ours)
Qwen2.5-1.5B
77.54
78.35
Qwen3-4B
90.35
91.04
Appendix
Table 10: SV-Rescue with the precomputed correctness direction (ours) vs a same-norm random Gaussian direction on GSM8K. The SV direction is what carries the rescue gain.
Student
Temperature schedule only
SV-Explore (full)
Qwen2.5-1.5B
78.72
79.18
Qwen3-4B
91.79
91.99
Appendix
Table 11: SV-Explore (full) vs SV-Explore with the SV-Rescue advantage disabled (temperature schedule only) on GSM8K. SV-Rescue is not redundant once exploration is added.
Figure 7: Per-rollout Pearson correlation between the SV alignment score at each transformer block and the binary verifier reward, on a 2000 -rollout GSM8K hold-out, both before RL (init) and after RL (SV-Rescue). The vertical dashed line marks the canonical injection layer ℓ⋆=L/2+1 used by SV-Rescue and the dotted line marks the peak-correlation layer. The correlation peaks slightly above L/2+1 on both students, but the canonical L/2+1 retains 80 - 90% of the peak.
Method
Accuracy
Δ vs. Plain
Plain SFT
58.81±0.37
0
DFT (original hyperparameters)
58.88±0.80
+0.07
DFT (re-tuned for LoRA)
61.09†
+2.28
Hidden-state distillation
53.68†
−5.13
SV (Per-sample Slerp)
62.75±0.50
+3.94
DFT (original) + SV
62.64±0.73
+3.83
Appendix
Table 12: SFT-side baselines on Qwen2.5-1.5B with GSM8K (mean ± std over three seeds unless marked † ).
Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.
Recently, steering vectors (SVs) have emerged as an effective and lightweight approach to steer behaviors of large language models (LLMs), among which fine-tuned SVs are more effective than optimization-free ones. However, current approaches to fine-tuned SVs suffer from two limitations. First, they require careful selection of steering factors on a per-SV basis to balance steering effectiveness and generation quality at inference time. Second, they operate as full-sequence SVs (FSSVs), which can sacrifice generation quality regardless of factor selection due to excessive intervention on the model generation process. To address the first limitation, we propose joint training of steering factors and directions, such that post-hoc factor selection is no longer required. Using neural network scaling theory, we find that moderately large initialization sizes and learning rates for steering factors are essential for stability and efficiency of joint training. To tackle the second limitation, we draw inspiration from representation fine-tuning and introduce Prompt-only SV (PrOSV), an SV that intervenes only on a few prompt tokens. Our empirical results show that PrOSV outperforms traditional FSSVs on AxBench when using our joint training scheme. We also find that PrOSV achieves a better tradeoff between general model utility and adversarial robustness than FSSV.
Yuntai Bao, Qinfeng Li, Xinyan Yu +6
Zhejiang University · Innovation and Management Center, School of Software Technology (Ningbo), Zhejiang University · Ant Group
Steering vectors are a lightweight tool for controlling LLM behavior. However, emerging evidence shows that steering vectors can unintentionally compromise a model's safety mechanisms and increase compliance with harmful requests, while no effective mitigation yet exists. In this work, we show that this safety degradation arises from a separable component in the vector that disrupts the model's safety mechanisms but contributes little to the steering objective. We identify and remove this safety-degrading component, formulating the task as a constrained optimization problem solved through primal-dual updates, subject to preserving the intended steering effect and bounding false refusal. The resulting solution is both interpretable and surgical: the optimization recovers a single direction whose ablation from the steering vector restores model safety with minimal utility cost. Across models, steering behaviors, and attack suites, including unseen attacks types, our method substantially reduces steering-induced safety degradation while preserving the original steering effect with minimal impact on false refusal. Our method offers a post-hoc correction to steering vectors that mitigates their safety cost, and more broadly, it provides a general recipe for applying activation-level model interventions without paying a safety tax.
Yuxiao Li, Gjergji Kasneci
Technical University of Munich Munich Center for Machine Learning (MCML)
Applying steering vectors to large language models (LLMs) is an efficient and effective model alignment technique, but we lack an interpretable explanation for how it works--specifically, what internal mechanisms steering vectors affect and how this results in different model outputs. To investigate the causal mechanisms underlying the effectiveness of steering vectors, we conduct a comprehensive case study on refusal. We propose a multi-token activation patching framework and discover that different steering methodologies leverage functionally interchangeable circuits when applied at the same layer. These circuits reveal that steering vectors primarily interact with the attention mechanism through the OV circuit while largely ignoring the QK circuit. Freezing all attention scores during steering drops performance by only 8.83% across three model families. A mathematical decomposition of the steered OV circuit further reveals semantically interpretable concepts, even in cases where the steering vector itself does not. Leveraging the activation patching results, we show that steering vectors can be sparsified by up to 85-96% while retaining most performance, and that different steering methodologies agree on a subset of important dimensions.