Steering vectors (SVs) are residual-stream directions that shift the behavior of LLMs when added to hidden states at inference time. While prior work has mainly used SVs as post-hoc controls, we ask whether the same directions can also guide parameter updates during post-training. We study this question across supervised learning, including SFT and distillation, and reinforcement learning with verifiable rewards (RLVR). Our framework considers two intervention regimes: a hidden-state hook during teacher-forced supervised training, and an advantage-repair mechanism on wrong rollouts in GRPO. For each regime, we analyze three design axes, which are the steering direction, its placement, and the operation used to combine it with training. In supervised settings, SV injection improves over plain SFT and distillation, with gains scaling with the off-policy gap and reaching up to a 12% improvement when distilling from thinking-mode teachers. These gains hold across three random seeds. In RL settings, applying a correctness-contrast direction as advantage repair on wrong rollouts, including otherwise zero-gradient all-wrong groups, recovers a learning signal that transfers to held-out math benchmarks and acts as a composable auxiliary signal that adds to existing zero-variance remedies such as entropy-aware advantage shaping. We further identify three failure boundaries. In detail, code generation can break the direction, cross-family layer transfer can break the placement, and inference-time application on an SV-trained checkpoint can break the intervention regime. Together, these results establish steering vectors as a conditional but effective training signal for LLM post-training. Our code is available here.
Figures & tables
Figure 1: Overall framework. Panel (A) precomputes the off-policy displacement vector, which serves as the injection signal for the Supervised Regime in (B). In parallel, the Reinforcement Learning Regime in (C) applies the steering mechanism independently to rescue zero-variance rollout groups.
Model
Dataset
Plain
Mean-SV
Per-sample Slerp
Qwen2.5-1.5B
GSM8K
59.14
62.85
64.14
MATH
24.60
27.00
27.20
MBPP
47.00
45.80
45.60
StrategyQA
62.74
64.48
63.61
Qwen2.5-7B
GSM8K
76.88
80.14
80.21
MATH
43.20
44.00
47.40
Table 1: Comparison of test accuracy (%) after SFT across four models and datasets. The best result per row is shown in bold . SV-augmented SFT improves over plain SFT in general.
Figure 2: Analysis of our SV-augmented SFT on Qwen2.5-1.5B with GSM8K.
Method
α
Accuracy
Δ
Plain LoRA (no SV)
–
59.14
0
Random per-sample Gaussian (control)
0.50
54.82
−4.32
PCA SV
0.50
59.29
+0.15
Multi-layer (L=10/15/20)
0.50/0.50/0.50
60.85
+1.71
Multi-layer (L=10/15/20)
0.10/0.50/0.10
62.62
+3.48
Per-sample additive
0.50
62.93
+3.79
Table 2: Four representative SV injection variants for SV SFT on Qwen2.5-1.5B with GSM8K. The full set of variants we tested is reported in Appendix Appendix C . The best result is shown in bold . Per-sample Slerp outperforms every alternative SV construction we tested.
Student
Dataset
Qwen3-4B-Instruct
Qwen3-8B
Gemma-4-26B (no-thinking)
Gemma-4-26B (thinking)
Plain
Mean-SV
Slerp
Plain
Mean-SV
Slerp
Plain
Mean-SV
Slerp
Plain
Mean-SV
Slerp
Qwen2.5-1.5B
GSM8K
74.83
74.83
74.68
65.81
67.55
67.40
70.58
71.11
70.43
49.58
60.42
60.50
MATH
41.60
41.40
42.20
21.20
31.20
33.60
36.60
37.60
37.40
11.80
15.00
14.00
MBPP
49.00
48.80
50.00
46.00
45.60
45.40
48.80
48.00
48.80
27.00
31.80
37.40
StrategyQA
56.91
58.08
55.31
55.31
59.83
60.84
61.86
60.99
58.52
20.82
19.07
21.11
Qwen3-4B
GSM8K
87.87
88.02
87.95
76.42
75.51
78.62
86.88
87.34
86.73
62.70
52.77
67.63
Table 3: Comparison of test accuracy (%) across three students, four teachers, and four datasets. The best result is shown in bold . SV-augmented distillation improves over plain distillation on math, with the largest gains on thinking-mode teachers.
Figure 3: Analysis on distillation settings. SV-augmented distillation closes the off-policy gap, and the SV direction is reshaped more by switching teacher mode than by switching teacher family.
1.5B GSM8K
7B GSM8K
1.5B MATH
Qwen3-4B MATH
7B MATH
1.5B MATH (Distill)
1.5B MBPP
Plain
58.81±0.37
77.66±1.73
25.80±0.53
42.47±2.20
42.47±1.22
18.13±2.30
46.13±0.31
Per-sample Slerp
62.75±0.50
80.74±0.49
27.00±0.20
47.07±1.70
46.20±1.40
33.47±1.14
46.47±1.01
Δ
+3.94
+3.08
+1.20
+4.60
+3.73
+15.34
+0.34
Table 4: Accuracy (%, mean ± std over three seeds) of Plain and Per-sample Slerp. “Distill” uses the Qwen3-8B thinking teacher.
Table 8
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model, dataset
Plain
SV during training
SV at inference
Qwen2.5-1.5B, GSM8K
59.14
64.14 ( +5.00 )
44.43 ( −14.71 )
Qwen2.5-1.5B, MATH
24.60
27.20 ( +2.60 )
19.60 ( −5.00 )
Qwen2.5-7B, GSM8K
76.88
80.21 ( +3.33 )
71.87 ( −5.01 )
Qwen3-4B, GSM8K
80.06
84.61 ( +4.55 )
77.03 ( −3.03 )
Qwen2.5-1.5B, distill from Qwen3-8B, GSM8K
65.81
67.55 ( +1.74 )
63.53 ( −2.28 )
Qwen2.5-1.5B, distill from Qwen3-8B, MATH
21.20
31.20 ( +10.00 )
10.80 ( −10.40 )
Appendix
Table 7: The same SV used during training versus at inference. “SV at inference” injects the training direction ( α=0.5 ) into the Plain-SFT checkpoint at test time. Training with the SV helps in every cell; injecting it at inference hurts in every cell.
Method
α
Accuracy
Δ
Plain LoRA (no SV)
–
59.14
0
Negated Mean-SV additive (sign-flip control)
−0.50
54.58
−4.56
Random per-sample Gaussian (control)
0.50
54.82
−4.32
α decay schedule (linear 0.5→0 )
0.5→0
58.07
−1.07
Gated Slerp (drop cos<0.7 )
0.10
58.45
−0.69
Slerp first- K=64 (extended scope)
0.10
58.61
−0.53
Appendix
Table 8: Full sweep of SV injection variants on Qwen2.5-1.5B with GSM8K at ℓ⋆=15 . Methods are sorted by accuracy.
Figure 4: Full off-policy mitigation sweep on all 10 cells (eight distillation cells of the headline grid plus the two SFT cells), in two complementary scales. Eight of ten cells are positive at frac(ΔNLLsum>0)≥73% ; the two near-zero/negative cells in panel (a) are exactly the Qwen2.5-1.5B × non-thinking-teacher distillation cells that also show no SV-distill accuracy gain in Table 3 . The linear-probability metric in panel (b) is non-negative on every cell because the per-token probability mass moved onto label tokens is positive even on the cells where token-level NLL gains and losses cancel.
Figure 5: Three statistics of the SV direction at ℓ⋆=15 broken down by teacher mode and averaged over the 12 (student, dataset) cells of each teacher, namely the direction norm in panel (a), the per-sample alignment in panel (b), and Cohen’s d for the GT-versus-Pred projection in panel (c). Both thinking teachers dominate the two non-thinking teachers on every axis, while the two non-thinking teachers (different families) are statistically indistinguishable on all three quantities.
Figure 6: Accuracy under a normalized Mean-SV α -sweep on the Qwen3-thinking-teacher MATH cells, where the steering direction is unit-normalized so that α is interpretable as the literal injection magnitude rather than a Slerp rotation fraction. Plain SFT is shown as a dashed grey line and the Per-sample Slerp α=0.1 result from Table 3 as a dotted orange line. Both panels show that a tuned α∈[10,30] on the normalized direction outperforms Per-sample Slerp at α=0.1 .
Cell (student × teacher × dataset)
∥v∥
mean cos
Cohen’s d
SV Δ (pp)
7B × Qwen3-thinking × MBPP
22.39
0.801
7.10
−1.80
Qwen3-4B × Qwen3-thinking × MBPP
12.93
0.814
6.77
0.00
1.5B × Qwen3-thinking × MBPP
18.33
0.835
5.85
−0.60
Qwen3-4B × Qwen3-thinking × GSM8K
5.51
0.783
5.84
−2.20
7B × Qwen3-thinking × GSM8K
10.57
0.778
5.79
+3.94
7B × Qwen3-thinking × MATH
11.23
0.749
4.76
+19.80
Appendix
Table 9: The six (student, teacher, dataset) cells with the largest Cohen’s d in the distillation grid, alongside the direction norm ∥v∥ , the per-sample alignment cos , and the Per-sample Slerp gain over Plain. A high Cohen’s d on the training distribution does not imply a positive gain at test time, as illustrated by the three top-ranked MBPP cells whose gain is at most zero. The cell with the largest gain in this subset is shown in bold .
Student
Random direction
SV-Rescue (ours)
Qwen2.5-1.5B
77.54
78.35
Qwen3-4B
90.35
91.04
Appendix
Table 10: SV-Rescue with the precomputed correctness direction (ours) vs a same-norm random Gaussian direction on GSM8K. The SV direction is what carries the rescue gain.
Student
Temperature schedule only
SV-Explore (full)
Qwen2.5-1.5B
78.72
79.18
Qwen3-4B
91.79
91.99
Appendix
Table 11: SV-Explore (full) vs SV-Explore with the SV-Rescue advantage disabled (temperature schedule only) on GSM8K. SV-Rescue is not redundant once exploration is added.
Figure 7: Per-rollout Pearson correlation between the SV alignment score at each transformer block and the binary verifier reward, on a 2000 -rollout GSM8K hold-out, both before RL (init) and after RL (SV-Rescue). The vertical dashed line marks the canonical injection layer ℓ⋆=L/2+1 used by SV-Rescue and the dotted line marks the peak-correlation layer. The correlation peaks slightly above L/2+1 on both students, but the canonical L/2+1 retains 80 - 90% of the peak.
Method
Accuracy
Δ vs. Plain
Plain SFT
58.81±0.37
0
DFT (original hyperparameters)
58.88±0.80
+0.07
DFT (re-tuned for LoRA)
61.09†
+2.28
Hidden-state distillation
53.68†
−5.13
SV (Per-sample Slerp)
62.75±0.50
+3.94
DFT (original) + SV
62.64±0.73
+3.83
Appendix
Table 12: SFT-side baselines on Qwen2.5-1.5B with GSM8K (mean ± std over three seeds unless marked † ).
Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.