cs.LGOct 29, 2026

Steering Vectors as a Training Signal in LLM Post-Training

Authors: Tiejin Chen, Maunil Vyas, Huaiyuan Yao, Hua Wei

Organizations: Arizona State University · Amazon

Abstract

Steering vectors (SVs) are residual-stream directions that shift the behavior of LLMs when added to hidden states at inference time. While prior work has mainly used SVs as post-hoc controls, we ask whether the same directions can also guide parameter updates during post-training. We study this question across supervised learning, including SFT and distillation, and reinforcement learning with verifiable rewards (RLVR). Our framework considers two intervention regimes: a hidden-state hook during teacher-forced supervised training, and an advantage-repair mechanism on wrong rollouts in GRPO. For each regime, we analyze three design axes, which are the steering direction, its placement, and the operation used to combine it with training. In supervised settings, SV injection improves over plain SFT and distillation, with gains scaling with the off-policy gap and reaching up to a 12%12\% improvement when distilling from thinking-mode teachers. These gains hold across three random seeds. In RL settings, applying a correctness-contrast direction as advantage repair on wrong rollouts, including otherwise zero-gradient all-wrong groups, recovers a learning signal that transfers to held-out math benchmarks and acts as a composable auxiliary signal that adds to existing zero-variance remedies such as entropy-aware advantage shaping. We further identify three failure boundaries. In detail, code generation can break the direction, cross-family layer transfer can break the placement, and inference-time application on an SV-trained checkpoint can break the intervention regime. Together, these results establish steering vectors as a conditional but effective training signal for LLM post-training. Our code is available here.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Towards Steering without Sacrifice: Principled Training of Steering Vectors for Prompt-only Interventions

    May 7, 2026Yuntai Bao, Qinfeng Li, Xinyan Yu +6Language Model SteeringCausal Interventions in Language Models

  2. Safety Cost of Steering Vectors Is Separable and Reducible

    Aug 9, 2026Yuxiao Li, Gjergji KasneciLanguage Model SteeringLLM Safety Alignment

  3. What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal

    Apr 9, 2026Stephen Cheng, Sarah Wiegreffe, Dinesh ManochaLanguage Model SteeringLLM Interpretability