cs.LGNov 17, 2026

Get Hint-Augmented GRPO Done Right Through Activation Steering

Authors: Khiem Le, Ting Hua, Nitesh V. Chawla

Organizations: University of Notre Dame, IN, USA

Abstract

GRPO has become the dominant reinforcement learning method for eliciting reasoning capability in LLMs using verifiable rewards. However, it suffers from a fundamental failure mode: when a training question exceeds the base model's current capability, all generated responses fail and receive zero reward, yielding no gradient signal and stalling further capability expansion. An emerging class of GRPO extensions, termed Hint-Augmented GRPO (HA-GRPO), adds a "hint" to the context to reduce the difficulty of the question, thereby producing a usable gradient signal. Nevertheless, existing HA-GRPO methods share a key issue: no hint is available at inference time to append to the question, creating a training-inference context mismatch that leads to inconsistent performance gains. In this work, we reinterpret the function of hinting -- not as reducing question difficulty, but as steering model behavior -- and propose ASH (Activation Steering for Hinting), a novel HA-GRPO method built on this view. During training, the model is presented with both hinted and plain questions, while a trainable steering vector is optimized to widen the margins between the model's generation probabilities for responses to the hinted version relative to the plain version, thereby capturing the question-agnostic component of the behavioral shift induced by the hint. During inference, the trained steering vector is directly injected into the model's activation, reproducing the hint-conditioned behavior without requiring any hint in the context. Experiments with four LLMs (Qwen3-1.7B/4B-Base, Llama-3.2-1B/3B-Instruct) show that ASH consistently improves pass@kk on competition-level mathematics benchmarks (AMC, OlympiadBench, AIME24, AIME25) and out-of-distribution benchmarks (Minerva, GPQA-Diamond). Notably, it improves average pass@8 of Qwen3-1.7B-Base and Qwen3-4B-Base by 2.60% and 2.21%.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Teach to Learn: Hint Annealing for Self-improving LLM Reasoning

    Sep 28, 2026Zile Wang, Zijian Li, Haodong Wang +5Reinforcement LearningSelf-Training for Language Models

  2. Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models

    Jan 30, 2026Khiem Le, Phuc Nguyen, Youssef Mroueh +4RL for Language Model ReasoningGroup Relative Policy Optimization

  3. Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts

    Oct 10, 2025Xinyi Wang, Jinyi Han, Zishang Jiang +7Reinforcement LearningRL for Language Model Reasoning