Get Hint-Augmented GRPO Done Right Through Activation Steering
Organizations: University of Notre Dame, IN, USA
Abstract
GRPO has become the dominant reinforcement learning method for eliciting reasoning capability in LLMs using verifiable rewards. However, it suffers from a fundamental failure mode: when a training question exceeds the base model's current capability, all generated responses fail and receive zero reward, yielding no gradient signal and stalling further capability expansion. An emerging class of GRPO extensions, termed Hint-Augmented GRPO (HA-GRPO), adds a "hint" to the context to reduce the difficulty of the question, thereby producing a usable gradient signal. Nevertheless, existing HA-GRPO methods share a key issue: no hint is available at inference time to append to the question, creating a training-inference context mismatch that leads to inconsistent performance gains. In this work, we reinterpret the function of hinting -- not as reducing question difficulty, but as steering model behavior -- and propose ASH (Activation Steering for Hinting), a novel HA-GRPO method built on this view. During training, the model is presented with both hinted and plain questions, while a trainable steering vector is optimized to widen the margins between the model's generation probabilities for responses to the hinted version relative to the plain version, thereby capturing the question-agnostic component of the behavioral shift induced by the hint. During inference, the trained steering vector is directly injected into the model's activation, reproducing the hint-conditioned behavior without requiring any hint in the context. Experiments with four LLMs (Qwen3-1.7B/4B-Base, Llama-3.2-1B/3B-Instruct) show that ASH consistently improves pass@ on competition-level mathematics benchmarks (AMC, OlympiadBench, AIME24, AIME25) and out-of-distribution benchmarks (Minerva, GPQA-Diamond). Notably, it improves average pass@8 of Qwen3-1.7B-Base and Qwen3-4B-Base by 2.60% and 2.21%.
Figures & tables
| Model | Method | AMC | OlympiadBench | AIME24 | AIME25 | Average | Minerva | GPQA-Diamond | Average | |
| Qwen3-1.7B-Base | Base | - | 55.77 | 46.48 | 10.24 | 13.84 | 31.58 | 39.10 | 39.79 | 39.45 |
| GRPO | - | 56.06 | 50.14 | 21.44 | 14.99 | 35.66 | 48.99 | 61.10 | 55.05 | |
| HA-GRPO | QUESTA | 56.30 | 51.04 | 17.30 | 14.72 | 34.84 | 47.69 | 69.78 | 58.73 | |
| SAGE | 58.65 | 51.56 | 20.24 | 14.88 | 36.33 | 48.77 | 63.54 | 56.16 | ||
| HILL | 59.13 | 52.75 | 21.72 | 16.51 | 37.53 | 49.23 | 64.76 | 57.00 | ||
| ASH | 60.73 | 54.59 | 23.79 | 18.91 | 39.50 | 51.52 | 70.15 | 60.84 |
| Model | AMC | OlympiadBench | AIME24 | AIME25 | Average | Minerva | GPQA-Diamond | Average | |
| Qwen3-1.7B-Base | w/o activation steering | 57.35 | 51.51 | 19.28 | 14.74 | 35.72 | 48.31 | 69.92 | 59.12 |
| w/ o activation steering | 60.73 | 54.59 | 23.79 | 18.91 | 39.50 | 51.52 | 70.15 | 60.84 | |
| Qwen3-4B-Base | w/o activation steering | 75.15 | 64.68 | 33.45 | 31.82 | 51.28 | 59.88 | 75.74 | 67.81 |
| w/ o activation steering | 79.21 | 67.71 | 39.11 | 37.31 | 55.83 | 62.65 | 78.28 | 70.47 | |
| Llama-3.2-1B-Instruct | w/o activation steering | 39.31 | 24.38 | 11.22 | 00.80 | 18.93 | 18.81 | 16.94 | 17.88 |
| w/ o activation steering | 43.57 | 27.18 | 14.33 | 05.45 | 22.63 | 21.05 | 20.97 | 21.01 |
| Model | LogiQA | LiveCodeBench-v5 | |
| Qwen3-1.7B-Base | w/o activation steering | 79.57 | 16.13 |
| w/ o activation steering | 88.69 | 18.77 | |
| Qwen3-4B-Base | w/o activation steering | 81.87 | 30.47 |
| w/ o activation steering | 89.10 | 33.47 | |
| Llama-3.2-1B-Instruct | w/o activation steering | 54.84 | 05.74 |
| w/ o activation steering | 59.10 | 08.20 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| max prompt length | 1088 |
| max response length | 4096 |
| temperature | 1.0 |
| step size | 1e-6 |
| warmup steps ratio | 0.1 |
| warmup style | cosine |
| KL loss coefficient | 0.0 |
| Model | Method | Training Times | |
| Qwen3-1.7B-Base | GRPO | - | 10 hours |
| HA-GRPO | QUESTA | 12 hours | |
| SAGE | 18 hours | ||
| HILL | 20 hours | ||
| ASH | 15 hours | ||
| Qwen3-4B-Base | GRPO | - | 20 hours |
| Model | Question: is an integer, its representation in base is 777. Find the smallest positive integer such that is a fourth power of an integer in decimal notation. Ground Truth: 18 |
| Qwen3-1.7B-Base | To solve this problem, we need to find the smallest positive integer such that the number represented in base as is a perfect power of an integer in decimal notation. Step 1: Convert from base to decimal notation Step 2: Set equal to a perfect power We need to be a perfect square, so let for some integer : Step 3: Divisibility by 7 Since 7 is prime and , we have . Let . Then: Step 4: Test small values of ( ): Since is a positive integer, it is the smallest possible base. Step 5: Verify the solution Since is a perfect square, satisfies the condition. Final Answer: 2 Why it’s wrong : The problem asks for a fourth power, not a square. Base 2 also cannot contain the digit 7. |
| Qwen3-4B-Base | To solve this problem, we need to follow these steps: Step 1: Convert the base representation to decimal Step 2: Set up the fourth-power condition Dividing both sides by 7: Since is an integer, , so . The smallest such is : Step 3: Solve the quadratic equation We discard the negative root, so , which is greater than 7 and therefore a valid base. Step 4: Compute Since is the smallest possible value of , is the smallest fourth power of the form . Therefore, the answer to the problem is . Final Answer: 2401 Why it’s wrong : The question asks for the smallest base , not the value of . The model finds along the way but reports instead. |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,862 papers