Learning to Hint for Reinforcement Learning

Yu Xia; Canwen Xu; Zhewei Yao; Julian McAuley; Yuxiong He

cs.LGApr 1, 2026

Learning to Hint for Reinforcement Learning

Authors: Yu Xia, Canwen Xu, Zhewei Yao, Julian McAuley, Yuxiong He

Organizations: University of California, San Diego · Snowflake AI Research

Abstract

Group Relative Policy Optimization (GRPO) is widely used for reinforcement learning with verifiable rewards, but it often suffers from advantage collapse: when all rollouts in a group receive the same reward, the group yields zero relative advantage and thus no learning signal. For example, if a question is too hard for the reasoner, all sampled rollouts can be incorrect and receive zero reward. Recent work addresses this issue by adding hints or auxiliary scaffolds to such hard questions so that the reasoner produces mixed outcomes and recovers a non-zero update. However, existing hints are usually fixed rather than adapted to the current reasoner, and a hint that creates learning signal under the hinted input does not necessarily improve the no-hint policy used at test time. To this end, we propose Hint Learning for Reinforcement Learning (HiLL), a framework that jointly trains a hinter policy and a reasoner policy during RL. For each hard question, the hinter generates hints online conditioned on the current reasoner's incorrect rollout, allowing hint generation to adapt to the reasoner's evolving errors. We further introduce hint reliance, which measures how strongly correct hinted trajectories depend on the hint. We derive a transferability result showing that lower hint reliance implies stronger transfer from hinted success to no-hint success, and we use this result to define a transfer-weighted reward for training the hinter. Therefore, HiLL favors hints that not only recover informative GRPO groups, but also produce signals that are more likely to improve the original no-hint policy. Experiments across multiple benchmarks show that HiLL consistently outperforms GRPO and prior hint-based baselines, demonstrating the value of adaptive and transfer-aware hint learning for RL. The code is available at https://github.com/Andree-9/HiLL.

Figures & tables

Figure 1: Overview of our HiLL framework. Given a question

q

with an all-incorrect group, the hinter

\mathcal{H}_{\phi}

takes the question, a failed rollout

\tau_{k}

, and the reference solution

z^{\star}

as input, and generates

M

candidate hints. The reasoner

\pi_{\theta}

re-samples

G

rollouts under each hinted input

q{+}h_{j}

. Each hint is then scored by signal creation (Sec. 4.2 ) and signal transfer (Sec. 4.3 ). The best hinted group is selected reasoner GRPO update, while all candidate hints form the group for the hinter GRPO update.

Method	In-distribution						Out-of-distribution
Method	AIME24 / 25	AMC23	MATH-500	Minerva	Olympiad	Avg.	GPQA	MMLU-Pro	Avg.
Llama-3.2-3B-Instruct
Base	6.5 / 0.6	22.8	44.7	17.8	14.2	17.8	17.9	27.0	22.5
GRPO	6.7 / 0.8	29.5	52.1	20.5	21.8	21.9	26.3	39.8	33.1
LUFFY	4.4 / 0.4	18.6	38.9	14.3	11.9	14.7	16.0	26.7	21.4
Scaf-GRPO	7.7 / 2.3	28.8	51.7	19.4	19.5	21.5	24.1	38.0	31.0

Table 1: Main results on in-distribution and out-of-distribution benchmarks. Best results are in bold and second-best are underlined. HiLL consistenctly outperforms GRPO and hint-based baselines.

Panel of Figure 2 — Figure 2: All-incorrect ratio (left two) and hint reliance (right two) over training steps. Both HiLL variants substantially reduce the fraction of degenerate all-incorrect groups compared to GRPO. Transfer weighting (HiLL vs. HiLL ${}_{\text{w/o TW}}$ ) keeps hint reliance consistently lower throughout training.

Figure 3: Effect of transfer temperature

T

on average signal creation, signal transfer, and in-distribution accuracy for HiLL with Llama-3.2-3B-Instruct. Dashed lines show HiLL

{}_{\text{w/o TW}}

Question (excerpt)	Hint from HiLL	Hint from HiLL ${}_{\text{w/o TW}}$
A line segment $AB$ of length 1 moves on the $x$ -axis. Point $P(0,1)$ is connected to $A$ by a line, and point $Q(1,2)$ is connected to $B$ by a line. Find the equation of the trajectory of the intersection point $R$ .	Consider expressing the intersection in terms of a parameter $a$ , then eliminate it to reveal the hidden trajectory. (18 words, hinted pass rate 0.13)	Let $A=(a,0)$ and $B=(a{+}1,0)$ for a real parameter $a\neq 0$ . Write the equation of line $PA$ , where $P=(0,1)$ , and the equation of line $QB$ , where $Q=(1,2)$ , both in terms of $a$ . Find the coordinates $(x,y)$ of their intersection point $R$ as a function of $a$ .… (108 words, hinted pass rate 0.38)
In trapezoid $ABCD$ , $AB\parallel CD$ , $AB{=}33$ , $CD{=}21$ , the non-parallel sides are 10 and 14, and $\angle A$ , $\angle B$ are both acute. Find the length of the shorter diagonal.	The shorter diagonal emerges from a triangle where 14, 12, and 10 form a critical configuration, with the angle between them yielding a result just above 25. (27 words, hinted pass rate 0.13)	Draw a line $DE\parallel BC$ from point $D$ to side $AB$ , intersecting $AB$ at point $E$ . This creates triangle $ADE$ with $DE=10$ , $AE=12$ , and $AD=14$ . Use the Law of Cosines in triangle $ADE$ to find $\cos A$ . Then apply the same angle $\angle A$ in … (97 words, hinted pass rate 0.50)

Table 2: Example hints generated by HiLL and HiLL

{}_{\text{w/o TW}}

on the same question. HiLL provides conceptual strategies while HiLL

{}_{\text{w/o TW}}

tends to set up the computation directly.

Figure 4: Hint length and math expressions per hint.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Sep 28, 2026

Teach to Learn: Hint Annealing for Self-improving LLM Reasoning

Group Relative Policy Optimization (GRPO) improves language-model reasoning by comparing verified rewards among multiple solution rollouts for each query. However, difficult training queries can yield only incorrect rollouts, leaving GRPO with no reward contrast or learning signal. Prior hint-based methods construct auxiliary hints from solution evidence and use them to re-solve failed queries, recovering learning signal. Yet the resulting trajectories are typically treated as ordinary solution trajectories despite being generated under an assisted condition unavailable at evaluation. We discover hinted reward shift: recovered reward contrast can concentrate policy updates on hinted trajectories, limiting improvement without hints. This also creates a trade-off: increasing hinted trajectories can accelerate early learning but intensify reward shift later. To address this problem, we propose HATCH (Hint-Annealed Self-Teaching), an online single-policy framework that learns from both generating and using its own hints to improve reasoning without assistance. To mitigate hinted reward shift, we introduce online weighting to anneal the contribution of hinted trajectories. However, learning to generate hints can conflict with improving query solving. We therefore use gradient projection to remove the opposing component of hint-generation updates. Together, these designs support self-improvement by enabling the policy to create learning opportunities for itself and turn them into stronger reasoning without hints. We evaluate our method on mathematical reasoning benchmarks and outperform state-of-the-art methods by 1.02 pp on Llama-3.2-1B-Instruct, 2.84 pp on Qwen3-1.7B, and 4.32 pp on Qwen3-8B.

Zile Wang, Zijian Li, Haodong Wang +5

The Hong Kong University of Science and Technology · Yuanbao Team, Tencent

Reinforcement Learning Self-Training for Language Models RL for Language Model Reasoning LLM Reasoning

Ask PDF

Oct 10, 2025

Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts

Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs). However, prevalent methods like GRPO often fail when task difficulty exceeds model capacity, leading to reward sparsity and inefficient training. Prior work attempts to mitigate this with off-policy data, but such methods often induce severe distributional mismatches that destabilize policy updates. In this work, we identify a core issue underlying these failures, which we term low training affinity, and introduce Affinity, the first quantitative metric for monitoring the compatibility between external guidance and the model's intrinsic policy. To address this, we propose HINT, an adaptive framework designed to enhance reasoning capabilities while explicitly preserving high Affinity. First, instead of revealing partial answers, HINT supplies Meta-Hints, which act as abstract cognitive scaffolding to guide the model in articulating solutions independently. Second, to ensure stability, we integrate Affinity-Aware Policy Optimization (AAPO), which dynamically modulates the learning objective based on the Affinity. Extensive experiments across diverse benchmarks demonstrate that HINT consistently outperforms strong baselines, while exhibiting superior stability and robust generalization to out-of-distribution tasks. Code is available at https://github.com/ViviqwerAsd/HINT.

Xinyi Wang, Jinyi Han, Zishang Jiang +7

School of Data Science, Fudan University · 2Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1

Reinforcement Learning RL for Language Model Reasoning LLM-Guided RL RL Post-Training

Ask PDF

Apr 14, 2026

KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance

RLVR improves reasoning in large language models, but its effectiveness is often limited by severe reward sparsity on hard problems. Recent hint-based RL methods mitigate sparsity by injecting partial solutions or abstract templates, yet they typically scale guidance by adding more tokens, which introduce redundancy, inconsistency, and extra training overhead. We propose KnowRL (Knowledge-Guided Reinforcement Learning), an RL training framework that treats hint design as a minimal-sufficient guidance problem. During RL training, KnowRL decomposes guidance into atomic knowledge points (KPs) and uses Constrained Subset Search (CSS) to construct compact, interaction-aware subsets for training. We further identify a pruning interaction paradox -- removing one KP may help while removing multiple such KPs can hurt -- and explicitly optimize for robust subset curation under this dependency structure. We train KnowRL-Nemotron-1.5B from OpenMath-Nemotron-1.5B. Across eight reasoning benchmarks at the 1.5B scale, KnowRL-Nemotron-1.5B consistently outperforms strong RL and hinting baselines. Without KP hints at inference, KnowRL-Nemotron-1.5B reaches 70.08 average accuracy, already surpassing Nemotron-1.5B by +9.63 points; with selected KPs, performance improves to 74.16, establishing a new state of the art at this scale. The model, curated training data, and code are publicly available at https://github.com/Hasuer/KnowRL.

Linhao Yu, Tianmeng Yang, Siyu Ding +8

TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · Baidu Inc. · Institute of Information Engineering, Chinese Academy of Sciences.

RL for Language Model Reasoning LLM Reasoning RL for Language Models LLM-Guided RL

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Teach to Learn: Hint Annealing for Self-improving LLM Reasoning

Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts

KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance