H$^2$R: Hierarchical Hint-Based Reward for Mathematical Reasoning via Reinforcement Learning

He Tu; Siye Li; Hongyi Wen

cs.LGOct 29, 2026

H $^2$ R: Hierarchical Hint-Based Reward for Mathematical Reasoning via Reinforcement Learning

Authors: He Tu, Siye Li, Hongyi Wen

Organizations: New York University Shanghai

Abstract

Reinforcement Learning with Verifiable Rewards has shown great promise in enhancing mathematical reasoning in large language models. However, existing approaches provide limited intermediate supervision for complex, multi-step reasoning tasks. To bridge this gap, we propose Hierarchical Hint-based Reward, a framework that leverages the intrinsic sub-question structure of mathematical problems as structured intermediate conditioning signals during RLVR training. Unlike chain-of-thought prompting, these hierarchical hints serve as logically grounded intermediate milestones that effectively reduce the reasoning horizon toward the final answer. To validate this framework, we introduce HiMath, a benchmark of hierarchically structured mathematical problems. Across diverse LLMs, hierarchical hints consistently improve reasoning performance even without task-specific training. When integrated into RL training via H $^2$ R, they boost Qwen2.5-Math-7B accuracy on HiMath from 30.4% to 39.1% using hierarchical hints, and up to 45.6% with Oracle Hierarchical Hints. This pronounced improvement demonstrates that incorporating reliable intermediate checkpoints into reward design is essential for sustaining complex mathematical reasoning.

Figures & tables

Figure 1: Overview of the H 2 R framework. Five hint conditions are constructed by varying the <hints> field: No-Hint provides no intermediate context; Sub 1 -Hint and Sub 2 -Hint provide the ground-truth solution to $Q_{1}$ or $Q_{2}$ , respectively; Hier.-Hint provides model-generated solutions to both sub-questions; and Oracle Hier.-Hint provides both ground-truth solutions as an oracle upper bound. At each iteration, the policy receives a hint-conditioned prompt and H 2 R reward, and GRPO updates the policy toward high-reward trajectories.

Figure 2: Inference-only accuracy of eight untrained checkpoints under five hint conditions, with hints given at inference time and no RL training. Hints consistently improve performance across model scales and architectures. Unlike these results, H 2 R-trained policies receive no hints at inference. Differences between adjacent Qwen model scales are not statistically significant (Appendix D.3 ).

Model	Method	Training Hint	HiMath	MATH500	AIME24 / 25
			(avg@3)	(pass@1)	(avg@5)
Llama-3.2-3B-Instruct	Initial checkpoint	—	11.11	44.7	6.5 / 0.6
	GRPO	No-Hint	11.67	54.6	10.0 / 0.7
	DAPO	No-Hint	13.04	57.0	6.7 / 0.5
	SFT	Oracle Hier.-Hint	13.04	53.8	9.5 / —
	SAGE-Light	Self-Hint	8.69	54.1 †	8.8 / 0.9

Table 1: Performance on HiMath and public benchmarks across three model families, where all models are evaluated without hints at test time. GRPO, DAPO, and SFT are controlled baselines trained on the same 105 HiMath problems, while SAGE and SAGE-Light are included as external reference points trained under a different data regime. The MATH500 results marked with † are sourced from Liao et al. (2026) , while all other results come from our own evaluation. For the hint conditions, Hier.-Hint uses hints generated by o3-mini, whereas Oracle Hier.-Hint relies on expert-annotated ground-truth solutions to establish an upper bound only. The metrics are reported as the mean accuracy over three independent training seeds for HiMath avg@3, a single greedy decoding for MATH500 pass@1, and the mean over five samples at temperature

0.7

for AIME avg@5.

Gate	Acc.	Avg.P	Incorr. $p_{i}{\geq}3$ (%)	Grad.
No Gate	39.13	3.21	34.2	None
Soft Gate	41.30	3.45	18.7	Partial
Hard Gate	45.65	3.89	0.0	Full

Table 2: Gate design ablation on Qwen2.5-Math-7B with Oracle Hier.-Hint. “Incorr.

p_{i}\geq 3

” measures the fraction of incorrect responses receiving a high process score and directly captures the reward-hacking rate. The hard gate eliminates this false training signal while achieving the highest accuracy.

Model	Benchmark	No-Hint	Random	Unordered	Hier.
Llama-3.2-3B-Instruct	HiMath	$11.67$	$10.87$	$15.22$	$19.57$
	MATH500	$54.6$	$54.8$	$57.4$	$59.5$
	AIME24	$10.0$	$10.1$	$11.0$	$12.1$
Qwen2.5-7B-Instruct	HiMath	$19.57$	$21.74$	$26.09$	$34.78$
	MATH500	$77.8$	$78.2$	$80.8$	$84.5$
	AIME24	$15.0$	$15.3$	$17.7$	$22.0$

Table 3: Ablation of hint relevance and hierarchical organization across three model families and three benchmarks. Hints are used only during training and removed at inference.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Training Hint	Test Hint	Accuracy (%)	Avg. Reward	Avg. Process Score
Qwen2.5-Math-7B	No-Hint (GRPO Baseline)	None	30.43	0.22	2.81
	$+$ Sub 1 -Hint	None	32.61	0.23	2.96
	$+$ Sub 2 -Hint	None	36.96	0.28	3.25
	$+$ Hier.-Hint	None	39.13	0.31	3.48
	$+$ Oracle Hier.-Hint	None	45.65	0.35	3.89
Qwen2.5-7B-Instruct	No-Hint (GRPO Baseline)	None	19.57	0.18	2.43

Appendix

Table 5: Performance of models trained under different hint conditions, all evaluated without hints at test time . Accuracy: avg@3 over 3 independent random seeds; std

\leq

2 problems across all conditions. Oracle Hier.-Hint serves as an oracle upper bound using expert-curated sub-question solutions. Avg. Reward and Avg. Process Score are averaged over all test samples.

Judge	No-Hint	Hier.-Hint	Oracle Hier.-Hint
o3-mini	$9/46$	$16/46$	$19/46$
GPT-4o	$8/46$	$14/46$	$18/46$
DeepSeek-R1	$7/46$	$13/46$	$17/46$

Appendix

Table 7: Cross-judge evaluation of the same 138 responses across the No-Hint, Hier.-Hint, and Oracle Hier.-Hint conditions. The independent judges preserve the same ordering across conditions, while differing slightly in absolute strictness.

Stratum	$\mathbf{n}$	o3-mini agrees with human
Random (unbiased)	$20$	$19/20=95.0\%$
o3-mini/GPT-4o disagree	$5$	$4/5=80.0\%$
Pooled	$25$	$23/25=92.0\%$

Appendix

Table 8: Human audit of the automated correctness judgments. The random stratum provides the unbiased estimate of agreement, while the disagreement stratum is deliberately over-sampled to stress-test the judge.

Metric	No-Hint	H 2 R	$\Delta$ (95% CI)
Length (tokens)	$598\pm 87$	$1089\pm 203$	$+491$ $[413,570]$
Step markers	$1.3\pm 0.8$	$3.1\pm 1.2$	$+1.8$ $[1.3,2.3]$
Interm. conclusions	$2.8\pm 1.4$	$5.7\pm 2.1$	$+2.9$ $[2.0,3.8]$
Sub-goal phrases	$0.4\pm 0.5$	$1.8\pm 0.9$	$+1.4$ $[1.0,1.8]$

Appendix

Table 9: Lexical and structural characteristics of Qwen2.5-7B-Instruct responses on AIME24 and AIME25 (60 problems, avg@5; 300 responses per condition). No-Hint denotes the GRPO baseline, while H 2 R denotes the Hier.-Hint-trained policy. Both are evaluated without hints at inference time. Confidence intervals are computed using paired cluster bootstrap resampling with problems as clusters; all four contrasts have

p<0.001

Figure 5: $Q_{3}$ accuracy under Hier.-Hint, dropping from $57.1\%$ (both correct) to $23.8\%$ (both wrong).

LLM Mathematical Reasoning

Large language model mathematical reasoning studies how AI systems solve, explain, verify, and generalize across mathematical problems and proofs.

See all 273 papers

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning
Jun 2, 2026Ziyue Wang, Aomufei Yuan, Yongfu Zhu +10LLM Alignment RL for Language Model Reasoning
1Peking University · 2JINGDONG · 3Shanghai Innovation Institute +2
HintMR: Eliciting Stronger Mathematical Reasoning in Small Language Models
Apr 14, 2026Jawad Hossain, Xiangyu Guo, Jiawei Zhou +1LLM Mathematical Reasoning Small Language Models
University at Albany · University at Buffalo · Stony Brook University
Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealing
Apr 9, 2026Pei-Xi Xie, Che-Yu Lin, Cheng-Lin YangReinforcement Learning LLM Mathematical Reasoning
CyCraft AI Lab, Taiwan

Jun 2, 2026J/K move · Enter open · S save

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

Ziyue Wang, Aomufei Yuan, Yongfu Zhu, Shuai Dong+9

1Peking University · 2JINGDONG · 3Shanghai Innovation Institute+2

LLM Alignment RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards LLM Mathematical Reasoning

Open paper AskPDF arXiv

Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states. Investigating this structure, we find that at the anchor token (the position immediately before the answer marker), correct rollouts converge naturally because they must produce the same answer (cosine similarity ~0.84), yet each retains residual variance from its unique reasoning path. Encouraging full alignment at this point pushes the model to extract a unified "correct decision" representation, reducing sensitivity to which reasoning path was taken. Based on this observation, we propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference. On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales, supported by ablations on loss type, anchor position, layer depth, and loss weight.

Abstract

LLM Mathematical Reasoning

Explore similar work

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

HintMR: Eliciting Stronger Mathematical Reasoning in Small Language Models

Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealing