Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR

Yuhang He; Haodong Wu; Siyi Liu; Hongyu Ge; Hange Zhou; Keyi Wu; Zhuo Zheng; Qihong Lin

cs.LGApr 13, 2026

Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR

Authors: Yuhang He, Haodong Wu, Siyi Liu, Hongyu Ge, Hange Zhou, Keyi Wu, Zhuo Zheng, Qihong Lin, +2 more

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Ltd.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning ability of Large Language Models (LLMs), but sparse outcome rewards make token-level credit assignment difficult. We study token-level credit as a reward-conditioned shift from the behavior policy to a hindsight posterior. In autoregressive RLVR, this shift can be expressed through Conditional Mutual Information (CMI), which shows that token entropy upper-bounds possible hindsight credit. Entropy, however, indicates capacity rather than update direction, so we introduce the Four Quadrant Decomposition to separate updates by reward polarity and token entropy. Controlled interventions show that these two factors jointly shape token updates. Sustained reasoning gains concentrate in signed high-entropy quadrants, whereas low-entropy updates saturate quickly. Based on this analysis, we propose Hindsight-Aware Policy Optimization (HAPO), a sign-preserving modification to GRPO that performs capacity-guided advantage reallocation. Experiments on mathematical reasoning benchmarks in two model settings show that HAPO achieves competitive performance among entropy-aware baselines.

Figures & tables

Figure 1: Tokens in Bold exhibit high entropy. From an information-theoretic view, high-entropy and low-entropy tokens differ in how much outcome information they can carry. We decompose token updates in RLVR into four quadrants to study the mechanism of token-level credit assignment. See Appendix I for full entropy examples.

Figure 2: Training dynamics and evaluation for four-quadrant diagnostic interventions. (a) Training solve rate. (b) Training entropy. (c) Pass@32 on AIME24. (d) Avg@32 on AIME24. The x-axis denotes training steps.

Method	$\text{Avg@}32$	$\text{Pass@}32$	$\text{Avg@}32$	$\text{Pass@}32$	$\text{Avg@}32$	$\text{Pass@}32$	$\text{Avg@}4$	$\text{Pass@}4$	$\text{Avg@}4$	$\text{Pass@}4$	$\text{Avg@}4$	$\text{Pass@}4$	avg
Method	AIME24		AIME25		AMC		MATH		Minerva		Olympiad
Classic RLVR Method
Base-7B	10.0	39.3	5.1	23.5	32.7	85.2	46.9	66.3	14.3	25.9	15.5	27.1	20.8
PRIME	15.3	54.5	6.1	26.3	42.4	90.5	56.6	74.1	15.7	37.8	20.8	34.7	30.1
W-REINF	31.2	58.2	10.1	34.0	58.1	87.1	76.2	84.5	34.2	51.2	38.9	49.3	41.6
GRPO	34.0	50.8	9.3	25.2	58.6	80.3	79.9	85.1	38.1	43.4	42.8	51.1	43.8

Table 1: Main results in percentage (%) on mathematics reasoning benchmarks with Qwen2.5-Math-7B . The best results are in bold , and the second-best results are underlined . All subsequent tables follow this format. The avg is computed by averaging the

\text{Avg@}k

across all benchmarks.

Classic RLVR Method
Method	AIME24		AIME25		AMC		MATH		Minerva		Olympiad
Method	$\text{Avg@}32$	$\text{Pass@}32$	$\text{Avg@}32$	$\text{Pass@}32$	$\text{Avg@}32$	$\text{Pass@}32$	$\text{Avg@}4$	$\text{Pass@}4$	$\text{Avg@}4$	$\text{Pass@}4$	$\text{Avg@}4$	$\text{Pass@}4$	avg
Base-1.5B	32.5	71.6	24.3	51.9	69.4	89.9	85.7	91.9	37.5	46.7	55.2	63.8	50.7
W-REINF	35.0	66.1	25.0	49.3	69.8	91.0	87.5	92.6	37.8	37.8	55.9	65.2	51.8
GRPO	38.1	64.8	27.1	49.1	71.4	92.4	88.9	93.4	39.4	48.0	58.7	67.4	53.9
DAPO	40.9	75.6	28.6	53.6	73.6	90.8	89.9	94.0	39.1	48.0	58.4	67.1	55.1

Table 2: Main results in percentage (%) on mathematics reasoning benchmarks with DeepScaleR-1.5B-8k .

Figure 3: Performance–efficiency trade-off for 1.5B models on AIME24. We plot HAPO’s Pareto frontier under different max response length budget.

	AIME24/25	AMC	MATH	Minerva	Olympiad	avg
HAPO	39.8 / 17.2	62.1	83.7	43.9	45.1	48.6
w/o PHR	35.6/15.6	63.4	83.9	42.9	43.2	47.4
w/o PLR	39.2/15.8	63.1	83.2	43.1	44.5	48.1
w/o NLR	36.7/17.1	61.2	83.2	43.9	43.8	47.7
w/o NHR	35.7/16.3	61.8	82.8	44.0	42.9	47.2

Table 3: Ablation study of HAPO with 7B model. Avg@

32

for AIME24, AIME25, AMC, and Avg@

4

for other benchmarks are reported in percentage (%).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Token Criterion	Polarity-Aware	CMI-Bound Grounding	Modulation Type	Auxiliary Model
GRPO	None (uniform)	✗	✗	Uniform advantage	✗
Forking	Entropy (top-20%)	✗	✗	Binary mask	✗
EntroAdv	Entropy (additive)	✗	✗	Additive entropy bonus	✗
KL-Cov	KL divergence	✗	✗	Continuous	✗
PRIME	Learned reward	✗	✗	Token-level reward	✓
A3PO	Probability (high/low)	✓	✗	Adaptive time-decaying	✗

Appendix

Table 4: Comparison of token-level update allocation methods. “Polarity-Aware” indicates whether the method applies different modulations to positive vs. negative samples. “CMI-Bound Grounding” denotes whether the design is motivated by the entropy capacity bound used in this work.

Config	Qwen2.5-Math-7B
lr	1e-6
kl_coef	0.0
max_prompt_length	1024
max_response_length	3072
train_batch_size	256
ppo_mini_batch_size	32

Appendix

Table 5: Training hyperparameters for the four-quadrant experiments.

Config	Qwen2.5-Math-7B	DeepScaleR-1.5B-8k
lr	1e-6	1e-6
kl_coef	0.0	0.0
max_prompt_length	1024	2048
max_response_length	3072	24576
train_batch_size	512	64
ppo_mini_batch_size	32	32

Appendix

Table 6: Training hyperparameters for HAPO.

Benchmark	#Questions	Level
AIME24	30	Olympiad
AIME25	30	Olympiad
AMC	83	Intermediate
MATH	500	Advanced
Minerva	272	Graduate
Olympiad	675	Olympiad

Appendix

Table 7: Dataset statistics.

Figure 4: Token entropy distribution. Low-entropy distributions remain aligned, but a sharp divergence emerges between positive and negative samples in the high-entropy quadrant.

Figure 5: Test time compute performance of four-quadrant training objectives. The x-axis represents varying $k\in\{1,2,4,8,16,32,64,128,256\}$ values, covering the full spectrum of $\text{Pass@}k$ .

Figure 6: Entropy dynamics analysis of 7B models.

Method	MATH Avg@4	MATH Tokens	Olympiad Avg@4	Olympiad Tokens
GRPO	88.9	3172	58.7	5721
DAPO	89.9	2690	58.4	5113
Forking	87.6	2769	59.1	5193
KL-Cov	90.0	3396	59.9	6629
EntroAdv	90.2	3963	59.1	7752
HAPO	91.1	3569	60.8	7353

Appendix

Table 8: Extended efficiency evidence on MATH and Olympiad for DeepScaleR-1.5B-8k . Higher Avg@4 is better; lower token counts indicate lower inference cost. The table is derived from the additional rebuttal measurements used to corroborate the AIME24 Pareto analysis.

Figure 7: Hyperparameter analysis for $\alpha\in\{0.1,0.2,0.3,0.4,0.5,1\}$ on 7B models. Left : training efficiency comparison. Right : evaluation on various benchmarks.

$\alpha$	AIME24	AIME25	AMC	MATH	Minerva	Olympiad	avg
1	35.1	15.7	63.5	84.3	43.5	43.5	47.6
0.5	34.9	15.2	62.9	83.5	43.3	43.2	47.1
0.4	36.2	16.6	62.7	83.8	43.1	44.4	47.8
0.3	35.7	17.1	61.8	82.8	43.1	43.8	47.3
0.2	39.8	17.2	62.1	83.7	43.9	45.1	48.6
0.1	39.4	17.5	61.2	83.1	42.8	44.3	48.0

Appendix

Table 9:

\text{Avg@}k

results on different benchmarks with 7B model. Bold and underline numbers denote the best and second-best results.

Domain	Base	GRPO	HAPO
Overall (Mean)	18.2	33.1	36.6
Computer Science	31.0	41.0	43.2
Math	32.9	34.1	37.9
Chemistry	28.9	30.4	29.2
Engineering	16.4	29.8	29.8
Law	7.1	12.8	16.2

Appendix

Table 10: Per-domain accuracy (%) on MMLU-Pro with Qwen2.5-Math-7B . Best and tied-best results are in bold .

Figure 8: Advantage value distributions. Comparison of token-level advantage allocation among different methods.

Panel of Figure 9 — Figure 9: High-entropy token word cloud.

Figure 11: Visualization of token entropy in incorrect and correct responses.

Explore similar work

Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
Oct 14, 2026Wei Chen, Yubing Wu, Zhou Xu +9RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
South China University of Technology · Tsinghua University · The Hong Kong University of Science and Technology +3
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Jul 3, 2026Zijun Xie, Yuyang You, Yongzhi Li +6Token-Level Credit Assignment RL for Language Model Reasoning
Peking University · Baidu Inc. · Kuaishou Inc.
On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
Mar 23, 2026Kexin Huang, Haoming Meng, Junkang Wu +10RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Alibaba Group

Oct 14, 2026J/K move · Enter open · S save

Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling

Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang+8

South China University of Technology · Tsinghua University · The Hong Kong University of Science and Technology+3

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Step-Level Credit Assignment in RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts, with no reference solution or learned critic. In paired Qwen3-8B runs over a shared

100

-step schedule, HP-RLVR trains

1.21\times

1.46\times

, and

2.30\times

faster than GRPO on mathematics, code generation, and ALFWorld, using one eighth the rollouts per prompt. Accuracy is comparable overall, with HP-RLVR ahead on mathematics at all three sizes and GRPO ahead on code and ALFWorld. In ALFWorld,

43.2\%

(4B) and

49.0\%

(8B) of GRPO's gradient-norm records are zero, versus none for HP-RLVR.

Abstract

Explore similar work

Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling

ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy

On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation