Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling

Wei Chen; Yubing Wu; Zhou Xu; Haoyue Zhang; Delu Zeng; Jiahao Wang; Junmei Yang; Pan tiewen

cs.LGOct 14, 2026

Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling

Authors: Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan tiewen, +4 more

Organizations: South China University of Technology · Tsinghua University · The Hong Kong University of Science and Technology · Huawei 2012 Laboratories · Shenzhen Technology University · LinkSprite

Abstract

Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts, with no reference solution or learned critic. In paired Qwen3-8B runs over a shared $100$ -step schedule, HP-RLVR trains $1.21\times$ , $1.46\times$ , and $2.30\times$ faster than GRPO on mathematics, code generation, and ALFWorld, using one eighth the rollouts per prompt. Accuracy is comparable overall, with HP-RLVR ahead on mathematics at all three sizes and GRPO ahead on code and ALFWorld. In ALFWorld, $43.2\%$ (4B) and $49.0\%$ (8B) of GRPO's gradient-norm records are zero, versus none for HP-RLVR.

Figures & tables

Figure 1: Qwen3-8B training cost at four prompts per step and

100

steps. HP-RLVR uses group size

G{=}1

, versus GRPO’s

G{=}8

. Bars are relative to GRPO, with labels above each pair showing speedup.

Figure 2: Overview of HP-RLVR’s single-rollout training pipeline with training-only hindsight. After generation, the frozen generating policy scores the trajectory under causal and hindsight contexts, without additional sampling. The paired profiles show full-vocabulary distributions at the same token position. Normalized JSD weights the historically centered verifier reward to produce token credit. Bars are schematic, and dashed frames group stages.

		Time (min)		NPU-hours
Model	Task	GRPO	HP-RLVR	GRPO	HP-RLVR	Ratio
1.7B	Mathematics	$92.70$	$140.27$	$12.36$	$18.70$	$0.66\times$
	Code	$67.55$	$\mathbf{45.80}$	$9.01$	$\mathbf{6.11}$	$\mathbf{1.48\times}$
	ALFWorld	$3242.85$	$\mathbf{1003.25}$	$432.4$	$\mathbf{133.8}$	$\mathbf{3.23\times}$
4B	Mathematics	$175.66$	$\mathbf{125.47}$	$23.42$	$\mathbf{16.73}$	$\mathbf{1.40\times}$
	Code	$88.80$	$\mathbf{61.09}$	$11.84$	$\mathbf{8.15}$	$\mathbf{1.45\times}$

Table 1: Training-cost records for completed

100

-step runs. NPU-hours are separately logged run totals. Ratio: GRPO/HP-RLVR training time, so values below one favor GRPO. The 4B mathematics pair uses

32

16

prompts per step for HP-RLVR/GRPO.

Phase	Time (s)	Share (%)
Rollout generation	$7{,}678.48$	$72.9$
Policy training	$1{,}343.70$	$12.7$
Behavior-policy replay	$576.85$	$5.5$
Weight synchronization	$141.96$	$1.3$
Residual time	$798.51$	$7.6$
Total	$10{,}539.50$	$100.0$

Table 2: Qwen3-4B GRPO mathematics timing over

100

steps at

16

prompts and

128

trajectories per step. Residual time is the total minus the four listed phase measurements.

		Mathematics (Avg@ $12$ )					Code		ALFWorld
Size	Method	AIME24	AIME25	HMMT25	Avg.	Pass@ $12$	Pass@ $1$	Pass@ $8$	ID	OOD	Avg.
1.7B	GRPO ( $G{=}8$ )	$51.90$	$38.10$	$25.00$	$37.80$	$65.56$	$25.78$	$32.04$	$40.00$	$39.55$	$38.71$
1.7B	HP-RLVR ( $G{=}1$ )	$52.20$	$40.30$	$27.20$	$38.90$	$66.67$	$25.11$	$31.75$	$32.14$	$31.34$	$30.64$
4B	GRPO ( $G{=}8$ )	$71.39$	$64.44$	$46.39$	$60.09$	$76.67$	$37.29$	$45.31$	$70.00$	$66.42$	$68.21$
4B	HP-RLVR ( $G{=}1$ )	$76.67$	$67.50$	$46.39$	$62.32$	$78.89$	$37.24$	$44.55$	$62.14$	$63.43$	$61.72$
8B	GRPO ( $G{=}8$ )	$74.20$	$72.20$	$45.00$	$63.07$	$81.10$	$40.65$	$49.29$	$65.71$	$64.93$	$65.32$

Table 3: Qwen3-1.7B, 4B, and 8B results under the current protocol (%). Each cell is the best observed value for that metric across the four checkpoints. Avg@

12

averages correctness over

12

samples per problem and the mathematics Avg. is the mean over the three benchmarks. Code reports Pass@

1

and Pass@

8

1{,}055

LiveCodeBench problems. ALFWorld reports ID/OOD success (

140

134

games) and their mean.

Figure 3: Best observed task performance versus complete-run training time over

100

steps. Each method and model size uses the highest whole-task score at steps

25/50/75/100

. Filled and hollow points are non-dominated and dominated within each task, respectively, and gray steps trace the observed budget envelope. Code uses Pass@

1

over

1{,}055

problems. Single-seed runs differ in hardware and prompt budgets, including

32

16

prompts per step for 4B mathematics (HP-RLVR/GRPO). The 1.7B HP-RLVR ALFWorld timing includes a restart.

Figure 4: ALFWorld zero gradient-norm records as a fraction of all logged records.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
${\bm{x}},{\bm{y}}$	Prompt and generated trajectory.
$\pi_{\bm{\theta}},\pi_{\theta_{\mathrm{old}}}$	Current policy and rollout snapshot.
$i,j,k$	Trajectory, token, and batch indices.
$R_{i},O$	Verifier reward and outcome descriptor.
$G$	Group size: rollouts per prompt.
$h_{j},y_{j}$	History and token at position $j$ .

Appendix

Table 4: Core notation.

Backbone	Method	Zero-gradient records	Fraction
Qwen3-4B	GRPO ( $G{=}8$ )	$1{,}207/2{,}792$	$43.2\%$
Qwen3-4B	HP-RLVR ( $G{=}1$ )	$\mathbf{0/353}$	$\mathbf{0.0\%}$
Qwen3-8B	GRPO ( $G{=}8$ )	$1{,}343/2{,}739$	$49.0\%$
Qwen3-8B	HP-RLVR ( $G{=}1$ )	$\mathbf{0/293}$	$\mathbf{0.0\%}$

Appendix

Table 5: ALFWorld zero gradient-norm records, counted with the same parser. Fractions use all logged gradient records, rather than distinct optimizer steps. The figure reports the same fractions.

Explore similar work

Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR
Apr 13, 2026Yuhang He, Haodong Wu, Siyi Liu +7RL for Language Model Reasoning Token-Level Credit Assignment
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Ltd.
Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards
Sep 29, 2026Fanchao Chen, Hengyu Fu, Shivaram Venkataraman +1RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
University of Wisconsin–Madison · University of California, Berkeley · ETH Zurich +1
Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR
May 8, 2026Tao Wang, Shuo Li, Yan Sun +2RL for Language Model Reasoning Group Relative Policy Optimization
University of Pennsylvania · New Jersey Institute of Technology · University of Tennessee, Knoxville

Apr 13, 2026J/K move · Enter open · S save

Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR

Yuhang He, Haodong Wu, Siyi Liu, Hongyu Ge+6

The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Ltd.

RL for Language Model Reasoning Token-Level Credit Assignment Credit Assignment Reinforcement Learning with Verifiable Rewards LLM Reasoning Group Relative Policy Optimization

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning ability of Large Language Models (LLMs), but sparse outcome rewards make token-level credit assignment difficult. We study token-level credit as a reward-conditioned shift from the behavior policy to a hindsight posterior. In autoregressive RLVR, this shift can be expressed through Conditional Mutual Information (CMI), which shows that token entropy upper-bounds possible hindsight credit. Entropy, however, indicates capacity rather than update direction, so we introduce the Four Quadrant Decomposition to separate updates by reward polarity and token entropy. Controlled interventions show that these two factors jointly shape token updates. Sustained reasoning gains concentrate in signed high-entropy quadrants, whereas low-entropy updates saturate quickly. Based on this analysis, we propose Hindsight-Aware Policy Optimization (HAPO), a sign-preserving modification to GRPO that performs capacity-guided advantage reallocation. Experiments on mathematical reasoning benchmarks in two model settings show that HAPO achieves competitive performance among entropy-aware baselines.

Abstract

Explore similar work

Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR

Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards

Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR