Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards

Fanchao Chen; Hengyu Fu; Shivaram Venkataraman; Jiantao Jiao

cs.LGSep 29, 2026

Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards

Authors: Fanchao Chen, Hengyu Fu, Shivaram Venkataraman, Jiantao Jiao

Organizations: University of Wisconsin–Madison · University of California, Berkeley · ETH Zurich · NVIDIA

Abstract

Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling complete trajectories is costly and does not explicitly explore the decision space at critical positions. Feedback on a completed trajectory can reveal which earlier choices the policy reconsiders, suggesting where to sample alternative continuations. We introduce Hindsight-Divergence Localization (HDL), which uses hindsight-induced changes in token log-likelihoods to select branch points. HDL generates a small number of complete root trajectories and fills each training group with continuations from the selected positions under the original task context. Each continuation reuses its root prefix and contributes policy updates only through its newly generated suffix, reducing generation cost while focusing additional exploration and learning on decisions after branching. Experiments with three models across math, code, and agent tasks show gains in both rollout efficiency and task performance. Compared with GRPO at matched group sizes and training steps, HDL yields up to a 2.5 $\times$ reduction in generated tokens and a 1.8 $\times$ speedup in rollout wall-clock time. Despite this reduced generation budget, HDL improves performance across all three domains, with gains of up to 12.5 points on agent tasks.

Figures & tables

Figure 1: Hindsight-divergence localization on a coding example. The root’s primality test incorrectly accepts

1

as prime. Verifier feedback prompts a reflection identifying the missing guard for

x<2

. Hindsight re-scoring produces the largest absolute log-likelihood change at for , which HDL selects as a branch point. Fresh continuations are sampled under the original task context while reusing the preceding prefix. The illustrated continuations introduce alternative guards before the loop, increasing the reward from 0.7 to 1.0.

Figure 2: HDL reduces generated tokens by 35–61% relative to GRPO. Bars show mean generated tokens per training step, normalized to HDL within each model–task pair; annotations give the corresponding cost ratios.

Figure 3: HDL reduces end-to-end rollout time by 18–45% relative to GRPO. Bars show mean seconds per training step on identical hardware; annotations give time ratios relative to HDL. Each task uses a separate vertical scale.

Qwen3-4B
Method	AIME24	AIME25	AIME26	HMMT	Minerva	Olympiad	Avg.
GRPO	$\mathbf{74.51}_{\scriptscriptstyle\pm 0.26}$	$65.62_{\scriptscriptstyle\pm 0.74}$	$66.46_{\scriptscriptstyle\pm 0.29}$	$\underline{19.44}_{\scriptscriptstyle\pm 0.39}$	$\underline{31.13}_{\scriptscriptstyle\pm 0.45}$	$52.25_{\scriptscriptstyle\pm 0.16}$	$51.57_{\scriptscriptstyle\pm 0.07}$
DAPO	$\underline{74.51}_{\scriptscriptstyle\pm 0.60}$	$\mathbf{68.82}_{\scriptscriptstyle\pm 0.35}$	$\mathbf{68.96}_{\scriptscriptstyle\pm 0.51}$	$18.37_{\scriptscriptstyle\pm 0.54}$	$\mathbf{31.20}_{\scriptscriptstyle\pm 0.32}$	$\underline{53.04}_{\scriptscriptstyle\pm 0.10}$	$\mathbf{52.48}_{\scriptscriptstyle\pm 0.02}$
HDL	$73.12_{\scriptscriptstyle\pm 0.68}$	$\underline{67.01}_{\scriptscriptstyle\pm 0.80}$	$\underline{67.57}_{\scriptscriptstyle\pm 1.52}$	$\mathbf{19.89}_{\scriptscriptstyle\pm 1.07}$	$30.38_{\scriptscriptstyle\pm 0.21}$	$\mathbf{53.18}_{\scriptscriptstyle\pm 0.45}$	$\underline{51.86}_{\scriptscriptstyle\pm 0.14}$
Qwen3-8B
GRPO	$\underline{75.62}_{\scriptscriptstyle\pm 0.34}$	$68.40_{\scriptscriptstyle\pm 0.64}$	$68.33_{\scriptscriptstyle\pm 2.78}$	$21.09_{\scriptscriptstyle\pm 1.17}$	$\mathbf{32.72}_{\scriptscriptstyle\pm 0.10}$	$52.86_{\scriptscriptstyle\pm 0.29}$	$53.17_{\scriptscriptstyle\pm 0.13}$

Table 1: Math. Scores (%,

\uparrow

) on six benchmarks and their unweighted average (Avg.). Entries show mean ±std over the three checkpoints with the highest Avg. Bold and underlining mark the best and second-best means within each model, respectively.

Table 5

Task	HDL	Entropy	Reflection
Math	$\mathbf{54.16}_{\scriptscriptstyle\pm 0.10}$	$\underline{54.02}_{\scriptscriptstyle\pm 0.25}$	$53.12_{\scriptscriptstyle\pm 0.05}$
Code	$\mathbf{55.56}_{\scriptscriptstyle\pm 0.09}$	$\underline{55.37}_{\scriptscriptstyle\pm 0.26}$	$53.95_{\scriptscriptstyle\pm 0.10}$
Agent	$\mathbf{71.96}_{\scriptscriptstyle\pm 1.55}$	$\underline{66.29}_{\scriptscriptstyle\pm 1.59}$	$65.42_{\scriptscriptstyle\pm 0.66}$

Table 4: Localization-signal comparison on Qwen3-8B. Scores (%,

\uparrow

) are reported as mean ±std over the three checkpoints with the highest average score in each domain. Bold and underlining mark the best and second-best means.

Figure 5: Proportions of HDL and Entropy branch points on opening verbs and arguments in successful and failed roots.

Figure 6: Localization signals and verb selectivity. Left and middle: mean signal at positions selected by HDL and Entropy, respectively. Right: verb selectivity, measured by total variation distance (TVD). Higher values indicate stronger preferences for particular opening verbs.

Roots $\times$ Points	Continuations per root	Score (%) $\uparrow$	Tokens (M) $\downarrow$	Time (s) $\downarrow$
$2\times 2$ (default)	$3+4$	$\mathbf{71.96}_{\pm 1.55}$	0.27	70.46
$2\times 1$	$7$	$71.09_{\pm 0.29}$	0.25	59.97
$4\times 2$	$1+2$	$67.87_{\pm 0.44}$	0.31	68.07

Table 5: Branching configurations on Qwen3-8B Agent. Points are counted per root. Tokens and rollout time are per-step means.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Table 10

Explore similar work

Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
Oct 14, 2026Wei Chen, Yubing Wu, Zhou Xu +9RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
South China University of Technology · Tsinghua University · The Hong Kong University of Science and Technology +3
Diversifying RLVR Rollouts via First-Token Exploration
May 27, 2026Soeun Kim, Albert NoRL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Department of Artificial Intelligence, Yonsei University
Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR
May 8, 2026Tao Wang, Shuo Li, Yan Sun +2RL for Language Model Reasoning Group Relative Policy Optimization
University of Pennsylvania · New Jersey Institute of Technology · University of Tennessee, Knoxville

Oct 14, 2026J/K move · Enter open · S save

Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling

Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang+8

South China University of Technology · Tsinghua University · The Hong Kong University of Science and Technology+3

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Step-Level Credit Assignment in RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts, with no reference solution or learned critic. In paired Qwen3-8B runs over a shared

100

-step schedule, HP-RLVR trains

1.21\times

1.46\times

, and

2.30\times

faster than GRPO on mathematics, code generation, and ALFWorld, using one eighth the rollouts per prompt. Accuracy is comparable overall, with HP-RLVR ahead on mathematics at all three sizes and GRPO ahead on code and ALFWorld. In ALFWorld,

43.2\%

(4B) and

49.0\%

(8B) of GRPO's gradient-norm records are zero, versus none for HP-RLVR.

Abstract

Explore similar work

Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling

Diversifying RLVR Rollouts via First-Token Exploration

Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR