A Sign Law for Entropy Drift under Sampler Mismatch in RLVR

Zhiyuan Wang; Jacob Steeves; Chang Liu

cs.LGOct 25, 2026

A Sign Law for Entropy Drift under Sampler Mismatch in RLVR

Authors: Zhiyuan Wang, Jacob Steeves, Chang Liu

Abstract

In reinforcement learning with verifiable rewards (RLVR), whether entropy rises or falls can depend on choices that matter little under exact sampling, such as whether over-length responses are masked or counted as failures. We trace this to sampler mismatch: rollouts are drawn with top-k/top-p truncation or at a temperature the learner does not apply. We prove that a constant advantage shift then changes the entropy of a softmax policy, at first order and in expectation, with a sign fixed by the sampler. Positive shifts sharpen the policy under any sampler whose likelihood ratio to the policy increases with token probability, including top-k, top-p, min-p, and temperatures below one, and flatten it under flatter samplers. In group-based RLVR with token-level losses, the baseline-dependent part of this drift is set by the token-weighted net advantage, which masking and failing can push to opposite signs. The rest comes from a covariance between advantages and the sampler's sharpness gap and feeds back on itself: in a two-level model, a push away from a sharper sampler takes off in finite time, while a pull toward it saturates. On a 2B model, the entropy slope is linear in shifts of the net advantage under truncated sampling and indistinguishable from flat under exact sampling. Token-level importance ratios leave 98% of the drift, and four clipped minibatch updates per batch amplify it sevenfold. In 100-step runs, a net-zero baseline only delays an entropy explosion, and entropy raised at the start keeps rising only under truncation; cancelling the first-order drift at every step keeps entropy bounded without changing the learner. On a 9B reasoning model forked from an uncorrected run, zeroing the net advantage or renormalizing the learner raises held-out accuracy after ten steps by 8.2 and 8.8 points over the uncorrected rule, with no difference under exact sampling.

Figures & tables

Figure 1: (a) A truncated sampler

\mu

renormalizes the learner’s policy

\pi

over the most probable tokens, so (b) a constant advantage shift

c

moves the logits by

\eta c(\mu-\pi)

in expectation, sharpening the policy for

c>0

and flattening it for

c<0

. (c) In an illustrative group (width: tokens; hatched: masked but counted in the baseline), the shift a token-level loss applies is the net advantage

N

. (d) Entropy change of a 2B model under truncated sampling (3 seeds); with exact sampling or a renormalized learner all rules stay in the gray band.

Figure 2: (a) Dose-response on 2B: advantages of group

g

shifted by

-\lambda N_{g}

, one point per run. (b) Paired Mask

-

Fail slope difference against the measured sharpness gap (symmetric log scale; small dots: seeds; temperatures untruncated with a learner at

T=1

; open green markers: corrections, offset horizontally from the gap of their sampler for visibility).

Sampler	Rule	Held-out	Fixed	Bench.	$\Delta H$
Trunc.	Fail + Pen	75.3	72.3	56.4	$+0.215$
	+ NZ	83.5	83.6	67.6	$+0.016$
Exact	Fail + Pen	84.0	83.3	66.6	$+0.007$
	+ NZ	84.2	82.7	66.3	$+0.009$
Renorm.	Fail + Pen	84.2	81.3	66.4	$+0.035$

Table 1: Accuracy (%) of the 9B model 10 steps after the fork (78.7% held-out), mean over 4 seeds, evaluated with the truncated sampler. Held-out: 300 problems

\times 1

; fixed: 32

\times 8

; bench.: mean of four benchmarks ( Table 9 ).

\Delta H

: change of probe-set entropy. Renorm.: renormalized learner.

Figure 3: Long 2B runs under truncated sampling (top-

k

20, top-

p

0.95; Fail to step 50; Table 10 ). (a) Probe-set entropy (mean over seeds; bands: range). (b) Mean gap per token of the training batch (renormalized learner: untruncated softmax). (c) Held-out accuracy. (d) 50-step runs from logits divided by 1.2 (two seeds; dotted: initial entropy without the kick).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Rule	$N_{g}$	Drift
Group mean	$\mathrm{Cov}_{g}(r,L)/\bar{L}_{g}$	by sign
Mask	$\tfrac{\mathrm{Cov}_{M}}{\bar{L}_{M}}+\tfrac{n_{\mathrm{mask}}}{G}\bar{r}_{M}$	$H\!\downarrow$
Fail , Pen	$\mathrm{Cov}_{g}/\bar{L}_{g}$	$H\!\uparrow$
Length-weighted ( NZ )	$0$	none
Seq.-mean agg., Fail	$0$	none
Seq.-mean agg., Mask	$\tfrac{n_{\mathrm{mask}}}{G}\bar{r}_{M}$	$H\!\downarrow$

Appendix

Table 2: Net advantage of one group (uniform token weights except in the sequence-mean rows) and the direction of the net-advantage drift under a sharper sampler in our runs. Under Mask , masked responses enter the baseline with zero reward;

M

is the set of unmasked responses,

n_{\mathrm{mask}}=G-|M|

, and

\mathrm{Cov}_{M}

\bar{L}_{M}

\bar{r}_{M}

are taken over

M

Figure 4: Proposition 2 in the two-level model (

V=151{,}000

k=20

x_{0}=0.03

): discarded mass and entropy change under a push (

c<0

) and a pull (

c>0

) of the same size, with time in units of

1/(\eta|c|\kappa k)

. The push reaches the uniform policy in finite time; the pull slows down. Dashed: the cap on the discarded mass under top-

p

0.95.

	2B	9B
Model	Qwen3.5-2B (non-thinking)	MiMo-V2.6-Distill-Qwen-9B
Start	released checkpoint (kicked runs: logits divided by 1.2)	step 20 of an earlier run with the same trainer (weights and Adam state)
Groups per step $\times$ group size	$16\times 8$	$32\times 8$
Steps	15 (temperature runs: 10–15; long runs: 100; kicked: 50)	10 (step 20 $\to$ 30)
Generation limit	2,048	8,192
Reward	binary	binary with Pen ( $r=\mathbf{1}[\text{correct}]-\max(0,(L-6000)/2192)$ for completed responses)

Appendix

Table 3: Training and evaluation configuration.

Batch	Rule	Mean $N_{g}$	$\sum_{g}W_{g}N_{g}\bar{\delta}_{g}$	$\sum_{g}W_{g}\mathrm{Cov}_{g}(A,\bar{\delta})$	Net share
1	Fail	$-0.077$	$-3.6\times 10^{-4}$	$-1.2\times 10^{-4}$	76%
	Pen	$-0.079$	$-3.7\times 10^{-4}$	$-1.1\times 10^{-4}$	76%
	Mask	$+0.164$	$+6.9\times 10^{-4}$	$-1.1\times 10^{-4}$	119%
2	Fail	$-0.054$	$-2.4\times 10^{-4}$	$-0.7\times 10^{-4}$	78%
	Pen	$-0.058$	$-2.5\times 10^{-4}$	$-0.7\times 10^{-4}$	78%
	Mask	$+0.119$	$+6.1\times 10^{-4}$	$-0.6\times 10^{-4}$	110%

Appendix

Table 4: Mismatch drift on three 9B training batches sampled with top-

k

20 and top-

p

0.95 at the fork point, split as in Proposition 1 into the net-advantage part and the gap covariance, computed from the measured gap at every sampled prefix with prompt-mean weights. Columns 4–5 are the sums in Equation 5 ; the entropy change is

-\eta

times them, so negative entries raise entropy. The same rollouts are scored under each rule.

Probe	Model	Truncated	Exact	Ratio
SGD deriv.	9B	$-10.3\pm 1.1$	$-0.41\pm 0.14$	4%
Adam deriv.	9B	$-125\pm 13$	$-4.7\pm 1.9$	4%
SGD deriv.	2B	$-10.7\pm 0.4$	$-0.03\pm 0.30$	${<}6\%$
Odd part	2B	$-0.079$	$-0.0010$	1.3%

Appendix

Table 5: Probes of a pure advantage shift. Derivative: first-order change of probe-set entropy along the update of

A\equiv 1

on one training batch, in nats per unit SGD learning rate, or in

10^{-3}

nats per Adam step at learning rate

10^{-6}

with the optimizer state of the fork (mean

\pm

s.e. over four sub-batches). Odd part:

\frac{1}{2}[H(+1)-H(-1)]

after five steps with

A\equiv\pm 1

Sampler	Mask	Fail + NZ	Fail	Rel.
Mean $N$	$+0.30$	$0$	$-0.08$
Truncated	$-9.4\pm 0.2$	$+2.4\pm 0.4$	$+11.9\pm 0.6$	100%
Exact	$-0.4\pm 0.5$	$-1.0\pm 0.3$	$-1.1\pm 0.2$	$-3\%$
Renormalized	$+0.1\pm 0.1$	$-0.3\pm 0.1$	$-0.7\pm 0.3$	$-4\%$

Appendix

Table 6: Entropy slope (

10^{-3}

nats/step) of 2B runs under three advantage rules and three samplers (mean

\pm

s.e. over 3 seeds). Rel.: paired Mask

-

Fail difference relative to its value under truncated sampling. Under truncation the arms do not differ in batch accuracy (0.56–0.59) or over-length rate (0.27–0.31).

Figure 5: Logged first-order mismatch drift (mean over the first 15 steps) against the 15-step entropy slope, 35 runs of 2B under top-

k

20 and top-

p

0.95. Solid: fit to all runs; dashed: fit without Mask . Open markers: Mask with advantages shifted by

-\lambda N_{g}

\lambda\in\{0.5,1,1.5\}

, run after the fit.

$\lambda$	$-1$	$-0.5$	$0$	$0.5$	$1$	$1.5$	$2$
Mean $N$	$-0.157$	$-0.121$	$-0.080$	$-0.037$	$0$	$+0.034$	$+0.066$
Entropy slope ( $10^{-3}$ nats/step)	$+18.8$	$+15.9$	$+11.9$	$+6.9$	$+2.4$	$-1.5$	$-4.1$

Appendix

Table 7: Dose-response on 2B under truncated sampling ( Figure 2 a): mean over 3 seeds.

Sampler	Mask	Fail + NZ	Fail	Mask $-$ Fail
$T=0.7$	$-13.6$ , $-13.4$	$+4.9$ , $+3.7$	$+21.5$ , $+19.1$	$-35.2$ , $-32.6$
$T=1.2$	$+20.3$ , $+21.4$ , $+22.8$	$-7.3$ , $-5.8$ , $-6.8$	$-12.1$ , $-11.1$ , $-11.5$	$+32.4$ , $+32.5$ , $+34.3$
Truncated (mean)	$-9.38$	$+2.43$	$+11.94$	$-21.3$
Exact (mean)	$-0.38$	$-0.95$	$-1.08$	$+0.70$

Appendix

Table 8: Temperature mismatch on 2B ( Figure 2 b): entropy slope (

10^{-3}

nats/step) per seed, with the truncated and exact means of Table 6 for reference.

Sampler	Rule	Held-out	Fixed	MATH-500	AIME24	AIME25	AMC23	$\Delta H$
Truncated	Fail + Pen	75.3	72.3	85.0	36.9	32.1	71.7	$+0.215$
	Fail + Pen + NZ	83.5	83.6	92.4	48.8	41.7	87.5	$+0.016$
	difference	$\mathbf{+8.2}$	$\mathbf{+11.3}$	$+7.5$	$+11.9$	$+9.6$	$+15.8$
Exact	Fail + Pen	84.0	83.3	92.2	49.6	40.2	84.4	$+0.007$
	Fail + Pen + NZ	84.2	82.7	91.8	50.6	40.6	82.2	$+0.009$
	difference	$+0.2$	$-0.6$	$-0.3$	$+1.0$	$+0.4$	$-2.2$

Appendix

Table 9: Per-benchmark accuracy (%) of the 9B model after 10 steps from the fork, mean over 4 seeds; 95% intervals over seeds are given in the next paragraph. In-distribution sets use an 8,192-token limit (held-out: 300 problems

\times 1

; fixed: 32 problems

\times 8

); public benchmarks use 16,384 tokens (MATH-500

\times 1

; AIME and AMC

\times 4

). All evaluations use the truncated sampler.

\Delta H

: change of probe-set entropy (nats). Differences are computed before rounding. Renorm.: truncated sampling with the learner renormalized over the sampled support. The fork point has 78.7% held-out accuracy.

Figure 6: 100-step controls on 2B under top-

k

20 and top-

p

0.95 unless noted (mean over two or three seeds; Table 10 ), with the truncated Fail + NZ runs of Figure 3 . (a) Probe-set entropy. (b) Mean gap per token of the training batch (none under exact sampling).

Rule	Seed	Last step	Entropy	Held-out accuracy (%)
Fail	0 / 1 / 2	50	4.44 / 4.95 / 5.52	26.0 / 0.0; 28.0 / 0.0; 30.3 / 0.0
Fail + NZ	0 / 1	100 / 100	4.50 / 6.45	56.0 / 31.0 / 36.7 / 37.3; 56.3 / 28.0 / 26.0 / 16.0
	2	100	6.21	54.7 / 10.0 / 29.0 / 13.3
Mask	0 / 1	100 / 100	0.070 / 0.072	66.3 / 68.7 / 73.0 / 65.0; 66.3 / 65.0 / 65.3 / 60.7
	2	100	0.068	61.3 / 61.7 / 65.7 / 68.7
Renormalized	0 / 1	100 / 100	0.262 / 0.213	58.0 / 61.3 / 67.7 / 69.0; 53.7 / 62.0 / 68.7 / 69.3

Appendix

Table 10: Long 2B runs, by seed, under truncated sampling unless noted. Entropy: probe-set entropy at the last step (initial 0.303; kicked runs 0.402). Held-out accuracy at steps 25 / 50 / 75 / 100 (initial 44–47%; kicked runs: steps 25 / 50, initial 41–43%). Peak entropy of the truncated Fail + NZ runs: 4.53, 6.45, and 7.30. Fail runs stop at step 50, once their entropy has saturated.

Framework (commit)	Default $T$ / top- $p$ / top- $k$	Learner divides by $T$	Truncation correction
verl (d430157)	1 / 1 / off	yes	score centering on support, optional (off)
OpenRLHF (dc2a7ad)	1 / 1 / off	yes	none
TRL GRPOTrainer (e2dc408)	1 / 1 / off	yes	none
TRL AsyncGRPOTrainer (e2dc408)	1 / 1 / off	yes	mask replay, automatic (top- $k$ /top- $p$ )
Unsloth (71d24f7)	1 / 1 / off	yes	recomputed support, optional (off)
NeMo-RL (f0639c6)	1 / 1 / off	yes	recomputed support, automatic (no min- $p$ )

Appendix

Table 11: RL frameworks at the listed commits: default rollout sampler, whether the learner applies the sampling temperature, and correction for truncation.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,742 papers

Jun 30, 2026

Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index

Reinforcement learning (RL) has become a powerful tool for propelling Large Language Models (LLMs) beyond imitation-based training towards more robust reasoning capabilities. Among existing approaches, RL with Verifiable Rewards (RLVR) has emerged as a pivotal paradigm for advancing LLM reasoning. Despite its empirical success, recent studies have offered different insights. One line of inquiry advocates prioritizing high-entropy token positions during training, while another perspective cautions against allowing low-probability tokens to dominate gradient updates. Notably, although high-entropy tokens are usually correlated with low probability, both paradigms empirically yield substantial performance gains. In this work, we argue that evaluating sampled-token probability or entropy in isolation is insufficient to capture the policy optimization dynamics. To resolve this tension, we introduce the Relative Surprisal Index (RSI), a principled, information-theoretic metric that naturally couples the token's entropy with the probability of the selected token. We show that, under mild conditions, RSI is related to the local ratio between the first-order variations of the logit-gradient norm and predictive entropy under a selected-logit perturbation. Building on RSI, we propose RSI Selection (RSI-S), an entropy-adaptive token filtering method that retains tokens within a stable RSI interval. RSI-S successfully reconciles previous contradictory paradigms and filters out both redundant low-surprisal tokens and unstable high-surprisal tail tokens. Empirical evaluations show that RSI-S achieves higher avg@32 accuracy across different model scales (Qwen2.5-1.5B, 3B, and 7B) on AIME and AMC benchmarks: RSI-S improves avg@32 accuracy by 2--3 percentage points over GRPO. Overall, RSI offers a promising perspective for RLVR improvement.

Outongyi Lv, Yanzhao Zheng, Yuanwei Zhang +5

1ModelScope Team, 2Alibaba Group, Hangzhou, China. · 3Shanghai Jiao Tong University, Shanghai, China.

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF

Apr 6, 2026

Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning ability of large language models (LLMs), but it often suffers from restricted exploration, where the policy rapidly concentrates on a narrow set of solutions. A common remedy is entropy regularization, which attempts to preserve exploration by increasing policy entropy. However, for LLM-RL, this intervention is highly sensitive to its coefficient, can introduce semantically weak uncertainty, and often yields limited accuracy gains. This motivates a more precise question: which entropy helps reasoning, and which entropy should be reduced? To study this, we parameterize the advantage estimator in Group Relative Policy Optimization (GRPO) into positive and negative outcome-conditioned channels and analyze their entropy dynamics. Our results show that positive-channel modulation raises productive entropy associated with successful reasoning trajectories, while negative-channel modulation removes noisy entropy associated with failed rollouts and reduces interference with correct paths. Guided by this channel-wise view, we propose AsymGRPO, which decouples the modulation strengths of positive and negative advantages. This enables flexible control over how the model updates across prompt difficulty levels, allowing stronger reinforcement of rare successes on harder prompts or stronger suppression of residual failures on easier prompts without forcing the two channels to share the same modulation strength. Experiments on five mathematical reasoning benchmarks show that AsymGRPO outperforms strong RLVR baselines, with consistent gains across model backbones.

Hengrui Gu, Xiaotian Han, Yujing Bian +2

North Carolina State University · Case Western Reserve University · Oak Ridge National Laboratory

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Group Relative Policy Optimization Entropy Regularization

Ask PDF

Jun 7, 2026

PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR

Reinforcement learning with verifiable rewards (RLVR) improves large language model reasoning but often suffers from rapid policy-entropy collapse, where the policy prematurely concentrates on narrow high-probability reasoning paths. While global entropy regularization can encourage exploration, uniformly increasing entropy across all token positions is inefficient for long reasoning trajectories, where many tokens are not decision-relevant. We propose Position-Aware Entropy Calibration (PAEC), a token-level entropy-management framework that constructs a soft mask from local top-p entropy and top-two candidate competition, and applies an anchor-based lower-bound penalty to prevent selected-position entropy collapse. Experiments on five mathematical reasoning benchmarks show that PAEC improves macro-average majority-vote performance over strong RLVR baselines, with clear gains on AIME-style tasks. Our results suggest that entropy management in reasoning RL should be formulated as selective exploration allocation over decision-sensitive positions rather than uniform randomness injection.

Shumeng Yang, Yisu Liu, Jiayi Zheng +2

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · School of Artificial Intelligence, Beijing University of Posts and Telecommunications +2

RL for Language Model Reasoning Maximum Entropy RL Reinforcement Learning with Verifiable Rewards LLM Reasoning

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index

Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR

PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR