Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs

Haoming Meng; Kexin Huang; Shaohang Wei; Chiyu Ma; Shuo Yang; Xue Wang; Guoyin Wang; Bolin Ding

cs.CLMar 23, 2026

Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs

Authors: Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, +1 more

Organizations: Alibaba Group

Abstract

Reinforcement learning with verifiable rewards (RLVR) has significantly improved reasoning in large language models (LLMs), yet the token-level mechanisms underlying these improvements remain unclear. We present a systematic empirical study of RLVR's distributional effects organized around three main analyses: (1) token-level characterization of distributional shifts between base and RL models, (2) the impact of token-level distributional shifts on sequence-level reasoning performance through cross-sampling interventions, and (3) fine-grained mechanics of these shifts at the token level. We find that RL fine-tuning induces highly sparse and targeted changes, with only a small fraction of token distributions exhibiting meaningful divergence between the base and RL policies. We further characterize the structure and evolution of these shifts through analyses of token entropy, positional concentration, and reallocation of probability mass. To assess the functional importance of these sparse changes, we conduct cross-sampling experiments that selectively swap token choices between the base and RL models with varying intervention budgets. We show that inserting only a small fraction of RL-sampled tokens into base generations progressively recovers RL performance gains, while injecting a similarly small number of base token choices into otherwise RL-generated sequences collapses performance to base levels, isolating a small set of token-level decisions directly responsible for RLVR's performance gains. Finally, we explore divergence-weighted variants of the advantage signal as a diagnostic intervention, finding that they can yield improvements over baselines. Together, our results shed light on the distributional changes induced by RLVR and provide a fine-grained, token-level lens for understanding RLVR fine-tuning as a targeted refinement process.

Figures & tables

Panel of Figure 1 — Figure 1: Overview: RLVR acts as sparse, high-impact token-level refinement. RL fine-tuning induces sparse distributional shifts: divergence between base and RL token distributions remains near zero at most positions, with only a small subset of tokens exhibiting substantial changes.

Panel of Figure 2 — Figure 2: JS divergence distributions for Qwen2.5 32B DAPO and SimpleRL on AIME 2024.

Panel of Figure 3 — Figure 3: Mean and median JS divergence by normalized token position, with percentile bands. Both methods concentrate updates at the start and, to a lesser degree, at the end of responses.

Panel of Figure 4 — Figure 4: Entropy distributions for low and high divergence distributions for DAPO . Low-divergence tokens are generally low-entropy, while high-divergence tokens span both high- and low-entropy regions, indicating that DAPO can modify even initially confident predictions.

Panel of Figure 5 — Figure 5: Word clouds of high and low divergence tokens under DAPO.

Panel of Figure 6 — Figure 6: JS divergence distributions for Qwen2.5 32B fine-tuned with SFT on AIME 2024.

Panel of Figure 7 — Figure 7: Forward cross-sampling results (Qwen2.5 32B SimpleRL): injecting RL tokens into base generations progressively recovers RL accuracy.

Panel of Figure 8 — Figure 8: Reverse cross-sampling results (Qwen2.5 32B SimpleRL): swapping RL tokens with base tokens in RL generations causes near-monotonic degradation toward base performance.

Dataset	Method	Eff. %	Eff. #	Initial ( $\pi_{\text{prim}}$ )	Final ( $\pi_{\text{mix}}$ )
		Tokens	Tokens	Acc. (%)	Acc. (%)
AIME24	SimpleRL	$3.86$ %	$38$	$8.23$	$>25$
	SimpleRL Reverse	$5$ %	$29$	$25.52$	$<8.3$
	DAPO	$7.8$ %	$280$	$8.23$	$>44$
	DAPO Reverse	$10.1$ %	$173$	$44.8$	$<8.5$
AIME25	SimpleRL	$1.53$ %	$13$	$5.3$	$>14$

Table 1: Summary of cross-sampled tokens required to reach approximate RL-level performance (forward) or base-level performance (reverse) for Qwen2.5-32B on AIME 2024 and AIME 2025 with a token generation budget of 8000. Effective token counts/percentages exclude identity swaps during cross-sampling. Token percentages are computed at the sequence level.

Panel of Figure 9 — Figure 9: Top- $k$ token overlap between base and RL models at divergent positions (JS > 0.1). Computed as the size of the intersection divided by $k$ . High overlap for $k\geq 2$ shows that distributional shifts occur mostly within shared candidate sets.

Panel of Figure 10 — Figure 10: Analysis of tail behavior under DAPO for divergent token distributions ( $\mathrm{JS}>0.1$ ). (a) shows the percentage of divergent tokens whose RL top-1 choice had base probability below a given threshold. (b) shows the distribution of RL probabilities for the subset with base probability $<0.01$ .

Panel of Figure 11 — Figure 11: Distributional shifts grow increasingly focused and stable. Most tokens remain unchanged; updates concentrate in a sparse set late in training.

Configuration	AIME24	AIME25	AMC	Overall Avg
Baseline DAPO	33.61	18.75	75.08	42.48 $\pm$ 1.35
Low-KL boost	35.90	19.90	78.97	44.92 $\pm$ 0.05
High-KL boost	36.74	20.00	78.40	45.05 $\pm$ 0.79

Table 2: Accuracy (%) under divergence-weighted configurations on Qwen2.5-Math-7B. Results shown for KL divergence with

\pi_{\theta_{\text{old}}}

and sigmoid weighting scheme across AIME 2024, AIME 2025, and AMC datasets. The results displayed are the Mean@32 scores (or, equivalently, the pass@1 scores computed using 32 samples). Results are each averaged over 3 runs.

Appendix figures & tables48 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Qwen2.5-32B	Qwen2.5-Math-7B
$\sigma$ after RL	0.00143	0.00136
$\sigma$ after SFT	0.00347	0.00944

Appendix

Table 3: Relative gap ratio (

\sigma

) after RL and SFT fine-tuning.

Panel of Figure 12 — Figure 12: JS divergence distributions comparing supervised fine-tuning and RLVR on AIME 2024. RLVR exhibits sparser distributional shifts than SFT, suggesting more targeted refinement.

Panel of Figure 13 — Figure 13: JS divergence distributions computed using top- $p=1$ distribution (while sampling is still with top- $p=0.7$ ) for Qwen2.5-32B SFT on AIME 2024.

Panel of Figure 14 — Figure 14: Mean JS divergence by normalized token position comparing SFT and RLVR-trained models on AIME 2024. The positional patterns reveal differences in how SFT and RLVR concentrate their updates.

Panel of Figure 15 — Figure 15: Entropy distributions across divergence bins using full vocabulary for Qwen2.5-32B with SFT on AIME 2025.

Panel of Figure 16 — Figure 16: Top- $k$ token overlap between base and refined models at divergent positions ( $\mathrm{JS}_{t}>0.1$ ) comparing SFT and RLVR-trained models on AIME 2024.

Panel of Figure 17 — Figure 17: Distribution of base-model ranks for fine-tuned models’ top-3 tokens at high-divergence positions ( $\mathrm{JS}>0.1$ ) comparing SFT and RLVR-trained models on AIME 2024.

Panel of Figure 18 — Figure 18: Percentage of divergent tokens whose RL top-1 choice had base probability below a given threshold comparing SFT and RLVR-trained models on AIME 2024.

Panel of Figure 19 — Figure 19: JS divergence distributions for Qwen2.5-32B SFT on AIME 2025.

Panel of Figure 20 — Figure 20: JS divergence distributions computed using top- $p=1$ for Qwen2.5-32B SFT on AIME 2025.

Panel of Figure 21 — Figure 21: Entropy distributions across divergence bins for SimpleRL . Low-divergence tokens are mostly low-entropy, while high-divergence tokens are concentrated in higher-entropy regions, reflecting a more conservative update strategy.

Panel of Figure 22 — Figure 22: Distribution of base-model ranks for RL’s top-3 tokens at high-divergence positions ( $\mathrm{JS}>0.1$ ). Most RL-selected tokens were already highly ranked in the base model, especially under SimpleRL.

Panel of Figure 23 — Figure 23: Histogram of divergences for frequent high JS tokens and frequent low JS tokens (Qwen2.5 32B with DAPO).

Panel of Figure 24 — Figure 24: Percentage of divergent tokens whose RL top-1 choice had base probability below a given threshold.

Panel of Figure 25 — Figure 25: Probability differences and ratios for top-3 tokens under DAPO and SimpleRL among divergent distributions ( $\mathrm{JS}>0.1$ ).

Panel of Figure 26 — Figure 26: Local averages of JS divergence as a function of distance from key regions (prompt beginning and answer) for Qwen2.5-32B models on AIME 2024. Average divergence peaks occur in the same early and late windows highlighted by the positional analysis.

Panel of Figure 27 — Figure 27: Per-sequence scatter plots relating entropy to JS divergence for Qwen2.5-32B DAPO and SimpleRL on AIME 2024. DAPO exhibits a broader entropy spread among divergent tokens, whereas SimpleRL concentrates divergence in higher-entropy regions.

Panel of Figure 28 — Figure 28: JS divergence analysis for Qwen2.5-32B with DAPO on GPQA-Diamond. The sparsity patterns and positional concentration are consistent with findings on AIME datasets.

Panel of Figure 29 — Figure 29: Entropy distributions across divergence bins for Qwen2.5-32B with DAPO on GPQA-Diamond. Patterns are consistent with those observed on AIME datasets.

Panel of Figure 30 — Figure 30: JS divergence percentile curves for Qwen2.5-32B with DAPO on AIME 2024 under different top- $p$ sampling settings. The sparsity patterns remain consistent across different sampling top- $p$ values, indicating robustness to the specific sampling configuration.

Panel of Figure 31 — Figure 31: JS divergence distributions for Qwen2.5-32B with DAPO and SimpleRL on AIME 2025. The sparsity patterns are consistent with those observed on AIME 2024, confirming the robustness of our findings across datasets.

Panel of Figure 32 — Figure 32: JS divergence distributions computed using top- $p=1$ for Qwen2.5-32B with DAPO and SimpleRL on AIME 2025. The sparsity patterns are consistent with those observed using top- $p$ truncated distributions, confirming that truncation does not significantly impact our findings.

Panel of Figure 33 — Figure 33: JS divergence distributions for Qwen2.5-Math-7B trained with DAPO under different clip-higher settings on AIME 2024 and AIME 2025. The more restrictive clip-high=0.2 setting leads to sparser distributional shifts compared to the default 0.28 setting across both datasets, with a smaller proportion of tokens exhibiting nonnegligible divergence. However, on its divergent token set, the JS values are higher as indicated by the higher upper percentiles.

Panel of Figure 34 — Figure 34: Mean JS divergence by normalized token position for DAPO variants with different clip-higher settings on AIME 2024 and AIME 2025.

Panel of Figure 35 — Figure 35: Entropy distributions across divergence bins for Qwen2.5-Math-7B with DAPO variants on AIME 2024. Top row: DAPO (clip-higher=0.28); bottom row: DAPO (clip-higher=0.2). Patterns are consistent with those observed in the main text, confirming the relationship between entropy and divergence across different clip-higher settings.

Panel of Figure 36 — Figure 36: Entropy distributions across divergence bins for Qwen2.5-Math-7B with DAPO variants on AIME 2025. Top row: DAPO (clip-higher=0.28); bottom row: DAPO (clip-higher=0.2).

Panel of Figure 37 — Figure 37: Percentage of divergent tokens whose RL top-1 choice had base probability below a given threshold for Qwen2.5-Math-7B with DAPO variants. Top row: DAPO (clip-higher=0.28); bottom row: DAPO (clip-higher=0.2). Consistent with findings in the main text, RL rarely promotes tokens with very low base probability, even under more exploratory settings with clip-higher. We further observe a distinction between the two clip-high settings, with the more restrictive setting (0.2) promoting fewer tokens with low base probability.

Panel of Figure 38 — Figure 38: Top- $k$ token overlap between base and RL models at divergent positions ( $\mathrm{JS}_{t}>0.1$ ) for DAPO variants on AIME 2024 and AIME 2025.

Panel of Figure 39 — Figure 39: Distribution of base-model ranks for RL’s top-3 tokens at high-divergence positions ( $\mathrm{JS}>0.1$ ) for DAPO variants on AIME 2024 and AIME 2025.

Panel of Figure 40 — Figure 40: JS divergence distributions for DAPO variants of Qwen2.5-Math-7B on fine-tuning data (computed using approximated full distributions instead of truncated ones)

Panel of Figure 41 — Figure 41: Mean JS divergence by normalized token position for DAPO variants on fine-tuning data.

Panel of Figure 42 — Figure 42: Top- $k$ token overlap between base and RL models at divergent positions ( $\mathrm{JS}_{t}>0.1$ ) for DAPO variants on fine-tuning data.

Panel of Figure 43 — Figure 43: JS divergence distributions for Qwen3-8B with DAPO on AIME 2024 and AIME 2025. Sparse distributional shifts persist even under extended training.

Panel of Figure 44 — Figure 44: Mean JS divergence by normalized token position for Qwen3-8B with DAPO on AIME 2024 and AIME 2025. Divergences remain concentrated at early and late positions, consistent with other models.

Panel of Figure 45 — Figure 45: Entropy distributions across divergence bins for Qwen3-8B with DAPO on AIME 2024.

Panel of Figure 46 — Figure 46: Percentage of divergent tokens whose RL top-1 choice had base probability below a given threshold for Qwen3-8B with DAPO on AIME 2024 and AIME 2025.

Panel of Figure 47 — Figure 47: Example of base vs RL distributions at differing divergence levels.

Panel of Figure 48 — Figure 48: JS divergence distributions for Mistral-Small-24B with SimpleRL on AIME 2024 and AIME 2025. Sparse distributional shifts are consistent with findings in the main text across both datasets.

Panel of Figure 49 — Figure 49: Mean JS divergence by normalized token position for Mistral-Small-24B with SimpleRL on AIME 2024 and AIME 2025. Consistent with findings for other models, divergences are concentrated at the start and end of responses.

Panel of Figure 50 — Figure 50: Entropy distributions across divergence bins for Mistral-Small-24B with SimpleRL on AIME 2024.

Panel of Figure 51 — Figure 51: Percentage of divergent tokens whose RL top-1 choice had base probability below a given threshold for Mistral-Small-24B with SimpleRL on AIME 2024 and AIME 2025.

Dataset	Method	Eff. %	%	Eff. #	#	Initial	Final
		Tokens	Tokens	Tokens	Tokens	Acc.	Acc.
AIME24	SimpleRL	$3.86$ %	$7.58$ %	$38$	$75$	$8.23$	$>25$
	SimpleRL Rev.	$5$ %	$8.3$ %	$29$	$51$	$25.52$	$<8.3$
	DAPO	$7.8$ %	$11.9$ %	$280$	$410$	$8.23$	$>44$
	DAPO Rev.	$10.1$ %	$14.9$ %	$173$	$258$	$44.8$	$<8.5$
AIME25	SimpleRL	$1.53$ %	$2.97$ %	$13$	$26$	$5.3$	$>14$

Appendix

Table 4: Summary of cross-sampled tokens required to reach approximate RL-level performance (forward) or base-level performance (reverse) for Qwen2.5-32B on AIME 2024 and AIME 2025 with a token budget of 8000. Effective token counts/percentages exclude identity swaps during cross-sampling. Token percentages are computed at the sequence level.

Panel of Figure 52 — Figure 52: Cross-sampling token pair histograms of the form (primary sampled token) -> (intervention sampled token) at cross-sampling intervention points (excluding identity token swaps).

Panel of Figure 53 — Figure 53: Cross-sampling results (DAPO on AIME 2025): injecting RL tokens into base generations progressively recovers RL accuracy, while reverting RL tokens with base tokens causes near-monotonic degradation toward base performance.

Panel of Figure 54 — Figure 54: Cross-sampling results (DAPO on AIME 2024): injecting RL tokens into base generations progressively recovers RL accuracy, while reverting RL tokens with base tokens causes near-monotonic degradation toward base performance.

Panel of Figure 55 — Figure 55: Comparison of random baseline and DAPO cross-sampling on AIME 2024: average number of tokens (including identity swaps) replaced versus accuracy. The random baseline shows slow performance improvement, demonstrating that targeted RL token selection is critical for performance gains. Performing random replacement may skip critical positions for the reasoning trajectories.

Panel of Figure 56 — Figure 56: Cross-sampling results (Mistral-Small-24B SimpleRL on AIME 2024): injecting RL tokens into base generations progressively recovers RL accuracy, while reverting RL tokens with base tokens causes near-monotonic degradation toward base performance.

Figure 57: AIME24 performance of Qwen3-32B-Base trained with DAPO.

Explore similar work

Mar 23, 2026

On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the magnitude of these updates, largely overlooking their direction. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference

Δ\log p

between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that

Δ\log p

more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (\eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a test-time extrapolation method that amplifies the policy along the learned

Δ\log p

direction to improve reasoning accuracy without further training; (2) a training-time reweighting method that focuses learning on low-probability (corresponding to higher

Δ\log p

) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.

Kexin Huang, Haoming Meng, Junkang Wu +10

Alibaba Group

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards RL Post-Training

Ask PDF

Jun 18, 2026

Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced Large Language Model (LLM) reasoning; however, it faces a fundamental optimization instability: uniform token updates precipitate entropy collapse, leading to premature convergence to suboptimal strategies, whereas excessive Shannon Entropy maximization can cause entropy explosion, driving blind exploration toward incoherent reasoning chains. To resolve this dichotomy, we introduce the Independent Combinatorial Tokens (ICT) framework, which shifts the optimization focus from scalar uncertainty to the distributional properties of token logits. By leveraging the Jensen-Shannon (JS) divergence between token logits distributions, ICT identifies tokens with distinctive distributional patterns as critical branching points for guiding effective exploration in LLM reasoning. Our theoretical analysis, grounded in both Shannon and second-order Rényi entropy, proves that selectively updating on these tokens regulates policy concentration: it reduces the overall distribution uncertainty measured by Shannon entropy, while controlling probability concentration captured by second-order Rényi entropy. This dual effect prevents over-concentrated token generation from weakening exploration and effectively stabilizes the training landscape. Empirical results demonstrate that updating only the top 10% of unique tokens on Qwen2.5 (0.5B/1.5B/7B) models yields an average pass@4 improvement of 4.58%, with a maximum gain of 14.9%, over GRPO, 20-Entropy, and STAPO baselines across seven benchmarks spanning math, commonsense, and Olympiad-level problems.

Xuanzhi Feng, Zhengyang Li, Zeyu Liu +6

The Hong Kong University of Science and Technology, Hong Kong, China · Sichuan University, Chengdu, China · The Hong Kong Polytechnic University, Hong Kong, China

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF

Jun 30, 2026

Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index

Reinforcement learning (RL) has become a powerful tool for propelling Large Language Models (LLMs) beyond imitation-based training towards more robust reasoning capabilities. Among existing approaches, RL with Verifiable Rewards (RLVR) has emerged as a pivotal paradigm for advancing LLM reasoning. Despite its empirical success, recent studies have offered different insights. One line of inquiry advocates prioritizing high-entropy token positions during training, while another perspective cautions against allowing low-probability tokens to dominate gradient updates. Notably, although high-entropy tokens are usually correlated with low probability, both paradigms empirically yield substantial performance gains. In this work, we argue that evaluating sampled-token probability or entropy in isolation is insufficient to capture the policy optimization dynamics. To resolve this tension, we introduce the Relative Surprisal Index (RSI), a principled, information-theoretic metric that naturally couples the token's entropy with the probability of the selected token. We show that, under mild conditions, RSI is related to the local ratio between the first-order variations of the logit-gradient norm and predictive entropy under a selected-logit perturbation. Building on RSI, we propose RSI Selection (RSI-S), an entropy-adaptive token filtering method that retains tokens within a stable RSI interval. RSI-S successfully reconciles previous contradictory paradigms and filters out both redundant low-surprisal tokens and unstable high-surprisal tail tokens. Empirical evaluations show that RSI-S achieves higher avg@32 accuracy across different model scales (Qwen2.5-1.5B, 3B, and 7B) on AIME and AMC benchmarks: RSI-S improves avg@32 accuracy by 2--3 percentage points over GRPO. Overall, RSI offers a promising perspective for RLVR improvement.

Outongyi Lv, Yanzhao Zheng, Yuanwei Zhang +5

1ModelScope Team, 2Alibaba Group, Hangzhou, China. · 3Shanghai Jiao Tong University, Shanghai, China.

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF