On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

Kexin Huang; Haoming Meng; Junkang Wu; Jinda Lu; Chiyu Ma; Ziqian Chen; Xue Wang; Bolin Ding

cs.LGMar 23, 2026

On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

Authors: Kexin Huang, Haoming Meng, Junkang Wu, Jinda Lu, Chiyu Ma, Ziqian Chen, Xue Wang, Bolin Ding, +5 more

Organizations: Alibaba Group

Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the magnitude of these updates, largely overlooking their direction. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference $Δ\log p$ between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that $Δ\log p$ more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (\eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a test-time extrapolation method that amplifies the policy along the learned $Δ\log p$ direction to improve reasoning accuracy without further training; (2) a training-time reweighting method that focuses learning on low-probability (corresponding to higher $Δ\log p$ ) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.

Figures & tables

Figure 1: (a) Token-level metrics for analyzing RLVR updates. (b) Histograms of each metric on responses generated by base and RLVR models. With a log-scale y-axis, most values concentrate near zero for all metrics, but only $\Delta\log p$ shows a directional shift distinguishing RLVR from the base model. (c) Token - replacement performance: replacing base tokens with RLVR choices at positions selected by each metric, where $\Delta\log p$ recovers RLVR performance with the fewest replacements.

Figure 2: Token - replacement performance across metrics and model pairs. While all metrics can recover RLVR - level accuracy,

\Delta\log p

does so with the fewest replacements , demonstrating its precision in isolating the reasoning-critical minor tokens changed by RLVR training.

Panel of Figure 3 — Figure 3: (a) Token probability and gradient norm coefficient $1-\pi_{\theta}(\cdot)$ of a DAPO step, where the gradient concentrates on rare, low-probability tokens. (b) Token probability within different $\Delta\log p$ bins, where higher $\Delta\log p$ bins contain lower probability for both base and RLVR models. (c) Effect of top-p filtering on RLVR training performance. Performance declines with more filtering.

Figure 4: Extrapolation Performance

Replace Ratio	0.0%	1.8%	5.2%	20.0%
Avg@32	52.50	53.96	55.31	5 5.10
Threshold $\tau$	N/A	-0.5	-0.2	0.0

Table 1: Selective Extrapolate (

\gamma=0.1

) on the RLVR model (DAPO-32B) instead of the base model.

Model	Method	AIME24		AIME25		AMC		Average
Model	Method	Avg@32	Pass@16	Avg@32	Pass@16	Avg@32	Pass@16	Avg@32	Pass@16
Qwen2.5- Math-7B	Base	14.79	47.46	6.67	27.84	40.62	79.25	20.69	51.52
	DAPO	35.73	54.09	17.6	30.45	73.04	89.03	42.12	57.86
	Ours	39.06	60.58	18.54	36.72	73.64	89.69	43.75	62.33
Qwen3- 8B-Base	Base	5.42	30.63	5.73	32.8	27.64	78.09	12.93	47.17
	DAPO	36.98	72.3	26.67	46.76	69.13	88.51	44.26	69.19

Table 2: Comparison of our reweighting method and DAPO on math reasoning benchmarks.

Method		PPL	Dominate	Ours
AIME24	Avg@32	35.63	3 6.35	39.06
AIME24	Pass@16	61.95	55.27	6 0.58
AIME25	Avg@32	1 6.46	13.02	18.54
AIME25	Pass@16	3 2.19	20.69	36.72
AMC	Avg@32	72.06	79.97	7 3.64
AMC	Pass@16	8 9.1	84.93	89.69

Table 3: Results of various reweighting methods.

Figure 5: Training curves for different reweighting methods on Qwen2.5-Math-7B.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Model	ORZ	UniReason	DAPO
Threshold $\tau$ for $f_{\mathrm{logp}}^{\gamma}$	-0.4	-0.35	-0.3
Replaced Ratio	10.1%	7.5%	11.4%
$\gamma$ in $\pi_{\mathrm{Extra}}^{\gamma}$	0.1	0.1	0.05

Appendix

Table 4: Hyperparameters for the extrapolation results (Fig. 4 ).

Figure 6: Selective token replacement results with additional criteria for DAPO.

Figure 7: Top 50 tokens for replacing the base model’s choice under different metrics’ selection.

Panel of Figure 8 — Figure 8: Per-problem accuracy changes on AIME24 during each model’s selective token replacement experiment. We report the results with $\Delta\log p$ being the selection criterion.

Table 5: Hyperparameter sensitivity analysis for the selective extrapolation experiment. The ∗ sign marks the reported value for extrapolation results in Fig. 4 , while the † sign corresponds to the end point in token replacement of Fig. 2 .

Figure 9: Reproducibility analysis. The learning curves across 4 independent runs on Qwen2.5-Math-7B with our reweighting method (Eq. 8 ) show consistent convergence and performance.

Table 6: Performance of RLVR-trained models on Minerva.

Figure 10: Extrapolation results on Minerva.

	DAPO		ORZ		UniReason
Threshold $\tau$	-1.0	-0.9	-1.0	-0.9	-1.0	-0.9
Avg replace ratio	6.5%	7.0%	9.2%	9.6%	1.8%	2.2%
Replace w/ $\pi_{\mathrm{RL}}$	56.63	56.43	56.41	56.39	54.00	54.14
Extrapolate w/ $\gamma=0.05$	56.8	57.22	57.17 ∗	57.08	54.50	54.50
Extrapolate w/ $\gamma=0.1$	58.27 ∗	56.57	55.51	55.28	54.32	56.16 ∗

Appendix

Table 7: Hyperparameters and Avg@8 performance on Minerva benchmark. The ∗ sign marks the tuned value in Fig. 10 .

Panel of Figure 11 — Figure 11: Word clouds of top $\Delta\log p$ tokens, measured w/ different RLVR-trained models.

Panel of Figure 12 — Figure 12: Logp Difference histograms of different RLVR models, comparing the RLVR and base model’s generations.

Panel of Figure 13 — Figure 13: Divergence and entropy histograms of UniReason and its corresponding base model measured on UniReason or the base model’s generations.

Panel of Figure 14 — Figure 14: Divergence and entropy histograms of DAPO and its corresponding base model measured on DAPO or the base model’s generations.

Panel of Figure 15 — Figure 15: Divergence and entropy histograms of ORZ and its corresponding base model measured on ORZ or the base model’s generations.

Explore similar work

What are Key Factors for Updates in RL for LLM Reasoning?
Jun 21, 2026Peidong Wang, Demi Wang, Xufang Luo +5Reinforcement Learning RL for Language Model Reasoning
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · *Work done during internship at Microsoft Research Asia. · 2Microsoft Research +1
Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
Mar 23, 2026Haoming Meng, Kexin Huang, Shaohang Wei +6RL for Language Model Reasoning RL Fine-Tuning
Alibaba Group
Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
May 7, 2026Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1RL for Language Model Reasoning Efficient Language Model Reasoning
University of Southern California · 2DEVCOM ARL

Jun 21, 2026J/K move · Enter open · S save

What are Key Factors for Updates in RL for LLM Reasoning?

Peidong Wang, Demi Wang, Xufang Luo, Jiahang Xu+4

School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · *Work done during internship at Microsoft Research Asia. · 2Microsoft Research+1

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Proximal Policy Optimization Off-Policy RL

Open paper AskPDF arXiv

Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a promising framework for enhancing the reasoning ability of large language models. However, much of the existing work is guided by heuristic intuition, leading to divergent algorithmic choices, even contradictory ones that nevertheless report empirical gains. To better understand this phenomenon, we conduct a theoretical analysis of RLVR updates. Our study reveals that differences in off-policy degree, determined by the number of gradient steps per rollout, substantially affect the distribution of importance sampling ratios and their clipping behavior, thereby altering which tokens dominate the update. Building on this insight, we characterize gradient expectation as the central quantity governing update dynamics and analyze the roles of token probability, advantage, and importance sampling ratio. Motivated by these findings, we propose Adaptive Clip Policy Optimization (ACPO), which adjusts clipping boundaries across token groups according to the empirical variance of their importance sampling ratios. Experiments on 3B and 7B models across diverse reasoning benchmarks, spanning mathematical problem solving, tabular QA, and logic puzzles, demonstrate that ACPO outperforms strong baselines such as DAPO and CISPO. These results demonstrate that principled, analysis-driven approaches yield more robust and effective RLVR methods. Code is available in: https://github.com/Control-derek/ACPO

Abstract

Explore similar work

What are Key Factors for Updates in RL for LLM Reasoning?

Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning