Organizations: Mohamed bin Zayed University of Artificial Intelligence · Renmin University of China · Carnegie Mellon University · Hong Kong University of Science and Technology (Guangzhou)
Group Relative Policy Optimization (GRPO) has proven effective in RLVR by using outcome-based rewards. While fine-grained dense rewards can theoretically improve performance, we reveal that under practical sampling budgets, Monte Carlo estimation yields high-variance and sign-inconsistent advantages for early reasoning tokens, paradoxically underperforming outcome-only GRPO. We propose Skip-Connected Optimization (SKPO), which decomposes reasoning into upstream and downstream phases: upstream receives dense rewards from downstream Monte Carlo sampling with single-stream optimization; downstream maintains group-relative optimization, where a skip connection concatenates the upstream segment with the original problem, enabling the model to leverage helpful upstream reasoning while preserving the freedom to bypass flawed reasoning through direct problem access. Experiments demonstrate improvements of 3.91% and 6.17% relative gains over the strongest baselines on Qwen2.5-Math-7B and Llama-3.2-3B respectively across mathematical benchmarks and out-of-domain tasks including general reasoning and code generation. Further analysis reveals an implicit advantage: SKPO generates trajectories with higher intermediate-step quality even when matched for final correctness.
Figures & tables
Figure 1: Monte Carlo estimation of token-level rewards under practical sampling budgets ( K=8 ). Top: true values (left) vs. estimates (right). Bottom: sign accuracy and MAE, showing that early tokens suffer from higher estimation error. By the nature of MC sampling, true token rewards converge to the policy’s average accuracy on the prompt at early positions, and to the trajectory’s binary correctness at late positions; the former makes accurate advantage estimation difficult at early steps, while the latter explains why GRPO achieves reasonable posterior estimation. Additional examples across more correctness distributions (0/8 to 8/8) are provided in Section A.3 .
Figure 2: Overview of the SKPO algorithm. We illustrate three components: (1) upstream sampling process and components for upstream advantage computation; (2) downstream sampling process and components for downstream advantage computation; (3) engineering optimization that completes both upstream and downstream sampling within a single GPU batch through KV cache rewriting.
Qwen2.5-Math-7B
Llama-3.2-3B-Instruct
In-Domain (Math)
Out-of-Domain
In-Domain (Math)
Out-of-Domain
Method
AIME24
AIME25
AMC
MATH
Minerva
Olympiad
MMLU
LiveCode
Avg.
AIME24
AIME25
AMC
MATH
Olympiad
MMLU
LiveCode
Avg.
Base
12.1
5.4
43.4
58.1
20.6
34.1
18.6
13.8
25.8
3.4
0.4
20.1
36.3
18.1
29.5
12.4
17.2
GRPO
22.4
10.1
50.2
63.8
23.5
35.8
21.3
14.6
30.2
4.7
0.3
24.8
39.6
19.8
30.2
13.1
18.9
GSPO
31.1
14.8
60.3
74.2
28.1
43.6
25.2
16.2
36.7
6.9
0.5
30.4
42.1
22.3
30.5
13.8
21.0
SPO
28.8
13.6
56.8
65.9
29.2
40.1
24.1
15.7
34.3
5.8
0.4
26.1
39.2
20.3
29.8
12.9
19.2
Table 1: Performance comparison on In-Domain (Math) and Out-of-Domain (General/Code) benchmarks. We evaluate on two base models: Qwen2.5-Math-7B and Llama-3.2-3B-Instruct . The best results for each model are bolded , and the second best are underlined .
Figure 3: Upstream impact analysis comparing three conditioning strategies across Qwen and Llama models. Top : Diversity (average distinct answers per 8 rollouts); Middle : Advantage zero rate (probability of homogeneous outcomes); Bottom : Response length (excluding prefix). Continual conditioning leads to diversity collapse and elevated advantage zero rate, while Skip maintains properties comparable to Unconditional.
Setting
AIME24
AIME25
AMC
MATH
Minerva
Olympiad
w/o Skip
32.8
15.4
67.8
74.6
29.8
46.3
w/o MC Estimation
33.6
17.4
68.6
78.3
32.9
48.7
Split@Fixed
29.5
14.1
58.2
67.3
30.0
41.5
Selection:Extreme
28.2
13.3
55.9
65.1
28.6
39.4
Ours
35.7
18.7
71.4
80.8
34.7
51.0
Table 2: Ablation study on In-Domain benchmarks comparing different structural settings. We report accuracy (%) for Qwen2.5-Math-7B. The best results are bolded .
Figure 4: Segment-level implicit advantage analysis using GPT-5-nano as external evaluator across both Qwen and Llama models. For each method’s correct responses, we split into 100 segments by relative position and estimate rewards via Monte Carlo continuation sampling. Inter-method relative advantages are computed at each position. SKPO consistently maintains higher advantages in the 20–50% position range, indicating superior intermediate reasoning quality during the problem-setup phase.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Probability of correct advantage signs for all G=8 samples across varying group spreads. Reliable credit assignment for narrow spreads requires prohibitive sample counts ( N>512 ).
Figure 6: Monte Carlo variance analysis. (a) Classification accuracy for individual samples. (b) Distribution shift and sign volatility under sampling noise.
Figure 7: Advantage estimation accuracy across different correctness distributions. Key observation : Only extreme cases (all correct or all incorrect) achieve reliable sign consistency. Mixed distributions suffer from severe sign errors throughout the trajectory, with accuracy often below 50%. Additional examples (1/8, 2/8, 4/8, 6/8, 7/8) are available in figures/mc/ .
Figure 8: MC estimation accuracy across different sampling budgets ( K=8 to K=8192 ). Left column shows true token rewards and advantages. Middle columns show MC estimates at each K value. Right column summarizes win rate, variance, and MAE trends. With sufficient samples ( K≥512 ), MC consistently outperforms GRPO, but the required sampling cost (up to 1000 × ) is impractical for real-world training.
Figure 9: Mean accuracy at 32 samples across training steps.
Figure 10: Actor policy entropy throughout training.
Figure 11: Training time per step across different methods. Critique-GRPO exhibits significantly higher per-step time due to its two-batch rollout strategy: first generating 7 direct responses, then spawning 1 self-critique response in a separate GPU batch. Despite the two-phase upstream-downstream architecture, SKPO achieves comparable wall-clock time to single-phase baselines through our engineering optimization that performs mid-generation KV cache rewriting within a single rollout pass.
Reinforcement Learning with Verifiable Rewards (RLVR) has become the standard paradigm for LLM mathematical reasoning, where Group Relative Policy Optimization (GRPO) serves as the mainstream algorithm. We point out two understudied inefficiencies existing in GRPO. First, the fixed KL penalty coefficient overly restricts policy exploration at stages where the model requires significant deviation from the reference policy. Second, uniform sampling of training questions ignores that moderately difficult problems provide the most informative gradient signals for optimization. We propose Exploration-Prioritized Policy Optimization (EXPO) with two lightweight plug-in modules. The Accuracy-Conditioned KL Scaling (AKL) dynamically adjusts KL regularization strength through a smooth nonlinear function of batch average accuracy, relaxing the penalty when the model underperforms and strengthening it when the model achieves good results. The Gaussian Curriculum Sampling (GCS) assigns sampling weights to questions following a Gaussian distribution centered at moderate accuracy around 0.5, focusing training on the model's learning frontier. We conduct extensive experiments on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base over six mathematical reasoning benchmarks. The results show EXPO steadily surpasses vanilla GRPO. It obtains an absolute gain of 13.34 on AIME 2025 pass@32, rising from 63.33 percent to 76.67 percent, and achieves an average pass@32 improvement of 2.66 on the 8B model. The much larger performance gains on pass@32 compared with pass@1 demonstrate that EXPO effectively enlarges the model's exploration boundary under a fixed inference cost budget.
Group Relative Policy Optimization (GRPO) has significantly advanced the reasoning ability of large language models (LLMs), particularly in their mathemat ical reasoning performance. However, GRPO and related entropy regularization methods still struggle with token-level sparse-rewards, which is an inherent chal lenge in chain-of-thought (CoT) reasoning. These approaches often rely on undifferen tiated token-level entropy regularization, which easily leads to entropy collapse or model degradation under sparse token rewards. In this work, we propose TEPO, a novel token-level framework that (1) leverages sequence-level likelihood to link group-level rewards with individual tokens via token-level aggregation, and (2) introduces a token-level KL-Divergence mask constraint that targets tokens with positive advantages and decreasing entropy to mitigate abrupt policy updates. Experiments demonstrate that TEPO not only achieves state-of-the-art performance on mathematical reasoning benchmarks but also markedly enhances training stability, reducing convergence time by 50% compared with GRPO/DAPO.
Xingyu Lin, Yilin Wen, Du Su +5
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University · Baidu Inc. +2
Group Relative Policy Optimization (GRPO) has become a standard reinforcement learning method for post-training language models. Recent work shows that GRPO can reduce the base model's reasoning capacity and underperform it in Pass@k when k is large, indicating reduced coverage of reasoning paths. We find that this reduction is associated with GRPO concentrating on responses that the base model already generates with high probability. We trace this concentration to two mechanisms in the GRPO update. At the response level, high-probability responses dominate the group gradient through repeated occurrence. At the token level, GRPO's importance ratio scales gradients, further reinforcing tokens that become more likely under the current policy. We propose ReCo, a reweighting method that addresses both effects. Response contributions are normalized by their expected occurrence within the rollout group, and the token-level importance ratio is replaced with a variance-based ratio that gives larger update scale to non-saturated decision points where alternative token choices remain plausible. Across Qwen2.5-Math-1.5B/7B and Llama-3.1-8B-Instruct on five mathematical reasoning benchmarks, ReCo improves Pass@k for large values of k and is comparable to GRPO for small values of k.
Junoh Park, Junseo Hwang, Wonguk Cho +1
Graduate School of Data Science, Seoul National University