Organizations: Mohamed bin Zayed University of Artificial Intelligence · Renmin University of China · Carnegie Mellon University · Hong Kong University of Science and Technology (Guangzhou)
Group Relative Policy Optimization (GRPO) has proven effective in RLVR by using outcome-based rewards. While fine-grained dense rewards can theoretically improve performance, we reveal that under practical sampling budgets, Monte Carlo estimation yields high-variance and sign-inconsistent advantages for early reasoning tokens, paradoxically underperforming outcome-only GRPO. We propose Skip-Connected Optimization (SKPO), which decomposes reasoning into upstream and downstream phases: upstream receives dense rewards from downstream Monte Carlo sampling with single-stream optimization; downstream maintains group-relative optimization, where a skip connection concatenates the upstream segment with the original problem, enabling the model to leverage helpful upstream reasoning while preserving the freedom to bypass flawed reasoning through direct problem access. Experiments demonstrate improvements of 3.91% and 6.17% relative gains over the strongest baselines on Qwen2.5-Math-7B and Llama-3.2-3B respectively across mathematical benchmarks and out-of-domain tasks including general reasoning and code generation. Further analysis reveals an implicit advantage: SKPO generates trajectories with higher intermediate-step quality even when matched for final correctness.
Figures & tables
Figure 1: Monte Carlo estimation of token-level rewards under practical sampling budgets ( K=8 ). Top: true values (left) vs. estimates (right). Bottom: sign accuracy and MAE, showing that early tokens suffer from higher estimation error. By the nature of MC sampling, true token rewards converge to the policy’s average accuracy on the prompt at early positions, and to the trajectory’s binary correctness at late positions; the former makes accurate advantage estimation difficult at early steps, while the latter explains why GRPO achieves reasonable posterior estimation. Additional examples across more correctness distributions (0/8 to 8/8) are provided in Section A.3 .
Figure 2: Overview of the SKPO algorithm. We illustrate three components: (1) upstream sampling process and components for upstream advantage computation; (2) downstream sampling process and components for downstream advantage computation; (3) engineering optimization that completes both upstream and downstream sampling within a single GPU batch through KV cache rewriting.
Qwen2.5-Math-7B
Llama-3.2-3B-Instruct
In-Domain (Math)
Out-of-Domain
In-Domain (Math)
Out-of-Domain
Method
AIME24
AIME25
AMC
MATH
Minerva
Olympiad
MMLU
LiveCode
Avg.
AIME24
AIME25
AMC
MATH
Olympiad
MMLU
LiveCode
Avg.
Base
12.1
5.4
43.4
58.1
20.6
34.1
18.6
13.8
25.8
3.4
0.4
20.1
36.3
18.1
29.5
12.4
17.2
GRPO
22.4
10.1
50.2
63.8
23.5
35.8
21.3
14.6
30.2
4.7
0.3
24.8
39.6
19.8
30.2
13.1
18.9
GSPO
31.1
14.8
60.3
74.2
28.1
43.6
25.2
16.2
36.7
6.9
0.5
30.4
42.1
22.3
30.5
13.8
21.0
SPO
28.8
13.6
56.8
65.9
29.2
40.1
24.1
15.7
34.3
5.8
0.4
26.1
39.2
20.3
29.8
12.9
19.2
Table 1: Performance comparison on In-Domain (Math) and Out-of-Domain (General/Code) benchmarks. We evaluate on two base models: Qwen2.5-Math-7B and Llama-3.2-3B-Instruct . The best results for each model are bolded , and the second best are underlined .
Figure 3: Upstream impact analysis comparing three conditioning strategies across Qwen and Llama models. Top : Diversity (average distinct answers per 8 rollouts); Middle : Advantage zero rate (probability of homogeneous outcomes); Bottom : Response length (excluding prefix). Continual conditioning leads to diversity collapse and elevated advantage zero rate, while Skip maintains properties comparable to Unconditional.
Setting
AIME24
AIME25
AMC
MATH
Minerva
Olympiad
w/o Skip
32.8
15.4
67.8
74.6
29.8
46.3
w/o MC Estimation
33.6
17.4
68.6
78.3
32.9
48.7
Split@Fixed
29.5
14.1
58.2
67.3
30.0
41.5
Selection:Extreme
28.2
13.3
55.9
65.1
28.6
39.4
Ours
35.7
18.7
71.4
80.8
34.7
51.0
Table 2: Ablation study on In-Domain benchmarks comparing different structural settings. We report accuracy (%) for Qwen2.5-Math-7B. The best results are bolded .
Figure 4: Segment-level implicit advantage analysis using GPT-5-nano as external evaluator across both Qwen and Llama models. For each method’s correct responses, we split into 100 segments by relative position and estimate rewards via Monte Carlo continuation sampling. Inter-method relative advantages are computed at each position. SKPO consistently maintains higher advantages in the 20–50% position range, indicating superior intermediate reasoning quality during the problem-setup phase.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Probability of correct advantage signs for all G=8 samples across varying group spreads. Reliable credit assignment for narrow spreads requires prohibitive sample counts ( N>512 ).
Figure 6: Monte Carlo variance analysis. (a) Classification accuracy for individual samples. (b) Distribution shift and sign volatility under sampling noise.
Figure 7: Advantage estimation accuracy across different correctness distributions. Key observation : Only extreme cases (all correct or all incorrect) achieve reliable sign consistency. Mixed distributions suffer from severe sign errors throughout the trajectory, with accuracy often below 50%. Additional examples (1/8, 2/8, 4/8, 6/8, 7/8) are available in figures/mc/ .
Figure 8: MC estimation accuracy across different sampling budgets ( K=8 to K=8192 ). Left column shows true token rewards and advantages. Middle columns show MC estimates at each K value. Right column summarizes win rate, variance, and MAE trends. With sufficient samples ( K≥512 ), MC consistently outperforms GRPO, but the required sampling cost (up to 1000 × ) is impractical for real-world training.
Figure 9: Mean accuracy at 32 samples across training steps.
Figure 10: Actor policy entropy throughout training.
Figure 11: Training time per step across different methods. Critique-GRPO exhibits significantly higher per-step time due to its two-batch rollout strategy: first generating 7 direct responses, then spawning 1 self-critique response in a separate GPU batch. Despite the two-phase upstream-downstream architecture, SKPO achieves comparable wall-clock time to single-phase baselines through our engineering optimization that performs mid-generation KV cache rewriting within a single rollout pass.
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University · Baidu Inc. +2