In reinforcement learning with verifiable rewards, a language model is rewarded for a correct final answer, and GRPO passes that reward to every token of the solution in equal measure. We instead place the credit where the model hesitated, in a method we call MCPO (Margin-Credit Policy Optimization). We measure hesitation by the top-2 margin M=p1−p2 of the policy's next-token distribution, map it to a score with two fixed thresholds, and spread each solution's advantage over its tokens so that the mean stays unchanged. The margin is read from a forward pass the trainer already performs, so the method needs no extra model evaluation and no batch statistics. We train MiniCPM5-2B-Base with LoRA for 200 steps on 2,048 DAPO-Math problems and evaluate on six competition-math benchmarks. MCPO beats an unmodified GRPO baseline for all five data seeds, by 6.55 points of macro avg@32 on average (95% CI [4.1,9.0]), and a Qwen3-4B-Base replication improves for all three seeds (+4.75). The gain is largest on problems that GRPO solves only some of the time and grows with k in pass@k, reaching 13.8 points at pass@32. MCPO models also write two to three times longer answers, and runs that write longer score higher; our experiments do not separate the two effects.
Figures & tables
Figure 1: Overview. (a) Relative to the GRPO baseline, the method changes only the token-level advantages; the new components are orange. The old-policy forward pass that the trainer already runs also gives the top-2 margin M=p1−p2 at every position, and a fixed-threshold map turns it into a score h∈[0,1] . (b) An illustrative completion. Tokens with a clear winner get h=0 and close calls get h≈1 . In a positive completion the close calls receive more of the advantage; in a negative one they receive less of the penalty. The mean token advantage stays Ai .
Figure 2: Entropy vs. top-2 margin for 4,000 synthetic next-token distributions (random Gaussian logits over 2,000 tokens with 0–3 boosted candidates; not model outputs). Dotted lines: the thresholds 0.3 and 0.9 . For M∈[0.3,0.4) the 5th–95th percentile of the entropy is 1.5–4.8 nats.
Figure 3: (a) The fixed-threshold score h(M) of Equation 3 . (b,c) Macro avg@32 over six benchmarks per data seed, GRPO vs. MCPO, for MiniCPM5-2B-Base (5 seeds) and Qwen3-4B-Base (3 seeds). Each line is one data seed.
Setting
Value
MCPO
κ=ln4 , ℓ=0.1 , u=0.7
Adapter
LoRA r=32 , α=64
Learning rate / steps
10−5 / 200
Rollout
16 prompts × 8 samples
Steps per rollout
2 (64 samples each)
Max. completion
10,240 tokens
Table 1: Settings shared by the GRPO and MCPO runs.
Data seed
GRPO
Margin
Gain
43
31.04
34.16
+3.12
45
29.83
36.71
+6.87
100
34.67
40.32
+5.65
1024
26.79
34.37
+7.57
31415
24.65
34.18
+9.53
Mean
29.40
35.95
+6.55
Table 2: MiniCPM5-2B-Base: macro avg@32 (%) over six benchmarks. Bootstrap 95% CI of the mean gain [+4.14,+8.98] ; t -test p=0.004 ( df=4 ).
Data seed
GRPO
Margin
Gain
42
22.93
24.97
+2.04
43
22.95
30.00
+7.05
45
23.25
28.41
+5.16
Mean
23.04
27.79
+4.75
Table 3: Qwen3-4B-Base: macro avg@32 (%). Bootstrap 95% CI [+2.06,+7.65] ; t -test p=0.083 ( df=2 ; three seeds, so the t -test has little power).
MiniCPM5-2B-Base
Qwen3-4B-Base
Benchmark
n
GRPO
Margin
Gain
95% CI
GRPO
Margin
Gain
95% CI
AIME24
30
15.77
27.90
+12.13
[+8.4, +16.2]
11.94
16.63
+4.69
[+1.7, +8.2]
AIME25
30
15.96
21.10
+5.15
[+2.3, +8.4]
8.16
14.13
+5.97
[+2.1, +10.7]
AIME26
30
14.58
21.23
+6.65
[+3.6, +10.2]
7.19
13.23
+6.04
[+2.3, +10.6]
HMMT26
33
12.01
14.98
+2.97
[ − 1.2, +7.1]
5.78
8.96
+3.19
[0.0, +7.0]
AMC23
40
59.59
64.80
+5.20
[+2.5, +8.3]
53.67
55.57
+1.90
[ − 1.6, +5.5]
Table 4: avg@32 (%) per benchmark, averaged over data seeds. The per-benchmark intervals are bootstrap 95% intervals over problems for the seed-averaged gain (10,000 draws) and therefore do not include seed variation; the macro-average intervals are the two-level bootstrap intervals of Tables 2 and 3 .
GRPO acc.
n
GRPO
Margin
Diff
MiniCPM5-2B-Base
0
71
0.3
2.9
+2.6
(0,25%]
68
8.8
21.3
+12.5
(25%,75%]
72
48.4
60.1
+11.6
>75%
86
92.2
93.3
+1.1
Qwen3-4B-Base
Table 5: Accuracy (%) by difficulty bin. Bins use GRPO’s first 16 samples per problem, pooled over data seeds; all accuracies use the remaining 16 samples (held out from binning). Each problem counts equally.
Figure 4: Mean answer length over all evaluation answers, per data seed.
MiniCPM
Qwen3-4B
Benchmark
GRPO
Margin
GRPO
Margin
AIME24
1,511
4,785
2,704
5,887
AIME25
1,404
4,364
1,984
4,596
AIME26
1,364
4,314
2,142
5,473
HMMT26
1,329
4,922
2,054
5,463
AMC23
1,272
2,911
1,444
2,773
Table 6: Mean answer length in tokens per benchmark, averaged over data seeds (mean of per-benchmark means).
Figure 5: Macro avg@32 vs. mean answer length for every run (one point per method and data seed).
Figure 6: (a,b) Macro pass@ k over the six benchmarks (mean over data seeds; band: min–max over seeds). (c) Per-problem accuracy, GRPO vs. MCPO, MiniCPM5-2B-Base, pooled over seeds (small jitter added); points above the dashed diagonal favor MCPO.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Seed
AIME24
AIME25
AIME26
HMMT26
AMC23
M500
MiniCPM5-2B-Base
43
+8.85
+1.46
+3.02
+1.04
+1.33
+3.01
45
+10.63
+6.25
+7.08
+2.27
+7.19
+7.81
100
+11.15
+6.04
+3.65
+2.18
+4.45
+6.44
1024
+12.60
+6.46
+9.69
+3.79
+5.00
+7.91
31415
+17.40
+5.52
+9.79
+5.59
+8.05
+10.82
Appendix
Table 7: Gain (margin − GRPO) per data seed and benchmark (M500 = MATH500 level 5). All 48 entries are positive.
Reinforcement learning from verifiable rewards assigns one terminal reward to an entire reasoning trajectory, obscuring how individual reasoning steps contributed to the outcome. GRPO broadcasts a group-normalized advantage to every response token, while finer-grained approaches often rely on dense auxiliary rollouts or branching schemes that can also alter the data used for policy updates. We cast the missing middle as budgeted measurement: with a limited rollout budget, which partial solutions should be evaluated, and how should their value estimates refine credit? ProbePO selects partial-solution states with a reward-free uncertainty score and estimates their expected verifier rewards under a capped continuation budget. ProbePO redistributes, rather than replaces, GRPO's trajectory-level advantage across tokens while preserving each trajectory's mean token-level advantage; probe continuations serve only as measurements. Across three DeepSeekMath and Qwen-family models spanning 1.5B-7B parameters, ProbePO consistently outperforms GRPO and achieves the highest accuracy among the four evaluated methods in 16 of 18 model-benchmark comparisons. Against dense-probing VinePPO on DeepSeekMath-7B, ProbePO achieves higher accuracy on all six benchmarks with 76% less end-to-end training time.
Diyi Hu, Ruizhong Qiu, Lin Qiu +2
Independent Researcher · University of Illinois Urbana-Champaign · Meta
Verifier-guided reinforcement learning has become a powerful paradigm for improving LLM reasoning. In multi-turn settings, models receive a verifier score after each turn and iteratively refine their outputs. Although such scores provide dense feedback, they do not directly provide dense credit: a score measures the quality of the current output, while credit should measure how the current turn changes the refinement trajectory. We propose TCPO, a turn-level credit assignment method for verifier-guided multi-turn RL. TCPO casts credit assignment as score-to-credit conversion and constructs turn-level advantages through reference-based comparisons: retrospective credit captures immediate progress and regression relative to the best prior state; hindsight delayed credit identifies non-improving turns with later payoff; and selective fixed-history counterfactual estimation refines high-surprisal turns under the same history. Experiments on math reasoning, code generation, and AppWorld agent tasks show that TCPO improves or matches the strongest baselines across model scales, task domains, and verifier types. TCPO achieves the best or tied-best best-turn Pass@8 on Qwen3-4B and DeepSeek-R1-Distill-Llama-8B, reduces turns to success, and improves multi-turn agent performance. These results highlight score-to-credit conversion as a central ingredient for verifier-guided multi-turn policy optimization.
Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps. Existing token-level credit assignment methods require resources beyond the model's own rollouts. GRPO variants rely on process reward models or ground-truth answers. Knowledge distillation assigns credit through per-token divergence but requires external teachers (On-Policy Distillation) or privileged information (On-Policy Self Distillation). However, these dependencies limit applicability in the pure RLVR setting. We observe that conditioning the model on its own verified trajectories induces a measurable per-token KL divergence between the original and conditioned distributions, and prove that distilling from a self-teacher constructed by verified trajectories leads to infeasible weighted-average solutions when multiple verified trajectories exist. We propose SC-GRPO (Self-Conditioned GRPO), which uses KL divergence mentioned before as a multiplicative weight on GRPO gradients. Across five benchmarks spanning math, code, and agentic tasks, SC-GRPO consistently outperforms 8.1% over GRPO and 5.9% over DAPO with stronger OOD performance. Moreover, SC-GRPO achieves higher performance than OPD.
Yingyu Shan, Yuhang Guo, Zihao Cheng +7
Beijing Institute of Technology · Beihang University · Independent Researcher