In reinforcement learning with verifiable rewards, a language model is rewarded for a correct final answer, and GRPO passes that reward to every token of the solution in equal measure. We instead place the credit where the model hesitated, in a method we call MCPO (Margin-Credit Policy Optimization). We measure hesitation by the top-2 margin M=p1−p2 of the policy's next-token distribution, map it to a score with two fixed thresholds, and spread each solution's advantage over its tokens so that the mean stays unchanged. The margin is read from a forward pass the trainer already performs, so the method needs no extra model evaluation and no batch statistics. We train MiniCPM5-2B-Base with LoRA for 200 steps on 2,048 DAPO-Math problems and evaluate on six competition-math benchmarks. MCPO beats an unmodified GRPO baseline for all five data seeds, by 6.55 points of macro avg@32 on average (95% CI [4.1,9.0]), and a Qwen3-4B-Base replication improves for all three seeds (+4.75). The gain is largest on problems that GRPO solves only some of the time and grows with k in pass@k, reaching 13.8 points at pass@32. MCPO models also write two to three times longer answers, and runs that write longer score higher; our experiments do not separate the two effects.
Figures & tables
Figure 1: Overview. (a) Relative to the GRPO baseline, the method changes only the token-level advantages; the new components are orange. The old-policy forward pass that the trainer already runs also gives the top-2 margin M=p1−p2 at every position, and a fixed-threshold map turns it into a score h∈[0,1] . (b) An illustrative completion. Tokens with a clear winner get h=0 and close calls get h≈1 . In a positive completion the close calls receive more of the advantage; in a negative one they receive less of the penalty. The mean token advantage stays Ai .
Figure 2: Entropy vs. top-2 margin for 4,000 synthetic next-token distributions (random Gaussian logits over 2,000 tokens with 0–3 boosted candidates; not model outputs). Dotted lines: the thresholds 0.3 and 0.9 . For M∈[0.3,0.4) the 5th–95th percentile of the entropy is 1.5–4.8 nats.
Figure 3: (a) The fixed-threshold score h(M) of Equation 3 . (b,c) Macro avg@32 over six benchmarks per data seed, GRPO vs. MCPO, for MiniCPM5-2B-Base (5 seeds) and Qwen3-4B-Base (3 seeds). Each line is one data seed.
Setting
Value
MCPO
κ=ln4 , ℓ=0.1 , u=0.7
Adapter
LoRA r=32 , α=64
Learning rate / steps
10−5 / 200
Rollout
16 prompts × 8 samples
Steps per rollout
2 (64 samples each)
Max. completion
10,240 tokens
Table 1: Settings shared by the GRPO and MCPO runs.
Data seed
GRPO
Margin
Gain
43
31.04
34.16
+3.12
45
29.83
36.71
+6.87
100
34.67
40.32
+5.65
1024
26.79
34.37
+7.57
31415
24.65
34.18
+9.53
Mean
29.40
35.95
+6.55
Table 2: MiniCPM5-2B-Base: macro avg@32 (%) over six benchmarks. Bootstrap 95% CI of the mean gain [+4.14,+8.98] ; t -test p=0.004 ( df=4 ).
Data seed
GRPO
Margin
Gain
42
22.93
24.97
+2.04
43
22.95
30.00
+7.05
45
23.25
28.41
+5.16
Mean
23.04
27.79
+4.75
Table 3: Qwen3-4B-Base: macro avg@32 (%). Bootstrap 95% CI [+2.06,+7.65] ; t -test p=0.083 ( df=2 ; three seeds, so the t -test has little power).
MiniCPM5-2B-Base
Qwen3-4B-Base
Benchmark
n
GRPO
Margin
Gain
95% CI
GRPO
Margin
Gain
95% CI
AIME24
30
15.77
27.90
+12.13
[+8.4, +16.2]
11.94
16.63
+4.69
[+1.7, +8.2]
AIME25
30
15.96
21.10
+5.15
[+2.3, +8.4]
8.16
14.13
+5.97
[+2.1, +10.7]
AIME26
30
14.58
21.23
+6.65
[+3.6, +10.2]
7.19
13.23
+6.04
[+2.3, +10.6]
HMMT26
33
12.01
14.98
+2.97
[ − 1.2, +7.1]
5.78
8.96
+3.19
[0.0, +7.0]
AMC23
40
59.59
64.80
+5.20
[+2.5, +8.3]
53.67
55.57
+1.90
[ − 1.6, +5.5]
Table 4: avg@32 (%) per benchmark, averaged over data seeds. The per-benchmark intervals are bootstrap 95% intervals over problems for the seed-averaged gain (10,000 draws) and therefore do not include seed variation; the macro-average intervals are the two-level bootstrap intervals of Tables 2 and 3 .
GRPO acc.
n
GRPO
Margin
Diff
MiniCPM5-2B-Base
0
71
0.3
2.9
+2.6
(0,25%]
68
8.8
21.3
+12.5
(25%,75%]
72
48.4
60.1
+11.6
>75%
86
92.2
93.3
+1.1
Qwen3-4B-Base
Table 5: Accuracy (%) by difficulty bin. Bins use GRPO’s first 16 samples per problem, pooled over data seeds; all accuracies use the remaining 16 samples (held out from binning). Each problem counts equally.
Figure 4: Mean answer length over all evaluation answers, per data seed.
MiniCPM
Qwen3-4B
Benchmark
GRPO
Margin
GRPO
Margin
AIME24
1,511
4,785
2,704
5,887
AIME25
1,404
4,364
1,984
4,596
AIME26
1,364
4,314
2,142
5,473
HMMT26
1,329
4,922
2,054
5,463
AMC23
1,272
2,911
1,444
2,773
Table 6: Mean answer length in tokens per benchmark, averaged over data seeds (mean of per-benchmark means).
Figure 5: Macro avg@32 vs. mean answer length for every run (one point per method and data seed).
Figure 6: (a,b) Macro pass@ k over the six benchmarks (mean over data seeds; band: min–max over seeds). (c) Per-problem accuracy, GRPO vs. MCPO, MiniCPM5-2B-Base, pooled over seeds (small jitter added); points above the dashed diagonal favor MCPO.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Seed
AIME24
AIME25
AIME26
HMMT26
AMC23
M500
MiniCPM5-2B-Base
43
+8.85
+1.46
+3.02
+1.04
+1.33
+3.01
45
+10.63
+6.25
+7.08
+2.27
+7.19
+7.81
100
+11.15
+6.04
+3.65
+2.18
+4.45
+6.44
1024
+12.60
+6.46
+9.69
+3.79
+5.00
+7.91
31415
+17.40
+5.52
+9.79
+5.59
+8.05
+10.82
Appendix
Table 7: Gain (margin − GRPO) per data seed and benchmark (M500 = MATH500 level 5). All 48 entries are positive.