GRPO assigns the same group-relative advantage to equally rewarded responses within a question, leaving their generation uncertainty unused. We introduce MGE-PO, centered on four-quadrant (4Q) advantage shaping that combines verified correctness with rollout generation entropy. Correctness determines the sign of an additional correction, while entropy controls its magnitude: lower-entropy successes receive larger bonuses and higher-entropy failures receive larger penalties. An online entropy-based question curriculum supports this credit rule using cached rollout measurements. At the final checkpoint after eight training epochs on Qwen3-8B-Base, adding 4Q across three policy optimization settings yields absolute gains of 3.23-3.69% in six-benchmark macro avg@32 and 4.24-11.06% in macro pass@32. On Qwen2.5-7B-Base, the complete configuration raises four-benchmark macro avg@32 from 41.5% to 45.4%.
Figures & tables
Figure 1: GRPO-based MGE-PO centered on 4Q advantage shaping. Verified outcomes and cached token entropy determine continuous corrections: lower-entropy successes receive larger bonuses, higher-entropy failures larger penalties. Top- K pooling and running normalization produce ui . Corrections precede batch recentering and the clipped update; gi=0 disables them for truncated rollouts. High-entropy successes and low-entropy failures remain active. DynZPD and EA-KL support the complete configuration.
Figure 2: Continuous 4Q corrections for gi=1 , γ+=0.5 , and γ−=0.3 (Qwen2.5). The horizontal axis is standardized rollout entropy zi (Section 4.1). Lower entropy increases the success bonus; higher entropy increases the failure penalty. PLR/PHR and NLR/NHR mark within-outcome entropy regions; dashed lines mark zero. Corrections precede batch recentering, and PHR/NLR remain active.
Method
AIME2025 (%)
AIME2024 (%)
AMC2023 (%)
MATH500 (%)
Minerva (%)
Olympiad Bench (%)
Macro (%)
GRPO
13.44 / 33.33
5.52 / 13.33
61.02 / 85.00
76.45 / 89.40
46.66 / 60.29
44.78 / 66.47
41.31 / 57.97
DAPO
16.35 / 40.00
10.73 / 23.33
65.47 / 92.50
78.79 / 92.60
44.47 / 74.63
45.87 / 75.22
43.61 / 66.38
Clip-Cov
15.52 / 43.33
10.83 / 33.33
63.59 / 92.50
78.40 / 93.20
45.23 / 72.79
45.92 / 72.85
43.25 / 68.00
MGE-PO
23.96 / 53.33
13.02 / 40.00
64.53 / 97.50
81.31 / 93.00
47.48 / 68.01
52.47 / 74.33
47.13 / 71.03
Clip-Cov + 4Q
20.42 / 56.67
16.25 / 43.33
67.81 / 92.50
81.11 / 94.00
44.16 / 70.96
49.11 / 75.96
46.48 / 72.24
4Q
18.85 / 46.67
12.60 / 36.67
62.66 / 92.50
79.66 / 92.60
46.90 / 69.49
49.33 / 76.26
45.00 / 69.03
Table 1: Qwen3-8B-Base at final controller step 624 (eight epochs). Cells report avg@32 / pass@32 (%) using 32 responses per question, temperature 0.6, and matched test sets and decoding. Avg@32 equals mean@32. Macro scores weight six benchmarks equally. Bold marks each metric’s maximum across seven configurations, including ties. Displayed scores use two decimals; macro averages retain reported precision before rounding.
Figure 3: Adding 4Q to GRPO, DAPO, and Clip-Cov on Qwen3-8B-Base at step 624. Bars show absolute macro mean@32 (avg@32) and pass@32 changes (%), computed before rounding. Pairings are those in Section 6.2; values are point estimates without uncertainty intervals.
Initialization
Method
Macro avg@32 (%)
Macro pass@32 (%)
AIME24 pass@32 (%)
AIME25 pass@32 (%)
Qwen2.5-Math-1.5B-Base
GRPO
33.7
60.1
30.3
30.7
DAPO
35.3
59.2
32.7
26.7
KL-Cov
35.8
61.5
31.8
35.7
MGE-PO
36.0
63.3
33.3
38.3
Qwen2.5-Math-7B-Base
GRPO
41.4
63.2
36.7
35.4
DAPO
42.7
65.3
43.3
36.7
Table 2: Four-benchmark summary (%): macro scores are unweighted means over MATH-500, AMC23, AIME24, and AIME25 (full scores: Appendix D.1). Each model-method cell uses one training run. MGE-PO-GRPO comparisons include a global KL-coefficient difference (Appendices C.3 and D.10).
Figure 4: Conditional 4Q gains on Qwen2.5-7B-Base at β=0.001 (Table 15). Pairs share their supporting configuration; EA-KL off retains uniform KL. Scores are four-benchmark macro avg@32; annotations give percentage-point gains. Each configuration uses one training run; uncertainty intervals are unavailable.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Quantity
Reported value or status
Training
Reported global batch size
128; question/response unit unspecified
Training
Completions per question
16
Training
Dataset-equivalent epochs
8
Training
Sampling temperature
1.0
Training
Maximum generation length
12,288 tokens
Training
Top- p during rollout
Not reported
Appendix
Table 3: Reported hyperparameters and configuration metadata. Values describe the original Qwen2.5 experiments and their supplementary settings. Appendix C.4 describes the separate Qwen3-8B-Base training and evaluation protocol.
Configuration
Value
Base model
Qwen3-8B-Base
Training data
Random 10,000-question subset of the Skywork-OR1 mathematical training set
Data-subsampling seed
1
Training questions after filtering
9,998
Train batch size
128 questions
Training rollouts per question
8
Appendix
Table 4: Training and evaluation protocol for the Qwen3-8B-Base experiments. The seed below controls dataset subsampling. All seven configurations share the learning rate, evaluation question sets, and decoding settings, and are evaluated at the final controller step 624.
Initialization
Method
A25 avg (%)
A25 pass (%)
A24 avg (%)
A24 pass (%)
AMC avg (%)
AMC pass (%)
M500 avg (%)
M500 pass (%)
Macro avg (%)
Macro pass (%)
Qwen2.5-Math-1.5B Base
Base
3.3
28.7
6.9
10.3
32.5
72.5
32.8
77.8
18.9
47.3
GRPO
6.3
30.7
9.5
30.3
52.7
90.0
66.5
89.4
33.7
60.1
DAPO
7.1
26.7
11.0
32.7
54.5
87.5
68.7
90.0
35.3
59.2
KL-Cov
8.3
35.7
10.9
31.8
55.1
87.5
68.8
90.8
35.8
61.5
MGE-PO
8.9
38.3
10.2
33.3
55.3
92.5
69.4
89.2
36.0
63.3
Qwen2.5-Math-7B Base
Base
3.5
26.7
9.0
30.0
38.3
85.3
47.1
86.4
24.5
57.1
Appendix
Table 5: Full original results under avg@32/pass@32. Each RL method cell is based on one training run; Base rows evaluate the pre-RL initialization. Pass@32 averages repeated inference evaluations of that checkpoint. Macro averages are retained as reported; rounding of displayed per-benchmark values can produce small differences from a recomputed mean. A25/A24 denote AIME 2025/2024, AMC denotes AMC23, and M500 denotes MATH-500; avg/pass denote avg@32/pass@32. All scores use %.
Method
MATH-500 avg@64 (%)
AMC23 avg@64 (%)
Minerva avg@64 (%)
Olympiad avg@64 (%)
AIME24 pass@256 (%)
AIME25 pass@256 (%)
GRPO
66.3
50.0
35.4
37.4
36.7
53.3
DAPO
68.3
57.7
22.7
30.1
30.0
53.3
KL-Cov
67.1
53.5
36.3
39.9
38.9
57.8
MGE-PO
68.6
55.6
38.5
41.3
40.0
60.0
Appendix
Table 6: Supplementary Qwen2.5-7B-Base evaluation. Accuracy uses 64 samples per question and AIME coverage uses 256. Methods share the initialization, reported training budget, and evaluation protocol. The KL-coefficient distinction is specified in Appendix D.10. All scores use %.
Method
Minerva-Math avg@64 (%)
Olympiad avg@64 (%)
GRPO
35.4
37.4
DAPO
22.7
30.1
KL-Cov
36.3
39.9
Clip-Cov
35.1
38.4
MGE-PO
38.5
41.3
Appendix
Table 7: Available supplementary larger-benchmark comparisons. Qwen2.5-7B-Base; 64 responses per question, maximum generation length 8,192 tokens. The available Clip-Cov comparison covers these two benchmarks. All scores use %.
Diagnostic quadrant
Initialization count
Initialization share (%)
GRPO count
GRPO share (%)
Low-entropy correct
173
4.5
492
12.8
High-entropy correct
74
1.9
96
2.5
Low-entropy incorrect
1747
45.5
1428
37.2
High-entropy incorrect
1846
48.1
1824
47.5
Appendix
Table 8: Aggregate quadrant counts in the original manuscript. Each column summarizes 3,840 rollouts, with a reporting threshold at ui=0.5 . The comparison is initialization versus ordinary GRPO, not GRPO versus MGE-PO.
Benchmark
Questions
Original protocol
Supplementary protocol
MATH-500
500
avg@32, pass@32
avg@64
AMC 2023
40
avg@32, pass@32
avg@64
AIME 2024
30
avg@32, pass@32
pass@256
AIME 2025
30
avg@32, pass@32
pass@256
Minerva-Math / OCW
272
Not evaluated
avg@64
Olympiad, text-only English exact-answer subset
674
Not evaluated
avg@64
Appendix
Table 9: Qwen2.5 evaluation datasets and reported metrics. Competition subset sizes and the text-only English exact-answer Olympiad subset follow the experimental record. Appendix C lists the reproducibility requirements for exact manifests, grader versions, and overlap filtering.
Method
MATH-500 (%)
AMC23 (%)
Minerva (%)
Olympiad (%)
AIME24 (%)
AIME25 (%)
GRPO
80.3
62.4
32.4
34.6
43.3
53.3
MGE-PO
81.0
63.9
34.5
36.3
46.7
63.3
Absolute difference: MGE-PO minus GRPO (%)
+0.7
+1.5
+2.1
+1.7
+3.4
+10.0
Appendix
Table 10: Qwen2.5-Math-7B-Instruct at the final reported checkpoint. Both methods start from the same Instruct initialization. The first four columns use avg@64; AIME columns use pass@256. The record identifies step 1080 after eight epochs in the setting described as “4k”; the role of this setting is unspecified. All scores and absolute differences use %.
Configuration
PLR
NHR
PHR
NLR
avg@32 (%)
pass@32 (%)
Default
0.5
0.3
0
0
23.1
50.0
PLR-weighted term removed
0
0.3
0
0
21.3
36.7
NHR-weighted term removed
0.5
0
0
0
22.2
36.7
Magnitudes swapped
0.3
0.5
0
0
22.6
40.0
Additional PHR term
0.5
0.3
0.3
0
21.8
36.7
NHR-weighted term replaced by NLR-weighted term
0.5
0
0
0.3
21.6
33.3
Appendix
Table 11: Quadrant-coefficient ablation on Qwen2.5-Math-7B-Base. AIME24 uses the original avg@32/pass@32 protocol. Coefficients are nonnegative magnitudes in the generalized shaping rule: PLR/PHR terms are added and NHR/NLR terms are subtracted. Test-set selection scope is discussed below. Accuracy and coverage use %; quadrant coefficients are unitless.
Diagnostic
GRPO / comparison value
MGE-PO / EA-KL value
Scope
Mean EA-KL weight in lower-entropy PLR token stratum
Not applicable
1.19
Active PLR tokens, nine updates
Mean EA-KL weight in higher-entropy PLR token stratum
Not applicable
0.35
Same token window
Lower-/higher-entropy weight ratio
Not applicable
3.1–3.4 across updates
Same token window
Late-training policy entropy (nats/token)
Approximately 0.001
Approximately 0.06
Original Qwen2.5-7B-Base training curves
Late-training response length (tokens)
Approximately 6,000
Approximately 4,500
Original Qwen2.5-7B-Base training curves
Appendix
Table 12: Token-weight and training-dynamics diagnostics. The token analysis covers approximately 1.78 million active PLR tokens over nine actor updates. Entropy and response-length entries summarize the original plain-7B training curves and have a different scope from the token window.
Initialization
Method
Wall-clock hours
Base setting
GRPO
22.8
Base setting
KL-Cov
29.6
Base setting
MGE-PO
22.6
Qwen2.5-Math-7B-Instruct
GRPO
23.6
Qwen2.5-Math-7B-Instruct
MGE-PO
22.8
Appendix
Table 13: End-to-end wall-clock time for complete single runs with matched training budgets. Matching refers to training budget rather than KL coefficients. The timing record identifies a base setting without a separate model-checkpoint or hardware identifier. These are wall-clock measurements; utilization and GPU-hour accounting are unavailable.
Diagnostic
GRPO, uniform
KL-Cov, uniform
MGE-PO, DynZPD
Relative ESS
100%
100%
91.5%
Top-10% question probability mass
10.0%
10.0%
18.3%
Sampling-distribution Gini
0
0
0.085
Maximum question share (definition unreported)
Not reported
Not reported
0.4%
In-window unique-question coverage
14.3%, expected
14.3%, expected
14.0%, observed
Appendix
Table 14: Curriculum diagnostics in a 21-step window. The window contains 43,008 rollouts and 2,440 unique MGE-PO questions from the 17,398-question training dataset. Relative ESS, mass, and Gini describe the sampling distribution; unique-question coverage distinguishes uniform expectations from observed draws.
EA-Adv
EA-KL
DynZPD
Reference coefficient β
Macro avg@32 (%)
Absolute difference from GRPO (%)
Off
Off
Off
0.04
41.5
0.0
On
Off
Off
0.001
43.3
+1.8
Off
On
Off
0.001
42.2
+0.7
Off
Off
On
0.001
41.6
+0.1
On
On
Off
0.001
44.3
+2.8
On
Off
On
0.001
43.9
+2.4
Appendix
Table 15: Component configurations on Qwen2.5-7B-Base. The score is four-benchmark macro avg@32. EA-KL off retains uniform reference KL.
Optimization setting
avg@32 without 4Q (%)
avg@32 with 4Q (%)
Absolute Δ avg@32 (%)
pass@32 without 4Q (%)
pass@32 with 4Q (%)
Absolute Δ pass@32 (%)
GRPO
41.31
45.00
+3.69
57.97
69.03
+11.06
DAPO
43.61
46.99
+3.38
66.38
71.69
+5.31
Clip-Cov
43.25
46.48
+3.23
68.00
72.24
+4.24
Appendix
Table 16: 4Q ablation across three Qwen3-8B-Base optimization settings at the final controller step 624. Each row compares a method with its 4Q-augmented configuration in Table 1. The GRPO comparison uses the row labeled 4Q. Scores are six-benchmark macro averages (%); differences are absolute score changes (%) computed before rounding the macro averages to two decimals.
Method
Signal and granularity
Advantage intervention
KL intervention
Data selection
DAPO (Yu et al., 2025)
Group outcomes
Group-relative advantage
Frozen-reference KL omitted
Retains nondegenerate reward groups
EntroAdv (Cheng et al., 2026)
Token predictive entropy
Detached entropy bonus, capped by advantage magnitude; Equation (E1)
—
—
SEED-GRPO (Chen et al., 2025)
Question semantic entropy SE(q)
Semantic-entropy attenuation of advantage magnitude; Equation (E2)
—
—
EAPO v1 (He et al., 2026)
Group-normalized token entropy zi,t and advantage polarity
Table 17: Comparison of intervention rules. Hi,t denotes token predictive entropy, hitop the top- K pooled rollout statistic, ui its decreasing logistic transform, ri the binary outcome, gi the truncation gate, and ξq the question EMA. Ai is the base GRPO advantage. 4Q is MGE-PO’s central credit-assignment rule; its curriculum and entropy-weighted reference penalty are included to specify the complete implemented configuration. The MGE-PO advantage expression precedes batch re-centering. An em dash indicates that the cited method introduces no separate intervention in that column.
Figure 5: Per-benchmark effect of adding 4Q on Qwen3-8B-Base at the final controller step 624. Each cell gives the augmented score minus its baseline score as an absolute difference (%), for mean@32 (top) and pass@32 (bottom). Rows identify GRPO, DAPO, and Clip-Cov; the corresponding augmented configurations are 4Q, 4Q + DAPO, and Clip-Cov + 4Q. All 36 contrasts share a zero-centered color scale, including negative and zero changes. Differences use the reported two-decimal benchmark scores. Evaluation samples 32 responses per question at temperature 0.6 on identical test sets. Cells describe point estimates without significance claims.
Figure 6: Original scores stratified by initial-model entropy. The pooled AIME24, AIME25, and AMC23 set contains 100 questions, divided into five equal-frequency buckets of 20 using top- K pooled entropy from 64 base-model completions per question (pooling fraction 0.1). Paired bars show GRPO and complete MGE-PO mean@32 (left) and pass@32 (right), both in %; annotations give absolute score differences (%). The source does not specify the backbone. This is a full-configuration diagnostic and is separate from the Qwen3-8B 4Q comparisons.
Figure 7: Original complete-configuration training trajectories on Qwen2.5-7B-Base over 1080 steps. Panels show four-benchmark mean@32 (avg@32), pass@32, response length, and token entropy for GRPO, DAPO, and MGE-PO. These trajectories are separate from Table 2’s summary scores; their checkpoint correspondence is undocumented. The curves are retained as supplementary diagnostics of complete configurations, not as a 4Q component comparison.
Reinforcement learning from verifiable rewards assigns one terminal reward to an entire reasoning trajectory, obscuring how individual reasoning steps contributed to the outcome. GRPO broadcasts a group-normalized advantage to every response token, while finer-grained approaches often rely on dense auxiliary rollouts or branching schemes that can also alter the data used for policy updates. We cast the missing middle as budgeted measurement: with a limited rollout budget, which partial solutions should be evaluated, and how should their value estimates refine credit? ProbePO selects partial-solution states with a reward-free uncertainty score and estimates their expected verifier rewards under a capped continuation budget. ProbePO redistributes, rather than replaces, GRPO's trajectory-level advantage across tokens while preserving each trajectory's mean token-level advantage; probe continuations serve only as measurements. Across three DeepSeekMath and Qwen-family models spanning 1.5B-7B parameters, ProbePO consistently outperforms GRPO and achieves the highest accuracy among the four evaluated methods in 16 of 18 model-benchmark comparisons. Against dense-probing VinePPO on DeepSeekMath-7B, ProbePO achieves higher accuracy on all six benchmarks with 76% less end-to-end training time.
Diyi Hu, Ruizhong Qiu, Lin Qiu +2
Independent Researcher · University of Illinois Urbana-Champaign · Meta
Reinforcement learning with verifiable rewards has become the standard recipe for improving LLM reasoning, but the dominant algorithm GRPO assigns a single trajectory-level advantage to every token, diluting the signal at pivotal reasoning steps and injecting noise at uninformative ones. Critic-free alternatives derived from on-policy distillation supply per-token signals through oracle-conditioned likelihood ratios, yet apply each signal in isolation from the trajectory-level evidence accumulated up to that position. We propose Oracle-Prompted Policy Optimization (OPPO), which rests on a single observation: the oracle signal used by prior distillation-style methods for local discrimination is also the natural Bayesian update of the model's belief about eventual success. Accumulating the signal along a trajectory yields, in closed form and at the cost of one extra forward pass, a running estimate of the success probability at every position, together with a token-level advantage that requires no learned value network and no additional rollouts. A first-order analysis factorizes the advantage into the per-token discrimination signal used by distillation methods modulated by a state weight that concentrates credit on genuinely pivotal tokens, with a directional variance-reduction guarantee. The framework admits two estimators differing only in which model scores the evidence: a self-oracle that reuses the student and recovers the on-policy distillation reward as a strict special case, and a teacher-oracle that delegates scoring to a stronger frozen model. On two base LLMs across seven mathematics, science, and code reasoning benchmarks, OPPO improves over GRPO, DAPO, and SDPO by up to +6.0 points on AMC'23 and +5.2 points on AIME'24, with gains that widen monotonically with response length.
Yu Li, Rui Miao, Tian Lan +1
1George Washington University · 2The University of Texas at Dallas
Reinforcement learning with verifiable rewards (RLVR) supervises mathematical reasoning through final-answer correctness, but provides little guidance on individual tokens. On-policy distillation (OPD) supplies dense feedback on student-generated responses, yet teacher preference need not reflect correctness. Recent hybrids combine OPD and verifier-derived advantages or reweight task credit using teacher ratios. However, teacher guidance enters after verifier-based group normalization, and token reweighting need not preserve the total task credit assigned to each response. We introduce Unified Entropy-Calibrated Credit Redistribution for GRPO (UECR-GRPO), which integrates verifier and teacher signals within a single GRPO-style update at both the response and token levels. Path-Utility Unification (PUU) combines verifier reward and a teacher-to-anchor path log-ratio in a single KL-regularized objective. Its on-policy implementation uses a length-normalized teacher score and combines both rewards before group normalization and PPO clipping, allowing teacher evidence to influence the response ranking. Entropy-Calibrated Redistribution (ECR) then uses the signed teacher--old-policy token gap to redistribute the verifier-derived component. Full-vocabulary teacher entropy attenuates uncertain guidance, while a response-wise zero-sum projection preserves the total task credit and its token-wise sign before clipping. Across five mathematical reasoning benchmarks, UECR-GRPO achieves average Avg@12 accuracies of 17.21% and 65.09% with Qwen3-1.7B and Qwen3-4B students, respectively, exceeding the strongest baseline at each scale by 0.89 and 0.56 percentage points.
Jie Zhang, Jingxiao Yang, Zhehao Huang +2
Shanghai Jiao Tong University · Zhejiang University