MGE-PO: Entropy-Aware Credit Assignment for Mathematical Reasoning

Jihao Huang; Guangjia Chai; Zhirui Chen; Lei Wang; Yao Liu; Liu Qi; Pengjie Ding; Xinyang Liu

cs.CLOct 16, 2026

MGE-PO: Entropy-Aware Credit Assignment for Mathematical Reasoning

Authors: Jihao Huang, Guangjia Chai, Zhirui Chen, Lei Wang, Yao Liu, Liu Qi, Pengjie Ding, Xinyang Liu,

Abstract

GRPO assigns the same group-relative advantage to equally rewarded responses within a question, leaving their generation uncertainty unused. We introduce MGE-PO, centered on four-quadrant (4Q) advantage shaping that combines verified correctness with rollout generation entropy. Correctness determines the sign of an additional correction, while entropy controls its magnitude: lower-entropy successes receive larger bonuses and higher-entropy failures receive larger penalties. An online entropy-based question curriculum supports this credit rule using cached rollout measurements. At the final checkpoint after eight training epochs on Qwen3-8B-Base, adding 4Q across three policy optimization settings yields absolute gains of 3.23-3.69% in six-benchmark macro avg@32 and 4.24-11.06% in macro pass@32. On Qwen2.5-7B-Base, the complete configuration raises four-benchmark macro avg@32 from 41.5% to 45.4%.

Figures & tables

Figure 1: GRPO-based MGE-PO centered on 4Q advantage shaping. Verified outcomes and cached token entropy determine continuous corrections: lower-entropy successes receive larger bonuses, higher-entropy failures larger penalties. Top- $K$ pooling and running normalization produce $u_{i}$ . Corrections precede batch recentering and the clipped update; $g_{i}=0$ disables them for truncated rollouts. High-entropy successes and low-entropy failures remain active. DynZPD and EA-KL support the complete configuration.

Figure 2: Continuous 4Q corrections for

g_{i}=1

\gamma_{+}=0.5

, and

\gamma_{-}=0.3

(Qwen2.5). The horizontal axis is standardized rollout entropy

z_{i}

(Section 4.1). Lower entropy increases the success bonus; higher entropy increases the failure penalty. PLR/PHR and NLR/NHR mark within-outcome entropy regions; dashed lines mark zero. Corrections precede batch recentering, and PHR/NLR remain active.

Method	AIME2025 (%)	AIME2024 (%)	AMC2023 (%)	MATH500 (%)	Minerva (%)	Olympiad Bench (%)	Macro (%)
GRPO	13.44 / 33.33	5.52 / 13.33	61.02 / 85.00	76.45 / 89.40	46.66 / 60.29	44.78 / 66.47	41.31 / 57.97
DAPO	16.35 / 40.00	10.73 / 23.33	65.47 / 92.50	78.79 / 92.60	44.47 / 74.63	45.87 / 75.22	43.61 / 66.38
Clip-Cov	15.52 / 43.33	10.83 / 33.33	63.59 / 92.50	78.40 / 93.20	45.23 / 72.79	45.92 / 72.85	43.25 / 68.00
MGE-PO	23.96 / 53.33	13.02 / 40.00	64.53 / 97.50	81.31 / 93.00	47.48 / 68.01	52.47 / 74.33	47.13 / 71.03
Clip-Cov + 4Q	20.42 / 56.67	16.25 / 43.33	67.81 / 92.50	81.11 / 94.00	44.16 / 70.96	49.11 / 75.96	46.48 / 72.24
4Q	18.85 / 46.67	12.60 / 36.67	62.66 / 92.50	79.66 / 92.60	46.90 / 69.49	49.33 / 76.26	45.00 / 69.03

Table 1: Qwen3-8B-Base at final controller step 624 (eight epochs). Cells report avg@32 / pass@32 (%) using 32 responses per question, temperature 0.6, and matched test sets and decoding. Avg@32 equals mean@32. Macro scores weight six benchmarks equally. Bold marks each metric’s maximum across seven configurations, including ties. Displayed scores use two decimals; macro averages retain reported precision before rounding.

Figure 3: Adding 4Q to GRPO, DAPO, and Clip-Cov on Qwen3-8B-Base at step 624. Bars show absolute macro mean@32 (avg@32) and pass@32 changes (%), computed before rounding. Pairings are those in Section 6.2; values are point estimates without uncertainty intervals.

Initialization	Method	Macro avg@32 (%)	Macro pass@32 (%)	AIME24 pass@32 (%)	AIME25 pass@32 (%)
Qwen2.5-Math-1.5B-Base	GRPO	33.7	60.1	30.3	30.7
	DAPO	35.3	59.2	32.7	26.7
	KL-Cov	35.8	61.5	31.8	35.7
	MGE-PO	36.0	63.3	33.3	38.3
Qwen2.5-Math-7B-Base	GRPO	41.4	63.2	36.7	35.4
	DAPO	42.7	65.3	43.3	36.7

Table 2: Four-benchmark summary (%): macro scores are unweighted means over MATH-500, AMC23, AIME24, and AIME25 (full scores: Appendix D.1). Each model-method cell uses one training run. MGE-PO-GRPO comparisons include a global KL-coefficient difference (Appendices C.3 and D.10).

Figure 4: Conditional 4Q gains on Qwen2.5-7B-Base at

\beta=0.001

(Table 15). Pairs share their supporting configuration; EA-KL off retains uniform KL. Scores are four-benchmark macro avg@32; annotations give percentage-point gains. Each configuration uses one training run; uncertainty intervals are unavailable.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Quantity	Reported value or status
Training	Reported global batch size	128; question/response unit unspecified
Training	Completions per question	16
Training	Dataset-equivalent epochs	8
Training	Sampling temperature	1.0
Training	Maximum generation length	12,288 tokens
Training	Top- $p$ during rollout	Not reported

Appendix

Table 3: Reported hyperparameters and configuration metadata. Values describe the original Qwen2.5 experiments and their supplementary settings. Appendix C.4 describes the separate Qwen3-8B-Base training and evaluation protocol.

Configuration	Value
Base model	Qwen3-8B-Base
Training data	Random 10,000-question subset of the Skywork-OR1 mathematical training set
Data-subsampling seed	1
Training questions after filtering	9,998
Train batch size	128 questions
Training rollouts per question	8

Appendix

Table 4: Training and evaluation protocol for the Qwen3-8B-Base experiments. The seed below controls dataset subsampling. All seven configurations share the learning rate, evaluation question sets, and decoding settings, and are evaluated at the final controller step 624.

Initialization	Method	A25 avg (%)	A25 pass (%)	A24 avg (%)	A24 pass (%)	AMC avg (%)	AMC pass (%)	M500 avg (%)	M500 pass (%)	Macro avg (%)	Macro pass (%)
Qwen2.5-Math-1.5B Base	Base	3.3	28.7	6.9	10.3	32.5	72.5	32.8	77.8	18.9	47.3
	GRPO	6.3	30.7	9.5	30.3	52.7	90.0	66.5	89.4	33.7	60.1
	DAPO	7.1	26.7	11.0	32.7	54.5	87.5	68.7	90.0	35.3	59.2
	KL-Cov	8.3	35.7	10.9	31.8	55.1	87.5	68.8	90.8	35.8	61.5
	MGE-PO	8.9	38.3	10.2	33.3	55.3	92.5	69.4	89.2	36.0	63.3
Qwen2.5-Math-7B Base	Base	3.5	26.7	9.0	30.0	38.3	85.3	47.1	86.4	24.5	57.1

Appendix

Table 5: Full original results under avg@32/pass@32. Each RL method cell is based on one training run; Base rows evaluate the pre-RL initialization. Pass@32 averages repeated inference evaluations of that checkpoint. Macro averages are retained as reported; rounding of displayed per-benchmark values can produce small differences from a recomputed mean. A25/A24 denote AIME 2025/2024, AMC denotes AMC23, and M500 denotes MATH-500; avg/pass denote avg@32/pass@32. All scores use %.

Method	MATH-500 avg@64 (%)	AMC23 avg@64 (%)	Minerva avg@64 (%)	Olympiad avg@64 (%)	AIME24 pass@256 (%)	AIME25 pass@256 (%)
GRPO	66.3	50.0	35.4	37.4	36.7	53.3
DAPO	68.3	57.7	22.7	30.1	30.0	53.3
KL-Cov	67.1	53.5	36.3	39.9	38.9	57.8
MGE-PO	68.6	55.6	38.5	41.3	40.0	60.0

Appendix

Table 6: Supplementary Qwen2.5-7B-Base evaluation. Accuracy uses 64 samples per question and AIME coverage uses 256. Methods share the initialization, reported training budget, and evaluation protocol. The KL-coefficient distinction is specified in Appendix D.10. All scores use %.

Method	Minerva-Math avg@64 (%)	Olympiad avg@64 (%)
GRPO	35.4	37.4
DAPO	22.7	30.1
KL-Cov	36.3	39.9
Clip-Cov	35.1	38.4
MGE-PO	38.5	41.3

Appendix

Table 7: Available supplementary larger-benchmark comparisons. Qwen2.5-7B-Base; 64 responses per question, maximum generation length 8,192 tokens. The available Clip-Cov comparison covers these two benchmarks. All scores use %.

Diagnostic quadrant	Initialization count	Initialization share (%)	GRPO count	GRPO share (%)
Low-entropy correct	173	4.5	492	12.8
High-entropy correct	74	1.9	96	2.5
Low-entropy incorrect	1747	45.5	1428	37.2
High-entropy incorrect	1846	48.1	1824	47.5

Appendix

Table 8: Aggregate quadrant counts in the original manuscript. Each column summarizes 3,840 rollouts, with a reporting threshold at

u_{i}=0.5

. The comparison is initialization versus ordinary GRPO, not GRPO versus MGE-PO.

Benchmark	Questions	Original protocol	Supplementary protocol
MATH-500	500	avg@32, pass@32	avg@64
AMC 2023	40	avg@32, pass@32	avg@64
AIME 2024	30	avg@32, pass@32	pass@256
AIME 2025	30	avg@32, pass@32	pass@256
Minerva-Math / OCW	272	Not evaluated	avg@64
Olympiad, text-only English exact-answer subset	674	Not evaluated	avg@64

Appendix

Table 9: Qwen2.5 evaluation datasets and reported metrics. Competition subset sizes and the text-only English exact-answer Olympiad subset follow the experimental record. Appendix C lists the reproducibility requirements for exact manifests, grader versions, and overlap filtering.

Method	MATH-500 (%)	AMC23 (%)	Minerva (%)	Olympiad (%)	AIME24 (%)	AIME25 (%)
GRPO	80.3	62.4	32.4	34.6	43.3	53.3
MGE-PO	81.0	63.9	34.5	36.3	46.7	63.3
Absolute difference: MGE-PO minus GRPO (%)	+0.7	+1.5	+2.1	+1.7	+3.4	+10.0

Appendix

Table 10: Qwen2.5-Math-7B-Instruct at the final reported checkpoint. Both methods start from the same Instruct initialization. The first four columns use avg@64; AIME columns use pass@256. The record identifies step 1080 after eight epochs in the setting described as “4k”; the role of this setting is unspecified. All scores and absolute differences use %.

Configuration	PLR	NHR	PHR	NLR	avg@32 (%)	pass@32 (%)
Default	0.5	0.3	0	0	23.1	50.0
PLR-weighted term removed	0	0.3	0	0	21.3	36.7
NHR-weighted term removed	0.5	0	0	0	22.2	36.7
Magnitudes swapped	0.3	0.5	0	0	22.6	40.0
Additional PHR term	0.5	0.3	0.3	0	21.8	36.7
NHR-weighted term replaced by NLR-weighted term	0.5	0	0	0.3	21.6	33.3

Appendix

Table 11: Quadrant-coefficient ablation on Qwen2.5-Math-7B-Base. AIME24 uses the original avg@32/pass@32 protocol. Coefficients are nonnegative magnitudes in the generalized shaping rule: PLR/PHR terms are added and NHR/NLR terms are subtracted. Test-set selection scope is discussed below. Accuracy and coverage use %; quadrant coefficients are unitless.

Diagnostic	GRPO / comparison value	MGE-PO / EA-KL value	Scope
Mean EA-KL weight in lower-entropy PLR token stratum	Not applicable	1.19	Active PLR tokens, nine updates
Mean EA-KL weight in higher-entropy PLR token stratum	Not applicable	0.35	Same token window
Lower-/higher-entropy weight ratio	Not applicable	3.1–3.4 across updates	Same token window
Late-training policy entropy (nats/token)	Approximately 0.001	Approximately 0.06	Original Qwen2.5-7B-Base training curves
Late-training response length (tokens)	Approximately 6,000	Approximately 4,500	Original Qwen2.5-7B-Base training curves

Appendix

Table 12: Token-weight and training-dynamics diagnostics. The token analysis covers approximately 1.78 million active PLR tokens over nine actor updates. Entropy and response-length entries summarize the original plain-7B training curves and have a different scope from the token window.

Initialization	Method	Wall-clock hours
Base setting	GRPO	22.8
Base setting	KL-Cov	29.6
Base setting	MGE-PO	22.6
Qwen2.5-Math-7B-Instruct	GRPO	23.6
Qwen2.5-Math-7B-Instruct	MGE-PO	22.8

Appendix

Table 13: End-to-end wall-clock time for complete single runs with matched training budgets. Matching refers to training budget rather than KL coefficients. The timing record identifies a base setting without a separate model-checkpoint or hardware identifier. These are wall-clock measurements; utilization and GPU-hour accounting are unavailable.

Diagnostic	GRPO, uniform	KL-Cov, uniform	MGE-PO, DynZPD
Relative ESS	100%	100%	91.5%
Top-10% question probability mass	10.0%	10.0%	18.3%
Sampling-distribution Gini	0	0	0.085
Maximum question share (definition unreported)	Not reported	Not reported	0.4%
In-window unique-question coverage	14.3%, expected	14.3%, expected	14.0%, observed

Appendix

Table 14: Curriculum diagnostics in a 21-step window. The window contains 43,008 rollouts and 2,440 unique MGE-PO questions from the 17,398-question training dataset. Relative ESS, mass, and Gini describe the sampling distribution; unique-question coverage distinguishes uniform expectations from observed draws.

EA-Adv	EA-KL	DynZPD	Reference coefficient $\beta$	Macro avg@32 (%)	Absolute difference from GRPO (%)
Off	Off	Off	0.04	41.5	0.0
On	Off	Off	0.001	43.3	+1.8
Off	On	Off	0.001	42.2	+0.7
Off	Off	On	0.001	41.6	+0.1
On	On	Off	0.001	44.3	+2.8
On	Off	On	0.001	43.9	+2.4

Appendix

Table 15: Component configurations on Qwen2.5-7B-Base. The score is four-benchmark macro avg@32. EA-KL off retains uniform reference KL.

Optimization setting	avg@32 without 4Q (%)	avg@32 with 4Q (%)	Absolute $\Delta$ avg@32 (%)	pass@32 without 4Q (%)	pass@32 with 4Q (%)	Absolute $\Delta$ pass@32 (%)
GRPO	41.31	45.00	+3.69	57.97	69.03	+11.06
DAPO	43.61	46.99	+3.38	66.38	71.69	+5.31
Clip-Cov	43.25	46.48	+3.23	68.00	72.24	+4.24

Appendix

Table 16: 4Q ablation across three Qwen3-8B-Base optimization settings at the final controller step 624. Each row compares a method with its 4Q-augmented configuration in Table 1. The GRPO comparison uses the row labeled 4Q. Scores are six-benchmark macro averages (%); differences are absolute score changes (%) computed before rounding the macro averages to two decimals.

Method	Signal and granularity	Advantage intervention	KL intervention	Data selection
DAPO (Yu et al., 2025)	Group outcomes	Group-relative advantage	Frozen-reference KL omitted	Retains nondegenerate reward groups
EntroAdv (Cheng et al., 2026)	Token predictive entropy	Detached entropy bonus, capped by advantage magnitude; Equation (E1)	—	—
SEED-GRPO (Chen et al., 2025)	Question semantic entropy $\mathrm{SE}(q)$	Semantic-entropy attenuation of advantage magnitude; Equation (E2)	—	—
EAPO v1 (He et al., 2026)	Group-normalized token entropy $z_{i,t}$ and advantage polarity	Signed, capped token-entropy correction; Equation (E3)	—	—
KL-Cov (Cui et al., 2025)	Token covariance between log probability and advantage	Base policy-gradient term	$\beta D_{\mathrm{KL}}(\pi_{\mathrm{old}}\\|\pi_{\theta})$ on top-covariance tokens	—
SENT (Cao et al., 2025)	Question semantic entropy; token entropy and covariance	Base GRPO advantage	Piecewise coefficients: zero on high-entropy tokens; larger on high-covariance low-entropy tokens	Initial-policy semantic-entropy sorting, easy-to-hard stages

Appendix

Table 17: Comparison of intervention rules.

H_{i,t}

denotes token predictive entropy,

h_{i}^{\mathrm{top}}

the top-

K

pooled rollout statistic,

u_{i}

its decreasing logistic transform,

r_{i}

the binary outcome,

g_{i}

the truncation gate, and

\xi_{q}

the question EMA.

A_{i}

is the base GRPO advantage. 4Q is MGE-PO’s central credit-assignment rule; its curriculum and entropy-weighted reference penalty are included to specify the complete implemented configuration. The MGE-PO advantage expression precedes batch re-centering. An em dash indicates that the cited method introduces no separate intervention in that column.

Figure 5: Per-benchmark effect of adding 4Q on Qwen3-8B-Base at the final controller step 624. Each cell gives the augmented score minus its baseline score as an absolute difference (%), for mean@32 (top) and pass@32 (bottom). Rows identify GRPO, DAPO, and Clip-Cov; the corresponding augmented configurations are 4Q, 4Q + DAPO, and Clip-Cov + 4Q. All 36 contrasts share a zero-centered color scale, including negative and zero changes. Differences use the reported two-decimal benchmark scores. Evaluation samples 32 responses per question at temperature 0.6 on identical test sets. Cells describe point estimates without significance claims.

Figure 6: Original scores stratified by initial-model entropy. The pooled AIME24, AIME25, and AMC23 set contains 100 questions, divided into five equal-frequency buckets of 20 using top- $K$ pooled entropy from 64 base-model completions per question (pooling fraction 0.1). Paired bars show GRPO and complete MGE-PO mean@32 (left) and pass@32 (right), both in %; annotations give absolute score differences (%). The source does not specify the backbone. This is a full-configuration diagnostic and is separate from the Qwen3-8B 4Q comparisons.

Figure 7: Original complete-configuration training trajectories on Qwen2.5-7B-Base over 1080 steps. Panels show four-benchmark mean@32 (avg@32), pass@32, response length, and token entropy for GRPO, DAPO, and MGE-PO. These trajectories are separate from Table 2’s summary scores; their checkpoint correspondence is undocumented. The curves are retained as supplementary diagnostics of complete configurations, not as a 4Q component comparison.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,103 papers

ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning
Oct 17, 2026Diyi Hu, Ruizhong Qiu, Lin Qiu +2RL for Language Models RL for Language Model Reasoning
Independent Researcher · University of Illinois Urbana-Champaign · Meta
OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning
May 21, 2026Yu Li, Rui Miao, Tian Lan +1Value Function Estimation Token-Level Credit Assignment
1George Washington University · 2The University of Texas at Dallas
When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment
Sep 23, 2026Jie Zhang, Jingxiao Yang, Zhehao Huang +2RL for Language Model Reasoning Group Relative Policy Optimization
Shanghai Jiao Tong University · Zhejiang University

Oct 17, 2026J/K move · Enter open · S save

ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning

Diyi Hu, Ruizhong Qiu, Lin Qiu, Hanghang Tong+1

Independent Researcher · University of Illinois Urbana-Champaign · Meta

RL for Language Models RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Group Relative Policy Optimization

Open paper AskPDF arXiv

Reinforcement learning from verifiable rewards assigns one terminal reward to an entire reasoning trajectory, obscuring how individual reasoning steps contributed to the outcome. GRPO broadcasts a group-normalized advantage to every response token, while finer-grained approaches often rely on dense auxiliary rollouts or branching schemes that can also alter the data used for policy updates. We cast the missing middle as budgeted measurement: with a limited rollout budget, which partial solutions should be evaluated, and how should their value estimates refine credit? ProbePO selects partial-solution states with a reward-free uncertainty score and estimates their expected verifier rewards under a capped continuation budget. ProbePO redistributes, rather than replaces, GRPO's trajectory-level advantage across tokens while preserving each trajectory's mean token-level advantage; probe continuations serve only as measurements. Across three DeepSeekMath and Qwen-family models spanning 1.5B-7B parameters, ProbePO consistently outperforms GRPO and achieves the highest accuracy among the four evaluated methods in 16 of 18 model-benchmark comparisons. Against dense-probing VinePPO on DeepSeekMath-7B, ProbePO achieves higher accuracy on all six benchmarks with 76% less end-to-end training time.

Abstract

RL for Language Model Reasoning

Explore similar work

ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning

OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning

When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment