Exploration-Preserving Policy Optimization

Hangzhan jin; Mohammad Hamdaqa; Doina Precup

cs.AIOct 2, 2026

Exploration-Preserving Policy Optimization

Authors: Hangzhan jin, Mohammad Hamdaqa, Doina Precup

Organizations: Mila - Quebec AI Institute · Polytechnique Montr´eal · McGill University · CIFAR AI Chair · Google DeepMind

Abstract

Reinforcement learning with verifiable rewards improves reasoning, while the allocation of learning signal shapes which solutions remain accessible under repeated sampling. Group-relative objectives assign equal advantages to equally rewarded responses, making aggregate credit proportional to sampled mode frequency. We introduce Exploration-Preserving Policy Optimization (ExPPO), a lightweight advantage-shaping rule that redistributes credit using prompt-relative, length-normalized response surprisal and prompt pass rate. ExPPO combines bounded shaping with shared normalization to preserve verifier polarity and approximately maintain each prompt group's total absolute sequence-advantage mass. Our analysis characterizes response-level credit allocation alongside sampled mode updates, deriving local conditions for gains in entropy and correct-mode discovery. Experiments show improved in-domain and out-of-domain reasoning coverage, higher aggregate response accuracy, and strong coverage at large sampling budgets. A controlled multi-answer evaluation further demonstrates increased correct-mode yield and gains in diversity among verified-correct responses. Code is available at https://github.com/jinhangzhan/ExPPO

Figures & tables

Response	$A_{i}>0$ : reinforcement	$A_{i}<0$ : penalty	Gate $g_{i}$
Rare ( $u_{i}>0$ )	Stronger	Softer	$1-\rho_{q}$
Common ( $u_{i}<0$ )	Weaker	Stronger	$\rho_{q}$

Table 1: Response-level credit allocation before shared normalization. Rarity is prompt-relative. The gate emphasizes rare responses in low-success groups and common responses in high-success groups; bounded shaping preserves the verifier sign.

Figure 1: Checkpoint-wise any-success coverage. Columns: Qwen2.5-7B, Qwen3-0.6B, Phi-3.5-mini; rows: ID and OOD four-task means (percentage points). Budgets:

k=16

for AIME24/25, 8 for Minerva, 4 otherwise. Dashed lines: initialization. Per-task trajectories: Section J.5 .

Figure 2: Next-token entropy through training. Faint/solid traces: per-step observations/step means; diamonds: initial states. Qwen3-0.6B uses a linear scale through 0.8 and continuous compression above 0.8, with ticks in entropy units. The other panels use model-specific linear ranges.

Figure 3: Large-

K

coverage at step 900, using the first

k

responses per prompt. Rows: the three backbones; columns: four ID tasks. Curves include initialization and the core training rules. At the largest budget, ExPPO matches or exceeds the other trained core methods on both smaller backbones and three Qwen2.5-7B tasks.

A. Fixed-budget correctness and correct-mode yield
Model	Method	Accuracy	Solved / 64	Observed yield	Effective yield	Correct / prompt
Qwen2.5-7B	GRPO	6.45	57	1.750	1.546	8.25
	GRPO-H	9.68	59	1.906	1.610	12.39
	F-GRPO	35.51	64	3.234	2.389	45.45
	REPO-R	15.42	64	3.578	2.820	19.73
	ExPPO	33.50	64	4.172	3.013	42.88

Table 2: Last-checkpoint controlled evaluation (

K=128

; 64 prompts). A: Accuracy/Solved: correct-response share (0–100)/successful prompts. Observed/effective yields: mean distinct correct modes/exponentiated mode entropy (unsolved: zero). Correct/prompt: successes/64. B: Shared: prompts with

\geq 2

successes per method;

D

: correct-pair mode disagreement averaged on Shared (0–100);

\Delta D

: ExPPO minus baseline; CI: 95% paired prompt-bootstrap interval. Qwen2.5-7B: updated ExPPO for A and F-GRPO/REPO-R; earlier for GRPO/GRPO-H ( Section J.7 ).

Domain	Method	AIME24	AIME25	Minerva	Olympiad	Avg.
ID	Initialization	36.70	36.70	55.10	51.00	44.88
	GRPO	$23.33{\pm}3.33$	$24.44{\pm}9.62$	$53.55{\pm}0.77$	$50.96{\pm}0.90$	$38.07{\pm}2.74$
	GRPO-H	$26.67{\pm}10.00$	$27.78{\pm}10.18$	$56.50{\pm}1.39$	$53.28{\pm}1.13$	$41.06{\pm}4.76$
	GRPO-UR	$25.56{\pm}1.92$	$21.11{\pm}1.92$	$56.74{\pm}1.49$	$47.95{\pm}0.87$	$37.84{\pm}0.88$
	REPO-R	$26.67{\pm}3.33$	$33.33{\pm}3.33$	$57.60{\pm}1.29$	$\mathbf{56.20{\pm}1.05}$	$43.45{\pm}0.90$
	F-GRPO	$27.78{\pm}3.85$	$\mathbf{36.67{\pm}3.33}$	$56.37{\pm}0.93$	$53.83{\pm}0.34$	$43.66{\pm}0.67$

Table 3: Qwen2.5-7B last-checkpoint coverage (percentage points): mean

\pm

sample standard deviation across three decoding seeds at one fixed step-900 checkpoint per method. Initialization: one evaluation. Budgets: Section 6.1 ; Avg.: within-run four-task mean. Bold: best trained mean.

	Coverage		Response accuracy
Method	ID	OOD	ID	OOD
F-GRPO	$13.88{\pm}5.76$	$42.91{\pm}25.37$	$6.42{\pm}0.81$	$36.57{\pm}22.19$
GRPO-UR	$21.66{\pm}0.75$	$62.76{\pm}5.48$	$9.79{\pm}1.11$	$\mathbf{52.85{\pm}3.62}$
REPO-R	$21.94{\pm}1.89$	$\mathbf{66.35{\pm}0.38}$	$9.64{\pm}0.19$	$52.16{\pm}0.15$
ExPPO	$\mathbf{25.67{\pm}1.31}$	$66.33{\pm}1.59$	$\mathbf{10.59{\pm}0.27}$	$52.26{\pm}1.17$

Table 4: Phi-3.5-mini at 300 steps: mean

\pm

sample standard deviation across three independent training seeds per method. Each score is an unweighted four-task suite mean (%). Coverage is the share of prompts with a verified success; response accuracy is the verified-response share. Bold: highest mean per column. Task and seed details: Section J.2 .

Figure 4: Phi-3.5-mini coverage (%) from the first

k

responses per prompt, for the methods in Table 4 . ExPPO leads on three of four tasks at their largest evaluated budgets (

k=256

for AIME24/25;

k=128

otherwise), while GRPO-UR leads on AIME24. At the shared budget

k=128

, ExPPO’s four-task macro coverage of 51.03 exceeds GRPO-UR’s 44.34, REPO-R’s 40.30, and F-GRPO’s 12.17.

	Coverage		Response accuracy
Method	Code	OOD	Code	OOD
Initialization	32.11	59.40	6.92	41.93
GRPO	$85.94{\pm}0.00$	$52.24{\pm}0.44$	$75.22{\pm}0.04$	$36.42{\pm}1.18$
GRPO-H	$86.46{\pm}0.08$	$60.51{\pm}1.20$	$74.62{\pm}0.02$	$42.10{\pm}0.34$
ExPPO	$\mathbf{86.69{\pm}0.33}$	$\mathbf{65.74{\pm}0.71}$	$\mathbf{75.46{\pm}0.05}$	$\mathbf{51.02{\pm}0.87}$

Table 5: Code adaptation and OOD generalization at the last checkpoint. Scores (%): mean

\pm

sample standard deviation across three independent Phi-3.5-mini training runs. Code: CodeReasoner; OOD: unweighted mean over GPQA, MMLU-Pro, Safety, and TruthfulQA-MC (

k=4

). Initialization: fixed base model. Bold: best trained mean.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Suite	Prompts
AIME24	ID	30
AIME25	ID	30
Minerva	ID	272
OlympiadBench	ID	675
GPQA	OOD	448
MMLU-Pro	OOD	70

Appendix

Table 6: Reasoning-evaluation suites and prompt counts. Benchmark: evaluation set; Suite: In-Domain (ID) adaptation or Out-of-Domain (OOD) generalization; Prompts: evaluated questions. Each suite contains four tasks, and suite averages are unweighted.

Method	GPU-hours	Samples/s	Steps/s	Peak alloc.	Peak reserved
GRPO	151.70	6.997	0.01367	88.9 GiB	101.2 GiB
GRPO-H	157.76	6.710	0.01311	88.5 GiB	100.8 GiB
GRPO-UR	136.18	8.107	0.01583	86.5 GiB	98.6 GiB
REPO-R	151.66	7.055	0.01378	87.8 GiB	100.0 GiB
F-GRPO	146.08	7.289	0.01424	87.3 GiB	99.6 GiB
ExPPO	150.12	7.230	0.01412	87.7 GiB	99.9 GiB

Appendix

Table 7: Qwen2.5-7B-Instruct training cost on one eight-H200 node. Although the configured training prompt batch is 64, the synchronous data loader uses a generation batch of 32 prompts with 16 responses each, yielding 512 responses per logged step; Samples/s therefore equals 512 times Steps/s. Each profile runs for 100 steps, and throughput and per-GPU peak memory use steps 11–100. Estimated GPU-hours scale the full eight-GPU allocation wall time of the corresponding 100-step job by nine;

\dagger

marks the separately profiled F-GRPO job. GRPO-H uses the unchunked entropy-loss profile with activation checkpointing disabled. Peak alloc./reserved are GiB per GPU; GRPO-UR abbreviates Rewarding the Unlikely.

Symbol	Meaning
Rollouts, objectives, and shaping
$q,\mathcal{D},G$	Prompt, training prompt distribution, and responses per prompt group.
$Y_{i},y_{i};\ Y,y$	Random rollout and realized rollout; fresh random response and a value in its support.
$T_{i},y_{i,t},x_{i,t}$	Realized response length, token $t$ , and context $(q,y_{i,<t})$ .
$\theta,\pi_{\theta},\pi_{\mathrm{old}},\pi_{\mathrm{ref}}$	Parameters; current, frozen rollout, and fixed reference policies.
$R(q,y),r_{i};\ C,c_{i}$	Verifier and realized reward; random and realized success indicators ( $c_{i}=r_{i}$ for binary rewards).

Appendix

Table 8: Notation for the method and analysis. Indexed quantities are evaluated on the realized rollout group.

Domain	Setting	AIME24	AIME25	Minerva	Olympiad	ID Avg.
ID	GRPO	3.33	3.33	40.81	19.26	16.68
	$\lambda=0.10$	10.00	10.00	41.54	25.19	21.68
	$\lambda=0.30$	13.33	3.33	41.91	28.74	21.83
	$\lambda=0.50$	16.67	3.33	43.38	26.81	22.55
	$\lambda=0.75$	16.67	3.33	43.38	27.85	22.81
	$\lambda=1.00$	16.67	6.67	44.49	26.96	23.70

Appendix

Table 9: Phi-3.5-mini coefficient and component ablations at checkpoint 300. Coverage (%) at

k=16

for AIME24/25,

k=8

for Minerva, and

k=4

otherwise. Full ExPPO, component variants, and the shuffled control use

\lambda=0.2

. Full ExPPO: three-seed mean; other settings: one run. Avg.: unweighted four-task mean; reported macros may differ by 0.01 from rounded task averages. Olympiad/TruthfulQA abbreviate OlympiadBench/TruthfulQA-MC. Bold: highest ExPPO domain mean.

A. Coverage (%)
Domain	Benchmark	$k$	F-GRPO	GRPO-UR	REPO-R	ExPPO
ID	AIME24	16	$5.55{\pm}6.94$	$12.22{\pm}1.92$	$14.44{\pm}5.09$	$\mathbf{17.78{\pm}3.85}$
	AIME25	16	$3.33{\pm}3.34$	$7.78{\pm}5.09$	$2.22{\pm}1.92$	$\mathbf{13.33{\pm}0.00}$
	Minerva	8	$33.58{\pm}10.99$	$41.79{\pm}2.50$	$\mathbf{46.20{\pm}1.12}$	$44.12{\pm}0.97$
	Olympiad	4	$13.04{\pm}3.71$	$24.84{\pm}1.77$	$24.89{\pm}0.40$	$\mathbf{27.46{\pm}0.68}$
	Avg.	—	$13.88{\pm}5.76$	$21.66{\pm}0.75$	$21.94{\pm}1.89$	$\mathbf{25.67{\pm}1.31}$

Appendix

Table 10: Phi-3.5-mini at checkpoint 300 under the ablation setting of Table 9 . Entries are mean

\pm

sample standard deviation across three training seeds per method. Avg.: unweighted four-task mean within each seed; Overall: eight-task mean. Task summaries use the reported seed scores; macro summaries retain author-reported precision, which can differ by 0.01 point from aggregation of rounded task entries. Individual seeds: Tables 11 and 12 . Bold: highest mean across the four methods. ExPPO uses

\lambda=0.2

Method	Benchmark / suite	Seed 42	Seed 43	Seed 44
F-GRPO	AIME24	3.33 / 3.33	0.00 / 0.00	13.33 / 1.88
	AIME25	0.00 / 0.00	3.33 / 0.21	6.67 / 0.42
	OlympiadBench	9.19 / 6.19	13.33 / 5.56	16.59 / 6.78
	Minerva	21.69 / 13.28	35.66 / 19.30	43.38 / 20.08
	ID macro	8.55 / 5.70	13.08 / 6.27	19.99 / 7.29
	MMLU-Pro	10.00 / 8.21	54.29 / 43.21	61.43 / 52.86

Appendix

Table 11: Individual F-GRPO and ExPPO training seeds for Phi-3.5-mini at checkpoint 300. Each cell is coverage / response accuracy (percent), at

k=16

for AIME24/25, 8 for Minerva, and 4 otherwise. Task scores and macro rows retain author-reported precision; aggregation from rounded task entries can differ by 0.01 point. The other six runs appear in Table 12 .

Method	Benchmark / suite	Seed 42	Seed 43	Seed 44
GRPO-UR	AIME24	13.33 / 1.88	10.00 / 1.46	13.33 / 2.71
	AIME25	6.67 / 0.83	3.33 / 0.63	13.33 / 0.83
	OlympiadBench	25.63 / 15.00	26.07 / 14.52	22.81 / 10.74
	Minerva	42.65 / 25.18	43.75 / 23.76	38.97 / 19.94
	ID macro	22.07 / 10.72	20.79 / 10.09	22.11 / 8.56
	MMLU-Pro	74.29 / 55.36	68.57 / 52.50	61.43 / 49.64

Appendix

Table 12: Individual GRPO-UR and REPO-R training seeds for Phi-3.5-mini at checkpoint 300. Each cell is coverage / response accuracy (percent), with budgets and rounding conventions as in Table 11 . Together, the two seed tables report all twelve runs.

Coverage
Domain	Method	AIME24	AIME25	Minerva	Olympiad	Avg.
ID	GRPO	$7.78{\pm}5.09$	$2.22{\pm}1.92$	$33.09{\pm}13.78$	$16.15{\pm}7.40$	$14.81{\pm}6.89$
ID	ExPPO	$\mathbf{14.44{\pm}5.09}$	$\mathbf{8.89{\pm}3.85}$	$\mathbf{44.85{\pm}2.24}$	$\mathbf{27.80{\pm}1.11}$	$\mathbf{24.00{\pm}0.62}$
Domain	Method	GPQA	MMLU-Pro	Safety	TruthfulQA	Avg.
OOD	GRPO	$41.00{\pm}6.19$	$58.57{\pm}6.23$	$70.53{\pm}4.37$	$64.57{\pm}2.05$	$58.67{\pm}3.07$
OOD	ExPPO	$\mathbf{49.78{\pm}3.72}$	$\mathbf{71.90{\pm}5.02}$	$\mathbf{75.63{\pm}0.67}$	$\mathbf{71.93{\pm}3.98}$	$\mathbf{67.31{\pm}3.25}$

Appendix

Table 13: Phi-3.5-mini performance across the original and two additional independently trained runs at the last checkpoint. Domain: In-Domain (ID) or Out-of-Domain (OOD); upper panel: any-success coverage; lower panel: mean verified-response share. Scores are in percentage points at

k=16

for AIME24/25,

k=8

for Minerva, and

k=4

otherwise. Avg.: unweighted four-task mean within each run, followed by the across-run mean

\pm

sample standard deviation. Olympiad/TruthfulQA abbreviate OlympiadBench/TruthfulQA-MC. Bold: stronger mean per column.

Coverage
Method	GPQA	MMLU-Pro	Safety	TruthfulQA-MC
Initialization	38.17	58.57	74.50	66.37
GRPO	$32.96{\pm}0.46$	$47.62{\pm}1.65$	$70.10{\pm}0.30$	$58.28{\pm}0.83$
GRPO-H	$37.57{\pm}0.90$	$59.52{\pm}3.60$	$72.37{\pm}0.40$	$72.56{\pm}0.97$
ExPPO	$\mathbf{44.35{\pm}1.65}$	$\mathbf{65.24{\pm}4.12}$	$\mathbf{76.37{\pm}0.25}$	$\mathbf{77.00{\pm}0.69}$
Response accuracy

Appendix

Table 14: OOD generalization after code adaptation, disaggregating Table 5 . Upper panel: any-success prompt coverage at

k=4

; lower panel: mean verified-response share. Scores are percentages. Initialization: fixed base model; trained entries: mean

\pm

sample standard deviation across three independent runs at the last checkpoint. Bold: best trained mean.

Figure 5: Per-task In-Domain (ID) coverage through the last checkpoint; disaggregation of Fig. 1 . Rows: Qwen2.5-7B, Qwen3-0.6B, Phi-3.5-mini; columns: AIME24, AIME25, Minerva, OlympiadBench. Horizontal axis: checkpoint; vertical axis: coverage in percentage points at

k=16,16,8,4

, respectively. Gray lines: Initialization; colored curves: GRPO, GRPO-H, ExPPO.

Figure 6: Per-task Out-of-Domain (OOD) coverage through the last checkpoint; disaggregation of Fig. 1 . Rows: Qwen2.5-7B, Qwen3-0.6B, Phi-3.5-mini; columns: GPQA, MMLU-Pro, Safety, TruthfulQA-MC. Horizontal axis: checkpoint; vertical axis: coverage in percentage points at

k=4

. Gray lines: Initialization; colored curves: GRPO, GRPO-H, ExPPO.

Figure 7: Qwen2.5-7B macro response-accuracy trajectories under DAPO. Left: unweighted mean over AIME24, AIME25, Minerva, OlympiadBench (In-Domain; ID); right: unweighted mean over GPQA, MMLU-Pro, Safety, TruthfulQA-MC (Out-of-Domain; OOD). Horizontal axis: checkpoint; vertical axis: response accuracy in percentage points. Gray dashed lines: Initialization; colored curves: DAPO, ExPPO.

ID
Method	AIME24	AIME25	Minerva	Olympiad	Avg.
Initialization	12.10	7.70	37.10	36.10	23.25
DAPO	10.42	7.92	41.31	35.78	23.85
ExPPO	14.38	10.21	40.85	38.67	26.05

Appendix

Table 15: Qwen2.5-7B DAPO response accuracy at the last checkpoint. Upper/lower panels: In-Domain (ID)/Out-of-Domain (OOD). Values: mean verified-response share in percentage points at

k=16

for AIME24/25,

k=8

for Minerva, and

k=4

otherwise. Avg.: unweighted four-task mean. Olympiad/TruthfulQA abbreviate OlympiadBench/TruthfulQA-MC. Initialization: fixed base model; bold: better trained method per column.

Model	Method	Correct / 8,192	$C\!\geq\!2$ / 64
Qwen2.5-7B	GRPO	528	51
	GRPO-H	793	55
	ExPPO	2,608	64
Qwen3-0.6B	GRPO	8,063	64
	GRPO-H	2	1
	ExPPO	8,007	64

Appendix

Table 16: Additional controlled-task diagnostics (

K=128

; 64 prompts; last checkpoint). Correct: verified responses among 8,192 samples;

C\!\geq\!2

: prompts with at least two correct responses, eligible for diversity evaluation. Qwen2.5-7B ExPPO uses the earlier evaluation underlying the GRPO/GRPO-H pairs in Table 2 and the GRPO-UR comparison in Table 18 . Other rows match the main evaluation.

Model	Baseline	Shared / 64	Baseline $D$	ExPPO $D$	$\Delta D$	CI
Qwen3-0.6B	GRPO-H	1	0.0000	0.7531	$+0.7531$	–
Phi-3.5-mini	GRPO	6	0.0370	0.1620	$+0.1249$	$[-0.0741,+0.3768]$
	GRPO-H	4	0.5701	0.1310	$-0.4391$	$[-0.8125,-0.1131]$

Appendix

Table 17: Additional core-baseline correct-pair comparisons from the evaluations in Table 16 ; other pairs appear in Table 2 . Shared: prompts with at least two correct responses per method.

D

: mean distinct-mode probability of correct pairs on that shared set;

\Delta D

: ExPPO minus baseline; CI: paired prompt-bootstrap 95% interval.

D

uses a 0–1 scale, with differences and intervals in the same units, versus the main table’s 0–100 scale. The one-prompt Qwen3-0.6B comparison is reported as a point estimate.

A. Fixed-budget correctness and correct-mode yield
Method	Accuracy	Solved / 64	Observed yield	Effective yield	Correct / prompt
GRPO-UR	17.77	64	3.0000	2.3021	22.75
ExPPO	31.84	64	3.9688	2.7277	40.75

Appendix

Table 18: Qwen2.5-7B controlled-task comparison with GRPO-UR (Rewarding the Unlikely) at the last checkpoint (

K=128

; 64 prompts). A: Accuracy: verified-response share (0–100); Solved: prompts with a success. Observed/effective yields: prompt means of distinct correct modes/exponentiated mode entropy, with zero for unsolved prompts. Correct/prompt: total verified successes divided by 64, including unsolved prompts. Bold: higher mean. B: Shared: prompts with at least two successes per method;

D

: mean distinct-mode probability of correct pairs on that set;

\Delta D

: ExPPO minus baseline; in probability units. ExPPO uses the earlier evaluation; the updated F-GRPO/REPO-R comparisons appear in Table 2 .

A. Held-out policy outcomes
Policy	Accuracy (%)	Diversity $D$
Shared ExPPO-900 checkpoint	33.74	0.6404
+ GRPO update	35.64	0.6135
+ ExPPO update	36.52	0.6208
+ GRPO-H update	34.45	0.6001

Appendix

Table 19: Qwen2.5-7B common-checkpoint continuation on 32 held-out prompts. All three update arms start from the same ExPPO-900 policy. Panel A reports accuracy in percent and diversity as a probability. Panel B reports ExPPO-minus-baseline differences in percentage points for both metrics.

Response	Surprisal class	$\Delta_{\mathcal{C}}$	Allocation effect
Correct	High ( $u_{i}>0$ )	$+6.245$	Stronger reinforcement
Correct	Low ( $u_{i}<0$ )	$-3.113$	Reduced reinforcement
Incorrect	High ( $u_{i}>0$ )	$+8.690$	Softer penalty
Incorrect	Low ( $u_{i}<0$ )	$-5.557$	Stronger penalty

Appendix

Table 20: Post-normalization coefficient audit on the Qwen2.5-7B update responses. Values are category sums of

\widetilde{A}_{i}-A_{i}

in sequence-advantage units. Surprisal classes are relative to each prompt group. Allocation effects describe the aggregate scalar coefficients used by the policy loss.

Figure 8: Qwen2.5-7B matched-input OOD representations. Each panel shows 64 shared questions for Base and three step-900 policies in a separately fitted PCA coordinate system. Axes report explained variance; legends report full-benchmark, original-space cosine distances to Base, not distances in the projection.

Figure 9: Qwen3-0.6B matched-input OOD representations, using the protocol in Section J.9 . Each panel has its own PCA coordinates and 64 shared questions across four checkpoints. Legend values measure full-benchmark cosine distance to Base before projection; the relative ordering is sensitive to answer format.

Figure 10: Phi-3.5-mini matched-input OOD representations, using the protocol in Section J.9 . Each panel has its own PCA coordinates and 64 shared questions across four checkpoints. Legends report full-benchmark cosine distances to the released Base reference, whose identity with the training initialization is unverified. Proximity does not establish preserved accuracy or diversity.

Explore similar work

May 11, 2026

expo: Exploration-prioritized policy optimization via adaptive kl regulation and gaussian curriculum sampling

Reinforcement Learning with Verifiable Rewards (RLVR) has become the standard paradigm for LLM mathematical reasoning, where Group Relative Policy Optimization (GRPO) serves as the mainstream algorithm. We point out two understudied inefficiencies existing in GRPO. First, the fixed KL penalty coefficient overly restricts policy exploration at stages where the model requires significant deviation from the reference policy. Second, uniform sampling of training questions ignores that moderately difficult problems provide the most informative gradient signals for optimization. We propose Exploration-Prioritized Policy Optimization (EXPO) with two lightweight plug-in modules. The Accuracy-Conditioned KL Scaling (AKL) dynamically adjusts KL regularization strength through a smooth nonlinear function of batch average accuracy, relaxing the penalty when the model underperforms and strengthening it when the model achieves good results. The Gaussian Curriculum Sampling (GCS) assigns sampling weights to questions following a Gaussian distribution centered at moderate accuracy around 0.5, focusing training on the model's learning frontier. We conduct extensive experiments on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base over six mathematical reasoning benchmarks. The results show EXPO steadily surpasses vanilla GRPO. It obtains an absolute gain of 13.34 on AIME 2025 pass@32, rising from 63.33 percent to 76.67 percent, and achieves an average pass@32 improvement of 2.66 on the 8B model. The much larger performance gains on pass@32 compared with pass@1 demonstrate that EXPO effectively enlarges the model's exploration boundary under a fixed inference cost budget.

Mingxiong Lin, Zhangquan Gong, Maowen Tang +6

OPPO AI Center

Reinforcement Learning KL-Regularized RL RL for Language Model Reasoning Group Relative Policy Optimization Curriculum RL RL Exploration+2

Ask PDF

May 28, 2026

VeriGate: Verifier-Gated Step-Level Supervision for GRPO

Group Relative Policy Optimization (GRPO) is an effective recipe for training reasoning models with verifier-based outcome rewards, but its supervision is sparse: when all sampled trajectories for a prompt receive the same verifier reward, the group-relative advantage collapses to zero and learning stalls. Outcome-only rewards also provide no step-level credit assignment, limiting exploration and making it harder to learn robust reasoning. We present VeriGate (Verifier-Gated Step-Level GRPO), a verifier-gated extension of GRPO that addresses these limitations with three design choices. First, VeriGate keeps the verifier in charge whenever verifier rewards induce a meaningful preference among sampled trajectories, and uses process supervision only when verifier rewards are degenerate. Second, instead of collapsing Process Reward Model (PRM) step scores into a single trajectory reward, VeriGate converts them into future-cumulated rewards to assign continuation-aware credit. Third, VeriGate transforms these rewards into group-normalized token-level advantages, restoring informative gradients and fine-grained credit assignment while remaining less susceptible to reward hacking than methods that optimize aggregated PRM scores. Empirically, training on MATH with 1.5B and 7B Qwen2.5-Instruct models and evaluating on six reasoning benchmarks, VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively, substantially reduces zero-gradient failures, decreases reward-hacking behavior, and improves reasoning quality relative to outcome-only GRPO and PRM-as-outcome baselines.

Aakriti Agrawal, Minghui Liu, Furong Huang

University of Maryland, College Park

Process Reward Models RL for Language Model Reasoning Group Relative Policy Optimization Credit Assignment in RL Reinforcement Learning with Verifiable Rewards Step-Level Credit Assignment

Ask PDF

Apr 22, 2026

GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning

Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Language Models (LLMs) by leveraging direct outcome verification instead of learned reward models. Building on this paradigm, Group Relative Policy Optimization (GRPO) eliminates the need for critic models but suffers from indiscriminate credit assignment for intermediate steps, which limits its ability to identify effective reasoning strategies and incurs overthinking. In this work, we introduce a model-free and verifiable process supervision via probing the model's belief in the correct answer throughout its reasoning trajectory. By segmenting the generation into discrete steps and tracking the conditional probability of the correct answer appended at each segment boundary, we efficiently compute interpretable segment-wise progress measurements to refine GRPO's trajectory-level feedback. This approach enables more targeted and sample-efficient policy updates, while avoiding the need for intermediate supervision derived from costly Monte Carlo rollouts or auxiliary models. Experiments on mathematical and general-domain benchmarks show consistent gains over GRPO across diverse models: up to 2.6-point accuracy improvements and 13.7% reasoning-length reductions on math tasks, and up to 2.4 points and 4% on general-domain tasks, demonstrating strong generalization.

Jingyi Wang, Lei Zhu, Tengjin Weng +8

Tsinghua University · Huawei Noah’s Ark Lab · Shenzhen University

RL for Language Model Reasoning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards Process Supervision

Ask PDF